This repository contains the implementation of the paper Contrastive Learning for Neural Topic Model.
Thong Nguyen, Luu Anh Tuan (NeurIPS 2021)
In this work, we target the problem of capturing meaningful representations through modeling the relations among samples from a mathematical perspective and propose a novel contrastive objective to train the neural topic model, along with the optimization of the variational lower bound. In our contrastive learning framework, we introduce a novel sampling strategy that is motivated by human behavior when comparing numerous documents. Our results show that capturing mutual information between the prototype and its positive sample provides a strong foundation for constructing coherent topics, while differentiating the prototype from the negative samples plays a less fundamental role.
@inproceedings{
nguyen2021contrastive,
title={Contrastive Learning for Neural Topic Model},
author={Thong Thanh Nguyen and Anh Tuan Luu},
booktitle={Advances in Neural Information Processing Systems},
editor={A. Beygelzimer and Y. Dauphin and P. Liang and J. Wortman Vaughan},
year={2021},
url={https://openreview.net/forum?id=NEgqO9yB7e}
}
- python3
- pandas
- gensim
- numpy
- torchvision
- pytorch
- scipy
more in requirements.txt
- Download and put the dataset in the
data
folder: https://drive.google.com/file/d/1JeeUCzBRQqJUvdWGDN7aMRvIoBAIbZIc/view?usp=sharing - process nips dataset:
python preprocess_data.py data/nips/train.jsonlist data/nips/processed/ --vocab-size 10000 --test data/nips/test.jsonlist
process 20ngpython preprocess_data.py data/20ng/20ng_all/train.jsonlist data/20ng/processed/ --vocab-size 2000 --label group --test data/20ng/20ng_all/test.jsonlist
process AG_news dataset:python preprocess_data.py data/ag_news/train.jsonlist data/ag_news/processed/ --vocab-size 10000 --label label --test data/ag_news/test.jsonlist
process IMDB dataset:python preprocess_data.py data/imdb/train.jsonlist data/imdb/processed/ --vocab-size 5000 --label sentiment --test data/imdb/test.jsonlist
- to use word2vec download GoogleNews-vectors-negative300.bin.gz from https://code.google.com/archive/p/word2vec/
- run
python build_graph_new.py
andpython get_bert_embeddings.py
- change permissions if be denied:
chmod u+x scripts/train_models/run_20ng_gcn.sh
chmod u+x scripts/evaluate/run_20ng_npmi_gcn.sh
- Train the model by running
scripts/train_models/run_20ng_gcn.sh
- Evaluate the model via executing
scripts/evaluate/run_20ng_npmi_gcn.sh
dataset | LR | epochs | label | word2vec |
---|---|---|---|---|
20NG | 0.001 / 0.002 | 300 400 500 / 200 300 | no | no |
IMDB | 0.001 | 300 400 | ||
NIPS | 0.004 / 0.005 | 400 500 |
Our implementation is based on the official code of SCHOLAR.