hkf / lbldm-demo Goto Github PK

This code runs the log-bilinear document model (lblDm) described
in:
Andrew L. Maas and Andrew Y. Ng. (2010). 
A Probabilistic Model for Semantic Word Vectors. 
NIPS 2010 Workshop on Deep Learning and Unsupervised Feature Learning.


How to run the code::
open matlab
if you can, open a matlabpool as it will help the code run faster.
In matlab on my multi-core system I do this with:
matlabpool open local;

run the learning procedure for the model:
run_lblDm;

run the visualization to see how learned representations cluster:
run_tsne;


Dependencies::
You need to have minFunc and t-sne, both of which are available online.
The code assumes that these libraries are setup in the root directory.
If you have them elsewhere, you simply need to find and modify the
relevant addpath calls.


Details::
The demo uses data from flickr tags. Each document is the set of tags
associated with an image in flickr. The dataset is a derivative of
the NUS-WIDE dataset:
http://lms.comp.nus.edu.sg/research/NUS-WIDE.htm

The demo is currently set to learn word vectors for 1k words using
100k documents. The top 1k words are used after ignoring the 50
most frequently occuring words (common stop word removal). This is
a fairly small demo, but it runs quickly. 

The code uses alternating optimization, where both subproblems are
optimized with the minFunc package. First, the MAP estimates for the
document coefficients theta are updated. This requires solving number
of documents small optimization problems. The code is set to use a
parfor loop, so if you have a matlabpool open it will likely run much
faster on a multi-core machine. In the second phase, the word
representations are updated which is a large, single optimization
problem. This alternating procedure contunues until the maximum number
of algorithm iterations is reached. For this demo, the model converges
much sooner than the maximum number of outer iterations. You can see
this happening when the word representation optimization stops within
a few iterations.

The visualization step clusters the learned word representations using
the t-sne algorithm. A 2-D plot is then displayed and gives some sense
of how the word representations define similarity among words.

References for supporting code:
MinFunc:
http://www.cs.ubc.ca/~schmidtm/Software/minFunc.html

t-SNE:
http://homepage.tudelft.nl/19j49/t-SNE.html
L.J.P. van der Maaten and G.E. Hinton. 
Visualizing High-Dimensional Data Using t-SNE. 
Journal of Machine Learning Research 9(Nov):2579-2605, 2008.