yuvalpinter / mimick Goto Github PK

View Code? Open in Web Editor NEW

149.0 9.0 34.0 20.33 MB

Code for Mimicking Word Embeddings using Subword RNNs (EMNLP 2017)

License: GNU General Public License v3.0

Python 100.00%

word-embeddings neural-network lstm convolutional-neural-networks part-of-speech-tagger multilingual

mimick's Introduction

Mimick

Code for Mimicking Word Embeddings using Subword RNNs (EMNLP 2017) and subsequent experiments.

tl;dr

Given a word embedding dictionary (with vectors from, e.g. FastText or Polyglot or GloVe), Mimick trains a character-level neural net that learns to approximate the embeddings. It can then be applied to infer embeddings in the same space for words that were not available in the original set (i.e. OOVs - Out Of Vocabulary).

Citation

Please cite our paper if you use this code.

@inproceedings{pinter2017mimicking,
  title={Mimicking Word Embeddings using Subword RNNs},
  author={Pinter, Yuval and Guthrie, Robert and Eisenstein, Jacob},
  booktitle={Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  pages={102--112},
  year={2017}
}

Dependencies

The main dependency for this project is DyNet. Get it here.

As of November 22, 2017, the code complies with Dynet 2.0. You may access the 1.0 version code via the commit log.

Create Mimick models

The mimick directory contains scripts relevant to the Mimick model: dataset creation, model creation, intrinsic analysis (see readme within). The models directory within contains models trained for all 23 languages mentioned in the paper. If you're using the pre-trained models, you don't need anything else from the mimick directory in order to run the tagging model. If you train new models, please add them here via pull request!

December 12, 2017 note: pre-trained model are now in DyNet 2.0 format (and employ early-stopping). The 1.0-compatible models are still available in a subdirectory.

CNN Version (November 2017)

As of the November 22 PR, there is a CNN version of Mimick available for training. It is currently a single-layer convolutional net (conv -> ReLU -> max-k-pool -> fully-connected -> tanh -> fully-connected) that performs the same function as the LSTM version.

Tag parts-of-speech and morphosyntactic attributes using trained models

The root directory of this repository contains the code required to perform extrinsic analysis on Universal Dependencies data. Vocabulary files are supplied in the vocabs directory.

The entry point is model.py, which can use tagging datasets created using the make_dataset.py script. Note that model.py accepts pre-trained Word Embedding models via text files with no header. For Mimick models, this exact format is output into the path in mimick/model.py script's --output argument. For Word2Vec, FastText, or Polyglot models, one can create such a file using the scripts/output_word_vectors.py script that accepts a model (.pkl or .bin) and the desired output vocabulary (.txt).

mimick's People

Contributors

Stargazers

Watchers

mimick's Issues

Add early stopping

Find best Trainer

Since the upgrade to DyNet 2.0, training loss doesn't seem to converge on the Mimick algorithm (fine in tagger code; models also make sense).

This seems to be due to the change in learning rate behavior in DyNet's trainers. The current implementation here uses AdamTrainer, but SGDTrainer and AdaGradTrainer have the same issues.

Char2Tag takes wrong representations from backward LSTM

This line should not be concatenating char_embs[-1], but rather dy.concatenate([char_embs[-1][:h],char_embs[0][h:]]) for the appropriate h.

The in-model Mimick code is fine, since it uses separate fwd and bwd char-level models rather than dynet's built-in BiRNNBuilder. The word-level BiLSTM is fine because it's sequence prediction.

Code runs very slow on GPU

I ran a mimick algorithm on a small data set and it is taking 5 mins on CPU. But when I run the same on GPU it is taking 40 mins to finish one epoch. Is there a way to fix this?

Can we increase the batch size of this?

Remove `START_TAG` and `STOP_TAG` from `model.py` and `make_dataset.py`

They're dead references, back from the ancient version of the code that had a CRF w/ Viterbi decoding on top of the word-leve LSTM.

Model arguments confusion!!

params in MomentumSGDTrainer

Hi I was trying out your demo when I run into error at line 166, Mimick/mimick/model.py
trainer = dy.MomentumSGDTrainer(model.model, options.learning_rate, 0.9, 0.1)

The error message shows that MomentumSGDTrainer takes 3 parameters, as in
MomentumSGDTrainer(ParameterCollection &m, real learning_rate = 0.01, real mom = 0.9)

wondering is there a version conflict? But I installed the v2.0 dynet, following your README.

So what is this last parameter 0.1? Do I just simply delete it?

Thanks in advance!

Word-level LSTM states are not reset between sentences (although the computation graph is renewed so there's no trans-backprop).
Char-level LSTM in char2tag or both mode keeps its state across words along the entire dataset. Within sentences, this means there is also backprop across word boundaries since there's no call to renew_cg(). This effect may be insignificant due to the <PAD> characters, but I don't know for sure.