What's the exact command/dataset to reproduce a BLEU of 30.42 on test set? about xmunmt HOT 4 CLOSED

Update: another model, which is trained on 2 GTX 1080 Ti cards and with batch_size 64, achieved a BLEU score of 23.09. I guess the result could be better if I train it longer.

from xmunmt.

The benchmark is trained on De-En direction which has a higher BLEU score. I think your result is reasonable. This basic RNNsearch model cannot match the result achieved by WMT winners. Their system usually uses other techniques like back-translation, model ensemble and deep models.

from xmunmt.

I see. I have pretty much experience in training NMT models with Theano, and have tried out a lot more modern techniques such like sequence-level knowledge distillation. I also implemented an encoder-decoder model with attention mechanism in pure C++ and deployed it in cellphones. I'm just looking for an NMT framework in TensorFlow (because Theano is not maintained any more, lol~) and found this one. So I'm shocked by the BLEU score reported in readme.md.

The instruction in the readme.md is for training English->German models, but the benchmark result is for German->English, which is inconsistent. I suggest you to update it to make it less confusing, i.e.: by including my result on En->De translation.

For my experiments, the RNNSearch model trained for 75k steps on 2 GTX 1080 Ti cards with batch_size=64 achieved a BLEU score of 0.2309 and an NIST score of 6.7258.
Continuing training it for 75k more steps (150k steps in total) leads to a BLEU score of 0.2348 and an NIST score of 6.7717.

from xmunmt.

And thanks for your helping these days.
I'm considering reading through all the source code in this repo and doing research based on this. Hopefully I'll try to reproduce CSGAN-NMT and VNMT and send a pull request to you to include these more advanced models into the project.

from xmunmt.