devsinghsachan / attention_is_all_you_need Goto Github PK

This repository is deprecated. Please refer to the updated codebase here: https://github.com/DevSinghSachan/multilingual_nmt

This repository implements the transformer model in pytorch framework which was introduced in the paper Attention is All you Need as described in their NIPS 2017 version: https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf

The overall model architecture is as shown in the figure:

The code in this repository implements the following features:

• Positional Encoding
• Multi-Head Dot-Product Attention
• Positional Attention from "Non-Autoregressive Neural Machine Translation"
• Label Smoothing
• Warm-up steps based training of Adam Optimizer
• Shared weights of embedding and softmax layers
• Beam Search with length normalisation

• Python 3.6
• Pytorch v0.4 (needs manual installation from source https://github.com/pytorch/pytorch)
• torchtext
• numpy

One can install the above packages using the requirements file.

pip install -r requirements.txt

python preprocess.py -i data/ja_en -s-train train-big.ja -t-train train-big.en -s-valid dev.ja -t-valid dev.en -s-test test.ja -t-test test.en --save_data demo

python train.py -i data/ja_en --data demo --wbatchsize 4096 --batchsize 60 --tied --beam_size 5 --epoch 40 --layers 6 --multi_heads 8 --gpu 0

python translate.py -i data/ja_en --data demo --batchsize 60 --beam_size 5 --model_file "results/model.ckpt" --src data/ja_en/test.ja --gpu 0

Dataset Statistics included in data directory are:

All the experiments were performed on a modern Titan-Xp GPU with 12GB RAM. BLEU Scores are computed using Beam Search.

Dataset URL

^*1 epoch get completed in around 180 seconds.

Dataset URL

Dataset URL . This dataset exists in tokenised form (using NLTK and lowercase).

• Thanks to the suggestions from Graham Neubig @gneubig and Matt Sperber @msperber
• The code in this repository was originally based and has been adapted from the Sosuke Kobayashi's implementation in Chainer "https://github.com/soskek/attention_is_all_you_need".
• Some parts of the code were borrowed from XNMT (based on Dynet) and OpenNMT-py (based on Pytorch).