Not necessarily needed, but nice to have. Might or might not implement.

• hybrid model: recurrent encoder and self-attentional decoder and vice versa
• checkpoint averaging
• transformer: output layer / embedding parameter sharing
• multi-GPU support
• log probs of hypotheses as additional output

• Explaining all hyperparameters
• Create a good default ini with working example
• Explaining the structure of a config file
• Explaining the architecture of the code base
  -> "Annotated Joeynmt"?

I am training isiXhosa > English NMT for a small (XhosaNavy Corpus)<45k.
Tested JoeyNMT with Luong and Badhanau with no errors, but ...

The produced:

1. Got errors when training the transformer_BPE approach for isiXhosa > English
2. configuration file:
   name: "xhen_transformer"
   data:
   src: "xh"
   trg: "en"
   train: "test/data/navy_xhen/train-bpe"
   dev: "test/data/navy_xhen/dev-bpe"
   test: "test/data/navy_xhen/test-bpe"
   level: "bpe"
   lowercase: True
   max_sent_length: 50
   src_vocab: "test/data/navy_xhen/vocab-bpe.xh"
   trg_vocab: "test/data/navy_xhen/vocab-bpe.en"
   ....
3. code call [!python -m joeynmt train configs/xhen_transformer_bpe.yaml]

Logged output
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/content/joeynmt/joeynmt/main.py", line 41, in
main()
File "/content/joeynmt/joeynmt/main.py", line 29, in main
train(cfg_file=args.config_path)
File "/content/joeynmt/joeynmt/training.py", line 535, in train
model = build_model(cfg["model"], src_vocab=src_vocab, trg_vocab=trg_vocab)
File "/content/joeynmt/joeynmt/model.py", line 226, in build_model
"Embedding cannot be tied since vocabularies differ.")
joeynmt.helpers.ConfigurationError: Embedding cannot be tied since vocabularies differ.

Expected behavior
output : Hello! This is Joey-NMT.
params: ...
config

System (please complete the following information):

• OS: Google Colab VM runtime
• GPU
• Python >=3.0

Additional context
I am a novice and just stated with NMT projects and JoeyNMT toolkit is a good start

Describe the bug
After training, when I used the model to generate the hypothesis on dev and test set, there are at the end of some generated sentences.

Logged output
One example of the output is:
but/and horses then- redup- tell s.o. - <pad> <pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad>

System (please complete the following information):

• MAC OS
• GPU
• Python Version 3.7.4

Missing RL models

Hi @sjarmero,

After your pull request #139 has been merged, I've got an error at

It should look like this, maybe?

assert len(sort_reverse_index) == batch.nseqs

and a small problem here, too.

because now you use append, not extend as before, valid_attention_scores wil become non-empty even if attention_scores is None, and it unexpectedly triggers this if-block:

thank you for your effort to include the nbest list!

@juliakreutzer, FYI

Check whether all elements in batch are finished is already implemented in beam search, but it's useful for greedy decoding during validation as well, since it will save some computation.

Describe the bug
When trying to use multiple GPUs to train RNN models, I get the following error:

AssertionError: Caught AssertionError in replica 1 on device 1.
Original Traceback (most recent call last):
File "/home/students/kiegeland/anaconda3/envs/nmt/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
output = module(*input, **kwargs)
File "/home/students/kiegeland/anaconda3/envs/nmt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/students/kiegeland/test/joeynmt/joeynmt/model.py", line 87, in forward
trg_mask=kwargs["trg_mask"])
File "/home/students/kiegeland/test/joeynmt/joeynmt/model.py", line 147, in _encode_decode
trg_mask=trg_mask)
File "/home/students/kiegeland/test/joeynmt/joeynmt/model.py", line 186, in _decode
trg_mask=trg_mask)
File "/home/students/kiegeland/anaconda3/envs/nmt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/students/kiegeland/test/joeynmt/joeynmt/decoders.py", line 340, in forward
prev_att_vector=prev_att_vector)
File "/home/students/kiegeland/test/joeynmt/joeynmt/decoders.py", line 192, in _check_shapes_input_forward
assert src_mask.shape[2] == encoder_output.shape[1]
AssertionError

It seems like the src_length is different for each device which results in different shapes for the encoder output.
Is there anything else I need to configure?

To Reproduce
Training the reverse experiment using 2 gpus on version=1.0
node_configuration.txt

Logged output
error_log.txt

Hi Julia and Joost!

Currently testing joeynmt for our MT class, verdict: awesome! Planning to use it for several exercises.

For the class I've put together example scripts in a small repo here:

https://github.com/bricksdont/joeynmt-toy-models

Just a collection of scripts that show how to make a virtualenv, install packages, download data, preprocess, train and evaluate. Your documentation mentions all of this of course, but the info is more distributed.

If this is useful in any way feel free to link to this somewhere. Otherwise, feel free to close :)

Describe the bug

I believe that the current inference code internally removes BPE tokens, which in my opinion is unexpected. This happens here:

since this function for validation is re-used for translation.

To Reproduce
Follow all the steps here :)

https://github.com/bricksdont/joeynmt-toy-models

especially:

https://github.com/bricksdont/joeynmt-toy-models/blob/master/scripts/evaluate.sh

where removing BPE from translations does not have any effect.

Expected behavior

Translation should not do any postprocessing in my opinion, since JoeyNMT also does not do any preprocessing during training.

Doing BPE postprocessing to show examples from the validation set during training is indeed helpful I think.

Problem
When training using Masakhane custom data notebook, and pip installing joeyNMT,
!cd joeynmt; python3 -m joeynmt train configs/transformer_$src$tgt.yaml uses CPU instead of GPU.

This occurs if I use version 1.1, but NOT if I pip install 1.0 specifically.

• If I use joeynmt==1.0 when pip installing, I get device:cuda
• If I use joeynmt when pip installing, it installs 1.1, and I get device:cpu

System (please complete the following information):

• Google Colab, https://github.com/masakhane-io/masakhane-mt/blob/master/starter_notebook-custom-data.ipynb
• joeyNMT pip installed

Describe the bug
Model training is encountering an error after a few steps. The training log is shown below:

Logged Output
2020-06-23 03:52:05,935 Epoch 1 Step: 3900 Batch Loss: 4.291916 Tokens per Sec: 2948, Lr: 0.000300

2020-06-23 03:52:16,157 Epoch 1 Step: 4000 Batch Loss: 5.365633 Tokens per Sec: 2889, Lr: 0.000300

Traceback (most recent call last):
File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)

File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)

File "/home/renz/joeynmt/joeynmt/main.py", line 41, in
main()
File "/home/renz/joeynmt/joeynmt/main.py", line 29, in main
train(cfg_file=args.config_path)

File "/home/renz/joeynmt/joeynmt/training.py", line 653, in train
trainer.train_and_validate(train_data=train_data, valid_data=dev_data)

File "/home/renz/joeynmt/joeynmt/training.py", line 378, in train_and_validate
batch_type=self.eval_batch_type

File "/home/renz/joeynmt/joeynmt/prediction.py", line 88, in validate_on_data
for valid_batch in iter(valid_iter):
File "/opt/conda/lib/python3.7/site-packages/torchtext/data/iterator.py", line 156, in iter
yield Batch(minibatch, self.dataset, self.device)

File "/opt/conda/lib/python3.7/site-packages/torchtext/data/batch.py", line 34, in init
setattr(self, name, field.process(batch, device=device))

File "/opt/conda/lib/python3.7/site-packages/torchtext/data/field.py", line 236, in process
padded = self.pad(batch)

File "/opt/conda/lib/python3.7/site-packages/torchtext/data/field.py", line 254, in pad
max_len = max(len(x) for x in minibatch)
ValueError: max() arg is an empty sequence

Config file used
data:

level: bpe

max_sent_length: 80

...

training:

random_seed: 42

optimizer: "adam"

normalization: "tokens"

adam_betas: [0.9, 0.999]

scheduling: "plateau"

patience: 5

decrease_factor: 0.7

loss: "crossentropy"

learning_rate: 0.0003

learning_rate_min: 0.00000001

weight_decay: 0.0

label_smoothing: 0.1

batch_size: 512

batch_type: "token"

eval_batch_size: 256

eval_batch_type: "token"

batch_multiplier: 1

early_stopping_metric: "ppl
"
epochs: 100

validation_freq: 4000

logging_freq: 100

eval_metric: "bleu"

model_dir: "models/one2many"

overwrite: False
shuffle: True

use_cuda: True

max_output_length: 100

print_valid_sents: [0, 1, 2, 3]

keep_last_ckpts: 3

model:
initializer: "xavier"

bias_initializer: "zeros"

init_gain: 1.0

embed_initializer: "xavier"

embed_init_gain: 1.0

tied_embeddings: True

tied_softmax: True

encoder:
type: "transformer"

num_layers: 6

num_heads: 8

embeddings:

embedding_dim: 512

scale: True

dropout: 0.

hidden_size: 512

ff_size: 2048

dropout: 0.1

decoder:

type: "transformer"

num_layers: 6

num_heads: 8

embeddings:

embedding_dim: 512

scale: True

dropout: 0.

hidden_size:
512

ff_size: 2048

dropout: 0.1

I tried to use different batch sizes for the tokens e.g. 4096, 2048, 1028, etc. but i keep on encountering the same error. I checked the dataset I used and it has been properly preprocessed according to the Sockeye paper so I am not sure where's the error is coming from.

Amazing tool! Really usefull for fast testing / prototyping. I use it to test different RNN cell's. Thanx!

I played with both the iwslt-envi-luong and the iwslt-envi-xnmt predefined configs and get the results as reported. But, when i change the rnn_type in both encoder/decoder in both configs from LSTM to GRU, a strange things happen. The iwslt-envi-xnmt still learns well, no problem.. But the iwslt-envi-luong fails to learn at all.

I've included the validations.txt files for both LSTM and GRU

What am i missing? Is there any explanation for this behaviour? Is it due to the luong attention mechanism/internals/implementation only working with LSTM?

validations-gru.txt
validations-lstm.txt

Hi @juliakreutzer,

I wanted to save ckpts at every validation step regardless of early-stopping-metric score, so I set keep_last_ckpts = -1, according to the description here:

But joeynmt didn't save ckpts at all in that case. Actually, the TrainManager doesn't call _save_checkpoint() func if keep_last_ckpts is less than or equal to zero (queue with infinite length: https://docs.python.org/3/library/queue.html).

What is the expected behavior? You indeed intended no save action if keep_last_ckpts = -1, that is, the description in config was wrong or can we change the code so that ckpts will be saved every time if keep_last_ckpts = -1?

Describe the bug
File translation results are different from interactive translation outputs.

To Reproduce
Steps to reproduce the behavior:

I used the following command for generating file translations:

python3 -m joeynmt translate configs/small.yaml < my_input.txt > out.

To generate translations the interactive way:

python3 -m joeynmt translate configs/small.yaml

The interactive translation outputs are better than the file translation outputs.

Describe the bug
A clear and concise description of what the bug is.
While running the testing, it throws an error like below saying that it is missing a required argument 'batch_class'

! cd joeynmt; python3 -m joeynmt test "$gdrive_path/models/${src}${tgt}_transformer/config.yaml"

2021-01-16 15:01:14,414 - INFO - root - Hello! This is Joey-NMT (version 1.0).
2021-01-16 15:01:14,418 - INFO - joeynmt.data - building vocabulary...
2021-01-16 15:01:26,060 - INFO - joeynmt.data - loading dev data...
2021-01-16 15:01:26,089 - INFO - joeynmt.data - loading test data...
2021-01-16 15:01:26,102 - INFO - joeynmt.data - data loaded.
2021-01-16 15:01:26,121 - INFO - joeynmt.prediction - Process device: cuda, n_gpu: 1, batch_size per device: 3600
2021-01-16 15:01:32,448 - INFO - joeynmt.prediction - Decoding on dev set (data/uzen/dev.bpe.en)...
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/content/joeynmt/joeynmt/main.py", line 41, in
main()
File "/content/joeynmt/joeynmt/main.py", line 32, in main
output_path=args.output_path, save_attention=args.save_attention)
File "/content/joeynmt/joeynmt/prediction.py", line 325, in test
bpe_type=bpe_type, sacrebleu=sacrebleu, n_gpu=n_gpu)
TypeError: validate_on_data() missing 1 required positional argument: 'batch_class'

To Reproduce
Steps to reproduce the behavior:

1. Translation task with over 500k examples
2. config file looks like this:

Create the config

config = """
name: "{name}_transformer"

data:
src: "{source_language}"
trg: "{target_language}"
train: "data/{name}/train.bpe"
dev: "data/{name}/dev.bpe"
test: "data/{name}/test.bpe" # change this to data/{name}/test2.bpe so that you can test it on Ted Talks
level: "bpe"
lowercase: False
max_sent_length: 128
src_vocab: "data/{name}/vocab.txt"
trg_vocab: "data/{name}/vocab.txt"

testing:
beam_size: 5
alpha: 1.0
sacrebleu: # sacrebleu options
remove_whitespace: True # remove_whitespace option in sacrebleu.corpus_chrf() function (defalut: True)
tokenize: "none" # tokenize option in sacrebleu.corpus_bleu() function (options include: "none" (use for already tokenized test data), "13a" (default minimal tokenizer), "intl" which mostly does punctuation and unicode, etc)

training:
load_model: "{gdrive_path}/models/{name}_transformer/best.ckpt" # if uncommented, load a pre-trained model from this checkpoint
random_seed: 42
optimizer: "adam"
normalization: "tokens"
adam_betas: [0.9, 0.998]
scheduling: "plateau"
patience: 5
learning_rate_factor: 0.5
learning_rate_warmup: 4000
decrease_factor: 0.7
loss: "crossentropy"
learning_rate: 0.0003
learning_rate_min: 0.00000001
weight_decay: 0.0
label_smoothing: 0.1
batch_size: 4096
batch_type: "token"
eval_batch_size: 3600
eval_batch_type: "token"
batch_multiplier: 8
early_stopping_metric: "ppl"
epochs: 100
validation_freq: 1000
logging_freq: 100
eval_metric: "bleu"
model_dir: "{gdrive_path}/models/{name}_transformer2"
overwrite: True # TODO: Set to True if you want to overwrite possibly existing models.
shuffle: True
use_cuda: True
fp16: True
max_output_length: 128
print_valid_sents: [0, 1, 2, 3]
keep_last_ckpts: 3

model:
initializer: "xavier"
bias_initializer: "zeros"
init_gain: 1.0
embed_initializer: "xavier"
embed_init_gain: 1.0
tied_embeddings: True
tied_softmax: True
encoder:
type: "transformer"
num_layers: 6
num_heads: 4
embeddings:
embedding_dim: 512
scale: True
dropout: 0.2
# typically ff_size = 4 x hidden_size
hidden_size: 512
ff_size: 2048
dropout: 0.3
decoder:
type: "transformer"
num_layers: 6
num_heads: 4
embeddings:
embedding_dim: 512
scale: True
dropout: 0.2
# typically ff_size = 4 x hidden_size
hidden_size: 512
ff_size: 2048
dropout: 0.3
"""

System (please complete the following information):

• Google Colab
• Python 3.6

Additional context
Installing from joeynmt from pip

Joey NMT framework is developed for educational purposes.

It would be great to extend joeynmt with more Tensorboard metrics and diagrams to gain better insights on the internals of the encoder/decoders.

• histograms of weights/bias per layer per timestamp
• histograms of gradients or gradient_norm per step
  Etc.

Visualisation helps a lot in understanding what's going on and how the networks learns or fails to learn in a particular setup.

Make clear in documentation and small.yaml that encoder (and decoder?) embedding section takes a separate dropout argument and defaults to encoder dropout argument if missing.

Will send a pull request myself eventually and just document all issues I find along the way of setting joey up for me.

Python 3.7 ; Windows ;

I get an error when trying to train a network on deen_bpe downloaded dataset task on a Windows machine.

Error
UnicodeEncodeError: 'charmap' codec can't encode character '\u201f' in position 0: character maps to

Suggested solution
In vocabulary.py the with open(file, "r") and with open(file,"w") lacks the encoding="utf-8" argument.

Currently the code assumes eval_metric to be maximized, but this might not always be the case
(e.g. if we add TER).

This enhances PR #4

We need to include tests (urgently)! Most important:

• unit tests for all modules
  • encoder
  • embeddings
  • decoder
  • attention
  • data creation
  • batch creation
  • data iterators
  • beam search
    • alpha values
  • greedy decoding
• tiny benchmarks for quality checks
• style checks, e.g. with pylint & pep8
• shape checks for
  • encoder input
  • decoder input
  • attention

We can take inspiration from the tests in Sockeye, OpenNMT, or Neuralmonkey or any other NMT toolkit.

Hi,
I have a question about the init_hidden in decoder.
I can understand "only feed the final state of the top-most layer to the decoder".
But, the hidden state and cell state of LSTM have different meaning ?
So is it appropriate to initialize the decoder with " (h, h) if isinstance(self.rnn, nn.LSTM) " ?

Currently the checkpoints have no way to continue training from the same point in the dataset that the checkpoint left off at. Instead, the dataset is reset to how it was at the beginning of training using the same state given by the random seed.

This causes the following training progressions when training for two epochs and interrupting training after the first epoch:

So the interrupted training plus the continued training causes the model to see the same batches in the same order twice. If the training continued from the checkpoint is allowed to continue for a third epoch, it will then see what the uninterrupted training saw during the second epoch. In order to make sure uninterrupted training is comparable to continuing from checkpoint, it would be good if the checkpoint started and saw batches [5, 6, 7, 8].

Additionally, if the config file specifies two epochs, uninterrupted training will see exactly two epochs, but if you see one epoch before interrupting and continue from that checkpoint, the training will forget that it already saw one epoch and attempt to train on two more epochs instead of one more.

The training iterator also has a state_dict that can be saved and restored, which we should add to the checkpoints.

Describe the bug
init_state for Decoder in case of LSTM rnn, should take context and hidden tensor. However, init_hidden returns (hidden, hidden). Shouldn't we change this to (hidden, context) ?

Reference: https://github.com/joeynmt/joeynmt/blob/master/joeynmt/decoders.py#L422

Does joyenmt support pre-trained wordembedding like the one in OpenNMT-py?

Thanks.

Describe the bug
I want to train in debug mode in PyCharm (or VS code) to primarily inspect the structure of training data. How do I do this?

To Reproduce

I added ../configs/small.yaml as argument for file training.py before running training.py in debug mode. My data is stored in the required directory, and the data is in pickle format, so I need to load it first. I get an error saying the file can't be found, which makes sense because the training.py can't see ./data/training_file.pkl.
This is line 625 in training.py.

 # load the data
 train_data, dev_data, test_data, src_vocab, trg_vocab = load_data(
               data_cfg=cfg["data"])

I am trying to replicate the results presented in the paper "Correct Me If You Can". I am using Google Colab. I have already cloned the Joey NMT repo as well as Downloaded the model into the same location. I notice that for the toy models a few shell scripts are present to evaluate the toy models. However, I am unclear how I can evaluate the WMT17 en-de model. Can someone please help?

Hello, I'm testing JoeyNMT to see if it works for my translation studies master's thesis (right now looking at different toolkits with which to traini several patent translation models in EN-JP)

Describe the bug

When trying to run training on with batch_multiplier set to 4 instead of 1 training fails before starting.

Using the same settings, but with "batch_multiplier 1" starts training.

To Reproduce
Steps to reproduce the behavior:

1. Task description: Training Patent Transformer NMT on trainingcorpus with approx 680.000 lines in both src and tgt.

2. configuration file [ntc7otrans.yaml]

name: "transformer"

data:
src: "jp.tk"
trg: "en.tk"
train: "/home/chris/NMT/data/ntc7o/traino"
dev: "/home/chris/NMT/data/ntc7o/val"
test: "/home/chris/NMT/data/ntc7o/test"
level: "word"
lowercase: False
max_sent_length: 100
src_vocab: "/home/chris/NMT/data/ntc7o/joeyNMT/vocabo.jp.tk"
trg_vocab: "/home/chris/NMT/data/ntc7o/joeyNMT/vocabo.en.tk"

testing:
beam_size: 5
alpha: 1.0

training:
random_seed: 3435
optimizer: "adam"
normalization: "tokens"
adam_betas: [0.9, 0.999]
scheduling: "plateau"
patience: 8
decrease_factor: 0.7
loss: "crossentropy"
learning_rate: 0.0002
learning_rate_min: 0.00000001
weight_decay: 0.0
label_smoothing: 0.1
batch_size: 512
batch_type: "token"
eval_batch_size: 32
eval_batch_type: "token"
batch_multiplier: 4
early_stopping_metric: "ppl"
epochs: 100
validation_freq: 5000
logging_freq: 100
eval_metric: "bleu"
model_dir: "models/ntc7otrans"
overwrite: True
shuffle: True
use_cuda: True
max_output_length: 100
print_valid_sents: [0, 1, 2, 3]
keep_last_ckpts: 5

model:
initializer: "xavier"
bias_initializer: "zeros"
init_gain: 1.0
embed_initializer: "xavier"
embed_init_gain: 1.0
tied_embeddings: False
tied_softmax: True
encoder:
type: "transformer"
num_layers: 6
num_heads: 8
embeddings:
embedding_dim: 512
scale: True
dropout: 0.
# typically ff_size = 4 x hidden_size
hidden_size: 512
ff_size: 2048
dropout: 0.1
decoder:
type: "transformer"
num_layers: 6
num_heads: 8
embeddings:
embedding_dim: 512
scale: True
dropout: 0.
# typically ff_size = 4 x hidden_size
hidden_size: 512
ff_size: 2048
dropout: 0.1

3. code call [python3 -m joeynmt train configs/ntc7otrans.yaml]

Logged output

2020-03-27 13:49:27,311 Data set sizes:
train 682849,
valid 914,
test 899
2020-03-27 13:49:27,311 First training example:
[SRC] そして 、 上記 関係 を 少なくとも 10 万 枚 通 紙 し て も 維持 し なけれ ば なら ない 。
[TRG] This relation must be maintained even after passing at least 100,000 sheets .
2020-03-27 13:49:27,312 First 10 words (src): (0) (1) (2) (3) (4) 、 (5) の (6) に (7) １ (8) を (9) 。
2020-03-27 13:49:27,312 First 10 words (trg): (0) (1) (2) (3) (4) the (5) , (6) . (7) of (8) a (9) is
2020-03-27 13:49:27,312 Number of Src words (types): 67918
2020-03-27 13:49:27,312 Number of Trg words (types): 89223
2020-03-27 13:49:27,313 Model(
encoder=TransformerEncoder(num_layers=6, num_heads=8),
decoder=TransformerDecoder(num_layers=6, num_heads=8),
src_embed=Embeddings(embedding_dim=512, vocab_size=67918),
trg_embed=Embeddings(embedding_dim=512, vocab_size=89223))
2020-03-27 13:49:27,378 EPOCH 1
Traceback (most recent call last):
File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/chris/NMT/joeyNMT/joeynmt/main.py", line 41, in
main()
File "/home/chris/NMT/joeyNMT/joeynmt/main.py", line 29, in main
train(cfg_file=args.config_path)
File "/home/chris/NMT/joeyNMT/joeynmt/training.py", line 650, in train
trainer.train_and_validate(train_data=train_data, valid_data=dev_data)
File "/home/chris/NMT/joeyNMT/joeynmt/training.py", line 317, in train_and_validate
if self.batch_multiplier > 1 and i == len(train_iter) -
File "/home/chris/.virtualenvs/joeyNMT/lib/python3.6/site-packages/torchtext/data/iterator.py", line 136, in len
raise NotImplementedError
NotImplementedError

Expected behavior
Training should start and wait for 4 batches to complete before update.

System (please complete the following information):

• OS: Manjaro Linux
  DISTRIB_ID=ManjaroLinux
  DISTRIB_RELEASE=19.0.2
  DISTRIB_CODENAME=Kyria
  DISTRIB_DESCRIPTION="Manjaro Linux"
• CPU / GPU
  CPU: Core i7 2600k
  GPU: RTX 2060 6GB vram
• Python Version
  3.6.6

Additional context
Add any other context about the problem here.

• Add tensorboardX to requirements, and use it to log quantities (e.g. loss, validation BLEU) during training.
• Check if tensorboard shows all logged values correctly.
• Also use it to export all logged values at the end of training to a json file.

If there is no validation during training, the program chooses the checkpoint 0.ckpt for testing. But it does not exist.

Describe the bug
Runtime error while training using bandit-joeynmt fork, acl19 branch.

To Reproduce
Steps to reproduce the behavior:

1. Run the train script with data from bandit-joeynmt/test/data/reverse
2. configuration file: reverse.yaml
3. code call: !python3 training.py /content/drive/My\ Drive/IIITH/regulator/bandit-joeynmt/configs/reverse.yaml

Logged output
Traceback (most recent call last):
File "training.py", line 1256, in
train(cfg_file=args.config)
File "training.py", line 1228, in train
beam_size=beam_size, beam_alpha=beam_alpha)
File "/usr/local/lib/python3.6/dist-packages/joeynmt/prediction.py", line 66, in validate_on_data
max_output_length=max_output_length)
File "/usr/local/lib/python3.6/dist-packages/joeynmt/model.py", line 1379, in run_batch
decoder=self.decoder)
File "/usr/local/lib/python3.6/dist-packages/joeynmt/search.py", line 208, in beam_search
[alive_seq.index_select(0, select_indices),
RuntimeError: index_select(): Expected dtype int64 for index

Additional Information
select_indices parameter in the call to index_select() is a tensor of float values, but a tensor of integers is expected.

System - Using Google Colab:

• GPU
• Python Version: 3.6.9

I've been using the transformer_small.yaml configuration to train a model.
During training the validate_on_data() method takes 5 times longer than in 'test' mode. I did adapt the test mode code a bit to load all the lines from a file and batch them equally as in training mode.
I can't find a good explanation for it since i'm using the same config file and same validation data.

Data set sizes:
train 90727,
valid 926,
test 926

Expected behavior
I would suspect it to take more or less the same time since the metric calculation is only done after that method.

System:

• Ubuntu 18
• CPU
• python 3.7.4

As my knowledge about transformers is rather limited i was hoping someone had some insight into this.
Thank you for this really nice code base!

I am wondering, must be some positive result after use python3 -m joeynmt train configs/small.yaml

In my case, I have only something like:
2019-07-23 10:00:47,923 Example #0 2019-07-23 10:00:47,923 Source: ich freue mich , dass ich da bin . 2019-07-23 10:00:47,923 Reference: i’m happy to be here . 2019-07-23 10:00:47,924 Hypothesis: <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> 2019-07-23 10:00:47,924 Example #1 2019-07-23 10:00:47,924 Source: ja , guten tag . 2019-07-23 10:00:47,924 Reference: yes , hello . 2019-07-23 10:00:47,924 Hypothesis: <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> 2019-07-23 10:00:47,924 Example #2 2019-07-23 10:00:47,924 Source: ja , also , was soll biohacking sein ? 2019-07-23 10:00:47,925 Reference: yes , so , what is biohacking ? 2019-07-23 10:00:47,925 Hypothesis: <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk> <unk>

I commented

src_voc_limit: trg_voc_limit:

as well changed

epochs: 1000.

But it don't help.
As well files my_model/valid out.dev out.test empty?

PyTorch 1.1 supports Tensorboard and we can use it instead of tensorboardX.

https://pytorch.org/docs/stable/tensorboard.html?highlight=tensorboard

Describe the bug
When I execute "python3 -m joeynmt translate hoge", a following error message is displayed.

Error message:
RuntimeError: Integer division of tensors using div or / is no longer supported, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.

To Reproduce
Steps to reproduce the behavior:

1. task description
   I want to train a formality style transfer system from informal to formal on GYAFC Dataset (Entertainment_Music).
2. configuration file
   ezyzip.zip
3. code call
   python3 -m joeynmt translate gyafc/Entertainment_Music/result/config.yaml

Logged output
train.log
validations.txt

Expected behavior
I expect this error message is not displayed and I can execute the code call.

System (please complete the following information):

• OS: mac OS Catalina
• CPU
• Python Version 3.7.6

For more transparent and flexible data handling, batching and vocabulary building.

The goal would be to be as efficient as torchtext, but have more flexibility for the following use cases:

• loading data without the target side (e.g. for inference)
• an interactive mode where the data cannot be loaded from a file
• shared vocabularies, e.g. for source and target side, for multiple encoders or decoders
• for multi-tasking where more than one data set is used

Certain learning rate schedulers expect to be called at every training step, such as ExponentialLR.
Others expect to be called at every evaluation with the evaluation result.
There is currently no logic for this distinction.

Describe the bug
Environment setup breaks when trying to run training code

To Reproduce

conda create --name joey python=3.8
conda activate joey
pip install git+https://github.com/joeynmt/joeynmt.git

The trainer uses tf.io, so let's try to call it:

The error:

AttributeError: module 'tensorflow' has no attribute 'io'

Expected behavior
No error

System (please complete the following information):

• OS: CentOS 7
• CPU / GPU: CPU
• Python Version: 3.8.6 (also tried 3.7)

Additional context
I'm not using Joey's CLI, but programmatically with python

On a multicore machine, translation can potentially use multithreading for a speedup.

Does Joey respect OMP_NUM_THREADS?

I am testing on a n1-standard-4 GCP instance and running a command such as

CUDA_VISIBLE_DEVICES="" OMP_NUM_THREADS=4 python -m joeynmt translate \
   $configs/transformer_wmt17_ende.yaml < $data/test.bpe.$src > \
   $translations/test.bpe.$model_name.$trg

CPU utilization is constantly at 200% - do you know why that is and if it could be higher?

Thanks so much for your help
Mathias

When configuring a model with eval_metric: 'sequence_accuracy' and early_stopping_metric: 'eval_metric' in the training section, Joey thinks a small sequence accuracy is desirable and will (usually) only save a checkpoint only at the first validation time since the accuracy is still small at this time and will grow with training the model further.

This is due to the following code in training.py:

        # if we schedule after BLEU/chrf, we want to maximize it, else minimize
        # early_stopping_metric decides on how to find the early stopping point:
        # ckpts are written when there's a new high/low score for this metric
        if self.early_stopping_metric in ["ppl", "loss"]:
            self.minimize_metric = True
        elif self.early_stopping_metric == "eval_metric":
            if self.eval_metric in ["bleu", "chrf"]:
                self.minimize_metric = False
            # eval metric that has to get minimized (not yet implemented)
            else:
                self.minimize_metric = True
        else:
            raise ConfigurationError(
                "Invalid setting for 'early_stopping_metric', "
                "valid options: 'loss', 'ppl', 'eval_metric'.")

By the way, actual early stopping (i.e. training begin stopped if the patience is exhausted) is not implemented yet, right? Is this an open TODO?

Rather than using .cuda() calls, it would be cleaner to use .to(device) and to specify a device.

Does joeynmt support multi-gpu training? After reading the source-code, I've found none code distributing graphs in different devices. I might be wrong. Thanks for the work!

Describe the bug
The model ran perfectly with some of my dataset. However, for one of the datasets, the following bug appears. The log is as below.

Logged output
2020-05-24 18:50:40,236 EPOCH 1 2020-05-24 18:50:40,706 Epoch 1 Step: 10 Batch Loss: 129.308456 Tokens per Sec: 6443, Lr: 0.000014 2020-05-24 18:50:41,086 Epoch 1 Step: 20 Batch Loss: 88.039078 Tokens per Sec: 8751, Lr: 0.000028 2020-05-24 18:50:41,476 Epoch 1 Step: 30 Batch Loss: 94.100029 Tokens per Sec: 7615, Lr: 0.000042 2020-05-24 18:50:41,863 Epoch 1 Step: 40 Batch Loss: 128.856537 Tokens per Sec: 7575, Lr: 0.000056 2020-05-24 18:50:42,255 Epoch 1 Step: 50 Batch Loss: 114.877518 Tokens per Sec: 8100, Lr: 0.000070 2020-05-24 18:50:42,737 Epoch 1 Step: 60 Batch Loss: 85.376068 Tokens per Sec: 8259, Lr: 0.000084 2020-05-24 18:50:43,267 Epoch 1 Step: 70 Batch Loss: 298.312286 Tokens per Sec: 5568, Lr: 0.000098 2020-05-24 18:50:43,812 Epoch 1 Step: 80 Batch Loss: 101.327324 Tokens per Sec: 6123, Lr: 0.000112 2020-05-24 18:50:44,323 Epoch 1 Step: 90 Batch Loss: 172.604233 Tokens per Sec: 6887, Lr: 0.000126 2020-05-24 18:50:44,822 Epoch 1 Step: 100 Batch Loss: 93.117615 Tokens per Sec: 5750, Lr: 0.000140 Traceback (most recent call last): File "/home/xingyuaz/anaconda3/envs/my_fisrt_env/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/home/xingyuaz/anaconda3/envs/my_fisrt_env/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/xingyuaz/test_joeynmt/joeynmt/joeynmt/__main__.py", line 41, in <module> main() File "/home/xingyuaz/test_joeynmt/joeynmt/joeynmt/__main__.py", line 29, in main train(cfg_file=args.config_path) File "/home/xingyuaz/test_joeynmt/joeynmt/joeynmt/training.py", line 650, in train trainer.train_and_validate(train_data=train_data, valid_data=dev_data) File "/home/xingyuaz/test_joeynmt/joeynmt/joeynmt/training.py", line 375, in train_and_validate batch_type=self.eval_batch_type File "/home/xingyuaz/test_joeynmt/joeynmt/joeynmt/prediction.py", line 98, in validate_on_data batch, loss_function=loss_function) File "/home/xingyuaz/test_joeynmt/joeynmt/joeynmt/model.py", line 133, in get_loss_for_batch trg_mask=batch.trg_mask) File "/home/xingyuaz/test_joeynmt/joeynmt/joeynmt/model.py", line 74, in forward src_mask=src_mask) File "/home/xingyuaz/test_joeynmt/joeynmt/joeynmt/model.py", line 92, in encode return self.encoder(self.src_embed(src), src_length, src_mask) File "/home/xingyuaz/anaconda3/envs/my_fisrt_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/xingyuaz/test_joeynmt/joeynmt/joeynmt/encoders.py", line 216, in forward x = self.pe(embed_src) # add position encoding to word embeddings File "/home/xingyuaz/anaconda3/envs/my_fisrt_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/xingyuaz/test_joeynmt/joeynmt/joeynmt/transformer_layers.py", line 159, in forward return emb + self.pe[:, :emb.size(1)] RuntimeError: The size of tensor a (9098) must match the size of tensor b (5000) at non-singleton dimension 1

Hi,

I am trying to run joeynmt model by this config file (transformer_iwslt14_deen_bpe.yaml). I ran sucessfull but the blue score is very low like 23.18. I supposed that might be my dataset is small thats why I am getting low blue score. I have few questions regarding this. I am very new user and dont have much experience.

1. what should be the size of train, valid and test file?.
2. I already have much rawdata for train file but from where i can download data for valid.
3. should I use same parameter that are give by joenmt in a config file or can i change parameter like batch_size, logging_freq, validation_freq, num_layers and num_heads.
4. is it above parameters are according to the size of data.

Also, currently the size of data that I am using is following after tokenization and bpe fomatting:
train: 769107 KB
valid: 120,705 kB
test: 441 KB

Thanks

Describe the bug
Available GPUS: 4 Tesla V100-SXM2 with 15.78GB RAM each. JoeyNMT sees it:
2020-08-25 21:36:56.896921: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0, 1, 2, 3

During self.forward CUDA out of memory pops up:
RuntimeError: CUDA out of memory. Tried to allocate 21.71 GiB (GPU 0; 15.78 GiB total capacity; 4.20 GiB already allocated; 10.50 GiB free; 4.21 GiB reserved in total by PyTorch)

How does JoeyNMT utilizes availability of multiple GPUs?

System (please complete the following information):

• OS: Ubuntu 18.04
• CPU / GPU
• Python Version: 3.6.9
• CUDA 10.1

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64       Driver Version: 440.64       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  Off  | 00000000:04:00.0 Off |                    0 |
| N/A   33C    P0    39W / 300W |      0MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  Off  | 00000000:05:00.0 Off |                    0 |
| N/A   30C    P0    38W / 300W |      0MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  Off  | 00000000:09:00.0 Off |                    0 |
| N/A   29C    P0    39W / 300W |      0MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  Off  | 00000000:89:00.0 Off |                    0 |
| N/A   28C    P0    38W / 300W |      0MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

The link that the script get_iwslt14_bpe.sh uses to download the data is broken and now leads to a 404 page. They moved the hosting of the data to a google drive, which I don't think you can access with curl. Additionally, the tgz file on google drive is all languages so some of the logic will need to be re-written. Since I need to use this data, I'll try to write a new script for the iwslt14 data.

At the moment, the "test" mode loads the training set as well.

Hi there,

I have tried loading a previous checkpoint and continuing my training but when I do this the training goes on without saving the checkpoints. It does the validation at every specified step but never saves the model in the directory. Is there a specific command in the config I should specify about continued training?
Thanks so much!

Following The Annotated Transformer