Hi Biao,

is there a convenient way to reproduce Table 1 from your paper "Improving Deep Transformer with Depth-Scaled Initialization and Merged Attention"?

Thanks a lot!

Cheers,
Stephan

Hi,

I'm trying to reproduce the results of your ACL 2020 paper "Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation".
I follow the instruction here
https://github.com/bzhangGo/zero/tree/master/docs/multilingual_laln_lalt

I use example_train.sh example_evaluation.sh and My experiments are conducted on the one-to-many data.

However, I find my reproduced results are far from what you reported in your paper, about 10 bleu score lower.

Do you have any idea what causes this drop? Thanks.

Here is my training script.

To reproduce the result in Share or Not? Learning to Schedule Language-Specific Capacity for Multilingual Translation (Zhang et al., ICLR2021), environment are required as :

• python==2.7
• tensorflow==1.13.1
  but the tensorflow 1.13.1 simply can not be imported for python 2.7

Hi I tried the steps presented here completely until I train the model with the command
python zero/config.py --mode train --parameters=hidden_size=1024,embed_size=512,dropout=0.1,label_smooth=0.1,max_len=80,batch_size=80,eval_batch_size=240,token_size=3000,batch_or_token='token',model_name="rnnsearch",buffer_size=3200,clip_grad_norm=5.0,lrate=5e-4,epoches=25,update_cycle=8,gpus=[8],disp_freq=100,eval_freq=10000,sample_freq=1000,checkpoints=5,caencoder=True,cell='atr',max_training_steps=100000000,nthreads=8,swap_memory=True,layer_norm=True,max_queue_size=100,random_seed=1234,src_vocab_file="$data_dir/vocab.en",tgt_vocab_file="$data_dir/vocab.de",src_train_file="$data_dir/train.40k.en",tgt_train_file="$data_dir/train.40k.de",src_dev_file="$data_dir/dev.40k.en",tgt_dev_file="$data_dir/dev.40k.de",src_test_file="",tgt_test_file="",output_dir="train",test_output=""

But I get the following error and training doesn't start

Traceback (most recent call last):
  File "zero/config.py", line 223, in <module>
    tf.app.run()
  File "/miniconda3/envs/myenv_s_m/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/miniconda3/envs/myenv_s_m/lib/python3.6/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/miniconda3/envs/myenv_s_m/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "zero/config.py", line 215, in main
    graph.train(params)
  File "/Sugeeth/translation_from_atr/zero/main.py", line 108, in train
    loss, gradients = tower_train_graph(train_features, optimizer, params)
  File "/Sugeeth/translation_from_atr/zero/main.py", line 35, in tower_train_graph
    params.gpus, use_cpu=(len(params.gpus) == 0))
  File "/Sugeeth/translation_from_atr/zero/utils/parallel.py", line 179, in parallel_model
    features, device_mask = shard_features(features, num_devices)
  File "/Sugeeth/translation_from_atr/zero/utils/parallel.py", line 132, in shard_features
    pieces = util.uniform_splits(tf.shape(features.values()[0])[0],
TypeError: 'dict_values' object does not support indexing

Did I miss something ?

I'm having trouble finding the Google Drive URL for pretrained models, and it appears that there isn't any preprocessed data available in https://data.statmt.org/bzhang/iclr2021_share_or_not/. Is there another way to download them?

I followed the README guide of multilingual_laln_lalt. And I download the provided sentencepiece model to encode the corpus. After that I try to build the vocab.zero.src and vocab.zero.tgt, but the output vocab size is 200k+ ? And I try to train a new sentencepiece model via the scripts but the BLEU score is lower than the result in the paper.

Hi,

The mask implementation for aan seems to be different to what you describe in the paper
https://arxiv.org/pdf/1805.00631.pdf?
Specifically in Figure 3.

Based on Figure 3, I would think the mask computation for sequence length L would be as below:

mask = tf.matrix_band_part(tf.ones([L, L]), -1, 0)
factor = tf.reciprocal(tf.cumsum(tf.ones_like(mask)))
masked_mask = factor * mask
masked_mask = tf.expand_dims(masked_mask, 0)

The mask computation you did at the following location seems to completely different, in that there is a gate and softmax computation.

Hi

I'm using your AFS ST model as a baseline for my thesis project. I'm trying to run the speech preprocessing script, but the tensorflow.contrib package has been deprecated in tensorflow 2.0. Any suggestions on how to adapt your code?

Thank you!