dennybritz / deeplearning-papernotes Goto Github PK

Hi, Denny! I have been following your notes for a long time since I was a college students.
Hope I can see some new thoughts from you!

Hi Denny,

Thank you for your interest on our reconstruction work. I am the first author of this paper, and try to answer your questions.

• I feel like "adequacy" is a somewhat strange description of what the authors try to optimize. Wouldn't "coverage" be more appropriate?

Adequacy and/or Fluency evaluations are regularly employed for assessing the quality of machine translation. Adequacy measures how much of the meaning expressed in the source is also expressed in the target translation. It is well known that NMT favors fluent but inadequate translations, which have not only the coverage problems (e.g., over-translation and under-translation) but also mis-translated (e.g., wrong sense and unusual usage) and spurious translation (i.e., translation segments without any reference in source) problems.

• In Table 1, why does BLEU score still decrease when length normalization is applied? The authors don't go into detail on this.

As shown in Table 1, likelihood with length normalization favors long translations, which may face over-translation problems.

• The training curves are a bit confusing/missing. I would've liked to see a standard training curve that shows the MLE objective loss and the finetuning with reconstruction objective side-by-side.

We try to show that the increase of translation performance is indeed due to the improvement of reconstruction over time. We care more about the improvement of translation performance, since it is the ultimate goal of NMT.

• The training procedure somewhat confusing. The say "We further train the model for 10 epochs" with reconstruction objective, byt then "we use a trained model at iteration 110k". I'm assuming they do early-stopping at 110k * 80 = 8.8M steps. Again, would've liked to see the loss curves for this, not just BLEU curves.

During the training, we validate the translation performance for every 10K iterations. We select models that yield best performances on the validation set for NMT models. This is a standard procedure to select a well trained model in NMT.

Again, we care more about the ultimate goal of NMT -- translation performance measured in BLEU scores.

• I would've liked to see model performance on more "standard" NMT datasets like EN-FR and EN-DE, etc.

We only test on the Chinese-English translation task, which uses the same data as in our previous work Modeling Coverage for Neural Machine Translation and Context Gates for Neural Machine Translation.

• Is there perhaps a smarter way to do reconstruction iteratively by looking at what's missing from the reconstructed output? Trainig with reconstructor with MLE has some of the same drawbacks as training standard enc-dec with MLE and teacher forcing.

Really good point! There should be a better way to model the reconstruction (e.g., focus only on the wrong part). We will study on this in the future.

MLE favors fluent but inadequate translations, thus is not a optimal metric for NMT. It is necessary to introduce a better objective, such as sentence-level BLEU (Shen et al., 2016), and an auxiliary objective of reconstruction (in this work) or coverage penalty (in GNMT paper).

Could it be useful to have also the code used in the paper, when provided by the authors, next to the paper link? What do you think about a format of this type for the link's variable name: webRepository_[framework/library|Language]_model/packageName?
(e.g. tf-seq2seq link variable will be: [GH_TF_Tf-Seq2Seq] )
Or if you prefer something in the KISS way: a simple code as variable name will be perfect.

These notes would be well suited to shortscience, the goal on that site being to provide concise summaries of papers in markdown.

I have read the paper, and I can not understand how model2 works, you said you thought you understand what it does, so could you please give me more explanations on the cond. model? Thanks very much.

Thanks for this useful summary page. Have you considered providing also links to the relevant GitXiv entries, when available?

If you are not familiar with GitXiv, have a look: http://www.gitxiv.com/

For example, here is the WaveNet entry: http://www.gitxiv.com/posts/W5ax2LzW9Ht3mtTk2/wavenet-a-generative-model-for-raw-audio

Thanks again!