Reproducing the EN-ZH results in Table 1 about muse HOT 9 CLOSED

I tried adding the refinement stage and it indeed greatly improved the performance. I will also try to reduce the epoch_size for getting better adversarial results.

Thank you so much for all the help. Closing for now.

from muse.

I think it only made a significant difference for en-zh and en-eo

from muse.

Hi,

Can you try to add --normalize_embeddings center and see if that helps?

from muse.

I will try that. Thanks for the prompt reply!

from muse.

BTW, should I add this option to other language pairs as well? Or are there hyperparameters that work better for a subset of languages?

from muse.

Adding --normalize_embeddings center indeed gave non-zero performance on EN-ZH, but I got only 6.9 P@1, which still seems far away from the 23.4 reported in Table 1.
Any other suggestions?

EDIT: My ES-EN, FR-EN, and DE-EN runs (without refinement since I just realized that was turned off by default) also turned out to be about 10% lower than the reported P@1. I was running with the default hyperparameters.

from muse.

Mmm this is maybe because we set a smaller number of adversarial training epochs in the default parameters. The idea is that the accuracy after adversarial training does not matter much, it just has to be high enough for the refinement to work. Are you obtaining the same results after refinement for en-fr and en-zh?

from muse.

No, I just realized that the refinement was turned off by default, and my experiments are still running with refinement turned on.

I attached a training log for my DE-EN run in case you could quickly spot anything that's apparently going wrong. I seem to notice that the performance is getting much worse (or even collapse) after the first epoch, and the best results were always obtained at the end of the first epoch (for all the language pairs I tried).

DE-EN log: https://gist.github.com/ccsasuke/78d975e565ffffe22d1b9af9ab261458

If the adversarial results do not matter much, I will wait for the refinement experiments to finish and report back.

from muse.

Interesting, you obtain the best results after the first epoch. 45% P@1 is good enough to start the refinement training, so you should be fine in the end.

We set the default epoch_size to 1000000, which might be too much. If you are interested in obtaining the best results with adversarial training, I guess you should use a smaller epoch size. Otherwise, if you only care about the post-refinement results (that will be the bests anyway), then this should be fine.

from muse.