Why initialize weights from Uniform[-0.08, 0.08]? about xmunmt HOT 6 CLOSED

random_uniform initializer is commonly used in the NMT literatures. Two very influential works have used the uniform initialization, which we list as follows:

• Sequence to Sequence Learning with Neural Networks
• Google’s Neural Machine Translation System

Unfortunately, there is no theoretical explanations on how to initialize a neural network. Maybe you can read some papers related to VNMT and see how they initialized their networks.

from xmunmt.

Thanks for your quick reply. My question may be misleading.
What I want to know is why 0.08 (how did you get this range), not why uniform. Could you please explain this?

from xmunmt.

We found it work well and no other reasons. If you have any other questions, I suggest you to send emails to me directly.

from xmunmt.

Fair enough.
I apologize if these too many questions bothered you.

from xmunmt.

You're welcome. I just think that using emails are more appropriate for some questions because it is not an issue of the codes. Some questions are really hard to answer.

from xmunmt.

Yep, I see. When you suggest me to send emails, I realized some of my questions are just too trifling and not critical for the whole project, thus are more suitable to be discussed privately.

For example, "Any benchmark result?" is a good issue, because it's more general and constructive and everyone who wants to use this library may have the same question.
But this issue itself is more specific and is about the very last detail, so it is not a good issue. Too many issues like this will overwhelm the more essential and common ones.

from xmunmt.