Why using two embedding layers? about pytorch_neg_loss HOT 2 CLOSED

Hi, @nasusu,

word2vec is defined oved model that takes input word, pick ups embedfing for it and then passes it through feed forward network with single layer in order to get some probabilities of some target over various words in corpus.

We can approximate loss of this model with NEG-loss — we multiply input embedding of input word with row of feed-forward network (output embeddding) and input embedding with random output embeddings.

The intuition behind this model is quite simple, if words “cat” and “dog” both occure in the context of word “pet”, then they have something common in their sense. Notice that if you would have one embedding while training, it is unlikely that you would catch this similarity since you would multiply embeddings of “cat” and “dog”, that are less likely to occure in the same context.

Hope I helped you.

from pytorch_neg_loss.

Thank you for your reply！

from pytorch_neg_loss.