The setting of Z_v1 and Z_v2 in class ContrastMemory? about repdistiller HOT 3 CLOSED

Hi, @HaoKun-Li ,

The short answer is that, these two can be viewed as two constants to scale up the dynamic range of the score function.

Typically, NCE can deal with unnormalized distribution and will automatically adjust the score range.
In this specific case, the score is produced by the inner product of two l2-normalized vector, which means it's range is [-1, 1]. This range might not be enough for NCE's score adjustment. So here Z_v1 and Z_v2 are very simple monte carlo estimation of the partition function of full softmax (see sec 2.4 in this paper and sec 3.4 in this paper) to help adjust the score range a bit.

from repdistiller.

Thanks for your reply!

from repdistiller.

@HaoKun-Li , you are welcomed. I just closed it, but feel free to reopen it if you would like to discuss more.

from repdistiller.