loss function in train() about born-again-neural-network HOT 4 OPEN

@xht033 it's supposed to be mse, correct? kl should work, just not quite as well!

from born-again-neural-network.

first, kl_divergence is asymmetrical, and I think it should be kl_divergence(log_softmax(output), softmax(output2)).

from born-again-neural-network.

I found @xht033 was right, just thinking whether this code could reproduce the results.

from born-again-neural-network.

I think the result should be

loss2 = kldivergence(F.log_softmax(outputs, dim=0), F.softmax(outputs2, dim=0))
loss = loss + loss2.item()

from born-again-neural-network.