An issue on loss function about knowledge-distillation-pytorch HOT 4 OPEN

I was wondering if the multiplication of T square is really helpful? Because if T=20, the soft loss will dominate the total loss. And there is no need to add extra softmax for the hard target as it is already embedded in nn.functional.cross_entropy. @lhyfst

from knowledge-distillation-pytorch.

As @erichhhhho pointed out, it's indeed no need to manually add extra softmax. From the reference paper, it looks like T^2 is only required when using BOTH hard/soft targets.

from knowledge-distillation-pytorch.

Thank you, everybody! So, why does the first part of the KD loss function in distill_mnist.py multiply 2?
https://github.com/peterliht/knowledge-distillation-pytorch/blob/e4c40132fed5a45e39a6ef7a77b15e5d389186f8/mnist/distill_mnist.py#L96-L97

from knowledge-distillation-pytorch.

Thank you, everybody! So, why does the first part of the KD loss function in distill_mnist.py multiply 2?

As per distiller KD_Loss is effectively the following equation:

α * kl_divergence + β * cross_entropy

And Hinton et al. 2015 originally used a weighted average, i.e. α = 1 - β, but this is not strictly necessary. α and β can also be arbitrary and don't need to sum to 1. In this particular MNIST example, the relationship is α = 2 * (1 - β), maybe they were experimenting with a stronger reliance on kl_div.

from knowledge-distillation-pytorch.