Selection of teacher about repdistiller HOT 2 CLOSED

Hi, @yaxingwang,

Thanks for you interest.

The short answer is that the student-teacher combinations in this paper are randomly picked. I did not tune student-teacher combination such that our method could perform the best.

Your question actually reveals my initial thoughts of extending previous benchmark. I think most previous methods do not evaluate the performance on variance teacher-student combinations. For instance, Attention Transfer (AT) mainly consider WideResNet, and ResNet, and recent paper Similarity Preserving (SP) mainly consider WideResNet, MobileNet, and ShuffleNet. Additionally, most methods consider the teacher and student models to be of the same architectural type.

So I was trying to test those state-of-the-art methods with randomly picked various student-teacher combinations, and wanted to see how the performance would change. And I also divided these combinations into two groups: (1) of the same architectural type, or (2) of different types. As you may see from the two tables from the paper, some method significantly drops when dealing with group (2).

Lastly, though our CRD is the best among these randomly picked combinations in our paper, I think it's with high probability that it could not consistently be the best in all exhaustive combinations. But in general, it should not be very sensitive to different models, as you can see that we do not require specific architectural inductive bias in our algorithm.

from repdistiller.

Thank you for your reply, @HobbitLong . It is good insight to compare the picked randomly student teacher pair. CRD mine additionally the structural knowledge, which is great complementary for KD.

from repdistiller.