Training another student model(s4b2/s4b1) will drop just several epoches about fast_human_pose_estimation_pytorch HOT 6 OPEN

That such accuracy drop is kind of weird. I did not meet such things before.

1. Have you checked the training loss and accuracy? if things go wrong, training loss would become very large.
2. It may relate to bn issue in pytorch. Have you followed readme to "Disable cudnn for batchnorm layer to solve bug in pytorch0.4.0".

from fast_human_pose_estimation_pytorch.

Thanks for your reply!

1. And actually, for s4b1, the validation loss would be very large but at that time training loss is normally I think, the validation loss will be larger after several epochs, and the validation accuracy has been dropped to almost zero already. See below:

Epoch LR Train Loss Val Loss Train Acc Val Acc

1.000000 0.000250 0.002626 0.009073 0.161942 0.291490
2.000000 0.000250 0.002116 0.009197 0.300657 0.397661
3.000000 0.000250 0.001956 0.004193 0.399708 0.420880
4.000000 0.000250 0.001839 0.067800 0.472296 0.016799
5.000000 0.000250 0.001757 9.869279 0.520801 0.000000
6.000000 0.000250 0.001726 678.551853 0.542042 0.007716
7.000000 0.000250 0.001753 3865.202707 0.544255 0.000042
8.000000 0.000250 0.001758 37764.363059 0.538761 0.000000
9.000000 0.000250 0.001917 933.443237 0.410885 0.000068

2. I forgot to do that step you mentioned, I will try to disable cudnn for the bn layer, thanks!

from fast_human_pose_estimation_pytorch.

And one more question, what do num_feature and inplane really influence? looks like they are the channel variables of the first several layers, but do they affect the accuracy or just affect the size of models? Thanks！

from fast_human_pose_estimation_pytorch.

I don't think the network structure has such big impact. The training acc looks good, while validation acc goes to zero. It looks like related to bn layer who behaves differently in train and eval mode. I suspect it has something to do with bn issue I mentioned.

from fast_human_pose_estimation_pytorch.

Thanks for your proposals! it is actually the BN layer eval bug on pytorch0.4.0, I disable the cudnn for batchnorm layer, the val loss and val acc seems normal then.

By the way, do you think if I use s8b1 teacher model train s4b2 student, and use s4b2 as the teacher train s1b2, will get better accuracy for s1b2 than directly use s8b1 teacher model train s1b2 student? That means steps KD. Thanks!

from fast_human_pose_estimation_pytorch.

Your idea looks interesting, but not sure if this performs better or not. KD is a kind of regulariation.

Ensemble(s8b1, s4b2 ..) to teach s1b2
Iterative KD, s8b1->s4b2->s1b2 .

Good luck!

from fast_human_pose_estimation_pytorch.