[Question] kaiming initialization when pretrained=False about practical-deep-learning-for-coders-2.0 HOT 4 CLOSED

I'm confused about the very last code cell in the notebook, but maybe I'm just overtired:

body = create_timm_body('efficientnet_b3a', pretrained=False)
head = create_head(3072, dls.c)
model = nn.Sequential(body, head)
apply_init(model[1], nn.init.kaiming_normal_)
learn = Learner(dls, model, loss_func=LabelSmoothingCrossEntropy(), 
                splitter=default_split, metrics=accuracy)
learn.freeze()
learn.fit_one_cycle(5, 3e-3)

I would think here since the net is being loaded with pretrained=False, that you would use apply_init(model, nn.init.kaiming_normal_) and not freeze the network. I could be missing something though, just trying to check my understanding.

from practical-deep-learning-for-coders-2.0.

Aha! Totally my fault, my bad :) yes you are right. We probably should be initializing the whole thing there, not just the head. (Along with not freezing) I can try to get to it here in the next few days, but a PR would be more than welcome 😊

from practical-deep-learning-for-coders-2.0.

Yes, but we also are using pretrained weights there so it doesn’t matter in the long run (notice we load old weights in), as we don’t train with the uninitialized body, we instead use the body from our other model

from practical-deep-learning-for-coders-2.0.

No problem, it took me a while to realize while playing with a very non-imagenet-like dataset :) Just glad I was understanding correctly! Will try to make a PR tomorrow.

from practical-deep-learning-for-coders-2.0.