How to save the weights learned by MorphNet about morph-net HOT 7 CLOSED

For retraining, we re-initialize from scratch for simplicity and find that is does not matter. You can try an experiment to reuse the weights during structure learning. Just be careful to copy the tensors correctly as the TF graph will be different (if you resize the convolutions).

from morph-net.

The weights trained during the MorphNet structure learning phase are not intended for use in the final inference. The retraining is (at present) a required step to achieve good performance from the pruned model.

Of course, you can still examine the weights trained during the structure learning phase (for example, to analyze them, or to come up with your own extensions to MorphNet). They are available as checkpoints in the Tensorflow training directory, where the trainer (usually) saves them. This is no different from how you'd save/restore weights without MorphNet.

from morph-net.

@pkch Thank you for your reply. The weights trained during the MorphNet structure learning phase has been also well optimized. So that may be used as initialization for retraining the new structure.

from morph-net.

You could try to reuse the weight, we did not have a very positive experience with that, but it could work for you. In addition there is some research that suggest that training from scratch is actually more useful.

I would also would like to point out that depending on the model architecture it could be a bit tedious to a new architecture.

from morph-net.

Just to add to the previous comments, reusing the previous weights can be tricky because the network structure has changed. Thus, the shapes of the weight tensors need to change as well. For example, if you remove a channel from a convolution, then that filter needs to be removed from the weights, AND the convolutions consuming the output of the first convolution need to have weights removed as well because one of the inputs has been removed. In theory this is possible, but it is hard to implement.

from morph-net.

When retraining, are you supposed to use the same initialization, or does it not matter?

from morph-net.

I have a general question for Resnet V1 50 on ImageNet dataset.

After stage 1, let's say I take the alive_1000 JSON file after training step 1000, figure out the best activation channels needed and update the model.py.
It is suggested from this discussion as to retrain from scratch. What hyper-parameters should be used for this case? Has anyone tried to retrain a new slimmed version of Resnet model using TF slim library?

I tried to run a few experiments with different hyper-parameters, seems like its difficult to converge. Hoping someone has achieved convergence for this model.

from morph-net.