model_best VS. Training from Scratch about normalized-autoencoders HOT 5 CLOSED

OMG I missed this thread. Sorry for the late reply.

For the last figure you attached, it is technically okay.

I think the absolute scale of energy depends on various hyperparameters and may be subjected to change.

I do have observed the red-like and the blue-like curves during my experiments. Still quite mysterious.

Please let me know if you have figured out anything regarding the issue.

Best,
Sangwoong.

from normalized-autoencoders.

Dear @GloryyrolG ,

Yes, NAE training does not need to start from the best epoch. You can surely use an early-stopped checkpoint, or any other checkpoint.

However, in practice, the value of ae_epoch does affect the quality of learning. It seems that there's a sweet spot.

• If ae_epoch is too small, the initial manifold learned by the autoencoder is very good, and on-manifold initialization becomes less effective (my guess).
• On the other hand, if ae_epoch is too large, then the norms of the weights of the autoencoder become large. This seems to make the autoencoder more "rigid" and hinders NAE training.

By the way, from the figure you have attached, what do the red and the blue lines indicate?

Always thank you for your questions.

from normalized-autoencoders.

Hi SW @swyoon ,

Thanks for your instant reply.

Q: What do the red and the blue lines indicate in the last figure?
A: The red one is resuming from model_best.pth and ae_epoch = 0 while the blue one is training from scratch, i.e., ae_epoch = 30.

Btw, I have some follow-up questions.

Q: So it seems model_best.pth is the best final NAE, right? Not the best AE checkpoint.

Q: May I confirm why a small ae_epoch leads to a good model manifold? To me, if the pre-trained AE is not trained sufficiently, it seems less helpful to the following NAE training?

Regards,

from normalized-autoencoders.

Dear @GloryyrolG ,

model_best.pth is supposed to be the autoencoder with the smallest validation reconstruction error, but in the current setting, that is happened to be the last autoencoder, because I set a moderately large ae_epoch.

I didn't mean that a small ae_epoch results in a good model manifold. I was gonna say that you should avoid too small or too large ae_epoch.

Best,

from normalized-autoencoders.

So may I confirm like, the variations of the energy the model reaches finally are okay in the last figure? @swyoon

from normalized-autoencoders.