Giter Club home page Giter Club logo

Comments (5)

swyoon avatar swyoon commented on July 30, 2024 1

OMG I missed this thread. Sorry for the late reply.

For the last figure you attached, it is technically okay.

I think the absolute scale of energy depends on various hyperparameters and may be subjected to change.

I do have observed the red-like and the blue-like curves during my experiments. Still quite mysterious.

Please let me know if you have figured out anything regarding the issue.

Best,
Sangwoong.

from normalized-autoencoders.

swyoon avatar swyoon commented on July 30, 2024

Dear @GloryyrolG ,

Yes, NAE training does not need to start from the best epoch. You can surely use an early-stopped checkpoint, or any other checkpoint.

However, in practice, the value of ae_epoch does affect the quality of learning. It seems that there's a sweet spot.

  • If ae_epoch is too small, the initial manifold learned by the autoencoder is very good, and on-manifold initialization becomes less effective (my guess).
  • On the other hand, if ae_epoch is too large, then the norms of the weights of the autoencoder become large. This seems to make the autoencoder more "rigid" and hinders NAE training.

By the way, from the figure you have attached, what do the red and the blue lines indicate?

Always thank you for your questions.

from normalized-autoencoders.

GloryyrolG avatar GloryyrolG commented on July 30, 2024

Hi SW @swyoon ,

Thanks for your instant reply.

Q: What do the red and the blue lines indicate in the last figure?
A: The red one is resuming from model_best.pth and ae_epoch = 0 while the blue one is training from scratch, i.e., ae_epoch = 30.

Btw, I have some follow-up questions.

Q: So it seems model_best.pth is the best final NAE, right? Not the best AE checkpoint.

Q: May I confirm why a small ae_epoch leads to a good model manifold? To me, if the pre-trained AE is not trained sufficiently, it seems less helpful to the following NAE training?

Regards,

from normalized-autoencoders.

swyoon avatar swyoon commented on July 30, 2024

Dear @GloryyrolG ,

model_best.pth is supposed to be the autoencoder with the smallest validation reconstruction error, but in the current setting, that is happened to be the last autoencoder, because I set a moderately large ae_epoch.

I didn't mean that a small ae_epoch results in a good model manifold. I was gonna say that you should avoid too small or too large ae_epoch.

Best,

from normalized-autoencoders.

GloryyrolG avatar GloryyrolG commented on July 30, 2024

So may I confirm like, the variations of the energy the model reaches finally are okay in the last figure? @swyoon

from normalized-autoencoders.

Related Issues (12)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.