Comments (5)
OMG I missed this thread. Sorry for the late reply.
For the last figure you attached, it is technically okay.
I think the absolute scale of energy depends on various hyperparameters and may be subjected to change.
I do have observed the red-like and the blue-like curves during my experiments. Still quite mysterious.
Please let me know if you have figured out anything regarding the issue.
Best,
Sangwoong.
from normalized-autoencoders.
Dear @GloryyrolG ,
Yes, NAE training does not need to start from the best epoch. You can surely use an early-stopped checkpoint, or any other checkpoint.
However, in practice, the value of ae_epoch
does affect the quality of learning. It seems that there's a sweet spot.
- If
ae_epoch
is too small, the initial manifold learned by the autoencoder is very good, and on-manifold initialization becomes less effective (my guess). - On the other hand, if
ae_epoch
is too large, then the norms of the weights of the autoencoder become large. This seems to make the autoencoder more "rigid" and hinders NAE training.
By the way, from the figure you have attached, what do the red and the blue lines indicate?
Always thank you for your questions.
from normalized-autoencoders.
Hi SW @swyoon ,
Thanks for your instant reply.
Q: What do the red and the blue lines indicate in the last figure?
A: The red one is resuming from model_best.pth
and ae_epoch = 0
while the blue one is training from scratch, i.e., ae_epoch = 30
.
Btw, I have some follow-up questions.
Q: So it seems model_best.pth
is the best final NAE, right? Not the best AE checkpoint.
Q: May I confirm why a small ae_epoch
leads to a good model manifold? To me, if the pre-trained AE is not trained sufficiently, it seems less helpful to the following NAE training?
Regards,
from normalized-autoencoders.
Dear @GloryyrolG ,
model_best.pth
is supposed to be the autoencoder with the smallest validation reconstruction error, but in the current setting, that is happened to be the last autoencoder, because I set a moderately large ae_epoch
.
I didn't mean that a small ae_epoch
results in a good model manifold. I was gonna say that you should avoid too small or too large ae_epoch
.
Best,
from normalized-autoencoders.
So may I confirm like, the variations of the energy the model reaches finally are okay in the last figure? @swyoon
from normalized-autoencoders.
Related Issues (12)
- where is cifar10_trainval_idx.npy? HOT 1
- VAE Comparison HOT 1
- Divergence of CD/PCD HOT 2
- Noise Annealing with the Step Size HOT 1
- Some Keyword missed and unexpected in .yml files HOT 2
- some key word missing HOT 2
- 안녕하세요. 최근에 딥러닝을 공부하고있는 직장인입니다. 로스함수 관련 궁금한게 있어 문의드립니다. HOT 2
- Some unexpected keyword augments when constructing the model HOT 1
- Not Good Quality of the Generated Images on CIFAR-10 HOT 3
- pos_out > neg_out HOT 1
- Availability of 2D Experiments? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from normalized-autoencoders.