Comments (3)
Hi @someitalian123. Could you try decreasing the learning rate? I have recently decreased it as well on default template configs due to some other users reporting instability in some instances. You can see the recommended values here.
Another suggestion: at the beginning of training, use a larger batch size, particularly if you're training from scratch (no pretrained model). Training without a pretrain is highly discouraged btw.
from neosr.
Hi @someitalian123. Could you try decreasing the learning rate? I have recently decreased it as well on default template configs due to some other users reporting instability in some instances. You can see the recommended values here. Another suggestion: at the beginning of training, use a larger batch size, particularly if you're training from scratch (no pretrained model). Training without a pretrain is highly discouraged btw.
I just copy and pasted the values that you linked and I have made it to epoch 80. I will have to run it a bit longer though to see if it happens again. As it's running though the console still says that the learning rate is 1.60e-03 instead of the the 1-e3 that I changed it to.
28-07-2024 02:00 PM | INFO: [ epoch: 80 ] [ iter: 73,900 ] [ performance: 1.065 it/s ] [ lr: 1.60e-03 ] [ eta: 7 days, 3:11:22 ] [ l_g_mssim: 9.9570e-02 ] [ l_g_percep: 5.0608e+00 ] [ l_g_color: 3.1832e-04 ] [ l_g_luma: 4.0132e-03 ] [ l_g_gan: 1.8426e+00 ] [ l_g_total: 7.0073e+00 ] [ l_d_real: 1.3882e-05 ] [ out_d_real: 1.2296e+01 ] [ l_d_fake: 2.2195e-03 ] [ out_d_fake: -6.1791e+00 ] [ l_d_total: 1.1167e-03 ]
I'm wondering if the change actually took effect or if something else is going on. Do I need to delete the current contents of the experiments folder and start the training over? Or is it fine to continue after changing the learning rate in the config file?
As far as increasing the batch size is concerned, I wasn't sure if it would be better to have a larger batch size or a larger patch size. As of right now while the training is running 7.1/8.0 GB of my dedicated VRAM is in use. I would likely need to decrease the patch size to 32 if I wanted to increase the batch size right?
from neosr.
As it's running though the console still says that the learning rate is 1.60e-03 instead of the the 1-e3
Yes, changing learning rate doesn't work right now if you resume from the state dict (such as when using --auto_resume
). If you want to change it, you must to use the current model as a pretrain and start training again.
However, if it's not giving you NaN, you should just proceed. The learning rate of 1.6e-3
should work perfectly well since the schedule-free optimizer blends the parameters (adapts). If you do want a more stable training though, decreasing it to 1.2e-3
or 1e-3
should do the trick.
I would likely need to decrease the patch size to 32 if I wanted to increase the batch size right?
Yes. Normally you should start with a larger batch and a patch_size of 32. Then later on training (>40-60k) you can start to decrease batch size and increase patch_size. This is a strategy called "curriculum learning" in ML literature.
from neosr.
Related Issues (20)
- wrapper around train.py, ... scripts and release on pypi
- `train_realesrgan.yml` is named "train_esrgan" HOT 1
- imwrite raise IOError when trying to write an image where the filename contains unicode characters HOT 2
- How to save a model fully in pth? Not state dict HOT 3
- Document which architectures have been modified and are incompatible with the original architecture HOT 4
- Add tile option HOT 2
- Modifications for CRAFT, SwinIR, and DCTLAS are not detectable HOT 8
- Automatically remove old checkpoints and visualizations HOT 4
- and again rgb_mean HOT 1
- assets HOT 3
- dataset_enlarge_ratio description HOT 1
- Mismatched resume state issue HOT 1
- All architectures with `rgb_mean = (0.5, 0.5, 0.5)` are incompatible HOT 9
- Python version problem HOT 4
- Question regarding net_d HOT 4
- model zoo HOT 1
- can not start train HOT 3
- GPU memory not fixed when training HOT 1
- NaN errors occurring constantly HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from neosr.