Comments (7)
Are you using Amp? If so, we're about to include some optimizations targeting Pytorch 1.0 (based on the fact that their C++ API is now stable) which will greatly reduce the cost of checking for infs and nans (necessary to implement dynamic loss scaling) performed as part of the optimization step.
I didn't write the FusedAdam optimizer, but I'll pass along your request.
from apex.
Thanks, currently I'm using FP16_Optimizer instead of Amp. My usecase is standard imagenet training, using AdamW instead of the SGD. I understand from the documentation that FP16_Optimizer is best, or should I be using Amp?
Basically I'm after highest throughput. Lower GPU memory usage is also nice, but throughput is my chief concern.
I gave dynamic loss scaling a go with FP16_Optimizer, it does seem to be quite a bit slower than using static loss scaling. I'm glad to hear it'll become faster. I'm just having a read of the mixed precision training paper again (https://arxiv.org/pdf/1710.03740.pdf), would it otherwise be possible to monitor the mean & standard deviation of the gradients? Perhaps you could choose a loss scale value such that the distribution stays within acceptable bounds. Such a check you could also probably just perform once per epoch?
from apex.
Hi @yaysummeriscoming ,
We are looking into adamW as well as some general improvement to fusedAdam.
Since the fused adam does not support weight decay at all. Do you think we should also support original pytorch adam weight decay?
from apex.
@FDecaYed , that's great news. Perhaps for completeness it might be a good idea, but in practice I'm always using the AdamW formulation for CNNs. I've found it to always be superior.
Edit: I have a python reference implementation, if that's helpful?
from apex.
@FDecaYed Any update on this? I'd also like a fusedAdamW.
from apex.
@yaysummeriscoming @EvenOldridge
The latest apex.optimizers.FusedAdam already support AdamW. You can use it with optimizers.fp16_optimizer(or amp O2 with manual cast? @mcarilli to confirm)
What it does not support is original adam with L2.
Sorry for not seeing and replying. I made the change ~ a week after last replied here and forgot to update this thread.
from apex.
Thanks for the update & putting this in!
from apex.
Related Issues (20)
- cannot import name 'AutoencoderKLTemporalDecoder' from 'diffusers.models'
- No module named 'torch._six' HOT 1
- install failure
- Unclear licensing for contrib/sparsity
- Up to date patch for Windows compilation with Visual Studio 2022, CUDA 12.1 and PyTorch 2.2.2 HOT 1
- apex not installing
- [ INSTALLATION ] - Not able to install apex on a Linux machine HOT 2
- Not Able to install apex.
- Contrib unit test failure in `openfold_triton/test_fused_adam_swa.py::FusedAdamSWATestCase::test_fused_update_on_random_data`
- ImportError: cannot import name '_library_root_logger' from 'apex' (unknown location) HOT 2
- Cannot import name 'UnencryptedCookieSessionFactoryConfig' HOT 3
- "packaging" library exists but not found HOT 3
- Error
- Win11+Visual Studio 2022,install successfully. HOT 2
- Unable to install Apex HOT 2
- Release GIL when calling C extensions
- unable to install HOT 3
- Unable to install Apex on Linux(debian) with CUDA 12.1 and torch 2.2.2 HOT 2
- Unknown CUDA arch (compute) or GPU not supported error while installing on docker ubuntu with cuda 12.1
- Install set.up
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from apex.