Feature request: FusedAdamW about apex HOT 7 CLOSED

Are you using Amp? If so, we're about to include some optimizations targeting Pytorch 1.0 (based on the fact that their C++ API is now stable) which will greatly reduce the cost of checking for infs and nans (necessary to implement dynamic loss scaling) performed as part of the optimization step.

I didn't write the FusedAdam optimizer, but I'll pass along your request.

from apex.

Thanks, currently I'm using FP16_Optimizer instead of Amp. My usecase is standard imagenet training, using AdamW instead of the SGD. I understand from the documentation that FP16_Optimizer is best, or should I be using Amp?

Basically I'm after highest throughput. Lower GPU memory usage is also nice, but throughput is my chief concern.

I gave dynamic loss scaling a go with FP16_Optimizer, it does seem to be quite a bit slower than using static loss scaling. I'm glad to hear it'll become faster. I'm just having a read of the mixed precision training paper again (https://arxiv.org/pdf/1710.03740.pdf), would it otherwise be possible to monitor the mean & standard deviation of the gradients? Perhaps you could choose a loss scale value such that the distribution stays within acceptable bounds. Such a check you could also probably just perform once per epoch?

from apex.

Hi @yaysummeriscoming ,
We are looking into adamW as well as some general improvement to fusedAdam.
Since the fused adam does not support weight decay at all. Do you think we should also support original pytorch adam weight decay?

from apex.

@FDecaYed , that's great news. Perhaps for completeness it might be a good idea, but in practice I'm always using the AdamW formulation for CNNs. I've found it to always be superior.

Edit: I have a python reference implementation, if that's helpful?

from apex.

@FDecaYed Any update on this? I'd also like a fusedAdamW.

from apex.

@yaysummeriscoming @EvenOldridge
The latest apex.optimizers.FusedAdam already support AdamW. You can use it with optimizers.fp16_optimizer(or amp O2 with manual cast? @mcarilli to confirm)
What it does not support is original adam with L2.
Sorry for not seeing and replying. I made the change ~ a week after last replied here and forgot to update this thread.

from apex.

Thanks for the update & putting this in!

from apex.