awesome-optimizers

A curated list of optimizers for machine learning.

Symbol	Meaning
✔️	Native support from PyTorch / TensorFlow
🟡	Non-native implementation
❌	No implementation found

TIP: Click on ✔️ or 🟡 to view the implementation of that optimizer!

Name	Date	Paper	PyTorch	TensorFlow
SGD			✔️	✔️
RPROP	1992	RPROP - A Fast Adaptive Learning Algorithm	✔️	❌
ASGD	1992	Acceleration of stochastic approximation by averaging	✔️	❌
Adagrad	2011	Adaptive Subgradient Methods for Online Learning and Stochastic Optimization	✔️	✔️
Adadelta	2012	ADADELTA: An Adaptive Learning Rate Method	✔️	✔️
RMSprop	2012	Neural Networks for Machine Learning Lecture 6a: Overview of mini-batch gradient descent	✔️	✔️
Adam	2014	Adam: A Method for Stochastic Optimization	✔️	✔️
Adamax	2014	Adam: A Method for Stochastic Optimization	✔️	✔️
Nadam	2015	Incorporating Nesterov Momentum into Adam	❌	✔️
AdamW	2017	Decoupled Weight Decay Regularization	✔️	🟡
Yogi	2018	Adaptive Methods for Nonconvex Optimization	🟡	🟡
AMSGrad	2018	On the Convergence of Adam and Beyond	✔️	❌
AdaBound	2019	Adaptive Gradient Methods with Dynamic Bound of Learning Rate	🟡	❌
AMSBound	2019	Adaptive Gradient Methods with Dynamic Bound of Learning Rate	🟡	❌
LAMB	2019	Large Batch Optimization for Deep Learning: Training BERT in 76 minutes	❌	🟡
RAdam	2019	On the Variance of the Adaptive Learning Rate and Beyond	🟡	🟡
SGDP	2020	Slowing Down the Weight Norm Increase in Momentum-based Optimizers	🟡	❌
AdamP	2020	Slowing Down the Weight Norm Increase in Momentum-based Optimizers	🟡	❌
Madam	2020	Learning compositional functions via multiplicative weight updates	🟡	❌
SuperAdam	2021	Learning compositional functions via multiplicative weight updates	✔️	❌