Does Adam Optimization depend on batch size? about ferminet HOT 3 CLOSED

The hyperparameters should not need to be changed with batch size. We use the mean of the local energies when computing the loss, so the gradient should not depend on batch size. Even so, methods like ADAM are self-tuning, and so should adapt to changes in batch size. If you are using the TensorFlow code, we would recommend using the KFAC optimizer rather than ADAM. If you only want to try ADAM, we would recommend using the JAX code (in the jax branch) as it is cleaner and easier to understand.

from ferminet.

Thank you for your suggestion. Since we are trying to test the code in pytorch and the converges in pytorch is not as quick as in the tensorflow code.

After some investigation, we think the optimizer.minimize function only combines the features ($\log|\psi|$) and grad_loss ($E_L - \mathbb{E}(E_L)$) in a dot product way and the denominator of the batch size is not included and that is why we ask the question.

from ferminet.

Ah yes, I think there's a slight difference between the JAX and TF implementations. But because of the way both ADAM and KFAC normalize the updates, it shouldn't matter.

from ferminet.