Use of AdaHessian with batched training data? about adahessian HOT 2 CLOSED

Hi,

Thanks for reaching out. Yes AdaHessian supports mini-batching, and that is actually the case for all the examples that we have provided in this repository.

Either the code under pytorch-optimizer or this repo works. If you want to use it for computer vision tasks and/or NLP tasks then it may be easier to use this repo since we already have demos that you can start with.

Best,
-Amir

from adahessian.

Thanks!
BTW, as you suggested, I just looked at your examples in this repo and noticed that adahessian/transformer/fairseq/optim/adahessian.py version of AdaHessian seems to use fp16. Is that correct? If so, I would like to use that version in my chatbot training for the memory efficiency and speed of fp16. It appears that I would only need two files from your repo: the above mentioned adahessian.py file and adahessian/transformer/fairseq/optim/fairseq_optimizer.py . Does that sound right? Do you have any lessons learned from using this version that you would could pass along?
Thanks for your help!

from adahessian.