Runtime: Adam vs AdamP about adamp HOT 2 CLOSED

Hi

Thank you for your interest in our work.

In our experiment , we use batch-size 256 and AdamP costs 8% training time than Adam.
MIRNet's batch-size is 16, which is a much smaller value than our setting.
It seems to be the cause of slow processing.
So, I recommend increasing the batch-size to improve the speed.
As the batch-size increases, the ratio of the optimizer operation in training decreases, and the training time per epoch can be reduced.

I know that increasing the batch-size is often infeasible and not a good solution.
But, I don't have any other solution besides that.
AdamP uses additional computation for the projection operation, which is an inevitable factor.
There may be ways to further optimize the projection operation, but I couldn't found it.

from adamp.

Thank you for a quick response.

from adamp.