Unstable training about rexnet HOT 3 CLOSED

Thank you for your interest in our backbone. In my experience, ReXNet has similar training behavior to other renowned architectures. I think the instability may stem from the training setup; please refer to the training setting here.

Adam and AdamW both are quite stable optimizers which are more stable than SGD, so may not cause any instability in my experience. Presumably, high LR or WD cause the problem, if there is no LR warming up.

from rexnet.

Thank you for your kindness, I take the Adam optimizer with lr=0.001,WD=1e-4, and make lr scheduler each epoch by ExponentialLR with gamma=0.9.

from rexnet.

@myt889 The setting you used may not cause instability in my experience. Presumably, dataset issues (e.g., labeling noise) affect the instability, and large LR without any LR warming-up harm the training stability as well.

from rexnet.