训练loss突增后变为nan about paddleclas HOT 6 CLOSED

训练的是什么数据集呢？其他的超参数可以提供一下吗？

from paddleclas.

我也碰到了这个问题，自己的数据集，身份证、医保卡、银行卡等数据分类，在pytorch上面跑过，没有问题
用的MobileNetV3_small_x0_35.yaml，就改了跟数据集有关的地方，比如类别数这些，其他都是官方提供的
2020-07-27 17:35:21 INFO: epoch:4 train step:170 loss: 0.1664 top1: 0.9688 top5: 1.0000 lr: 2.299093 elapse: 0.881s
2020-07-27 17:35:28 INFO: epoch:4 train step:180 loss: 0.3008 top1: 0.9297 top5: 0.9922 lr: 2.312960 elapse: 0.652s
2020-07-27 17:35:34 INFO: epoch:4 train step:190 loss: 0.2220 top1: 0.9141 top5: 1.0000 lr: 2.326827 elapse: 0.759s
2020-07-27 17:35:42 INFO: epoch:4 train step:200 loss: 0.2745 top1: 0.9375 top5: 0.9922 lr: 2.340693 elapse: 0.737s
2020-07-27 17:35:48 INFO: epoch:4 train step:210 loss: 0.2979 top1: 0.9219 top5: 1.0000 lr: 2.354560 elapse: 0.692s
2020-07-27 17:35:55 INFO: epoch:4 train step:220 loss: 0.2368 top1: 0.9297 top5: 1.0000 lr: 2.368427 elapse: 0.548s
2020-07-27 17:36:01 INFO: epoch:4 train step:230 loss: 0.3466 top1: 0.8828 top5: 1.0000 lr: 2.382293 elapse: 0.666s
2020-07-27 17:36:08 INFO: epoch:4 train step:240 loss: 0.2592 top1: 0.9141 top5: 1.0000 lr: 2.396160 elapse: 0.689s
2020-07-27 17:36:15 INFO: epoch:4 train step:250 loss: 0.3722 top1: 0.9531 top5: 0.9844 lr: 2.410027 elapse: 0.595s
2020-07-27 17:36:21 INFO: epoch:4 train step:260 loss: 0.7002 top1: 0.8828 top5: 0.9844 lr: 2.423893 elapse: 0.698s
2020-07-27 17:36:28 INFO: epoch:4 train step:270 loss: 57.0000 top1: 0.1094 top5: 0.9219 lr: 2.437760 elapse: 0.651s
2020-07-27 17:36:34 INFO: epoch:4 train step:280 loss: nan top1: 0.0312 top5: 0.8984 lr: 2.451627 elapse: 0.699s
2020-07-27 17:36:41 INFO: epoch:4 train step:290 loss: nan top1: 0.0156 top5: 0.8906 lr: 2.465493 elapse: 0.621s
2020-07-27 17:36:47 INFO: epoch:4 train step:300 loss: nan top1: 0.0156 top5: 0.9531 lr: 2.479360 elapse: 0.494s
2020-07-27 17:36:54 INFO: epoch:4 train step:310 loss: nan top1: 0.0234 top5: 0.9297 lr: 2.493227 elapse: 0.715s
2020-07-27 17:37:01 INFO: epoch:4 train step:320 loss: nan top1: 0.0156 top5: 0.8750 lr: 2.507093 elapse: 0.684s
2020-07-27 17:37:08 INFO: epoch:4 train step:330 loss: nan top1: 0.0312 top5: 0.9141 lr: 2.520960 elapse: 0.723s
2020-07-27 17:37:14 INFO: epoch:4 train step:340 loss: nan top1: 0.0156 top5: 0.9219 lr: 2.534827 elapse: 0.704s
2020-07-27 17:37:21 INFO: epoch:4 train step:350 loss: nan top1: 0.0156 top5: 0.9453 lr: 2.548693 elapse: 0.582s
2020-07-27 17:37:28 INFO: epoch:4 train step:360 loss: nan top1: 0.0078 top5: 0.9062 lr: 2.562560 elapse: 0.596s
2020-07-27 17:37:35 INFO: epoch:4 train step:370 loss: nan top1: 0.0000 top5: 0.8750 lr: 2.576427 elapse: 1.042s
2020-07-27 17:37:37 INFO: END epoch:4 train loss_avg: nan top1_avg: 0.6615 top5_avg: 0.9625 elapse_sum: 250.617s

@cuicheng01

from paddleclas.

多次测试，nan100%会出现，并非偶然

from paddleclas.

可以提供完整的log文件吗？出nan的话，可以减小学习率试下

from paddleclas.

@littletomatodonkey 单卡和多卡，在CosineWarmup模式下，初始学习率分别怎么设置合理点？

from paddleclas.

降低学习率work

from paddleclas.