Lr过小时会导致Loss为nan about plsc HOT 9 CLOSED

Training: 2022-03-24 15:02:19,865 - loss nan, lr: 0.010000, epoch: 11, step: 18000, eta: 1.72 hours, throughput: 433.87 imgs/sec
testing verification..
[[ 0.  0.  0. ...  0.  0.  0.]
 [ 0.  0.  0. ...  0.  0.  0.]
 [ 0.  0.  0. ...  0.  0.  0.]
 ...
 [nan nan nan ... nan nan nan]
 [nan nan nan ... nan nan nan]
 [nan nan nan ... nan nan nan]]
Traceback (most recent call last):
  File "/home/aistudio/PLSC/train_aio.py", line 304, in <module>
    train(args)
  File "/home/aistudio/PLSC/dynamic/train_aio.py", line 228, in train
    best_metric = callback_verification(global_step, backbone)
  File "/home/aistudio/PLSC/dynamic/utils/verification_aio.py", line 211, in __call__
    best_metric = self.ver_test(backbone, num_update)
  File "/home/aistudio/PLSC/dynamic/utils/verification_aio.py", line 143, in ver_test
    nfolds=10)
  File "<decorator-gen-287>", line 2, in test
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/base.py", line 351, in _decorate_function
    return func(*args, **kwargs)
  File "/home/aistudio/PLSC/dynamic/utils/verification_aio.py", line 89, in test
    embeddings = sklearn.preprocessing.normalize(embeddings)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/sklearn/utils/validation.py", line 63, in inner_f
    return f(*args, **kwargs)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/sklearn/preprocessing/_data.py", line 1905, in normalize
    estimator='the normalize function', dtype=FLOAT_DTYPES)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/sklearn/utils/validation.py", line 63, in inner_f
    return f(*args, **kwargs)
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/sklearn/utils/validation.py", line 721, in check_array
    allow_nan=force_all_finite == 'allow-nan')
  File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/sklearn/utils/validation.py", line 106, in _assert_all_finite
    msg_dtype if msg_dtype is not None else X.dtype)
ValueError: Input contains NaN, infinity or a value too large for dtype('float64').