训练过程中随机iter中出现报错： RuntimeWarning: Invalid value encountered in median
r = func(a,**kwargs) 然后损失会变成nan
2020-01-09 08:29:41,280-INFO: iter: 38900, lr: 0.000500, 'loss': '3.347605', time: 0.938, eta: 2 days, 20:04:31
2020-01-09 08:29:57,997-INFO: iter: 38920, lr: 0.000500, 'loss': '3.927342', time: 0.679, eta: 2 days, 1:18:34
2020-01-09 08:30:16,731-INFO: iter: 38940, lr: 0.000500, 'loss': '3.320120', time: 0.824, eta: 2 days, 11:46:39
/root/paddlejob/workspace/env_run/anaconda3/lib/python3.7/site-packages/numpy/lib/function_base.py:3405: RuntimeWarning: Invalid value encountered in median
r = func(a,**kwargs)
2020-01-09 08:30:33,687-INFO: iter: 38960, lr: 0.000500, 'loss': 'nan', time: 1.134, eta: 3 days, 10:18:00
2020-01-09 08:30:50,932-INFO: iter: 38980, lr: 0.000500, 'loss': 'nan', time: 0.859, eta: 2 days, 14:21:49
2020-01-09 08:31:08,486-INFO: iter: 39000, lr: 0.000500, 'loss': 'nan', time: 0.920, eta: 2 days, 18:43:11
2020-01-09 08:31:26,606-INFO: iter: 39020, lr: 0.000500, 'loss': 'nan', time: 1.346, eta: 4 days, 1:39:19