解决AssertionError: No inf checks were recorded for this optimizer

解决AssertionError: No inf checks were recorded for this optimizer

在torch.cuda.amp.grad_scaler.py里面有这段注释

``scaler`` dynamically estimates the scale factor each iteration.  To minimize gradient underflow,
    a large scale factor should be used.  However, ``float16`` values can "overflow" (become inf or NaN) if
    the scale factor is too large.  Therefore, the optimal scale factor is the largest factor that can be used
    without incurring inf or NaN gradient values.

大概意思即float16会溢出，全部换成float32即可。

查看数据类型可通过

print(x.dtype)

将float16转换为float32，可通过to()函数

x = x.to(torch.float32)

我把代码里的embedding数据类型都输出 一遍，全都改了，发现还是报错，才发现如果设置了trainer的这个yaml文件，也得改

trainer:
  max_steps: 100_000
  val_check_interval: 5000
  precision: 32    #这个原本是16，这会儿改成32即可运行

全文参考
链接: link.