RuntimeError: “LayerNormKernelImpl“ not implemented for ‘Half‘

报错信息

RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'

更加完整的报错信息：

/home/anaconda3/envs/py385/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1188         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1189                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1190             return forward_call(*input, **kwargs)
   1191         # Do not call functions when jit is used
   1192         full_backward_hooks, non_full_backward_hooks = [], []

/home/anaconda3/envs/py385/lib/python3.8/site-packages/torch/nn/modules/normalization.py in forward(self, input)
    188 
    189     def forward(self, input: Tensor) -> Tensor:
--> 190         return F.layer_norm(
    191             input, self.normalized_shape, self.weight, self.bias, self.eps)
    192 

/home/anaconda3/envs/py385/lib/python3.8/site-packages/torch/nn/functional.py in layer_norm(input, normalized_shape, weight, bias, eps)
   2513             layer_norm, (input, weight, bias), input, normalized_shape, weight=weight, bias=bias, eps=eps
   2514         )
-> 2515     return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
   2516 
   2517 

RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'

解决方法

最近在研究大模型，遇到这个问题，其实报错的原因只是因为我在推理的时候，忘记将模型放在GPU上，所以解决也很简单：

model.to('cuda:0')

把模型放在GPU上即可解决。

参考：https://github.com/huggingface/transformers/issues/21989

报错原因

更详细一点解释，是因为模型做了半精度，即fp16，也就是说在前面的代码中，你应该是执行过这一句：

model = model.half()

而fp16，在CPU上是不能生效的，如果用CPU推理，只能老老实实的用fp32。

补充

如果你是在部署stable-diffusion的时候遇到这个错误，可以参考hf的这个issue：
https://huggingface.co/CompVis/stable-diffusion-v1-4/discussions/64