RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`

最近在使用bert-base-Chinese模型做序列标注任务，出现报错，用以记录。

报错信息如下：

/pytorch/aten/src/ATen/native/cuda/Indexing.cu:699: indexSelectLargeIndex: block: [220,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "demo.py", line 368, in <module>
    online_test_coref(config, text)
  File "demo.py", line 304, in online_test_coref
    cluster_ids, sentence_map)
  File "/home/disk1/wuzhichao/py_env/uie_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/disk1/wuzhichao/code/Chinese_Coreference_Resolution-main/coreference.py", line 144, in forward
    emb_mention_doc, _ = self.bert(input_ids=input_ids, attention_mask=input_mask, output_all_encoded_layers=False)  # [batch_size, seg_len, hidden_size]
  File "/home/disk1/wuzhichao/py_env/uie_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/disk1/wuzhichao/code/Chinese_Coreference_Resolution-main/bert/modeling.py", line 741, in forward
    output_all_encoded_layers=output_all_encoded_layers)
  File "/home/disk1/wuzhichao/py_env/uie_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/disk1/wuzhichao/code/Chinese_Coreference_Resolution-main/bert/modeling.py", line 412, in forward
    hidden_states = layer_module(hidden_states, attention_mask)
  File "/home/disk1/wuzhichao/py_env/uie_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/disk1/wuzhichao/code/Chinese_Coreference_Resolution-main/bert/modeling.py", line 397, in forward
    attention_output = self.attention(hidden_states, attention_mask)
  File "/home/disk1/wuzhichao/py_env/uie_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/disk1/wuzhichao/code/Chinese_Coreference_Resolution-main/bert/modeling.py", line 355, in forward
    self_output = self.self(input_tensor, attention_mask)
  File "/home/disk1/wuzhichao/py_env/uie_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/disk1/wuzhichao/code/Chinese_Coreference_Resolution-main/bert/modeling.py", line 306, in forward
    mixed_query_layer = self.query(hidden_states)
  File "/home/disk1/wuzhichao/py_env/uie_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/disk1/wuzhichao/py_env/uie_env/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 103, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/disk1/wuzhichao/py_env/uie_env/lib/python3.7/site-packages/torch/nn/functional.py", line 1848, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`

方法一：调小batsh_size，这是我查阅资料中说的最多的解决方法。但是对于我来说无效。

方法二：设置的分类标签与实际不符，对我无效。

方法三：我使用了CUDA_LAUNCH_BLOCKING=1 python run.py查看到了报错底层，是embedding出了问题。发现是设置的文本长度超过了模型config中的最大长度，因为bert模型的输入token最大为512，而我把文本的最大长度设置为1024，因此导致了模型报错。


解决方法：1.调整配置文件中的max_length使其低于bert的最大长度。2.改用可以处理长序列的预训练模型如longbert等。

中文BERT最大长度不能超过512的原因：因为BERT做预训练任务时，只训练了512个长度的位置编码信息，想要扩长输入长度除非重新进行预训练。