pytorch报错：RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE

问题

运行torch代码时报错：

RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)

具体为：

File "/home/server/resources/nn/CLIP4Clip/modules/module_clip.py", line 253, in forward
    x = x + self.attention(self.ln_1(x))
  File "/home/server/resources/nn/CLIP4Clip/modules/module_clip.py", line 249, in attention
    return self.attn(x, x, x, need_weights=False, attn_mask=attn_mask_)[0]
  File "/root/miniconda3/envs/vclip/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/root/miniconda3/envs/vclip/lib/python3.9/site-packages/torch/nn/modules/activation.py", line 1153, in forward
    attn_output, attn_output_weights = F.multi_head_attention_forward(
  File "/root/miniconda3/envs/vclip/lib/python3.9/site-packages/torch/nn/functional.py", line 5066, in multi_head_attention_forward
    q, k, v = _in_projection_packed(query, key, value, in_proj_weight, in_proj_bias)
  File "/root/miniconda3/envs/vclip/lib/python3.9/site-packages/torch/nn/functional.py", line 4745, in _in_projection_packed
    return linear(q, w, b).chunk(3, dim=-1)
RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F, ldb, &fbeta, c, CUDA_R_16F, ldc, CUDA_R_32F, CUBLAS_GEMM_DFALT_TENSOR_OP)`
 

查询到可能存在的问题，及尝试的解决方案

1. 一个操作数太大而无法放入 int32（或负数）

减小batchsize

无效

2. 输入张量的维度与 nn.Linner 层的维度不匹配

经检查，我的维度没有问题

最终解决方案

在linux 命令行输入命令：

unset LD_LIBRARY_PATH