问题
运行torch代码时报错:
RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)
具体为:
File "/home/server/resources/nn/CLIP4Clip/modules/module_clip.py", line 253, in forward
x = x + self.attention(self.ln_1(x))
File "/home/server/resources/nn/CLIP4Clip/modules/module_clip.py", line 249, in attention
return self.attn(x, x, x, need_weights=False, attn_mask=attn_mask_)[0]
File "/root/miniconda3/envs/vclip/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/root/miniconda3/envs/vclip/lib/python3.9/site-packages/torch/nn/modules/activation.py", line 1153, in forward
attn_output, attn_output_weights = F.multi_head_attention_forward(
File "/root/miniconda3/envs/vclip/lib/python3.9/site-packages/torch/nn/functional.py", line 5066, in multi_head_attention_forward
q, k, v = _in_projection_packed(query, key, value, in_proj_weight, in_proj_bias)
File "/root/miniconda3/envs/vclip/lib/python3.9/site-packages/torch/nn/functional.py", line 4745, in _in_projection_packed
return linear(q, w, b).chunk(3, dim=-1)
RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F, ldb, &fbeta, c, CUDA_R_16F, ldc, CUDA_R_32F, CUBLAS_GEMM_DFALT_TENSOR_OP)`
查询到可能存在的问题,及尝试的解决方案
1. 一个操作数太大而无法放入 int32(或负数)
减小batchsize
无效
2. 输入张量的维度与 nn.Linner 层的维度不匹配
经检查,我的维度没有问题
最终解决方案
在linux 命令行输入命令:
unset LD_LIBRARY_PATH