Unable to find a valid cuDNN algorithm to run convolution

@cuDnn无法找到算法进行卷积

最近在复现一个项目，但是在运行训练文件时一直在报Unable to find a valid cuDNN algorithm to run convolution 这个问题，搜了一下貌似这个问题很小众，也没看到什么好的解决办法。目前已经尝试过更换cuDNN版本到8.0.3还是会有同样的错误，现在版本为7.6.5 ,还更改了pytoch版本1.5和1.6都试过了，依旧报错。
当前配置情况：
albumentations0.4.3
Cython>=0.25
future0.18.0
lmdb0.97
loguru0.3.2
mypy_extensions0.4.1
lvis0.5.1
numpy1.16.4
opencv-python4.1.2.30
scikit-learn0.21.3
sentencepiece0.1.85
torch1.6.0
torchvision0.7
tqdm==4.36.0

错误代码：
Traceback (most recent call last):
File “scripts/pretrain_virtex.py”, line 250, in
args=(_A, ),
File “/media/gc/168606691BA7B4F0/virtex/virtex/utils/distributed.py”, line 84, in launch
_job_worker(0, job_fn, 1, 1, 0, dist_url, args)
File “/media/gc/168606691BA7B4F0/virtex/virtex/utils/distributed.py”, line 118, in _job_worker
job_fn(*args)
File “scripts/pretrain_virtex.py”, line 158, in main
output_dict = model(batch)
File “/home/gc/anaconda3/envs/zlt/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in call
result = self.forward(*input, **kwargs)
File “/media/gc/168606691BA7B4F0/virtex/apex/amp/_initialize.py”, line 197, in new_fwd
**applier(kwargs, input_caster))
File “/media/gc/168606691BA7B4F0/virtex/virtex/models/captioning.py”, line 117, in forward
visual_features = self.visual(batch[“image”])
File “/home/gc/anaconda3/envs/zlt/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in call
result = self.forward(*input, **kwargs)
File “/media/gc/168606691BA7B4F0/virtex/virtex/modules/visual_backbones.py”, line 139, in forward
out = layer(image) if idx == 0 else layer(out)
File “/home/gc/anaconda3/envs/zlt/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in call
result = self.forward(*input, **kwargs)
File “/home/gc/anaconda3/envs/zlt/lib/python3.6/site-packages/torch/nn/modules/container.py”, line 100, in forward
input = module(input)
File “/home/gc/anaconda3/envs/zlt/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in call
result = self.forward(*input, **kwargs)
File “/home/gc/anaconda3/envs/zlt/lib/python3.6/site-packages/torchvision/models/resnet.py”, line 112, in forward
out = self.conv3(out)
File “/home/gc/anaconda3/envs/zlt/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in call
result = self.forward(*input, **kwargs)
File “/home/gc/anaconda3/envs/zlt/lib/python3.6/site-packages/torch/nn/modules/conv.py”, line 351, in forward
return self._conv_forward(input, self.weight)
File “/home/gc/anaconda3/envs/zlt/lib/python3.6/site-packages/torch/nn/modules/conv.py”, line 347, in _conv_forward
self.padding, self.dilation, self.groups)
RuntimeError: Unable to find a valid cuDNN algorithm to run convolution