解决 THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=405 error=11 : invalid argument
完整错误:
WARNING:root:Setting up a new session...
Training...
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=383 error=11 : invalid argument
Traceback (most recent call last):
File "train.py", line 161, in <module>
main(args)
File "train.py", line 155, in main
solver.train()
File "/home/ddy17/Speech-Transformer-plus-2DAttention/src/solver/solver.py", line 78, in train
tr_avg_loss = self._run_one_epoch(epoch)
File "/home/ddy17/Speech-Transformer-plus-2DAttention/src/solver/solver.py", line 158, in _run_one_epoch
pred, gold = self.model(padded_input, input_lengths, padded_target)
File "/home/ddy17/.conda/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/home/ddy17/Speech-Transformer-plus-2DAttention/src/transformer/transformer.py", line 31, in forward
padded_input = self.prenet(padded_input) # N x (Ti/4) x d_model 降维加映射
File "/home/ddy17/.conda/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/home/ddy17/Speech-Transformer-plus-2DAttention/src/transformer/attention.py", line 205, in forward
out = self.bn(self.downsample(inputs))
File "/home/ddy17/.conda/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
result = self.forward(*input, **kwargs)
File "/home/ddy17/Speech-Transformer-plus-2DAttention/src/transformer/conv2d.py", line 101, in forward
self.padding, self.dilation, self.groups)
File "/home/ddy17/Speech-Transformer-plus-2DAttention/src/transformer/conv2d.py", line 29, in conv2d_same_padding
dilation=dilation, groups=groups)
RuntimeError: cuda runtime error (11) : invalid argument at /pytorch/aten/src/THC/THCGeneral.cpp:383
显卡用的最新的RTX2080Ti,pytorch的安装不能直接pip install ,需要从源码编译或者下载wheel安装pt.
pip install https://download.pytorch.org/whl/cu100/torch-1.0.0-cp36-cp36m-linux_x86_64.whl
参考: