[debug]THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=405 error=11 : invalid argume

解决 THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=405 error=11 : invalid argument

完整错误：

WARNING:root:Setting up a new session...
Training...
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=383 error=11 : invalid argument
Traceback (most recent call last):
  File "train.py", line 161, in <module>
    main(args)
  File "train.py", line 155, in main
    solver.train()
  File "/home/ddy17/Speech-Transformer-plus-2DAttention/src/solver/solver.py", line 78, in train
    tr_avg_loss = self._run_one_epoch(epoch)
  File "/home/ddy17/Speech-Transformer-plus-2DAttention/src/solver/solver.py", line 158, in _run_one_epoch
    pred, gold = self.model(padded_input, input_lengths, padded_target)
  File "/home/ddy17/.conda/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ddy17/Speech-Transformer-plus-2DAttention/src/transformer/transformer.py", line 31, in forward
    padded_input = self.prenet(padded_input) # N x (Ti/4) x d_model 降维加映射
  File "/home/ddy17/.conda/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ddy17/Speech-Transformer-plus-2DAttention/src/transformer/attention.py", line 205, in forward
    out = self.bn(self.downsample(inputs))
  File "/home/ddy17/.conda/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ddy17/Speech-Transformer-plus-2DAttention/src/transformer/conv2d.py", line 101, in forward
    self.padding, self.dilation, self.groups)
  File "/home/ddy17/Speech-Transformer-plus-2DAttention/src/transformer/conv2d.py", line 29, in conv2d_same_padding
    dilation=dilation, groups=groups)
RuntimeError: cuda runtime error (11) : invalid argument at /pytorch/aten/src/THC/THCGeneral.cpp:383

显卡用的最新的RTX2080Ti，pytorch的安装不能直接pip install ，需要从源码编译或者下载wheel安装pt.

pip install https://download.pytorch.org/whl/cu100/torch-1.0.0-cp36-cp36m-linux_x86_64.whl

参考：

• https://discuss.pytorch.org/t/cuda-runtime-error-11/30080/13
• https://github.com/pytorch/pytorch/issues/15797#issuecomment-452021037