RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`

这个问题存在的其中一种可能就是模型的最后需要加sigmoid

 File ".\train.py", line 192, in train
    train_loss = solver.optimize()
  File "D:\wcs\seg_v2\framework.py", line 89, in optimize
    loss.backward()
  File "C:\Users\Administrator\Anaconda3\lib\site-packages\torch\tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "C:\Users\Administrator\Anaconda3\lib\site-packages\torch\autograd\__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`
00007FFF776675A200007FFF77667540 c10.dll!c10::Error::Error [<unknown file> @ <unknown line number>]
00007FFF2523B20800007FFF2523A1D0 torch_cuda.dll!at::cuda::getCurrentCUDASparseHandle [<unknown file> @ <unknown line number>]
00007FFF2523AB3800007FFF2523A1D0 torch_cuda.dll!at::cuda::getCurrentCUDASparseHandle [<unknown file> @ <unknown line number>]
00007FFF2523B9C700007FFF2523B500 torch_cuda.dll!at::cuda::getCurrentCUDABlasHandle [<unknown file> @ <unknown line number>]
00007FFF2523B5A700007FFF2523B500 torch_cuda.dll!at::cuda::getCurrentCUDABlasHandle [<unknown file> @ <unknown line number>]
00007FFF2523356700007FFF25232810 torch_cuda.dll!at::native::sparse_mask_cuda [<unknown file> @ <unknown line number>]
00007FFF23BAD8AD00007FFF23BAD810 torch_cuda.dll!THCudaBlas_Sgemm [<unknown file> @ <unknown line number>]
00007FFF240BDB3800007FFF240BD700 torch_cuda.dll!THNN_CudaSpatialConvolutionMM_updateGradInput [<unknown file> @ <unknown line number>]
00007FFF252BF06600007FFF2523E400 torch_cuda.dll!at::native::set_storage_cuda_ [<unknown file> @ <unknown line number>]
00007FFF245F64FC00007FFF245F6220 torch_cuda.dll!at::native::slow_conv2d_backward_cuda [<unknown file> @ <unknown line number>]
00007FFF252A9AE600007FFF2523E400 torch_cuda.dll!at::native::set_storage_cuda_ [<unknown file> @ <unknown line number>]
00007FFF252900D100007FFF2523E400 torch_cuda.dll!at::native::set_storage_cuda_ [<unknown file> @ <unknown line number>]
00007FFF4F109AC800007FFF4F07D9D0 torch_cpu.dll!at::native::mkldnn_sigmoid_ [<unknown file> @ <unknown line number>]
00007FFF4F1B276E00007FFF4F1B2660 torch_cpu.dll!at::thnn_conv2d_backward [<unknown file> @ <unknown line number>]
00007FFF505471A200007FFF503FE010 torch_cpu.dll!torch::autograd::GraphRoot::apply [<unknown file> @ <unknown line number>]
00007FFF4F0D194100007FFF4F07D9D0 torch_cpu.dll!at::native::mkldnn_sigmoid_ [<unknown file> @ <unknown line number>]
00007FFF4F109AC800007FFF4F07D9D0 torch_cpu.dll!at::native::mkldnn_sigmoid_ [<unknown file> @ <unknown line number>]
00007FFF4F1B276E00007FFF4F1B2660 torch_cpu.dll!at::thnn_conv2d_backward [<unknown file> @ <unknown line number>]
00007FFF503789C800007FFF50378630 torch_cpu.dll!torch::autograd::generated::ThnnConv2DBackward::apply [<unknown file> @ <unknown line number>]
00007FFF50337E9100007FFF50337B50 torch_cpu.dll!torch::autograd::Node::operator() [<unknown file> @ <unknown line number>]
00007FFF5089F9BA00007FFF5089F300 torch_cpu.dll!torch::autograd::Engine::add_thread_pool_task [<unknown file> @ <unknown line number>]
00007FFF508A03AD00007FFF5089FFD0 torch_cpu.dll!torch::autograd::Engine::evaluate_function [<unknown file> @ <unknown line number>]
00007FFF508A4FE200007FFF508A4CA0 torch_cpu.dll!torch::autograd::Engine::thread_main [<unknown file> @ <unknown line number>]
00007FFF508A4C4100007FFF508A4BC0 torch_cpu.dll!torch::autograd::Engine::thread_init [<unknown file> @ <unknown line number>]
00007FFF47F90A2700007FFF47F6A100 torch_python.dll!THPShortStorage_New [<unknown file> @ <unknown line number>]
00007FFF5089BF1400007FFF5089B780 torch_cpu.dll!torch::autograd::Engine::get_base_engine [<unknown file> @ <unknown line number>]
00007FFF9B011BB200007FFF9B011B20 ucrtbase.dll!configthreadlocale [<unknown file> @ <unknown line number>]
00007FFF9CFA703400007FFF9CFA7020 KERNEL32.DLL!BaseThreadInitThunk [<unknown file> @ <unknown line number>]
00007FFF9D8A265100007FFF9D8A2630 ntdll.dll!RtlUserThreadStart [<unknown file> @ <unknown line number>]