训练kaldi chian模型,出现如下错误
ERROR (nnet3-chain-train[5.5]:AllocateNewRegion():cu-allocator.cc:498) Failed to allocate a memory region of 5627707392 bytes. Possibly this is due to sharing the GPU. Try switching the GPUs to exclusive mode (nvidia-smi -c 3) and using the option --use-gpu=wait to scripts like steps/nnet3/chain/train.py. Memory info: free:10732M, used:440M, total:11172M, free/total:0.960617
同时伴随提示“ERROR (nnet3-am-copy[5.5]:Write():kaldi-matrix.cc:1403) Failed to write matrix to stream”
按照提示信息,需要设置GPU为exclusive mode,使用命令:
sudo nvidia-smi -c 3
重新运行脚本,即可正常训练