nvidia errors in TensorFlow

Go to check GPU memory usage, when encountering nvidia or cuda error.

I’ve been worked with tensorflow-gpu for a while. When run the codes, nvidia reports kinds of errors without clear instruction some time.

As far as my knowledge, met a nvidia error, one should check whether the gpu memory is occupied by other process by

nvidia-smi -l

As the following screenshot shows, the GPU-util is 0%, but the memory is nearly used out. If one run a new tensorflow process, errors are likely to be reported.
If you are coding with pycharm, there is scitific mode. In this mode, the process will not exit automatically, so the tensorflow session remains consuming the GPU memrory.

这里写图片描述

error example

E tensorflow/stream_executor/cuda/cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值