解决 Failed to get convolution algorithm. 报错问题
小白记录一下在深度学习问题上的报错代码,希望也能给予有相同问题的人一些帮助。
报错场景:
在训练完成了一个深度神经网络的前提下,希望输出中间层特征输出结果,于是使用Model构建了一个新的网络,原始的inputs是输入,model.get_layer().output提取的层是输出。对新网络使用model.predict()指令的时候报了如下错误。
报错代码:
E tensorflow/stream_executor/cuda/cuda_dnn.cc:328] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
E tensorflow/stream_executor/cuda/cuda_dnn.cc:328] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
Traceback (most recent call last):
File "G:/PycharmProjects/tensorflow/urban8k/netweights.py", line 70, in <module>
conv2d_output = M18NET_conv2d.predict(data)
File "J:\Anaconda\envs\tf2.1\lib\site-packages\tensorflow\python\keras\engine\training.py", line 130, in _method_wrapper
return method(self, *args, **kwargs)
File "J:\Anaconda\envs\tf2.1\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1599, in predict
tmp_batch_outputs = predict_function(iterator)
File "J:\Anaconda\envs\tf2.1\lib\site-packages\tensorflow\python\eager\def_function.py", line 780, in __call__
result = self._call(*args, **kwds)
File "J:\Anaconda\envs\tf2.1\lib\site-packages\tensorflow\python\eager\def_function.py", line 846, in _call
return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access
File "J:\Anaconda\envs\tf2.1\lib\site-packages\tensorflow\python\eager\function.py", line 1848, in _filtered_call
cancellation_manager=cancellation_manager)
File "J:\Anaconda\envs\tf2.1\lib\site-packages\tensorflow\python\eager\function.py", line 1924, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "J:\Anaconda\envs\tf2.1\lib\site-packages\tensorflow\python\eager\function.py", line 550, in call
ctx=ctx)
File "J:\Anaconda\envs\tf2.1\lib\site-packages\tensorflow\python\eager\execute.py", line 60, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node functional_3/conv2d/Conv2D (defined at G:/PycharmProjects/tensorflow/urban8k/netweights.py:70) ]] [Op:__inference_predict_function_1847]
Function call stack:
predict_function
原因分析:
通过查阅相关博客,发现原因是tensorflow的GPU使用和分配出现问题,需要对GPU的使用分配进行配置,之后不再报错。
解决方案:
gpus = tf.config.experimental.list_physical_devices(‘GPU’)指令获得运算设备类型
tf.config.experimental.set_memory_growth(gpu, True)设置显卡使用策略为“只在需要时申请显存空间”
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
for gpu in gpus:
tf.config.experimental.set_memory_growth(gpu, True)