CUBLAS_STATUS_ALLOC_FAILED CUDA_ERROR_OUT_OF_MEMORY

当我用keras多进程并发时，报错：

2018-02-26 23:05:31.843027: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:366] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED

解决方法：加上如下代码：

import tensorflow as tf
import keras.backend.tensorflow_backend as KTF

######################################
# TODO: set the gpu memory using fraction #
#####################################
def get_session(gpu_fraction=0.3):
    """
    This function is to allocate GPU memory a specific fraction
    Assume that you have 6GB of GPU memory and want to allocate ~2GB
    """
    num_threads = os.environ.get('OMP_NUM_THREADS')
    gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=gpu_fraction)

    if num_threads:
        return tf.Session(config=tf.ConfigProto(
            gpu_options=gpu_options, intra_op_parallelism_threads=num_threads))
    else:
        return tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))

KTF.set_session(get_session(0.9))  # using 40% of total GPU Memory

get_session的比例比较大时，就会报错：CUDA_ERROR_OUT_OF_MEMORY

解决方：把比例设置小一点，比如0.5

TensorFlow的方法：

这时你需要用下面的方法创建 Session：
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.333)

sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))