Tensorflow crash with CUDNN_STATUS_ALLOC_FAILED

could not create cudnn handle: cudnn_status_alloc_failed
tensorflow-gpu
cuda out of memory tensorflow
failed to create cublas handle: cublas_status_alloc_failed
tensorflow not using gpu
failed to get convolution algorithm
tensorflow error cudnn
tensorflow 2.1 failed to get convolution algorithm

Been searching the web for hours with no results, so figured I'd ask here.

I'm trying to make a self driving car following Sentdex's tutorial, but when running the model, I get a bunch of fatal errors. I've searched all over the internet for the solution, and many seem to have the same problem. However, none of the solutions I've found (Including this Stack-post), work for me.

Here is my software:

  • Tensorflow: 1.5, GPU version
  • CUDA: 9.0, with the patch
  • CUDnn: 7
  • Windows 10 Pro
  • Python 3.6

Hardware:

  • Nvidia 1070ti, with latest drivers
  • Intel i5 7600K

Here is the crash log:

2018-02-04 16:29:33.606903: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED 2018-02-04 16:29:33.608872: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED 2018-02-04 16:29:33.609308: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED 2018-02-04 16:29:35.145249: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED 2018-02-04 16:29:35.145563: E C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:352] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM 2018-02-04 16:29:35.149896: F C:\tf_jenkins\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\kernels\conv_ops.cc:717] Check failed: stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo<T>(), &algorithms)

Here's my code:

 import tensorflow as tf
    import numpy as np
    import cv2
    import time
    from PIL import ImageGrab
    from getkeys import key_check
    from alexnet import alexnet
    import os
    from sendKeys import PressKey, ReleaseKey, W,A,S,D,Sp

    import random

    WIDTH = 80
    HEIGHT = 60
    LR = 1e-3
    EPOCHS = 10
    MODEL_NAME = 'DiRT-AI-Driver-{}-{}-{}-epochs.model'.format(LR, 'alexnetv2', EPOCHS)

    def straight():
        PressKey(W)
        ReleaseKey(A)
        ReleaseKey(S)
        ReleaseKey(D)
        ReleaseKey(Sp)
    def left():
        PressKey(A)
        ReleaseKey(W)
        ReleaseKey(S)
        ReleaseKey(D)
        ReleaseKey(Sp)
    def right():
        PressKey(D)
        ReleaseKey(A)
        ReleaseKey(S)
        ReleaseKey(W)
        ReleaseKey(Sp)
    def brake():
        PressKey(S)
        ReleaseKey(A)
        ReleaseKey(W)
        ReleaseKey(D)
        ReleaseKey(Sp)
    def handbrake():
        PressKey(Sp)
        ReleaseKey(A)
        ReleaseKey(S)
        ReleaseKey(D)
        ReleaseKey(W)

    model = alexnet(WIDTH, HEIGHT, LR)
    model.load(MODEL_NAME)


    def main():
        last_time = time.time()
        for i in list(range(4))[::-1]:
            print(i+1)
            time.sleep(1)


    paused = False
    while(True):
            if not paused:
                screen = np.array(ImageGrab.grab(bbox=(0,40,1024,768)))
                screen = cv2.cvtColor(screen,cv2.COLOR_BGR2GRAY)
                screen = cv2.resize(screen,(80,60))
                print('Loop took {} seconds'.format(time.time()-last_time))
                last_time = time.time()
                print('took time')
                prediction = model.predict([screen.reshape(WIDTH,HEIGHT,1)])[0]
                print('predicted')
                moves = list(np.around(prediction))
                print('got moves')
                print(moves,prediction)

                if moves == [1,0,0,0,0]:
                    straight()
                elif moves == [0,1,0,0,0]:
                    left()
                elif moves == [0,0,1,0,0]:
                    brake()
                elif moves == [0,0,0,1,0]:
                    right()
                elif moves == [0,0,0,0,1]:
                    handbrake()

            keys = key_check()

            if 'T' in keys:
                if paused:
                    pased = False
                    time.sleep(1)
                else:
                    paused = True
                    ReleaseKey(W)
                    ReleaseKey(A)
                    ReleaseKey(S)
                    ReleaseKey(D)
                    ReleaseKey(Sp)
                    time.sleep(1)


main()

I've found that the line that crashes python and spawns the first three bugs is this line:

  • prediction = model.predict([screen.reshape(WIDTH,HEIGHT,1)])[0]

When running the code, the CPU goes up to a whopping 100%, suggesting that something is seriously off. GPU goes to about 40-50%

I've tried Tensorflow 1.2 and 1.3, as well as CUDA 8, to no good. When installing CUDA I do not install the specific drivers, since they are too old for my GPU. Tried different CUDnn's too, did no good.

In my case, the issue happened because another python console with tensorflow imported was running. Closing it solved the problem.

I have Windows 10, the main errors were :

failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED

Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED

Crash: Could not create cuDNN handle when convnets are , Tensorflow (GPU) was imported successfully, but when running a session that involves a neural network (CNN), Python crashes with the following message: E I have similar issue: CUDNN_STATUS_ALLOC_FAILED. For people searching the keyword CUDNN_STATUS_ALLOC_FAILED and finding this issue. You may check if there is another process allocating the gpu. In my case i got this error on terminal after previously running tensorflow from a jupyter notebook

Try to add the cuda path to environment variable. It's seems that the problem it's with cuda.

Set the CUDA Path in ~/.bashrc (edit with nano):

#Cuda Nvidia path
$ export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64"
$ export CUDA_HOME=/usr/local/cuda

Common issues, Python crashes - TensorFlow GPU¶. If you are using TensorFlow GPU and when you try to run some Python object detection script (e.g. could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED 2018-03-23 03:03:10.330475: E  Keras Tensorflow R Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED #30523

I encountered the same problem, then I found out that because I'm also using GPU for run other stuffs even it doesn't show on task manager (windows) using GPU. Maybe even things like (rendering videos, video encoding or play heavy workload game, coin mining...). If you think it's still using heavy GPU, just close it off and problem solve.

Crash: Could not create cuDNN handle when convnets , Tensorflow crash with CUDNN_STATUS_ALLOC_FAILED Tensorflow: 1.5, GPU version; CUDA: 9.0, with the patch; CUDnn: 7; Windows 10 Pro; Python 3.6. Sign up for GitHub By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement . We’ll occasionally send you account related emails.

Probably you're running out of GPU memory.


If you're using TensorFlow 1.x:

1st option) set allow_growth to true.

import tensorflow as tf    
config = tf.ConfigProto()
config.gpu_options.allow_growth=True
sess = tf.Session(config=config)

2nd option) set memory fraction.

# change the memory fraction as you want

import tensorflow as tf
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.3)
sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))

If you're using TensorFlow 2.x:

1st option) set set_memory_growth to true.

# Currently the ‘memory growth’ option should be the same for all GPUs.
# You should set the ‘memory growth’ option before initializing GPUs.

import tensorflow as tf
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
  try:
    for gpu in gpus:
      tf.config.experimental.set_memory_growth(gpu, True)
  except RuntimeError as e:
    print(e)

2nd option) set memory_limit as you want. Just change the index of gpus and memory_limit in this code below.

import tensorflow as tf
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
  try:
    tf.config.experimental.set_virtual_device_configuration(gpus[0], [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=1024)])
  except RuntimeError as e:
    print(e)

cuDNN crashes ever since an error during training, Ask questionsCrash: Could not create cuDNN handle when convnets are used. Tensorflow E tensorflow/stream_executor/cuda/cuda_dnn.cc:385] could not create cudnn handle: I have similar issue: CUDNN_STATUS_ALLOC_FAILED. System information Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04.2 TensorFlow installed from (source or bina

Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED , E tensorflow/stream_executor/cuda/cuda_event.cc:49] Error polling for event CUDNN failure Error: CUDNN_STATUS_ALLOC_FAILED mnistCUDNN.cpp:558. Stack Overflow for Teams is a private, secure spot for you and your coworkers to find and share information. Learn more TensorFlow: could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR but no other TF instances running

Machine learning education, Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED try to start training I get these error message: E tensorflow/stream_executor/cuda/cuda_blas​.cc:238] failed to create Re: Crash While Beginning Training. BTW, I was trying to build libtensorflow_cc.so, but I can't build it. Please see the last comment in #12747. I also tried building libtensorflow_inference.so with bazel, but my application has a lot of link errors: missing symbol errors, for example, SessionOptions class missing.

python, The Machine Learning Crash Course with TensorFlow APIs is a self-study guide for aspiring machine learning practitioners. It features a series of lessons with  If the version is not compatable with the CUDA version it won't throw the error while tensorflow installation but will interfere during memory allocation in the GPU. DO check your CUDA and CUDNN versions. Also follow the instructions about creation of sessions mentioned above.

Comments
  • "When running the code, the CPU goes up to a whopping 100%, suggesting that something is seriously off" – why so? High CPU loads are fine even when you use a GPU.
  • Only times I've seen spikes from idle to 100% CPU has been on infinite loops, but if you say it's normal in this case, it should be just fine, and shouldn't be a part of the problem.
  • I removed everything CUDA-related, went into the %PATH% and cleared all CUDA-variables. After reinstalling everything, it now finally works! The problem was the amount of paths I had, from previous attempts. They probably clashed with each other.
  • Well, seems I was too quick there! Now it works maybe 20% of the time. The other runs, I get the same crash. Better, but still not working as intended!
  • Sorry, but I forget to say that after edit .bashrc you may to do $ source ~/.bashrc . Make sure that you have only onde declaration of environment variable.
  • I have the same problem on Win10, so how can I add new environment variables on Win10?
  • There is no "lib64" folder in CUDA's root path on Win10.