Tensorflow Allocation Memory: Allocation of 38535168 exceeds 10% of system memory

tensorflow allocation of exceeds 10% of system memory killed
allocation exceeds 10% of system memory tensorflow
w tensorflow/core/framework/allocator.cc:108] allocation of exceeds 10% of system memory.
keras allocation exceeds 10% of system memory
allocation of exceeds 10% of system memory killed
allocation of 1811939328 exceeds 10 of system memory
w tensorflow core framework allocator cc 107 allocation of 822083584 exceeds 10 of system memory
allocation of 6400000000 exceeds 10% of system memory

Using ResNet50 pre-trained Weights I am trying to build a classifier. The code base is fully implemented in Keras high-level Tensorflow API. The complete code is posted in the below GitHub Link.

Source Code: Classification Using RestNet50 Architecture

The file size of the pre-trained model is 94.7mb.

I loaded the pre-trained file

new_model = Sequential()

new_model.add(ResNet50(include_top=False,
                pooling='avg',
                weights=resnet_weight_paths))

and fit the model

train_generator = data_generator.flow_from_directory(
    'path_to_the_training_set',
    target_size = (IMG_SIZE,IMG_SIZE),
    batch_size = 12,
    class_mode = 'categorical'
    )

validation_generator = data_generator.flow_from_directory(
    'path_to_the_validation_set',
    target_size = (IMG_SIZE,IMG_SIZE),
    class_mode = 'categorical'
    )

#compile the model

new_model.fit_generator(
    train_generator,
    steps_per_epoch = 3,
    validation_data = validation_generator,
    validation_steps = 1
)

and in the Training dataset, I have two folders dog and cat, each holder almost 10,000 images. When I compiled the script, I get the following error

Epoch 1/1 2018-05-12 13:04:45.847298: W tensorflow/core/framework/allocator.cc:101] Allocation of 38535168 exceeds 10% of system memory. 2018-05-12 13:04:46.845021: W tensorflow/core/framework/allocator.cc:101] Allocation of 37171200 exceeds 10% of system memory. 2018-05-12 13:04:47.552176: W tensorflow/core/framework/allocator.cc:101] Allocation of 37171200 exceeds 10% of system memory. 2018-05-12 13:04:48.199240: W tensorflow/core/framework/allocator.cc:101] Allocation of 37171200 exceeds 10% of system memory. 2018-05-12 13:04:48.918930: W tensorflow/core/framework/allocator.cc:101] Allocation of 37171200 exceeds 10% of system memory. 2018-05-12 13:04:49.274137: W tensorflow/core/framework/allocator.cc:101] Allocation of 19267584 exceeds 10% of system memory. 2018-05-12 13:04:49.647061: W tensorflow/core/framework/allocator.cc:101] Allocation of 19267584 exceeds 10% of system memory. 2018-05-12 13:04:50.028839: W tensorflow/core/framework/allocator.cc:101] Allocation of 19267584 exceeds 10% of system memory. 2018-05-12 13:04:50.413735: W tensorflow/core/framework/allocator.cc:101] Allocation of 19267584 exceeds 10% of system memory.

Any ideas to optimize the way to load the pre-trained model (or) get rid of this warning message?

Thanks!

Try reducing batch_size attribute to a small number(like 1,2 or 3). Example:

train_generator = data_generator.flow_from_directory(
    'path_to_the_training_set',
    target_size = (IMG_SIZE,IMG_SIZE),
    batch_size = 2,
    class_mode = 'categorical'
    )

Tensorflow Allocation Memory: Allocation of 38535168 exceeds 10 , Try reducing batch_size attribute to a small number(like 1,2 or 3). Example: train_generator = data_generator.flow_from_directory(  If you are using tensorflow to train a deep learning model, you may encouter this allocation memory error: Allocation exceeds 10% of system memory. In this tutorial, we will introduce some methods to fix it.

I was having the same problem while running Tensorflow container with Docker and Jupyter notebook. I was able to fix this problem by increasing the container memory.

On Mac OS, you can easily do this from:

       Docker Icon > Preferences >  Advanced > Memory

Drag the scrollbar to maximum (e.g. 4GB). Apply and it will restart the Docker engine.

Now run your tensor flow container again.

It was handy to use the docker stats command in a separate terminal It shows the container memory usage in realtime, and you can see how much memory consumption is growing:

CONTAINER ID   NAME   CPU %   MEM USAGE / LIMIT     MEM %    NET I/O             BLOCK I/O           PIDS
3170c0b402cc   mytf   0.04%   588.6MiB / 3.855GiB   14.91%   13.1MB / 3.06MB     214MB / 3.13MB      21

Memory problems with smaller CNN, W T:\src\github\tensorflow\tensorflow\core\framework\allocator.cc:101] Allocation of 1037238272 exceeds 10% of system memory. python - killed - Tensorflow Allocation Memory: Allocation of 38535168 exceeds 10% of system memory tf allocation exceeds 10% (3) Using ResNet50 pre-trained Weights I am trying to build a classifier.

Alternatively, you can set the environment variable TF_CPP_MIN_LOG_LEVEL=2 to filter out info and warning messages. I found that on this github issue where they complain about the same output. To do so within python, you can use the solution from here:

import os
import tensorflow as tf
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'

You can even turn it on and off at will with this. I test for the maximum possible batch size before running my code, and I can disable warnings and errors while doing this.

Problem with memory allocation in Keras TensorFlow =( : learnpython, W tensorflow/core/framework/allocator.cc:101] Allocation of 8589934592 exceeds 10% of system memory. Killed. I tried following the info here but I'm not  Training stuck on Allocation exceeds 10% of system memory stock example script provided in TensorFlow): Yes of 603979776 exceeds 10% of system memory.

I was running a small model on a CPU and had the same issue. Adding:os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' resolved it.

TensorFlow Allocation of 1511424000 exceeds 10% of system , Tensorflow: Allocation of 44302336 exceeds 10% of system memory problem on RasPi. Tue Jun 11, 2019 7:09 am. I am trying to run an image processing code  Thanks for contributing an answer to Stack Overflow! Please be sure to answer the question.Provide details and share your research! But avoid …. Asking for help, clarification, or responding to other answers.

Tensorflow: Allocation of 44302336 exceeds 10% of system memory , However Kera's Tensorflow Backend will allocate the whole GPU memory by Allocation Memory: Allocation of 38535168 exceeds 10% of system memory The​  @KoizumiNao This looks like a support issue. GitHub is mainly for Bugs and performance related issues. Your model might be big and cannot fit into the memory. Please post in stackoverflow and provide more details on your hardware.

Tensorflow log memory allocation, It seems to be a memory allocation problem. I reduced the size of my model and make smaller all the parameters but nothing has changed. I reduced the size of my model and make smaller all the parameters but nothing has changed.

Allocation of 18599850000 exceeds 10% of system memory. I get that the sparse matrix is not exactly small, but it is not excessively large either. Storing the coordinate form on disk is <2 gb. so What is with the rampant memory usage?

Comments
  • To clarify, does the model run after these messages?
  • Yes it run.....
  • In that case, take a look at stackoverflow.com/a/42121886/6824418 ? Unless there's some other reason you need to reduce the memory usage.