Tracking *maximum* memory usage by a Python function

python debug memory usage
python get total memory
python reduce memory usage
python check memory usage of object
python memory usage statistics
python show all memory usage
psutil memory usage
python print memory used by process

I want to find out what the maximum amount of RAM allocated during the call to a function is (in Python). There are other questions on SO related to tracking RAM usage:

Which Python memory profiler is recommended?

How do I profile memory usage in Python?

but those seem to allow you more to track memory usage at the time the heap() method (in the case of guppy) is called. However, what I want to track is a function in an external library which I can't modify, and which grows to use a lot of RAM but then frees it once the execution of the function is complete. Is there any way to find out what the total amount of RAM used during the function call was?

This question seemed rather interesting and it gave me a reason to look into Guppy / Heapy, for that I thank you.

I tried for about 2 hours to get Heapy to do monitor a function call / process without modifying its source with zero luck.

I did find a way to accomplish your task using the built in Python library resource. Note that the documentation does not indicate what the RU_MAXRSS value returns. Another SO user noted that it was in kB. Running Mac OSX 7.3 and watching my system resources climb up during the test code below, I believe the returned values to be in Bytes, not kBytes.

A 10000ft view on how I used the resource library to monitor the library call was to launch the function in a separate (monitor-able) thread and track the system resources for that process in the main thread. Below I have the two files that you'd need to run to test it out.

Library Resource Monitor - whatever_you_want.py

import resource
import time

from stoppable_thread import StoppableThread


class MyLibrarySniffingClass(StoppableThread):
    def __init__(self, target_lib_call, arg1, arg2):
        super(MyLibrarySniffingClass, self).__init__()
        self.target_function = target_lib_call
        self.arg1 = arg1
        self.arg2 = arg2
        self.results = None

    def startup(self):
        # Overload the startup function
        print "Calling the Target Library Function..."

    def cleanup(self):
        # Overload the cleanup function
        print "Library Call Complete"

    def mainloop(self):
        # Start the library Call
        self.results = self.target_function(self.arg1, self.arg2)

        # Kill the thread when complete
        self.stop()

def SomeLongRunningLibraryCall(arg1, arg2):
    max_dict_entries = 2500
    delay_per_entry = .005

    some_large_dictionary = {}
    dict_entry_count = 0

    while(1):
        time.sleep(delay_per_entry)
        dict_entry_count += 1
        some_large_dictionary[dict_entry_count]=range(10000)

        if len(some_large_dictionary) > max_dict_entries:
            break

    print arg1 + " " +  arg2
    return "Good Bye World"

if __name__ == "__main__":
    # Lib Testing Code
    mythread = MyLibrarySniffingClass(SomeLongRunningLibraryCall, "Hello", "World")
    mythread.start()

    start_mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
    delta_mem = 0
    max_memory = 0
    memory_usage_refresh = .005 # Seconds

    while(1):
        time.sleep(memory_usage_refresh)
        delta_mem = (resource.getrusage(resource.RUSAGE_SELF).ru_maxrss) - start_mem
        if delta_mem > max_memory:
            max_memory = delta_mem

        # Uncomment this line to see the memory usuage during run-time 
        # print "Memory Usage During Call: %d MB" % (delta_mem / 1000000.0)

        # Check to see if the library call is complete
        if mythread.isShutdown():
            print mythread.results
            break;

    print "\nMAX Memory Usage in MB: " + str(round(max_memory / 1000.0, 3))

Stoppable Thread - stoppable_thread.py

import threading
import time

class StoppableThread(threading.Thread):
    def __init__(self):
        super(StoppableThread, self).__init__()
        self.daemon = True
        self.__monitor = threading.Event()
        self.__monitor.set()
        self.__has_shutdown = False

    def run(self):
        '''Overloads the threading.Thread.run'''
        # Call the User's Startup functions
        self.startup()

        # Loop until the thread is stopped
        while self.isRunning():
            self.mainloop()

        # Clean up
        self.cleanup()

        # Flag to the outside world that the thread has exited
        # AND that the cleanup is complete
        self.__has_shutdown = True

    def stop(self):
        self.__monitor.clear()

    def isRunning(self):
        return self.__monitor.isSet()

    def isShutdown(self):
        return self.__has_shutdown


    ###############################
    ### User Defined Functions ####
    ###############################

    def mainloop(self):
        '''
        Expected to be overwritten in a subclass!!
        Note that Stoppable while(1) is handled in the built in "run".
        '''
        pass

    def startup(self):
        '''Expected to be overwritten in a subclass!!'''
        pass

    def cleanup(self):
        '''Expected to be overwritten in a subclass!!'''
        pass

Monitoring memory usage of a running Python program, At Survata, we do a lot of data processing using Python and its suite of data The easiest way to track memory usage is to use the operating system itself. you could write a monitoring script that uses ps to track memory usage of the maximum Resident Set Size — which is the amount of memory that is  Monitoring memory usage. When you invoke measure_usage() on an instance of this class, it will enter a loop, and every 0.1 seconds, it will take a measurement of memory usage. Any increase in

It is possible to do this with memory_profiler. The function memory_usage returns a list of values, these represent the memory usage over time (by default over chunks of .1 second). If you need the maximum, just take the max of that list. Little example:

from memory_profiler import memory_usage
from time import sleep

def f():
    # a function that with growing
    # memory consumption
    a = [0] * 1000
    sleep(.1)
    b = a * 100
    sleep(.1)
    c = b * 100
    return a

mem_usage = memory_usage(f)
print('Memory usage (in chunks of .1 seconds): %s' % mem_usage)
print('Maximum memory usage: %s' % max(mem_usage))

In my case (memory_profiler 0.25) if prints the following output:

Memory usage (in chunks of .1 seconds): [45.65625, 45.734375, 46.41015625, 53.734375]
Maximum memory usage: 53.734375

How to Profile Memory Usage in Python, Good developers will want to track the memory usage of their application The easiest way to profile a single method or function is the open  Tracking*maximum* memory usage by a Python function (4) Have been struggling with this task as well. After experimenting with psutil and methods from Adam, I wrote a function (credits to Adam Lewis) to measure the memory used by a specific function. People may find it easier to grab and use. 1) measure_memory_usage. 2) test measure_memory_usage.

This appears to work under Windows. Don't know about other operating systems.

In [50]: import os

In [51]: import psutil

In [52]: process = psutil.Process(os.getpid())

In [53]: process.get_ext_memory_info().peak_wset
Out[53]: 41934848

memory-profiler · PyPI, A module for monitoring memory usage of a python program. If a python script with decorator @profile is called using -m memory_profiler in the command be difficult to identify the part of the code that is causing the highest memory usage. python deep_getsizeof ( [x, x, x, x, x], set ()) 156. A list that contains the string x 5 times takes 156 bytes (72 + 5 * 8 + 44). The last example shows that deep_getsizeof () counts references to the same object (the x string) just once, but each reference’s pointer is counted.

You can use python library resource to get memory usage.

import resource
resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

It will give memory usage in kilobytes, to convert in MB divide by 1000.

How to measure memory used by a program in Python 3, Why did the print statement turn into a function for Python 3? Total memory used by Python process? How do I Worried that someone is tracking you online? The tracemalloc.start() function can be called at runtime to start tracing Python memory allocations. By default, a trace of an allocated memory block only stores the most recent frame (1 frame). To store 25 frames at startup: set the PYTHONTRACEMALLOC environment variable to 25 , or use the -X tracemalloc=25 command line option.

Improvement of the answer of @Vader B (as it did not work for me out of box):

$ /usr/bin/time --verbose  ./myscript.py
        Command being timed: "./myscript.py"
        User time (seconds): 16.78
        System time (seconds): 2.74
        Percent of CPU this job got: 117%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:16.58
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 616092   # WE NEED THIS!!!
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 432750
        Voluntary context switches: 1075
        Involuntary context switches: 118503
        Swaps: 0
        File system inputs: 0
        File system outputs: 800
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Different ways to get memory consumption or lessons learned from , The above function returns the memory usage of the current Python process in MiB. Depending on the platform it will choose the most accurate  A simple example to calculate the memory usage of a block of codes / function using memory_profile, while returning result of the function: import memory_profiler as mp def fun(n): tmp = [] for i in range(n): tmp.extend(list(range(i*i))) return "XXXXX" calculate memory usage before running the code then calculate max usage during the code:

Python Exception Handling - MemoryError, If you're using a 32-bit Python then the maximum memory allocation given to the method, which provides current memory usage stats when invoked. software provides real-time error monitoring and automatic exception  The easiest way to profile a single method or function is the open source memory-profiler package. It's similar to line_profiler , which I've written about before . You can use it by putting the @profile decorator around any function or method and running python -m memory_profiler myscript. You'll see line-by-line memory usage once your script

tracemalloc — Trace memory allocations, The tracemalloc module is a debug tool to trace memory blocks allocated by Python. The tracemalloc.start() function can be called at runtime to start tracing Python Get the maximum number of frames stored in the traceback of a trace. Get the memory usage in bytes of the tracemalloc module used to store traces of​  The first column represents the line number of the code that has been profiled, the second column ( Mem usage) the memory usage of the Python interpreter after that line has been executed. The third column ( Increment) represents the difference in memory of the current line with respect to the last one.

7 tips to Time Python scripts and control Memory & CPU usage, And also, you should also control the memory and CPU usage, as it can point you use the @profile decorator to mark which functions to track. La fonction memory_usage renvoie une liste de valeurs, celles-ci représentent l'utilisation de la mémoire au fil du temps (par défaut sur des portions de .1 seconde). Si vous avez besoin du maximum, prenez le maximum de cette liste.

Comments
  • Thanks for the detailed reply!
  • @astrofrog Glad to help. It's something I can also benefit from in the future.
  • I put this code up as a gist so it's easy to download the files: gist.github.com/b54fafd87634f017d50d
  • @Tom Are you sure that the data is in GB? Tracking it on OSX Lion made it look as if it were MB. Either way, thanks for putting it on github!
  • @AdamLewis I guess I'm not 100% sure, but I got numbers that were way too small (on Linux) and more reasonable if I multiplied by 1000. Seemed to match what I was seeing in the system monitor. Would be a bummer if getrusage doesn't account for OS differences, but the number is going to be used for relative comparisons while optimising so it's not a big deal.
  • brilliant solution for windows. pypi.python.org/pypi/memory_profiler but needs also to unstall psutil
  • What system are you using @Wajahat ?
  • Windows 8, with Python 2.7
  • In case when function f has arguments args and keywords arguments kw you can invoke it with memory_usage((f, args, kw)).
  • I believe that this should be memory_info_ex.peak_set
  • See docs. pythonhosted.org/psutil/#psutil.Process.memory_info Should be process.memory_info().rss to be cross plateform