How can you profile a Python script?

How can you profile a Python script?

cprofile python 3
python profiler gui
python profile decorator
python profiling pycharm
python cprofile sort
python line profiler
best python profiler
cprofile multiprocessing

Project Euler and other coding contests often have a maximum time to run or people boast of how fast their particular solution runs. With Python, sometimes the approaches are somewhat kludgey - i.e., adding timing code to __main__.

What is a good way to profile how long a Python program takes to run?


Python includes a profiler called cProfile. It not only gives the total running time, but also times each function separately, and tells you how many times each function was called, making it easy to determine where you should make optimizations.

You can call it from within your code, or from the interpreter, like this:

import cProfile
cProfile.run('foo()')

Even more usefully, you can invoke the cProfile when running a script:

python -m cProfile myscript.py

To make it even easier, I made a little batch file called 'profile.bat':

python -m cProfile %1

So all I have to do is run:

profile euler048.py

And I get this:

1007 function calls in 0.061 CPU seconds

Ordered by: standard name
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    1    0.000    0.000    0.061    0.061 <string>:1(<module>)
 1000    0.051    0.000    0.051    0.000 euler048.py:2(<lambda>)
    1    0.005    0.005    0.061    0.061 euler048.py:2(<module>)
    1    0.000    0.000    0.061    0.061 {execfile}
    1    0.002    0.002    0.053    0.053 {map}
    1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler objects}
    1    0.000    0.000    0.000    0.000 {range}
    1    0.003    0.003    0.003    0.003 {sum}

EDIT: Updated link to a good video resource from PyCon 2013 titled Python Profiling Also via YouTube.

Debugging and Profiling, Debugging and Profiling¶. These libraries help you with Python development: the debugger enables you to step through code, analyze stack frames and set  You can run it under a profiling tools and you have a choice of a number of such tools: The Python standard library provides two different implementations of the same profiling interface: cProfile is recommended for most users; it’s a C extension with reasonable overhead that makes it suitable for profiling long-running programs.


A while ago I made pycallgraph which generates a visualisation from your Python code. Edit: I've updated the example to work with 3.3, the latest release as of this writing.

After a pip install pycallgraph and installing GraphViz you can run it from the command line:

pycallgraph graphviz -- ./mypythonscript.py

Or, you can profile particular parts of your code:

from pycallgraph import PyCallGraph
from pycallgraph.output import GraphvizOutput

with PyCallGraph(output=GraphvizOutput()):
    code_to_profile()

Either of these will generate a pycallgraph.png file similar to the image below:

How can you profile a Python script?, Python includes a profiler called cProfile. It not only gives the total running time, but also times each function separately, and tells you how many times each  Python includes a profiler called cProfile. It not only gives the total running time, but also times each function separately, and tells you how many times each function was called, making it easy to determine where you should make optimizations. You can call it from within your code, or from the interpreter, like this: import cProfile


It's worth pointing out that using the profiler only works (by default) on the main thread, and you won't get any information from other threads if you use them. This can be a bit of a gotcha as it is completely unmentioned in the profiler documentation.

If you also want to profile threads, you'll want to look at the threading.setprofile() function in the docs.

You could also create your own threading.Thread subclass to do it:

class ProfiledThread(threading.Thread):
    # Overrides threading.Thread.run()
    def run(self):
        profiler = cProfile.Profile()
        try:
            return profiler.runcall(threading.Thread.run, self)
        finally:
            profiler.dump_stats('myprofile-%d.profile' % (self.ident,))

and use that ProfiledThread class instead of the standard one. It might give you more flexibility, but I'm not sure it's worth it, especially if you are using third-party code which wouldn't use your class.

Python Profilers: Learn the Basics of a profiler for Python in this blog, What part of the code should I profile? Now let's delve into profiling specifics. The term “profiling” is mainly used for performance testing, and the  To profile a Python script you can call it within your code, or from the interpreter, like as follows:-import cProfile . cProfile.run('foo()')


The python wiki is a great page for profiling resources: http://wiki.python.org/moin/PythonSpeed/PerformanceTips#Profiling_Code

as is the python docs: http://docs.python.org/library/profile.html

as shown by Chris Lawlor cProfile is a great tool and can easily be used to print to the screen:

python -m cProfile -s time mine.py <args>

or to file:

python -m cProfile -o output.file mine.py <args>

PS> If you are using Ubuntu, make sure to install python-profile

sudo apt-get install python-profiler 

If you output to file you can get nice visualizations using the following tools

PyCallGraph : a tool to create call graph images install:

 sudo pip install pycallgraph

run:

 pycallgraph mine.py args

view:

 gimp pycallgraph.png

You can use whatever you like to view the png file, I used gimp Unfortunately I often get

dot: graph is too large for cairo-renderer bitmaps. Scaling by 0.257079 to fit

which makes my images unusably small. So I generally create svg files:

pycallgraph -f svg -o pycallgraph.svg mine.py <args>

PS> make sure to install graphviz (which provides the dot program):

sudo pip install graphviz

Alternative Graphing using gprof2dot via @maxy / @quodlibetor :

sudo pip install gprof2dot
python -m cProfile -o profile.pstats mine.py
gprof2dot -f pstats profile.pstats | dot -Tsvg -o mine.svg

Python 102: How to Profile Your Code, Code profiling is an attempt to find bottlenecks in your code. Profiling is Python comes with three profilers built in: cProfile, profile and hotshot. To check time metrics given by each function and to better understand how much time is spent on functions, you can use the inbuilt cProfile in python. Going into more detailed metrics like performance, time is not the only metric. You can worry about memory, threads etc. Profiling options: 1. line_profiler is another profiler used commonly to find out timing metrics line-by-line.


@Maxy's comment on this answer helped me out enough that I think it deserves its own answer: I already had cProfile-generated .pstats files and I didn't want to re-run things with pycallgraph, so I used gprof2dot, and got pretty svgs:

$ sudo apt-get install graphviz
$ git clone https://github.com/jrfonseca/gprof2dot
$ ln -s "$PWD"/gprof2dot/gprof2dot.py ~/bin
$ cd $PROJECT_DIR
$ gprof2dot.py -f pstats profile.pstats | dot -Tsvg -o callgraph.svg

and BLAM!

It uses dot (the same thing that pycallgraph uses) so output looks similar. I get the impression that gprof2dot loses less information though:

Profiling in Python, It is better to optimize the code inorder to increase the efficiency of a program. So, perform some standard tests to ensure optimization and we can improve the  Directly using the Profile class allows formatting profile results without writing the profile data to a file: import cProfile , pstats , io from pstats import SortKey pr = cProfile . Profile () pr . enable () # do something pr . disable () s = io .


Use a High Level Approach for Quick Profiling in Python, Find the most quick and easy performance bottlenecks when profiling your code by using this high-level approach to profiling and speeding up  cProfile and profile provide deterministic profiling of Python programs. A profile is a set of statistics that describes how often and for how long various parts of the program executed. These statistics can be formatted into reports via the pstats module.


Profiling and optimizing your Python code, As soon as your code stops being trivial and runs on real-world data volumes, you will run into performance issues. You've heard that Python is  Common system administration tasks can take a huge amount of time, but with a Python script, you can automate these tasks so that they take no time at all. For example, Python can read the contents of your computer's file system and perform operations like printing an outline of your files and directories, moving folders from one directory to another, or renaming hundreds of files.


Profiling Python using cProfile: a concrete case, Here, I would like to show you how you can quickly profile and analyze your Python code to find what part of the code you should optimize. Whenever a Python program is called, Python will check, if a compiled version with the .pyc suffix exists. This file has to be newer than the file with the .py suffix. If such a file exists, Python will load the byte code, which will speed up the start up time of the script.