How to Profile Your Python Script/Module
source link: https://jdhao.github.io/2023/09/12/python_profiling_visualization/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
How to Profile Your Python Script/Module
In my previous post, I shared how to profile a single function using line_profiler. In this post, I want to share how to profile Python script/module and visualize the profile stat to pinpoint the slow part of the code.
Generate the raw profiling stat
In Python, we can use cProfile to generate the raw profile result:
python -m cProfile -o {script_name}.prof {your_script}.py [script_argument]
The -o
options tells cProfile to generate a profile stat file, which is in binary format and not human readable.
In order to interpret the profiling stat easily, we need to install some tools to visualize the result.
static profile graph generation
flameprof
flameprof can generate a flame graph in SVG format. It is not actively maintained anymore.
pip install flameprof
flameprof input.prof > output.svg
gprof2dot
gprof2dot can generate a directed call graph for your script with their running time.
pip install gprof2dot
gprof2dot -f pstats input.prof | dot -Tsvg -o output.svg
interactive profile graph generation
snakeviz
snakeviz can also generate flamegraph from cProfile stat. It also has great interactive UI to explore the profiling result.
pip install snakeviz
snakeviz input.prof
Tuna is like snakeviz, but it claims to solve some of the limitation of snakeviz. The UI is less polished compared to snakeviz, though.
Py-spy
py-spy can work with speedscope to generate flamegraph.
pip install py-spy
# on macOS, needs sudo to run py-spy
sudo py-spy record --format speedscope -r 500 -F -o output.speedscope python some_script.py
chown -R {your-user-name} output.speedscope
# use speedscope to open the file, https://www.speedscope.app/
py-spy can also profile the running program without interrupting it.
pyinstrument
pyinstrument is another tool you can use to generate profiling graph. You can directly show the result in terminal:
pip install pyinstrument
pyinstrument your_script.py
You can also generate other output format:
pyinstrument -r html your_script.py
# or generate speedscope file, then inspect with speedscope
pyinstrument -r speedscope your_script.py
references
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK