Optimizing Python Codes

Sometimes, when executing Python scripts, we encounter memory errors. These errors are primarily due to some variables that have high memory consumption.

In this tutorial, we will focus on profiling Python codes to optimize memory consumption. Memory profiling is a process using which we can dissect our code and identify variables that lead to memory errors.

1.) Memory Profiler

• The Memory Profiler is a python package that evaluates each line of Python code written within a function and correspondingly checks the usage of internal memory.
• We can either use pip or conda package managersto install this package. If using Anaconda distribution, I would recommend using conda install as it resolves dependencies and environment issues automatically.

#### Install Packages
conda install memory_profiler

• Once installed, the profiler can be loaded within the iPython environment using the following line of code:

#### Loading Memory Profiler Package
%load_ext memory_profiler

• After loading, use the following syntax to memory profile any predefined function:

#### Memory Profiling a Function
%mprun -f function_name_only Call_to_function_with_arguments

Note: To profile a function using a Memory Profiler, import it as a module which means, writing it in a separate .py file and then importing it into the main program like any other package.

2.) Working Example

For this tutorial, I have used the same function as available on Memory Profiler’s PyPi webpage. It creates two list variables with a large number of elements and then deletes one of them (refer to the function definition in the code below). We then save and import this function into the main program for profiling.

#### Function Definition (Saved as example.py file)
def my_func(): 
    a = [1] * (10 ** 6) 
    b = [2] * (2 * 10 ** 7) 
    del b 
    return a#### Loading the line profiler within ipython/jupyter environment
%load_ext memory_profiler
from example import my_func#### Profiling the function using line_profiler
%mprun -f my_func my_func()#### Memory Profile Output
Line #    Mem usage    Increment   Line Contents
================================================
     1     50.8 MiB     50.8 MiB   def my_func():
     2     58.4 MiB      7.6 MiB    a = [1] * (10 ** 6)
     3    211.0 MiB    152.6 MiB    b = [2] * (2 * 10 ** 7)
     4     58.4 MiB      0.0 MiB    del b
     5     58.4 MiB      0.0 MiB    return a

Explanation

From the output of the Memory Profiler, one can see that the output table has four columns:

• Column 1 (Line #) — Line number of code
• Column 2 (Mem Usage) — Total memory used by function till the line number
• Column 3 (Increment) — Memory used by a specific line of the program
• Column 4(Content) — Actual code content

Note: The memory consumption dropped significantly after dropping the variable “b” from the program.

From the table, it is clear that the creation of variable “b” has led to a sudden spike in memory usage. In real-time scenarios, once we are aware of such findings, we can always look at alternative ways to program our code such that the memory consumption stays within acceptable limits.

Closing note

In this short tutorial, we learned about memory profile our Python codes. To learn about time profiling your Python codes, please refer to this tutorial.

I hope this tutorial of the Did You Know series was informative and resourceful.

We will try and bring more engaging topics in future tutorials. Till then:

HAPPY LEARNING ! ! ! !