Working With the Python operator Module

Using the Python operator Module’s Basic Functions

In this section, you’ll learn about the operator module’s operator-equivalent functions that mimic built-in operators, and you’ll pass them as arguments to higher-order functions. You’ll also learn how to save them for later use. Finally, you’ll investigate the performance of the operator-equivalent functions and uncover why you should never use them where a built-in Python operator will do instead.

Learning How the Basic Functions Work

The Python operator module contains over forty functions, many of which are equivalent to the Python operators that you’re already familiar with. Here’s an example:

>>> import operator

>>> operator.add(5, 3)  # 5 + 3
8

>>> operator.__add__(5, 3)  # 5 + 3
8

Here, you add 5 and 3 together using both add() and __add__(). Both produce the same result. On the face of it, these functions provide you with the same functionality as the Python + operator, but this isn’t their purpose.

If you take a look at the list of functions that operator provides for you, then you’ll discover that they cover not only the arithmetic operators, but also the equality, identity, Boolean, and even bitwise operators. Try out a random selection of them:

>>> operator.truediv(5, 2)  # 5 / 2
2.5

>>> operator.ge(5, 2)  # 5 >= 2
True

>>> operator.is_("X", "Y")  # "X" is "Y"
False

>>> operator.not_(5 < 3)  # not 5 < 3
True

>>> bin(operator.and_(0b101, 0b110))  # 0b101 & 0b110
'0b100'

In the code above, you work with a selection from the five main categories. First, you use the equivalent of an arithmetic operator, and then you try out equality and identity operator examples in the second and third examples, respectively. In the fourth example, you try a Boolean logical operator, while the final example uses a bitwise operator The comments show the equivalent Python operators.

Before reading the rest of this tutorial, feel free to take some time to experiment with the range of operator-equivalent functions that Python’s operator module provides for you. You’ll learn how to use them next.

Passing Operators as Arguments Into Higher-Order Functions

You use the operator-equivalent functions most commonly as arguments for higher-order functions. You could write a higher-order function that performs a series of different tasks depending on the operator function passed to it. Suppose, for example, you wanted a single function that could perform addition, subtraction, multiplication, and division. One messy way of doing this would be to use an if … elif statement as follows:

>>> def perform_operation(operator_string, operand1, operand2):
...     if operator_string == "+":
...         return operand1 + operand2
...     elif operator_string == "-":
...         return operand1 - operand2
...     elif operator_string == "*":
...         return operand1 * operand2
...     elif operator_string == "/":
...         return operand1 / operand2
...     else:
...         return "Invalid operator."
...

In your perform_operation() function, the first parameter is a string representing one of the four basic arithmetic operations. To test the function, you pass in each of the four operators. The results are what you’d expect:

>>> number1 = 10
>>> number2 = 5
>>> calculations = ["+", "-", "*", "/"]

>>> for op_string in calculations:
...     perform_operation(op_string, number1, number2)
...
15
5
50
2.0

This code is not only messy, but also limited to the four operators defined in the elif clauses. Try, for example, passing in a modulo operator (%), and the function will return an "Invalid operator" message instead of the modulo division result that you were hoping for.

This is where you can make excellent use of the operator functions. Passing these into a function gives you several advantages:

>>> def perform_operation(operator_function, operand1, operand2):
...     return operator_function(operand1, operand2)
...

This time, you’ve improved your perform_operation() function so that the first parameter can accept any of the operator module’s functions that take exactly two arguments. The second and third parameters are those arguments.

The revised test code is similar to what you did before, except you pass in operator functions for your perform_operation() function to use:

>>> from operator import add, sub, mul, truediv

>>> number1 = 10
>>> number2 = 5
>>> calculations = [add, sub, mul, truediv]

>>> for op_function in calculations:
...     perform_operation(op_function, number1, number2)
...
15
5
50
2.0

This time, your calculations list contains references to the functions themselves. Note that you pass in function names and not function calls. In other words, you pass in add to perform_operation(), and not add(). You’re passing in the function object, not the result of its execution. Remember, the name of a function is actually a reference to its code. Using the () syntax calls the function.

There are two advantages to using your updated version of perform_operation(). The first is expandability. You can use the revised code with any of the other operator functions that require exactly two arguments. Indeed, you might like to experiment by passing the operator module’s mod(), pow(), and repeat() functions to both versions of your function. Your updated version works as expected, while your original version returns "Invalid operator".

The second advantage is readability. Take a look at both versions of your perform_operation() function, and you’ll notice that your second version is not only significantly shorter, but also more readable, than the original.

Passing functions as arguments to other functions is a feature that you’ll often use in functional programming. This is one of the main purposes of the operator module. You’ll study other examples of this later.

Serializing operator Module Functions

One way of saving objects, including functions, to disk is to serialize them. In other words, your code converts them into byte streams and stores them on disk for later use. Conversely, when you read serialized objects back from disk, you deserialize them, allowing them to be read from disk into a program for use.

There are several reasons why you might serialize functions, including to save them for future use in another program or to pass them between different processes running on one or more computers.

A common way to serialize functions in Python is by using the pickle module. This, along with its dictionary wrapper shelve, provides one of the most efficient ways of storing data. However, when you serialize a function using pickle, then you only serialize its fully qualified name, not the code in the function body. When you deserialize a function, the environment must provide access to the function’s code. The function can’t work otherwise.

To see an example, you’ll revisit the earlier perform_operation() example. You’ll call different operator functions to perform the different operations. The following code adds a dictionary that you’ll use to map a string operator to its matching function:

>>> import operator
>>> operators = {
...     "+": operator.add,
...     "-": operator.sub,
...     "*": operator.mul,
...     "/": operator.truediv,
... }

>>> def perform_operation(op_string, number1, number2):
...     return operators[op_string](number1, number2)
...

>>> perform_operation("-", 10, 5)
5

The operations supported by perform_operation() are the ones defined in operators. As an example, you run the "-" operation, which calls operator.sub() in the background.

One way to share the supported operators between processes is to serialize the operators dictionary to disk. You can do this using pickle as follows:

>>> import pickle
>>> with open("operators.pkl", mode="wb") as f:
...     pickle.dump(operators, f)
...

You open a binary file for writing. To serialize operators, you call pickle.dump() and pass the structure that you’re serializing and the handle of the destination file.

This creates the file operators.pkl in your local working directory. To demonstrate how to reuse operators in a different process, restart your Python shell and load the pickled file:

>>> import pickle
>>> with open("operators.pkl", mode="rb") as f:
...     operators = pickle.load(f)
...
>>> operators
{'+': <built-in function add>, '-': <built-in function sub>,
 '*': <built-in function mul>, '/': <built-in function truediv>}

Firstly, you import pickle again and reopen the binary file for reading. To read the operator structure, you use pickle.load() and pass in the file handle. Your code then reads in the saved definition and assigns it to a variable named operators. This name doesn’t need to match your original name. This variable points to the dictionary that references the different functions, assuming they’re available.

Note that you don’t need to explicitly import operator, although the module needs to be available for Python to import in the background.

You can define perform_operation() again to see that it can refer to and use the restored operators:

>>> def perform_operation(op_string, number1, number2):
...     return operators[op_string](number1, number2)
...

>>> perform_operation("*", 10, 5)
50

Great! Your code handles multiplication as you’d expect.

Now, there’s nothing special about operator supporting pickling of functions. You can pickle and unpickle any top-level function, as long as Python is able to import it in the environment where you’re loading the pickled file.

However, you can’t serialize anonymous lambda functions like this. If you implemented the example without using the operator module, then you’d probably define the dictionary as follows:

>>> operators = {
...     "+": lambda a, b: a + b,
...     "-": lambda a, b: a - b,
...     "*": lambda a, b: a * b,
...     "/": lambda a, b: a / b,
... }

The lambda construct is a quick way to define simple functions, and they can be quite useful. However, because pickle doesn’t serialize the function body, only the name of the function, you can’t serialize the nameless lambda functions:

>>> import pickle
>>> with open("operators.pkl", mode="wb") as f:
...     pickle.dump(operators, f)
...
Traceback (most recent call last):
  ...
PicklingError: Can't pickle <function <lambda> at 0x7f5b946cfba0>: ...

If you try to serialize lambda functions with pickle, then you’ll get an error. This is a case where you can often use operator functions instead of lambda functions.

Look back at your serialization code and notice that you imported both operator and pickle, while your deserialization code imported only pickle. You didn’t need to import operator because pickle did this automatically for you when you called its load() function. This works because the built-in operator module is readily available.

Investigating operator Function Performance Against the Alternatives

Now that you have an idea of how to use the operator-equivalent functions, you may wonder if you should use them instead of either the Python operators or lambda functions. The answer is no to the first case and yes to the second. The built-in Python operators are always significantly faster than their operator module counterparts. However, the operator module’s functions are faster than lambda functions, and they’re more readable as well.

If you wish to time the operator module’s functions against their built-in or lambda equivalents, then you can use the timeit module. The best way to do this is to run it directly from the command line:

PS> python -m timeit "(lambda a, b: a + b)(10, 10)"
5000000 loops, best of 5: 82.3 nsec per loop
PS> python -m timeit -s "from operator import add" "add(10, 10)"
10000000 loops, best of 5: 24.5 nsec per loop
PS> python -m timeit "10 + 10"
50000000 loops, best of 5: 5.19 nsec per loop

PS> python -m timeit "(lambda a, b: a ** b)(10, 10)"
1000000 loops, best of 5: 226 nsec per loop
PS> python -m timeit -s "from operator import pow" "pow(10, 10)"
2000000 loops, best of 5: 170 nsec per loop
PS> python -m timeit "10 ** 10"
50000000 loops, best of 5: 5.18 nsec per loop

The above PowerShell session uses the timeit module to compare the performance of various implementations of addition and exponentiation. Your results show that for both operations, the built-in operator is fastest, with the operator module function only outperforming the lambda function. The actual time values themselves are machine-specific, but their relative differences are significant.

Go ahead and try the other operator functions out for yourself. Although the exact timings will vary from machine to machine, their relative differences will still show that built-in operators are always faster than the operator module equivalents, which are always faster than lambda functions.

Now you’re familiar with the operator-equivalent functions from the operator module, but you might want to spend some time exploring the rest of these functions. Once you’re ready to move on, keep reading to investigate some of the other ways to use operator.