Python Does What?!?

We all have our favorite way of intentionally raising an exception in Python. Some like referencing an undefined variable to get a simple NameError, others might import a module that doesn't exist for a bold ImportError.

But the tasteful exceptioneer knows to reach for that classic computer-confounding conundrum: 1/0 for a satisfyingly descriptive DivisionByZero.

So, when does dividing by 0 not raise DivisionByZero?

Why, when you divide 0 by a Decimal(0), of course!

The numerator type doesn't seem to matter either:

>>> 0 / Decimal(0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
decimal.InvalidOperation: [<class 'decimal.DivisionUndefined'>]

"InvalidOperation" just doesn't quite have the same ring to it! Well, they can't all be heroes. :)

Consider a REPL with two tuples, a and b.

>>> type(a), type(b)
(<type 'tuple'>, <type 'tuple'>)
>>> a == b
True

So far, so good.  But let's dig deeper...

>>> a[0] == b[0]
False

The tuples are equal, but their contents is not.

>>> a is b
True


In fact, there was only ever one tuple.
What is this madness?

>>> a
(nan,)

Welcome to the float zone.

Many parts of python assume that a is b implies a == b, but floats break this assumption.  They also break the assumption that hash(a) == hash(b) implies a == b.

>>> hash(float('nan')) == hash(float('nan'))
True

Dicts handle this pretty elegantly:

>>> n = float('nan')
>>> {n: 1}[n]
1

>>> a = {float('nan'): 1, float('nan'): 2}
>>> a
{nan: 1, nan: 2}

... but here at PDW we abhor that kind of defeatism!

>>> import ctypes
>>> tup = (None,)
>>> ctypes.pythonapi.PyTuple_SetItem.argtypes = ctypes.c_void_p, ctypes.c_int, ctypes.c_void_p
>>> ctypes.pythonapi.PyTuple_SetItem(id(tup), 0, id(tup))
0

Showing the tuple itself is a little problematic
>>> tup
# ... hundreds of lines of parens ...
(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((
(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((
(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((
(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((
(((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((
((Segmentation fault

As a direct side effect of glom's 19.1.0 release, the authors here at PDW got to re-experience one of the more surprising behaviors of three of Python's most basic constructs:

Most experienced developers know the quickest way to combine a short list of short lists:

list_of_lists = [[1], [2], [3, 4]]
sum(list_of_lists, [])
# [1, 2, 3, 4]

Ah, nice and flat, much better.

But what happens when we throw a tuple into the mix:

list_of_seqs = [[1], [2], (3, 4)]
sum(list_of_seqs, [])
# TypeError: can only concatenate list (not "tuple") to list

This is kind of surprising! Especially when you consider this:

seq = [1, 2]
seq += (3, 4)
# [1, 2, 3, 4]

Why should sum() fail when addition succeeds?! We'll get to that.

new_list = [1, 2] + (3, 4)
# TypeError: can only concatenate list (not "tuple") to list

There's that error again!

The trick here is that Python has two addition operators. The simple "+" or "add" operator, used by sum(), and the more nuanced "+=" or "iadd" operator, add's inplace variant.

But why is ok for one addition to error and the other to succeed?

Symmetry. And maybe commutativity if you remember that math class.

"+" in Python is symmetric: A + B and B + A should always yield the same result. To do otherwise would be more surprising than any of the surprises above. list and tuple cannot be added with this operator because in a mixed-type situation, the return type would change based on ordering.

Meanwhile, "+=" is asymmetric. The left side of the statement determines the type of the return completely. A += B keeps A's type. A straightforward, Pythonic reason if there ever was one.

Going back to the start of our story, by building on operator.iadd, glom's new flatten() function avoids sum()'s error-raising behavior and works wonders on all manner of nesting iterable.

Args and kwargs are great features of Python.  There is a measurable (though highly variable) cost of them however:

>>> timeit.timeit(lambda: (lambda a, b: None)(1, b=2))
0.16460260000000204
>>> timeit.timeit(lambda: (lambda *a, **kw: None)(1, b=2))
0.21245309999999762

>>> timeit.timeit(lambda: (lambda *a, **kw: None)(1, b=2)) - timeit.timeit(lambda: (lambda a, b: None)(1, b=2))
0.14699769999992895

Constructing that dict and tuple doesn't happen for free:

>>> timeit.timeit(lambda: ((1,), {'b': 2})) - timeit.timeit(lambda: None)
0.16881599999999253

Specifically, it takes about 1/5,000,000th of a second.

Let's define a very simple class:

>>> class F(object):
...    @staticmethod
...    def f(): return "I'm such a simple function, nothing could go wrong"
...
>>> F.f()
"I'm such a simple function, nothing could go wrong"

Now, let's do a trivial no-op to this class:

>>> F.f = F.f

Surely nothing changed, right?

>>> F.f()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unbound method f() must be called with F instance as first argument (got nothing instead)

What happened?  staticmethod uses the descriptor protocol in order to return something other than itself when accessed as an attribute.  The assignment above is not a no-op, because it is not setting the value back to what it already was, but to what was returned by __get__ of the staticmethod object.

>>> class F(object):
...    @staticmethod
...    def f(): return "I'm not what I seem"
...
>>> F.f
<function f at 0x7f05eda596e0>
>>> F.__dict__['f']
<staticmethod object at 0x7f05eda5ce50>

Version note -- Python3 doesn't raise an exception, although the type still changes from staticmethod to function.

>>> class F:
...    @staticmethod
...    def f(): return "I'm protected by python3 wizardry"
...
>>> F.f()
"I'm protected by python3 wizardry"
>>> F.__dict__['f']
<staticmethod object at 0x7fd087b739b0>
>>> F.f = F.f
>>> F.__dict__['f']
<function F.f at 0x7fd087b5cae8>
>>> F.f()
"I'm protected by python3 wizardry"