Strange Python Behavior (can someone please explain to me what is going on here?)

Every once in a while, seemingly really simple Python code does something completely unexpected for me. Look at the following snippet of Python code. This is run straight from the 2.6.5 interpreter, with no other commands executed. Do you notice anything strange?

$python
Python 2.6.5 (r265:79359, Mar 24 2010, 01:32:55) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>; l = lambda i: a[i]
>>> l
<function at="" 0x39e7f0="">
>>> H = [(1, 2), (3, 4)]
>>> [l(0) + l(1) for a in H]
[3, 7]

Did you spot it? Here is a hint. Running a different but similar session:

$python
Python 2.6.5 (r265:79359, Mar 24 2010, 01:32:55) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> l = lambda i: a[i]
>>> l
<function at="" 0x39e7f0="">
>>> l(0)
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 1, in 
NameError: global name 'a' is not defined

Do you see it now? I defined the lambda function l in terms of a without defining first defining a! And furthermore, it just works when a is defined. This is actually independent of the fact that we are working in a list comprehension, as this continuation of the previous session shows:

>>> a = [3, 4, 5]
>>> l(0)
3

But I want to expand on the list comprehension example, because there even more bizzare things going on here. Restarting a new session again:

$python
Python 2.6.5 (r265:79359, Mar 24 2010, 01:32:55) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> l = lambda i: a[i]
>>> H = [(1, 2), (3, 4)]
>>> [l(0) + l(1) for a in H]
[3, 7]
>>> (l(0) + l(1) for a in H)
<generator object="" at="" 0x3a4350="">
>>> list((l(0) + l(1) for a in H))
[7, 7]

So, if you are astute and have been using Python for long enough, you should be able to catch what is going on here. If you don’t know, here is a hint (continuation of previous session):

>>> a
(3, 4)

So, as you may know, in Python 2.6 and earlier, list comprehension index variables “leek” into the local namespace. The strange thing here is that although the list comprehension would reset it, the generator version does not. Well, normally, it does do this:

>>> x = 1
>>> [x for x in range(10)]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> x
9
>>> del x
>>> list((x for x in range(10)))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> x
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'x' is not defined
>>> x = 1
>>> list((x for x in range(10)))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> x
1

So the above bit has something to do with the way the lambda function was defined with the a. By the way, here is what happens with the generator comprehension (is that what these are called?) if a is not defined:

>>> del a
>>> list((l(0) + l(1) for a in H))
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 1, in 
  File "", line 1, in 
NameError: global name 'a' is not defined

This is how I discovered this. I had defined a lambda function using an variable that was then passed to a list comprehension that used this variable as the index without realizing it. But then I tried converting this into a generator comprehension to see if it would be faster, and got the above error.

Finally, since the “feature” of leaking list comprehension loop variables into the local namespace is going away in Python 3, I expected things to behave at least a little differently in Python 3. I tried the above in a Python 3.1.2 interpreter and got the following:

$python3
Python 3.1.2 (r312:79147, Mar 23 2010, 22:02:05) 
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> l = lambda i: a[i]
>>> l
<function at="" 0x100585a68="">
>>> H = [(1, 2), (3, 4)]
>>> [l(0) + l(1) for a in H]
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 1, in 
  File "", line 1, in 
NameError: global name 'a' is not defined
>>> list((l(0) + l(1) for a in H))
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 1, in 
  File "", line 1, in 
NameError: global name 'a' is not defined

So in Python 3, both the list comprehension and the generator comprehensions act the same, which is not too surprising. I guess I should recode that piece of code to make it future proof, although this doesn’t seem easy at the moment, and it may require converting a one-liner into a six-liner. If you are interested, the piece of code is here.

So can anyone provide any insight into what is going on with that lambda function? Running it with the -3 switch to python2.6 didn’t give any warnings related to it.

Update: As I noted in a comment, I figured out how to make this future-proof. I need to convert it from

def residue_reduce_derivation(H, D, x, t, z):
    lambdafunc = lambda i: i*derivation(a[1], D, x, t).as_basic().subs(z, i)/ \
         a[1].as_basic().subs(z, i)
    return S(sum([RootSum(a[0].as_poly(z), lambdafunc) for a in H]))

to

def residue_reduce_derivation(H, D, x, t, z):
    return S(sum((RootSum(a[0].as_poly(z), lambda i: i*derivation(a[1], D, x, t).as_basic().subs(z, i)/ \
        a[1].as_basic().subs(z, i)) for a in H)))

Thanks to all the commenters for the explanations.

Also, you may have noticed that I discovered that if you use [code] instead of <code>, you get these nicer code blocks that actually respect indentation! Now I just need to figure out how to make them syntax highlight Python code.

Update 2: [code='py'] colors it! Sweet!

Update 3: I just discovered that SymPy has a Lambda() object that handles this better. In particular, it pretty prints the code, and is what is already being used for RootSum() in the rational function integrator, at least in Mateusz’s polys9.

>>> integrate(1/(x**5 + 1), x)
log(1 + x)/5 + RootSum(625*_t**4 + 125*_t**3 + 25*_t**2 + 5*_t + 1, Lambda(_t, _t*log(x + 5*_t)))                                                   

Still, this has been a very good learning experience.

About these ads