Strange Python Behavior (can someone please explain to me what is going on here?)

Every once in a while, seemingly really simple Python code does something completely unexpected for me. Look at the following snippet of Python code. This is run straight from the 2.6.5 interpreter, with no other commands executed. Do you notice anything strange?

$python
Python 2.6.5 (r265:79359, Mar 24 2010, 01:32:55) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>; l = lambda i: a[i]
>>> l
<function at="" 0x39e7f0="">
>>> H = [(1, 2), (3, 4)]
>>> [l(0) + l(1) for a in H]
[3, 7]

Did you spot it? Here is a hint. Running a different but similar session:

$python
Python 2.6.5 (r265:79359, Mar 24 2010, 01:32:55) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> l = lambda i: a[i]
>>> l
<function at="" 0x39e7f0="">
>>> l(0)
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 1, in 
NameError: global name 'a' is not defined

Do you see it now? I defined the lambda function l in terms of a without defining first defining a! And furthermore, it just works when a is defined. This is actually independent of the fact that we are working in a list comprehension, as this continuation of the previous session shows:

>>> a = [3, 4, 5]
>>> l(0)
3

But I want to expand on the list comprehension example, because there even more bizzare things going on here. Restarting a new session again:

$python
Python 2.6.5 (r265:79359, Mar 24 2010, 01:32:55) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> l = lambda i: a[i]
>>> H = [(1, 2), (3, 4)]
>>> [l(0) + l(1) for a in H]
[3, 7]
>>> (l(0) + l(1) for a in H)
<generator object="" at="" 0x3a4350="">
>>> list((l(0) + l(1) for a in H))
[7, 7]

So, if you are astute and have been using Python for long enough, you should be able to catch what is going on here. If you don’t know, here is a hint (continuation of previous session):

>>> a
(3, 4)

So, as you may know, in Python 2.6 and earlier, list comprehension index variables “leek” into the local namespace. The strange thing here is that although the list comprehension would reset it, the generator version does not. Well, normally, it does do this:

>>> x = 1
>>> [x for x in range(10)]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> x
9
>>> del x
>>> list((x for x in range(10)))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> x
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'x' is not defined
>>> x = 1
>>> list((x for x in range(10)))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> x
1

So the above bit has something to do with the way the lambda function was defined with the a. By the way, here is what happens with the generator comprehension (is that what these are called?) if a is not defined:

>>> del a
>>> list((l(0) + l(1) for a in H))
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 1, in 
  File "", line 1, in 
NameError: global name 'a' is not defined

This is how I discovered this. I had defined a lambda function using an variable that was then passed to a list comprehension that used this variable as the index without realizing it. But then I tried converting this into a generator comprehension to see if it would be faster, and got the above error.

Finally, since the “feature” of leaking list comprehension loop variables into the local namespace is going away in Python 3, I expected things to behave at least a little differently in Python 3. I tried the above in a Python 3.1.2 interpreter and got the following:

$python3
Python 3.1.2 (r312:79147, Mar 23 2010, 22:02:05) 
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> l = lambda i: a[i]
>>> l
<function at="" 0x100585a68="">
>>> H = [(1, 2), (3, 4)]
>>> [l(0) + l(1) for a in H]
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 1, in 
  File "", line 1, in 
NameError: global name 'a' is not defined
>>> list((l(0) + l(1) for a in H))
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 1, in 
  File "", line 1, in 
NameError: global name 'a' is not defined

So in Python 3, both the list comprehension and the generator comprehensions act the same, which is not too surprising. I guess I should recode that piece of code to make it future proof, although this doesn’t seem easy at the moment, and it may require converting a one-liner into a six-liner. If you are interested, the piece of code is here.

So can anyone provide any insight into what is going on with that lambda function? Running it with the -3 switch to python2.6 didn’t give any warnings related to it.

Update: As I noted in a comment, I figured out how to make this future-proof. I need to convert it from

def residue_reduce_derivation(H, D, x, t, z):
    lambdafunc = lambda i: i*derivation(a[1], D, x, t).as_basic().subs(z, i)/ \
         a[1].as_basic().subs(z, i)
    return S(sum([RootSum(a[0].as_poly(z), lambdafunc) for a in H]))

to

def residue_reduce_derivation(H, D, x, t, z):
    return S(sum((RootSum(a[0].as_poly(z), lambda i: i*derivation(a[1], D, x, t).as_basic().subs(z, i)/ \
        a[1].as_basic().subs(z, i)) for a in H)))

Thanks to all the commenters for the explanations.

Also, you may have noticed that I discovered that if you use [code] instead of <code>, you get these nicer code blocks that actually respect indentation! Now I just need to figure out how to make them syntax highlight Python code.

Update 2: [code='py'] colors it! Sweet!

Update 3: I just discovered that SymPy has a Lambda() object that handles this better. In particular, it pretty prints the code, and is what is already being used for RootSum() in the rational function integrator, at least in Mateusz’s polys9.

>>> integrate(1/(x**5 + 1), x)
log(1 + x)/5 + RootSum(625*_t**4 + 125*_t**3 + 25*_t**2 + 5*_t + 1, Lambda(_t, _t*log(x + 5*_t)))                                                   

Still, this has been a very good learning experience.

About these ads

11 Responses to Strange Python Behavior (can someone please explain to me what is going on here?)

  1. willem says:

    The lambda always refers to the value of A in the global (module) scope.

    A list comprehension iteration variable used to influence the current (here: module) scope, i.e. [.. for a in ..] would introduce/override variable A. Later on this was fixed,

    But with generator expressions, iteration variables are in a separate scope. Now using dots to ensure the indentation shows up correctly, think of:

    .. x = (l(0) + l(1) for a in H)

    as:

    .. def _generator(_H):
    …. for a in _H:
    …… yield a
    .. x = _generator(H)

    and you see the iteration variable A is local to the (hidden) function, and does not influence the global variable.

    It seems in Python 3, list comprehensions are implemented similar to generator expressions (so without leaking) except they collect all values in a list instead of yielding them.

    • asmeurer says:

      Ah, of course, that’s the reason. If Python lets you define a function as

      def f(x):
          return a*x
      

      and just use the a from the outer scope, then lambda i: a*i should work too. I guess I need to learn a bit more about how Python treats scoping.

  2. EOL says:

    Thank you for sharing! I found your discussion to be very interesting. I’m glad that Python 3.1 treats the list comprehension and the “list from generator” in a similar way, with respect to dummy variables.

  3. Ted Horst says:

    “I defined the lambda function l in terms of a without defining first defining a”

    Did you mean to do that, or were you just surprised that it behaved differently in different contexts? I think its considered bad form to define lambdas that operate on global variables especially if they haven’t been defined yet.

    Ted

    • asmeurer says:

      No, it was a complete accident. I only realized what I had done after the error had popped up and I had looked at it for a bit.

  4. Stepan says:

    Couldn’t you make it “future proof” by adding a parameter to the lambda function?

    In [2]: H = [(1, 2), (3, 4)]

    In [5]: l = lambda a, i: a[i]

    In [6]: [l(a, 0) + l(a, 1) for a in H]
    Out[6]: [3, 7]

    In [7]: list((l(a, 0) + l(a, 1) for a in H))
    Out[7]: [3, 7]

    • asmeurer says:

      No. The problem is that RootSum takes a function of 1 argument (see the link to the actual problem code in the blog post).

      Actually, I just figured out that the way to future proof it is to redue it from a two-liner to a one-liner. So, for example, this works:

      >>> list(((lambda i: a[i])(0) + (lambda i:a[i])(1) for a in H))
      [3, 7]
      

      I had pulled out the lambda function because it made the code a little easier to read, but I guess future compatibility and the ability to use a generator comprehension instead of a list comprehension outweigh that in this case.

      I will update the blog post pretty soon with this.

      • Stepan says:

        Oh, now I see what you meant. I think you could still split it in two lines, if you dont mind using lambda twice (untested):

        def residue_reduce_derivation(H, D, x, t, z):
            lfunc = lambda a, i: i*derivation(a[1], D, x, t).as_basic().subs(z, i)/ \
                 a[1].as_basic().subs(z, i)
            return S(sum([RootSum(a[0].as_poly(z), lambda i: lfunc(a, i)) for a in H]))
        
  5. Matt Curry says:

    Hey I just discovered yours and others’ comments on my blog (they were just kind of hidden to me at first…). Anyway, thanks! I’ll check into the whitespace errors. Keep on making these epic blog posts of yours.

  6. [...] for another one of my WTF Python blog posts. Yesterday, I randomly typed this in a Python session (it was late at [...]

  7. [...] of this blog know that I sometimes like to write about some strange, unexpected, and unusual things in Python that I stumble across. This post is another one of [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 124 other followers

%d bloggers like this: