List comprehension rebinds names even after scope of comprehension. Is this right?

python list comprehension variable not defined
list comprehension python 3
python list comprehension if
pdb self' is not defined

Comprehensions are having some unexpected interactions with scoping. Is this the expected behavior?

I've got a method:

def leave_room(self, uid):
  u = self.user_by_id(uid)
  r = self.rooms[u.rid]

  other_uids = [ouid for ouid in r.users_by_id.keys() if ouid != u.uid]
  other_us = [self.user_by_id(uid) for uid in other_uids]

  r.remove_user(uid) # OOPS! uid has been re-bound by the list comprehension above

  # Interestingly, it's rebound to the last uid in the list, so the error only shows
  # up when len > 1

At the risk of whining, this is a brutal source of errors. As I write new code, I just occasionally find very weird errors due to rebinding -- even now that I know it's a problem. I need to make a rule like "always preface temp vars in list comprehensions with underscore", but even that's not fool-proof.

The fact that there's this random time-bomb waiting kind of negates all the nice "ease of use" of list comprehensions.

List comprehensions leak the loop control variable in Python 2 but not in Python 3. Here's Guido van Rossum (creator of Python) explaining the history behind this:

We also made another change in Python 3, to improve equivalence between list comprehensions and generator expressions. In Python 2, the list comprehension "leaks" the loop control variable into the surrounding scope:

x = 'before'
a = [x for x in 1, 2, 3]
print x # this prints '3', not 'before'

This was an artifact of the original implementation of list comprehensions; it was one of Python's "dirty little secrets" for years. It started out as an intentional compromise to make list comprehensions blindingly fast, and while it was not a common pitfall for beginners, it definitely stung people occasionally. For generator expressions we could not do this. Generator expressions are implemented using generators, whose execution requires a separate execution frame. Thus, generator expressions (especially if they iterate over a short sequence) were less efficient than list comprehensions.

However, in Python 3, we decided to fix the "dirty little secret" of list comprehensions by using the same implementation strategy as for generator expressions. Thus, in Python 3, the above example (after modification to use print(x) :-) will print 'before', proving that the 'x' in the list comprehension temporarily shadows but does not override the 'x' in the surrounding scope.

Accessing class variables from a list comprehension in the class , their local variables bleeding over into the surrounding scope (see Python list comprehension rebind names even after scope of comprehension. Is this right?). 171 List comprehension rebinds names even after scope of comprehension. Is this right? Is this right? 131 What is the most pythonic way to check if an object is a number?

Yes, list comprehensions "leak" their variable in Python 2.x, just like for loops.

In retrospect, this was recognized to be a mistake, and it was avoided with generator expressions. EDIT: As Matt B. notes it was also avoided when set and dictionary comprehension syntaxes were backported from Python 3.

List comprehensions' behavior had to be left as it is in Python 2, but it's fully fixed in Python 3.

This means that in all of:

list(x for x in a if x>32)
set(x//4 for x in a if x>32)         # just another generator exp.
dict((x, x//16) for x in a if x>32)  # yet another generator exp.
{x//4 for x in a if x>32}            # 2.7+ syntax
{x: x//16 for x in a if x>32}        # 2.7+ syntax

the x is always local to the expression while these:

[x for x in a if x>32]
set([x//4 for x in a if x>32])         # just another list comp.
dict([(x, x//16) for x in a if x>32])  # yet another list comp.

in Python 2.x all leak the x variable to the surrounding scope.


UPDATE for Python 3.8(?): PEP 572 will introduce := assignment operator that deliberately leaks out of comprehensions and generator expressions! It's motivated by essentially 2 use cases: capturing a "witness" from early-terminating functions like any() and all():

if any((comment := line).startswith('#') for line in lines):
    print("First comment:", comment)
else:
    print("There are no comments")

and updating mutable state:

total = 0
partial_sums = [total := total + v for v in values]

See Appendix B for exact scoping. The variable is assigned in closest surrounding def or lambda, unless that function declares it nonlocal or global.

tensorflow/activity.py at master · tensorflow/tensorflow · GitHub, Copyright 2017 The TensorFlow Authors. All Rights Reserved. # targets leak in Python 2. # For details, see: # https://stackoverflow.com/questions/4198906/list-​comprehension-rebinds-names-even-after-scope-of-comprehension-is-this-righ. 171 List comprehension rebinds names even after scope of comprehension. Is this right? 131 What is the most pythonic way to check if an object is a number?

Yes, assignment occurs there, just like it would in a for loop. No new scope is being created.

This is definitely the expected behavior: on each cycle, the value is bound to the name you specify. For instance,

>>> x=0
>>> a=[1,54,4,2,32,234,5234,]
>>> [x for x in a if x>32]
[54, 234, 5234]
>>> x
5234

Once that's recognized, it seems easy enough to avoid: don't use existing names for the variables within comprehensions.

Comprehensions, List comprehensions, a shortcut for creating lists, have been in Python since version list expressions have their own scope: they are functions, just defined with a This is when a variable of the same name is set before the comprehension,  Beni Cherniavsky-Paskin. 48 List comprehension rebinds names even after scope of comprehension. Is this right? 1 Handling left-to-right inside right-to-left

Interestingly this doesn't affect dictionary or set comprehensions.

>>> [x for x in range(1, 10)]
[1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> x
9
>>> {x for x in range(1, 5)}
set([1, 2, 3, 4])
>>> x
9
>>> {x:x for x in range(1, 100)}
{1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9, 10: 10, 11: 11, 12: 12, 13: 13, 14: 14, 15: 15, 16: 16, 17: 17, 18: 18, 19: 19, 20: 20, 21: 21, 22: 22, 23: 23, 24: 24, 25: 25, 26: 26, 27: 27, 28: 28, 29: 29, 30: 30, 31: 31, 32: 32, 33: 33, 34: 34, 35: 35, 36: 36, 37: 37, 38: 38, 39: 39, 40: 40, 41: 41, 42: 42, 43: 43, 44: 44, 45: 45, 46: 46, 47: 47, 48: 48, 49: 49, 50: 50, 51: 51, 52: 52, 53: 53, 54: 54, 55: 55, 56: 56, 57: 57, 58: 58, 59: 59, 60: 60, 61: 61, 62: 62, 63: 63, 64: 64, 65: 65, 66: 66, 67: 67, 68: 68, 69: 69, 70: 70, 71: 71, 72: 72, 73: 73, 74: 74, 75: 75, 76: 76, 77: 77, 78: 78, 79: 79, 80: 80, 81: 81, 82: 82, 83: 83, 84: 84, 85: 85, 86: 86, 87: 87, 88: 88, 89: 89, 90: 90, 91: 91, 92: 92, 93: 93, 94: 94, 95: 95, 96: 96, 97: 97, 98: 98, 99: 99}
>>> x
9

However it has been fixed in 3 as noted above.

Review Python I: start - Parallel/Tuple/List Assignments, Python Review (everything you should have learned in ICS-31/32) When to this lecture: 1) The first step towards wisdom is calling things by their right names. We can even use this mechanism for rebinding multiple global names: gn1, gn2, Also, we can always use a comprehension (dicussed later in this lecture) to  172 List comprehension rebinds names even after scope of comprehension. Is this right? Is this right? 131 What is the most pythonic way to check if an object is a number?

some workaround, for python 2.6, when this behaviour is not desirable

# python
Python 2.6.6 (r266:84292, Aug  9 2016, 06:11:56)
Type "help", "copyright", "credits" or "license" for more information.
>>> x=0
>>> a=list(x for x in xrange(9))
>>> x
0
>>> a=[x for x in xrange(9)]
>>> x
8

[PDF] Untitled, with the names of the function's formal parameters (which are only bound rebinds predefined names, is legitimate but deplorable, in that it is not immediately Even though char is a basic type and hd and tl are functions that are normally Inside this list comprehension x is a local identifier and n is free but in scope. In Python 2, the list comprehension "leaks" the loop control variable into the surrounding scope: x = 'before' a = [x for x in 1, 2, 3] print x # this prints '3', not 'before' This was an artifact of the original implementation of list comprehensions; it was one of Python's "dirty little secrets" for years.

Python Comprehension and Generator Expressions, Python comprehension expressions and generator expressions have a At the left side of : there is an optional list of function parameters, while at the right side In fact, in Python 3 a comprehension can't rebind any name outside its scope. of the generator before it raises an exception with even.close() . Q&A for road warriors and seasoned travelers. Stack Exchange network consists of 176 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

(PDF) Comprehensive comprehensions, PDF | Abstract We propose,an extension to list comprehensions,that makes,it easy to express the kind of queries one would write in Keywords list comprehension, SQL, query, aggregate scope: before the group by each tuple contains a name, a depart- function has the right type to work with arbitrary encodings of tu-. Q&A for information security professionals. Stack Exchange network consists of 176 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

48 List comprehension rebinds names even after scope of comprehension. Is this right? 35 Algorithms FPGAs dominate CPUs on; 31 Default font set on Android;

Comments
  • -1: "brutal source of errors"? Hardly. Why choose such an argumentative term? Generally the most expensive errors are requirements misunderstandings and simple logic errors. This kind of error has been a standard problem in a lot of programming languages. Why call it 'brutal'?
  • It violates the principle of least surprise. It's also not mentioned in the python documentation on list comprehensions which does however mention several times how easy and convenient they are. Essentially it's a land-mine that existed outside my language model, and hence was impossible for me to foresee.
  • +1 for "brutal source of errors". The word 'brutal' is entirely justified.
  • The only "brutal" thing I see here is your naming convention. This isn't the 80s any more you're not limited to 3 character variable names.
  • Note: the documention does state that list-comprehension are equivalent to the explicit for-loop construct and for-loops leak variables. So it wasn't explicit but was implicitly stated.
  • I'll add that although Guido calls it a "dirty little secret", many considered it a feature, not a bug.
  • Also note that now in 2.7, set and dictionary comprehensions (and generators) have private scopes, but list comprehensions still don't. While this makes some sense in that the former were all back-ported from Python 3, it really makes the contrast with list comprehensions jarring.
  • I know this is an insanely old question, but why did some consider it a feature of the language? Is there anything in favour of this kind of variable leaking?
  • ugh just got bit by this...leaking this was definitely a surprise IMO. I guess not relevant anymore, but felt the need to mention :P
  • for: loops leaking has good reasons, esp. to access last value after early break — but irrelevant to comprehesions. I recall some comp.lang.python discussions where people wanted to assign variables in middle of expression. The less insane way found was single-value for clauses eg. sum100 = [s for s in [0] for i in range(1, 101) for s in [s + i]][-1], but just needs a comprehension-local var and works just as well in Python 3. I think "leaking" was the only way to set variable visible outside an expression. Everybody agreed these techniques are horrible :-)
  • That syntax doesn't work at all in Python 2.6. Are you talking about Python 2.7?
  • Python 2.6 has list comprehensions only as does Python 3.0. 3.1 added set and dictionary comprehensions and these were ported to 2.7. Sorry if that was not clear. It was meant to note a limitation to another answer, and which versions it applies to is not entirely straightforward.
  • While I can imagine making an argument that there are cases where using python 2.7 for new code makes sense, I can't say the same for python 2.6... Even if 2.6 is what came with your OS, you're not stuck with it. Consider installing virtualenv and using 3.6 for new code!
  • The point about Python 2.6 could come up though in maintaining existing legacy systems. So as an historical note it is not totally irrelevant. Same with 3.0 (ick)
  • Sorry if I sound rude, but this doesn't answer the question in any way. It's better suited as a comment.