Names not defined in list comprehension in the IPython shell #103
Comments
|
I think this is the same as ipython/ipython#136 and ipython/ipython#62. When you're already inside a function scope, list comprehensions effectively make a closure, but you can't define closures in dynamically compiled code (such as anything run in a shell). There might be a way round this by combining locals and globals in collections.ChainMap, and treating the combination the globals for dynamic execution. But since we've been encouraging things like Django's shell to use |
|
How is it working in the regular Python shell? |
|
As in outside a debugger? Global variables work differently to variables in nested non-local scopes, so we're not actually defining closures there. Functions have a reference to the global namespace, but when they're closures, they only keep a reference to the specific variables they've closed over, not the entire namespace of their creation scope. If you mean there's a way of embedding the plain Python shell that does work in this context, then I can only guess that either it uses the |
|
Within PuDB, you can choose what shell you use. The Python shell works just fine with this. So does the builtin shell. I don't know how it works. |
|
I guess it does do something like that. See https://github.com/inducer/pudb/blob/master/pudb/shell.py#L50. I'll have to see if this can be used with IPython. |
|
I think it should be possible to do that with IPython. |
|
A very irritating problem. Temp fix via: |
|
Is it acceptable to up-vote, 👍 ? This affects me as well. I can add my use-case if requested. |
|
I don't know how to do this. The IPython code is significantly more complicated than the python or bpython code (note that you need to look at the v11 function). |
|
4 years later, is there a solution yet? |
|
Still not solved I guess. Someone above asked if this occurs in the regular iPython REPL. It does not. Here is a related stack overflow question with an example: in iPython: In regular Python REPL: |
|
I does anyone know if this is fixed in version 7.1? It is not in 7.0: |
|
It's a PuDB problem, not an IPython problem. PuDB needs to properly fake the function locals dictionary as a globals dictionary when using the IPython shell. It does this for other shells (using |
|
For those who are using pdb, one of the way to avoid this, use |
|
The same for filter + lambda x = 5
list(filter(lambda s: x, [{}]))Command line: [Ctrl-X] gives exception: Tested in python 2.7, 3.7, pudb 2018.1 |
|
As-seen on the mailing list:
My usecase involves the Ctrl+X internal shell.
As noted the "normal python shell" works out of the box. I am invoking |
For context, this is with the builtin shell (but it's the same fundamental issue). It looks like the SetPropogatingDict is explicitly disabled in the builtin shell because of #166. Can we use the Regardless, I believe that issue only occurs in Python 2, so we should be able to use SetPropogatingDict in Python 3 to get the correct behavior there. |
|
I updated my above comment with info as-they-come. For the sake of keeping the timeline, however, I'll say I am using |
|
Maybe it works with the classic shell even in Python 2 because the |
|
I'm a little confused by https://stackoverflow.com/questions/12185110/subclassed-python-dictionary-for-custom-namespace-in-exec-method. As far as I can tell, eval and exec work with custom dictionaries just fine # test.py
class CustomDict(dict):
def __setitem__(self, key, val):
print('setting', key, val)
return dict.__setitem__(self, key, val)
def __getitem__(self, key):
print('getting', key)
return dict.__getitem__(self, key)
d = CustomDict()
eval(compile('a = 1', '<none>', 'single'), d)
print d['a']
d = CustomDict()
exec 'a = 1' in d
print d['a']But then again #166 was a very subtle bug, so maybe things break in nonobvious ways. |
|
OK, I can confirm the issue now. It only comes up when accessing global variables from a local scope. It seems to happen with both eval and exec, so I don't understand how the classic shell works. # test2.py
class CustomDict(dict):
def __getitem__(self, key):
if key == 'a':
return 1
return dict.__getitem__(self, key)
print 'doing eval'
try:
d = CustomDict()
eval(compile('def t():\n print a', '<none>', 'single'), d, d)
eval(compile('t()', '<none>', 'single'), d, d)
except Exception as e:
print '!!! Exception:', e
print 'doing exec'
try:
d = CustomDict()
exec 'def t():\n print a\n\nt()' in d
except Exception as e:
print '!!! Exception:', e# test3.py
class CustomDict(dict):
def __getitem__(self, key):
if key == 'a':
return 1
return dict.__getitem__(self, key)
print('doing eval')
try:
d = CustomDict()
eval(compile('def t():\n print(a)', '<none>', 'single'), d, d)
eval(compile('t()', '<none>', 'single'), d, d)
except Exception as e:
print('!!! Exception:', e)
print('doing exec')
try:
d = CustomDict()
exec('def t():\n print(a)\n\nt()', d, d)
except Exception as e:
print('!!! Exception:', e) |
|
Now I'm confused how the classic shell in Python 2 even is working # test2.py
class CustomDict(dict):
def __getitem__(self, key):
if key == 'a':
return 1
return dict.__getitem__(self, key)
print 'doing eval'
try:
d = CustomDict()
eval(compile('def t():\n print a', '<none>', 'single'), d, d)
eval(compile('t()', '<none>', 'single'), d, d)
except Exception as e:
print '!!! Exception:', e
print 'doing exec'
try:
d = CustomDict()
exec 'def t():\n print a' in d
exec 't()' in d
except Exception as e:
print '!!! Exception:', e
print 'code module'
from code import InteractiveConsole
d = CustomDict()
cons = InteractiveConsole(d)
cons.push('def t():')
cons.push(' print a')
cons.push('')
cons.push('t()')# test3.py
class CustomDict(dict):
def __getitem__(self, key):
if key == 'a':
return 1
return dict.__getitem__(self, key)
print('doing eval')
try:
d = CustomDict()
eval(compile('def t():\n print(a)', '<none>', 'single'), d, d)
eval(compile('t()', '<none>', 'single'), d, d)
except Exception as e:
print('!!! Exception:', e)
print('doing exec')
try:
d = CustomDict()
exec('def t():\n print(a)\n\nt()', d, d)
except Exception as e:
print('!!! Exception:', e)
print('code module')
from code import InteractiveConsole
d = CustomDict()
cons = InteractiveConsole(d)
cons.push('def t():')
cons.push(' print(a)')
cons.push('')
cons.push('t()') |
|
If I use instead from pudb.shell import SetPropagatingDict
l = {}
g = {'a': 1}
d = SetPropagatingDict([l, g], l)then all three work in Python 2. But as we saw in #166 the breakage was subtle. I wouldn't be surprised if the classic shell breaks things in Python 2 as well. Unless we can figure out exactly why things broke, I would suggesting leaving the code as is for Python 2 and using SetPropogatingDict for Python 3. Fewer people are using Python 2 anymore anyway. Actually using the |
The issues with using a custom dict subclass to eval/exec are fixed in Python 3. Without this, local functions in the shell cannot access global variables (for instance, inside of list comprehentions). See the discussion on issues inducer#166 and inducer#103.
The issues with using a custom dict subclass to eval/exec are fixed in Python 3. Without this, local functions in the shell cannot access global variables (for instance, inside of list comprehensions). See the discussion on issues inducer#166 and inducer#103.
|
Please add those (and mine and whatever else) as unit tests. The issue seems very delicate |
I agree that the issue seems delicate. Would you be willing to contribute some tests? |
|
I'll have to take a look to see how hard it is to test. The issue only occurs when the debugger is in a local function scope. Also, list comprehensions work just fine in Python 2 because they don't create their own scope in Python 2. But a lambda as in #103 (comment) will always reproduce the problem, if it exists. To clarify, the problem is that variables defined in the console (variables already defined in the file being debugged work just fine) cannot be used inside of something in the console that creates a new function scope, such as a comprehension or a lambda (or a |
|
There don't seem to be any existing tests for the debugger to model off of. Do you know how to create a Debugger to manipulate programmatically? Do we have to use pexpect, or somehow mock everything that manipulates stdout? Maybe it would be easier to test if the built in shell were factored out from the debugger. I do want to refactor it a bit at some point, to clean up some issues with it (like the |
I understand that; however, for the
>>> test = 'ab\r\nc\r\n\r\nde\r\nf'
>>> re.split(r'(?:\r?\n){2}', test)
['ab\r\nc', 'de\r\nf']
>>> [re.sub(r'\r?\n', ' ', x.strip()) for x in re.split(r'(?:\r?\n){2}', test)]
Traceback (most recent call last):
File "<pudb command line>", line 1, in <module>
File "<pudb command line>", line 1, in <listcomp>
NameError: name 're' is not defined
>>>@inducer I don't object contributing tests. Adding, however, as constrains:
My only idea would be to go down on the debugger/shell, obtain "the evaluator object", and then feed it data, but that may not be optimal or desirable. There aren't may "enforcable" boundaries in Python, so I can eventually reference any part of the program - but maybe that's not a nice way to test. I will first try locally to see if the commit I pulled actually fixes my problem. Then, I will try to provide something that looks like testing it (at least my 2 lines), but I cannot promise that "the reviewer" will like it. As @asmeurer has also noted, doing tests that also have/mock having a loaded program (to execute testing from that angle too), it might get even more complicated (i.e. going the |
|
Can you show a reduced version of the file you are debugging where the error occurs? |
|
I will unfortunately have to mock it 😞. I will try to provide everything by tomorrow (i.e.:
), minus the part of "fully working unit test(s)" |
Confirmed both ways i.e.: >>> test = 'ab\r\nc\r\n\r\nde\r\nf'
>>> re.split(r'(?:\r?\n){2}', test)
['ab\r\nc', 'de\r\nf']
>>> [re.sub(r'\r?\n', ' ', x.strip()) for x in re.split(r'(?:\r?\n){2}', test)]
TracebaConfck (most recent call last):
File "<pudb command line>", line 1, in <module>
File "<pudb command line>", line 1, in <listcomp>
NameError: name 're' is not defined
>>> import re
>>> [re.sub(r'\r?\n', ' ', x.strip()) for x in re.split(r'(?:\r?\n){2}', test)]
Traceback (most recent call last):
File "<pudb command line>", line 1, in <module>
File "<pudb command line>", line 1, in <listcomp>
NameError: name 're' is not definedRepeat with
Ditto
Unfortunately, I cannot replicate with a mock script. However, def main():
print(1) # Breakpoint-here
if __name__ == "__main__":
exit(main())already replicates the OPs issue (and it is also fixed with the fix). I guess my script, deep down, it has some global declaration that in the end it is not seeing? idk. You can in-place use the above script and OPs original discovery, to mimic the problem. |
|
Where are you importing By the way, I noticed a related issue that occurs in all shells. In a nested function, variables from the outer function scopes are not available unless the inner function uses them. This is expected Python behavior, due to the way local variables work, but it could be annoying. def test():
import re
def test2():
pass # <- breakpoint here
# re will not be available in the shell
test2()
test()I can't think of an easy way to fix it, so unless someone complains about it (as far as I know no one has), we shouldn't worry about it. |
|
I think all of us are stuck on this one https://bugs.python.org/issue21161 Note: debugging on ipdb! import logging
from robot.api.deco import keyword
class SubCompFull:
ROBOT_LIBRARY_SCOPE = 'GLOBAL'
ROBOT_LIBRARY_VERSION = '0.1'
ROBOT_LIBRARY_DOC_FORMAT = 'reST'
def __init__(self):
pass
def get_keyword_names(self):
return [name for name in dir(self) if
hasattr(getattr(self, name), 'robot_name')]
def run_keyword(self, name, args, kwargs):
logging.debug("Calling %s('%s', **kwargs='%s')", name, args, kwargs)
method = getattr(self, name)
return method(*args, **kwargs)
@keyword(name='Get this-that data')
def get_this_that_data_dashed(self) -> list:
print([name for name in dir(self) if
hasattr(getattr(self, name), 'robot_name')])
@keyword
def get_this_that_data(self) -> list:
ex_ca = [getattr(getattr(self, name), 'robot_name', name) or name
for name in dir(self) if
hasattr(getattr(self, name), 'robot_name')]
from pprint import pprint
from util.python_repl import interactive
interactive()
pprint(ex_ca)
print([name for name in dir(self) if
hasattr(getattr(self, name), 'robot_name')])Output: Example normal > preserve/dashed-kw/lib/com/sys/SubCompFull.py(35)get_this_that_data()
34 interactive()
---> 35 pprint(ex_ca)
36 print([name for name in dir(self) if
ipdb> ex_ca
['get_this_that_data', 'Get this-that data']
ipdb> [getattr(getattr(self, name), 'robot_name', name) or name for name in dir(self) if hasattr(getattr(self, name), 'robot_name')]
*** NameError: name 'self' is not defined
ipdb> interact
In : [getattr(getattr(self, name), 'robot_name', name) or name for name in dir(self) if hasattr(getattr(self, name), 'robot_name')]
['get_this_that_data', 'Get this-that data']And "free-running" it works too: pprint([getattr(getattr(self, name), 'robot_name', name)
for name in dir(self) if
hasattr(getattr(self, name), 'robot_name')])
Related issues: |
In the IPython shell, you can't seem to access names from within list comprehensions:
It doesn't matter if the name was defined within the session or if it comes from the function you are debugging. This is debugging within a function scope, if that matters.
The text was updated successfully, but these errors were encountered: