Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IPythonShellEmbed fails to recognize local variables #136

Open
prsteel opened this issue Jul 19, 2010 · 19 comments
Open

IPythonShellEmbed fails to recognize local variables #136

prsteel opened this issue Jul 19, 2010 · 19 comments
Assignees
Labels
Milestone

Comments

@prsteel
Copy link

prsteel commented Jul 19, 2010

Issue

Embedded IPython shells can lose track of local variables.

Test Case

Minimal test case:

class Foo(object):
    """ Container-like object """
    def __setattr__(self, obj, val):
        self.__dict__[obj] = val

    def __getattr__(self, obj, val):
        return self.__dict__[obj]

f = Foo()
f.indices = set([1,2,3,4,5])
f.values = {}
for x in f.indices:
    f.values[x] = x

def bar(foo):
    import IPython
    IPython.Shell.IPShellEmbed()()
    return sum(foo.values[x] for x in foo.indices)

print bar(f)

To see the error, first run the code in Python (or IPython) and exit from the spawned shell; the final print statement correctly displays '15'. Run the code again, but this time type sum(foo.values[x] for x in foo.indices) in the spawned shell, and we receive the error

" NameError: global name 'foo' is not defined".

@tomspur
Copy link
Contributor

tomspur commented Aug 1, 2010

Hmm, where did you define 'foo'?

If you just run call foo outside of the function bar, foo should not exist (and obvously it doesn't).

When you instead run "sum(f.values[x] for x in f.indices)" you get 15 again...

@prsteel
Copy link
Author

prsteel commented Aug 3, 2010

Quite correct. However, I'm refering to the use of foo inside the spawned IPython shell, which is in turn spawned inside the function definition where foo is a local variable.

@takluyver
Copy link
Member

Is this likely to be the same issue as #62?

@fperez
Copy link
Member

fperez commented Nov 28, 2011

No, @takluyver: this is a separate issue and indeed a real bug in our embedding code. I was hoping your recent work with namespaces would have fixed it, but it didn't. For reference, here's the example code to run with the current embedding api:

class Foo(object):
    """ Container-like object """
    def __setattr__(self, obj, val):
        self.__dict__[obj] = val

    def __getattr__(self, obj, val):
        return self.__dict__[obj]

f = Foo()
f.indices = set([1,2,3,4,5])
f.values = {}
for x in f.indices:
    f.values[x] = x

def bar(foo):
    import IPython
    IPython.embed()
    return sum(foo.values[x] for x in foo.indices)

print bar(f)

Then, in the spawned, embedded IPython, this fails:

In [1]: sum(foo.values[x] for x in foo.indices)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/home/fperez/tmp/junk/ipython/foo.py in <module>()
----> 1 sum(foo.values[x] for x in foo.indices)

/home/fperez/tmp/junk/ipython/foo.py in <genexpr>((x,))
----> 1 sum(foo.values[x] for x in foo.indices)

NameError: global name 'foo' is not defined

And it doesn't work even if we pass to the embed call user_ns=locals() explicitly, but in that case we get in addition a crash on exit:

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2702, in atexit_operations
    self.reset(new_session=False)
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 1100, in reset
    self.displayhook.flush()
  File "/home/fperez/usr/lib/python2.7/site-packages/IPython/core/displayhook.py", line 319, in flush
    self.shell.user_ns['_oh'].clear()
KeyError: '_oh'

It looks like our embedding machinery is in pretty bad shape...

@ghost ghost assigned takluyver Nov 28, 2011
@takluyver
Copy link
Member

Assigned this to myself to look at.

@fperez
Copy link
Member

fperez commented Nov 29, 2011

Excellent, thanks.

@takluyver
Copy link
Member

Unfortunately, I think this is a limitation of Python itself. It appears code compiled dynamically cannot define a closure, which is essentially what we're doing here with a generator expression. Here is a minimal test case:

def f():
   x = 1
   exec "def g(): print x\ng()"

f()

Which gives:

Traceback (most recent call last):
  File "scopetest.py", line 5, in <module>
    f()
  File "scopetest.py", line 3, in f
    exec "def g(): print x\ng()"
  File "<string>", line 2, in <module>
  File "<string>", line 1, in g
NameError: global name 'x' is not defined

Note that you can still see local variables in IPython - in the example given, plain print foo works. But you can't close a new scope over them.

@takluyver
Copy link
Member

I think it may be possible to make this work using collections.ChainMap from Python 3.3 so that IPython sees both local and global variables where it is embedded as globals. However, from the lack of noise about this over the last two years, I don't think it is high priority, so I'm retagging accordingly, and hopefully getting to this some time after 1.0.

@clebio
Copy link

clebio commented Nov 13, 2014

Is it acceptable to up-vote, 👍 ? This affects me as well. I can add my use-case if requested.

@minrk
Copy link
Member

minrk commented Nov 13, 2014

I would be 100% okay if the fix for this only worked on Python 3.

@amelio-dx
Copy link

I am also having the same problem under Python 3. Thanks for re-openning.

@amelio-vazquez-reina
Copy link

Same problem here 👍 both in Python 2 and 3.

@minrk minrk removed the prio-medium label Jan 14, 2015
@Erotemic
Copy link

I have this issue in both Python 2 and 3. It affects me on a daily basis.

@nikitakit
Copy link

nikitakit commented Jun 27, 2016

I looked into this issue a bit, and it's definitely fixable (though maintaining fixes for both Python 2 and 3 may be messy).

The ChainMap solution would be easiest to include into IPython proper. However, there's a slight catch that eval/exec require globals to be a dict. Creating a class MyChainMap(ChainMap, dict): pass can work around this.

I also wrote a Python 3.5+ fix based on a different strategy of simulating closure cells and forcing the python compiler to emit the correct bytecode to work with them. The relevant file is here, part of my xdbg demo. It works by replacing get_ipython().run_ast_nodes.

As far as I can tell, the two approaches differ only in their handling of closures. When xdbg is embedded at a scope that has closed over some variables, it can correctly access those variables by reference and mutate them. Additionally, if any functions are created in the interactive interpreter, they will close over any local variables they need while allowing the rest of the local scope to be garbage collected.

@ivanov
Copy link
Member

ivanov commented May 18, 2017

IPython 6.0 and later versions only work in Python 3, so if @nikitakit or anyone else wants to open a pulll request with a test case and a fix for this, that would be welcome.

@nikitakit
Copy link

It's been a year since my last comment here, and during this time my understanding of the issue has changed a bit.

Interactively inspecting local scope remains an important feature to me, but there are actually a number of inter-related issues regarding local variables and the embedding machinery. For example, modifying local variables inside an embedded shell doesn't work:

>>> import IPython
>>> def test():
...     x = 5
...     IPython.embed()
...     print('x is', x)
...
>>> test()
Python 3.5.1 |Continuum Analytics, Inc.| (default, Dec  7 2015, 11:24:55)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: x
Out[1]: 5

In [2]: x = 6

In [3]:
Do you really want to exit ([y]/n)? y

x is 5

A ChainMap-based approach wouldn't help in this situation.

Another question is what happens when closures defined inside an embedded shell are leaked out into the global scope. Consider running the same code as above, but inputting something along the lines of IPython.get_my_x = lambda: x in the embedded IPython shell. A ChainMap-based solution will avoid causing a NameError in this situation, at the expense of potentially introducing two simultaneous copies of x that exist independently of each other (one being the ChainMap, and the other the local variable/closure cell used by the python interpreter).

Given the complexity of the situation, I've decided to focus my efforts on a more comprehensive approach to the problem (which also better aligns with my own usage of IPython). This has led to the development of xdbg, which is essentially a debugger that integrates with IPython. The key idea is to extend the IPython shell by offering debugger commands via magics (e.g. %break to set breakpoints). The fact that breakpoints are set externally, rather than by calling the function embed in-place, has allowed an implementation that addresses many of these problems with local variables.

I don't currently plan on pull-requesting a narrowly-targeted bugfix to core IPython. However, I'm very interested to know what IPython users and devs think about using a debugger-inspired interface for local code inspection. IPython (and now also Jupyter) have had a huge impact on the ability to do interactive coding in Python, but there are still many improvements to be made in how it interacts with heavy encapsulation using classes/functions/modules.

@rmcgibbo
Copy link
Contributor

I've been burned a bunch of times by this bug. E.g.

[mcgibbon@xps13:~]$ cat test.py 
x = 1
def func():    
    x = 2
    import IPython
    IPython.embed()
    
if __name__ == "__main__":
    func()
[mcgibbon@xps13:~]$ python test.py 
Python 3.7.6 (default, Dec 18 2019, 19:23:55) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.12.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: [x for _ in range(2)]                                                                                                                                                                                                                 
Out[1]: [1, 1]  # lol whoops. looked up in wrong scope.

I'm not sure what can be done, but just adding my voice to the choir.

@rmcgibbo
Copy link
Contributor

Based on https://bugs.python.org/issue13557 and the fact that explicitly passing locals() and globals() to an exec fixes @takluyver's reproduction of the bug, it seems like this ought to be possible to fix in IPython by passing the correct local and global namespaces.

x = 1

def func():    
    x = 2
    exec("def g():\n print(x)\ng()")  # prints 1 :(
    exec("def g():\n print(x)\ng()", locals(), globals())  # prints 2 yay!

    
if __name__ == "__main__":
    func()

@kflu
Copy link

kflu commented Oct 23, 2021

As a work around, you can use globals().update(locals()) to carry embedding sessions' local into process globals.

Source: https://stackoverflow.com/a/67517617/695964

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests