Accessing variables from previous notebook cells

[nonatomiclabs]

Usually, one can access from the current notebook cell variables from previous cells (for instance, we define a in cell 2 and access it in 3). However, I can not get that to work when using heat (example in cell 4), which is sometimes annoying (especially when creating such variable can take up some time).

Full traceback (click to expand)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-4-6d5dc00d7528> in <module>()
----> 1 get_ipython().run_cell_magic('heat', '', 'print(a)')

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2085             magic_arg_s = self.var_expand(line, stack_depth)
   2086             with self.builtin_trap:
-> 2087                 result = fn(magic_arg_s, cell)
   2088             return result
   2089 

<decorator-gen-124> in heat(self, line, cell)

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    186     # but it's overkill for just that one bit of state.
    187     def magic_deco(arg):
--> 188         call = lambda f, *a, **k: f(*a, **k)
    189 
    190         if callable(arg):

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/heat.py in heat(self, line, cell)
     61 
     62         pyheat = PyHeat(tmp_file)
---> 63         pyheat.create_heatmap()
     64         pyheat.show_heatmap(output_file=filename)
     65         pyheat.close_heatmap()

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pyheat/pyheat.py in create_heatmap(self)
     44         """Method to define the heatmap using profile data."""
     45         # Profile the file.
---> 46         self.__profile_file()
     47         # Map profile stats to heatmap data.
     48         self.__fetch_heatmap_data_from_profile()

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pyheat/pyheat.py in __profile_file(self)
     71         self.line_profiler = pprofile.Profile()
     72         self.line_profiler.runfile(
---> 73             open(self.pyfile.path, 'r'), {}, self.pyfile.path)
     74 
     75     def __get_line_profile_data(self):

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pprofile.py in runfile(self, fd, argv, fd_name, compile_flags, dont_inherit, globals)
    606         try:
    607             sys.argv[:] = argv
--> 608             return self.runctx(code, ctx_globals, None)
    609         finally:
    610             sys.argv[:] = original_sys_argv

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pprofile.py in runctx(self, cmd, globals, locals)
    586         """Similar to profile.Profile.runctx ."""
    587         with self():
--> 588             exec(cmd, globals, locals)
    589         return self
    590 

/Users/Jean/Desktop/ipython_cell_input.py in <module>()
----> 1 print(a)

NameError: name 'a' is not defined

Would it be possible to add that feature? I have never worked on iPython extensions, but I would be glad to contribute to that one, if I'm pointed to the right part of code to improve.

---

Anyway, thank you for the great extension 👏

[csurfer]

@nonatomiclabs: I will definitely look into the what would make the above possible keeping the API contract as simple as it is now. In the meantime, you can use https://github.com/csurfer/pyheat as a command line and use it on an entire file that way all your variables and the methods which access them are in one single file and you get an approximate idea of the bottlenecks in code.

P.S https://github.com/csurfer/pyheat is a pet project of mine which powers pyheatmagic.

[nonatomiclabs]

Great, thank you very much.

About PyHeat, I'm not sure it's a real solution here, as:

• it's not as convenient as a simple iPython extension
• the reason I don't want to execute Heat/PyHeat on all of my code is that it can change the scale of the heat map. Let's say that I create a variable and it somehow takes 10 seconds to be created, but what is really interesting to me is to see timing of further code (which runs much quicker). Then, creating that variable will appear in dark red and all the other code will be in light yellow. If I could somehow exclude long operations from being included in the heat map, it wouldn't introduce such a bias in the heat map scale

[jaypeedevlin]

Are either of the solutions in this stackoverflow thread (@needs_local_scope decorator & from IPython.core.magic import Magics) useful?

[psychemedia]

I've noticed issues with trying to refer to variables that are used outside of the function (for examples, variables defined in previous cells that are then passed into the function).

Is one way round this to allow those variables to be passed into the magic?

eg using a pattern along the lines of:

%%heat -v $myvar

def myfun(v):
    pass

myfun(myvar)

where the -v switch lets you pass variables in to the block magic?

[msm1089]

@csurfer
Thanks for creating this extension. I am wondering if you ever found anything that might allow the variables in previous cells to be used?

[tyoc213]

The only way I see is to paste all you need to trace on the same cell.

[tyoc213]

You can also merge multiple cells just for doing "heat magic" selecting multiple cells and then shift+M, but you cant undo (well if you hit ctrl+shift+p you can search for undo, but it is not as good as others undoes in other programms.

The other is in git, save your work before doing the merge if you want to restore the cells as they where.

[christian-oreilly]

This issue is a definitive no-go for me (and I suspect many others), since any significantly complex analysis pipeline will start involving writing functions that you want to reuse from cells to cells (DRY code) and you don't want to start wrapping hundreds of lines within a single cell anytime you want to profile some snippet of code involving functions. That would just be a messy way to work. Seems to me that the problem is that the heat magic does not have access to local variables and functions. Maybe using the @needs_local_scope decorator may help? Example of usage: https://github.com/ipython/ipython/blob/23fb4ff6343e32b9d1921af4f9f1c58ffadd3376/IPython/core/magics/execution.py#L1063-L1084