Execute in reverse: Execute cells required for a cell to execute

[talwrii]

I looked into this package to deal with a problem: Big notebooks which take a while to run where I only want to do a bit of analysis.

It it possible to do the opposite of normal execution. Namely, find a cell and then find all the cells that this cell depends upon and execute them all in order as a kind of "recalc".

The API looks tantalisingly close to being able to do this... but I'm not sure how to trigger a recalc of an earlier cell.

[smacke]

Ah, "pull-based" reactivity rather than "push-based". I've thought about it before as well -- should be possible to support, in theory. It's a bit tricky because the previous dependency structure can change -- that's why for pushed-based, we don't decide up front everything that we want to execute, just what to execute next. I think for pull-based, the simplest thing to do would be to decide the full set of cells to execute ahead-of-time, and maybe stop if there's an an exception thrown in the middle.

[smacke]

Oh, I guess you are looking for static + pull-based, since the notebook hasn't been executed yet. ipyflow depends pretty heavily on dynamic analysis to determine the dependencies, but does have some limited static slicing capabilities that could potentially suffice here. Kind of up in the air whether it's good enough though.

[smacke]

Good news; the next version of ipyflow will support this (in JupyterLab and notebook 7 at least; maybe not in legacy notebook 6).

[talwrii]

Ooh exciting. I'll be keen to test this out once it is released!

[smacke]

This should be working now that notebook 7 is out and the next version of ipyflow is out. You can run cmd+k / ctrl+k to execute the backward slice up to and including a cell. I'll leave this issue open until it's properly documented.

EDIT: to upgrade pip install --upgrade ipyflow

[talwrii]

ooh exciting. Will try to give it a try when the need next arises.

[talwrii]

Had a play with this. worked as expected. I think I had to restart jupyter to get the cmd-k binding to work.

I noticed you had to do a complete recalc before dependencies got calculated. E.g. if you start jupyter go to the end of the document and press cmd-k it doesn't work. I imagine this might be a "deep plumbing issue however"...

[smacke]

E.g. if you start jupyter go to the end of the document and press cmd-k it doesn't work. I imagine this might be a "deep plumbing issue however"...

We can't infer dependencies as accurately as we do without watching the notebook execute, which is why it doesn't work when you start fresh. However, I think what we could do is persist the dependencies in a previous session in the notebook metadata, and we would likely get what you're looking for (tracked in #144).

[talwrii]

Yep, persisting dependencies would work. As a rule I'll always execute cells when I first write them, but I might want to reopen a notebook and execute just one cell and the things that it depends on.

[smacke]

Good news! In the latest release, we're now persisting the dependencies in the notebook metadata (for notebook7 and jupyterlab), so you should be able to resume a previous session and execute everything a downstream cell needs (with the cmd+k shortcut) without extraneous stuff now.

Still going to leave this issue open while I figure out the documentation story.

EDIT:

can be installed with pip install --upgrade ipyflow

One thing to mention: it won't work on older notebooks using nbformat < 4.5, since these don't persist the cell id with each cell. New notebooks created by the new versions of jupyter[lab] should all be >= 4.5 by default.