Feature request: Be able to receive cell output from a remote kernel after reconnecting

[ptwobrussell]

Once your browser disconnects from the kernel running on a remote notebook server, there's no way to recover the output once the notebook reconnects to the kernel. My understanding is that even though the output is actually shipped back to the browser, it's just unable to be sent to the right cell for display. If you are using IPython Notebook for long-running computation in a cell, the ability to re-connect and get cell output from the point of reconnection forward would be nice to have.

[michaelaye]

Do you condone +1 comments? (Sorry if not).

[eldad-a]

+1 (sorry if not...)
This feature will highly improve the ipython notebook interface, particularly for long runs!

[eldad-a]

There is a "half" of a work-around based on the ipycache extension.
Well, it may not be even a "half" but it is the best I found so far:
A cell can be "cached" such that running it again would load the output instead of executing the code again.

For example:
After executing a cell containing

%load_ext ipycache
%mkdir -vp ipycache
import time

A cell can be "decorated" by the %%cache magic:

%%cache ipycache/test_long 
i = 1
while i<30:
    print '{}, '.format(i),
    time.sleep(1)
    i+=1

The browser can be closed
Running the last cell again will not execute it again but load the cached output (and display it).
However, one has to wait for the execution to complete.
So this is will not help in monitoring the progress of a running code and I am not sure how it deals with errors.

[alexbw]

+1

[kristall]

+1

[mhgarci1]

This would be incredibly useful for creating reports or tests
+1

[jabooth]

+1, didn't actually realise this is a limitation of the notebook and lost some output that I incorrectly assumed would be waiting for me when I rejoined my server...

Any chance a core developer could weigh in and give a brief idea as to how feasible this is to fix?

Just to confirm I'm not missing something, whilst this goes unresolved the notebook is not good for running long jobs on remove servers that you can then rejoin to check state.

As a concrete example, a user is at their office workstation and is just finishing up a notebook that will take some hours to run. User leaves the office and on the train later on gets a (sporadic) SSH port forward back to the office workstation. They tweak the notebook and get it finished - they hit run. Connection will bounce in and out for the next few hours - later on the user will rejoin the SSH port forward and have no output waiting for them.

[Carreau]

Any chance a core developer could weigh in and give a brief idea as to how feasible this is to fix?

This is tough. Really. We need to move the M part of MVC to the server and get synchronization with client, so an architecture which is more MC-MVC with the dash being the network.
It's in progress but I wouldn't bet for less than in a year or so (unless we get lots more help, or lots more money)

[michaelaye]

So, I guess @jabooth 's work-flow could only be solved today with writing any and all results to disk, correct?

[Carreau]

If the kenel still have the value then you can re-display it.
The only problem is if anythong get displayed while no client is connected then the displayed result is just gone.
But the following is still possible.

sleep(5)
a = 1

close browser, wait 10 sec, open browser,

print(a)
# 1

You can probably alo do things with Futures also.

[jacobhinkle]

@Carreau, I am willing to help get this working, as this would be a big help to my coworkers and I who are used to working with screen for remote jobs. You say this is being worked on. Is there a fork or branch I could take a look at? Thanks.

[Carreau]

Thanks a lot for the proposal.

Hum, right now that will be though, but let me make a quick recap of future plan.

1. get 3.0 out of the door.
2. get https://github.com/jupyter/jupyter-drive out of the door (need 1)
3. Refactor IPython to support RT colaboration (technically help with what you ask, 2 will help a lot)
4. Make an Operational Ttransform server (google RT like) backend for RT communication (independant of 1-3 mostly, but knowing google RT API would help to make things compatible)
5. Make kernel speek with (4) instead of browser.

Depending on what you want to work on there are plenty to do.
Even just playing with IPython master, testing Pull request, or reviewing code
is of huge help as it free us time to work on more complex stuff.

Even if you are just better at communicating that we are, and know a bit about our future plan,
even proposing a news to be posted on main ipython.org website would be awesome !

I should probably make a blog post that details more what we are doing :-P

[malmaud]

What if the web UI could just show some kind of raw console with the output being sent from the kernel, even if it doesn't know what cell that output should go into? Obviously that's not the full long-term solution, but at least it would alleviate a critical usability issue for those of us using the notebook to monitor long-running jobs.

[Carreau]

#7087 should at least lay down the basics for that where there are hooks to receive this output. We still don't know exactly what to do with the output, core team is divised half half on "the right thing'" to do.

[jeanpijon]

This is dirty, DIRTY hack! But if you want to be able at least see the progress output (not cached though), put this code in your custom.js

$([IPython.events]).on('app_initialized.NotebookApp', function(){
      IPython.notebook.events.on('received_unsolicited_message.Kernel', function(evt, data) {
        cells = IPython.notebook.get_cells();
          ncells = cells.length
          idx = ncells-1
          for (var i = ncells-1; i >= 0; i--) {
                if (cells[i].get_text() != ""){
                    idx = i;
                    break;
                }
        }
        cells[idx].output_area.handle_output(data)
        }); 
});

It will append the output from the kernel to the last non-empty code cell. Not nice, but works for me...

[udibr]

+1

[chasewright]

+1

[pasky]

+1

[deepakrox]

+1

[mohamed-ezz]

+1

[sebastian-schlecht]

+1

[zhengbx]

+1

[andychisholm]

+1

[wangz10]

+1

[tachim]

+1

[ZephArvanitis]

+1

[jtanios]

+1

[paolodoors]

+1

[mlzxy]

+1

[mlzxy]

I think if the notebook could have another V layer to capture all the progress data. The browser will interact with this layer instead of raw python kernel. The problem could be solved. (Like a long running proxy browser on your server) This doesn't seem to be a small surgery.

But another idea I have is more like frontend point of view. If the jupyter notebook supports server rendering someday using react/angular2.js/etc... then this problem will be automatically solved.

This feature is very important, since jupyter notebook is extremely popular among machine learning development workflow. Some algorithm training programs could last a couple of days.

[Carreau]

Yes @benbbear, you seem to have came up with roughly one of the long-time planned architecture.

The extra question we still haven't solved is how to build real-time collaboration on top of that.

Currently if you used Python you can use Futures to hold intermediate results.

[mlzxy]

@Carreau

Great to see RT collaboration is on the road map. Maybe the collaboration could be easier if the proxy layer is websocket-enabled instead of just http? Just my rough idea.

Use Asynchronous computing to hold intermediate result, this is genius... 👍

[abrahamneben]

+1

[guillaumerosinosky]

+1

[mharradon]

+1

[francomarianardini]

+1

[Carreau]

If you can restrain from just posting +1 that woudl be great. You can or use emojis on the issue, github added them for that, as you can sort issues by number of +1 Emojis. Also if you just like to subscribe to the issue there is the "subscribe" button on the right.

We are aware of the issue and are doing our best. +1 ing won't make things faster. though reviewing open PRs and trying to write simple patches for issue tagged as "Need contributor" can free us some time to actually work on this :-) We'll also be happy to teach you jupyter internals to get new contributors !

Also at some point the issue probably need to be migrated to jupyter/notebook as this track only core IPython issue now.

Thanks !

[joernhees]

+1 (SCNR)

A connection drop today also made me fall into that trap :-/

I guess even at the moment something on server side must be able to notice that no client is connected anymore. Can't it in such cases modify the ipynb file directly? (Like an auto-saver server side fail-over loop in case no client is reachable.)

Obviously a proper, cool async solution that also allows multi-player edit would be awesome, but after nearly 4 years, maybe a simpler solution first?

[simon19891101]

+1

[wanglovesyang]

+100000000000

[lucasb-eyer]

This actually seems to happen now, though. When I leave the notebook open, put my laptop to sleep while there's output coming (even matplotlib images), then later resume my laptop, re-initiate the connection and reconnect to the kernel, I receive the output that happened while the connection was gone. There's actually two flags controlling this that I had to increase:

## (bytes/sec) Maximum rate at which stream output can be sent on iopub before
#  they are limited.
c.NotebookApp.iopub_data_rate_limit = 100000000

## (msgs/sec) Maximum rate at which messages can be sent on iopub before they are
#  limited.
c.NotebookApp.iopub_msg_rate_limit = 100000

## (sec) Time window used to  check the message and data rate limits.
c.NotebookApp.rate_limit_window = 3

[phizaz]

@lucasb-eyer You mean jupyter notebook or jupyterlab? It seems to be somewhat working in jupyterlab, but not perfectly.

Regrading your settings, could you justify that your settings relevant?

[lucasb-eyer]

Jupyter notebook, I didn't even know about jupyterlab 😄

Yes, the settings are relevant because when the notebook reconnects, all missed messages come streaming in, which kicks-in these rate-limits. Before adjusting the values as above, I only got the first bunch of missed messages upon reconnect exactly because of these rate-limits. Now they are large enough so that I usually get everything I've missed. That's in my typical workflows and tasks, at least. The notebook actually pointed me to those settings when starting to drop messages.

[AbhinavMehrotra]

Which version of notebook are you using? I couldn't get this working on 4.0.6 version.

[lucasb-eyer]

Uhm, seriously? 4.0.6 is 3 years old! I'm using 5.4.1 in all that I described above.

[francomarianardini]

Hello Lucas, thanks for your suggestion. I am using Jupyter Notebook 5.4.0. I put the three options you were mentioning in the config file. Unfortunately, I also couldn't get this working. I will update to 5.4.1 to test the same version you have. Best regards, Franco

…

On Fri, May 25, 2018 at 6:35 PM, Lucas Beyer ***@***.***> wrote: Uhm, seriously? 4.0.6 is 3 years old! I'm using 5.4.1 in all that I described above. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4140 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADmYKzmINkC0vF6AIHd-We1_5G-_CB0Qks5t2DLAgaJpZM4A8uyX> .

[CYHSM]

@lucasb-eyer I guess this only works when you reopen the browser, see jupyter/notebook#1647:

What #2871 did was buffer any outputs (and other messages) until the user reconnects -- it will only help you for cases where you're reconnecting the same tab.

So this fix does not really apply for a remote use case where you want to close the browser tab.

[lucasb-eyer]

@CYHSM Your statement is self-contradicting, I guess you wanted to write

this only works when you don't reopen the browser"

right? Because in that case, I think you are right: I always put my laptop on standby and thus the notebook stays in the same tab.

[tanmay-kulkarni]

Any update on this, please?

[mjkvaak]

+1, this feature would be extremely useful

[ThoSap]

+1

[Yiannis128]

Hi, has this been looked at yet?

[krassowski]

Yes (jupyterlab/jupyterlab#15448). The notebook codebase is no longer in IPython repository, instead it resides in https://github.com/jupyterlab/jupyterlab and https://github.com/jupyter/notebook.

Closing in favour of issues in the repositories where the relevant code is actually in:

• Restoring computation output after disconnect in Notebook jupyter/notebook#641 and
• Reconnect to running session: keeping output jupyterlab/jupyterlab#2833