Can't reconnect to running Python2 notebook

[peter-ch]

I installed the current git-master version of IPython, started a notebook server on a remote machine and then I made a notebook running a long calculation. It periodically outputs results (with clear_output(wait=True) and print()). So I left it running, saved it, and closed the browser. But when I returned to it, it didn't work. It says "Kernel starting" and it stays like that forever. The icon shows that the notebook is busy, the CPU usage is 100% as expected, but I can't see any output. Reconnect Kernel doesn't help.

[takluyver]

I think we made it wait for the kernel to be idle before it decided that the kernel had started. Which is problematic in the case you describe.

[jdfreder]

@takluyver does the kernel run in a separate process (or real thread) than the web server? If so, I'd think we could make it so the web server knew that the kernel exists and is busy- then it could relay that to the front-end which should fix this problem.

[takluyver]

@jdfreder you haven't completely forgotten our architecture, have you? Yes, kernels are separate processes from the web server.

As I remember, we made a deliberate decision that the frontend should wait for the kernel to be connected and idle before showing that it had started, so we could be sure it was ready to receive messages. In most cases, that makes sense, but it doesn't if you have a long-running function and you reconnect to a notebook.

@minrk probably recalls this stuff better than I do.

[jdfreder]

Maybe @minrk can shed some light on why keeping track of the kernel's state in the web server is a bad idea. It seems like it would be an 'easy' solution...

[Carreau]

The server never disconnect from the kernel and act only as a zmq-websocket bridge at this stage.
adding knowledge of kernel state to the server would break the all API/design.

The other thing that it won't help with is that disconnection browser and reopen won't forward output.
All that should be solve once we have notebook state on the server, OT, and browser beeing only a view of the server model.

[minrk]

I think there are two separate ways in which we wait for kernel_info:

1. on the server, we wait for kernel_info reply before opening the websocket. We only do this the first time someone connects to the kernel. We do this one, because we need to wait to do msg spec adaptation before any messages can be processed.
2. In the Javascript, we use a kernel_info ping to ensure that the initial busy state is correct.

I think the problem is caused here by the fact that we set the 'starting...' notification after finishing the websocket connection, which we did before the websocket connection waited for the kernel_info reply before opening. I think a small, but careful readjustments of kernel events and notifications should fix this one.

[minrk]

As @Carreau points out, the actual issue here is the 'lost output on reconnect' (#4140), but the current notification situation is misleading, and can be fixed.

[minrk]

#7664 should address the bad notifications. Closing this as duplicate of #4140.