Prevent kernels from even sending too much output to the notebook frontend

[michaelaye]

Due to case of PEBKAC we created a large notebook (4.6 MB) with a lot of print-outs that would not load anymore in Safari and Firefox. Safari does not get back to me at all, Firefox after a while tells me that the script is unresponsive. Chrome at the end could load it, so that I could use the clear output function to get rid of the megabytes of text.
So, I don't really need help anymore, but I was thinking that you maybe are interested in having a look at this script in the broken form to see if there's anything that can be done to have the notebook protect itself from such foolery.
I uploaded the broken script here: https://gist.github.com/8be664570dd2100310d6

[minrk]

Thanks, we'll look into it.

[Carreau]

cf #6516 for an anti while True loopextension, also you can run clear output from nbconvert command line and do ipynb -> ipynb if it ever happend again.

[dsblank]

It would be great if IPython could do a couple of things to help with this:

• "Your program is creating a large amount of output; continue running?"
• Loading notebook: "This notebook contains a lot of output; Clear output cells or Load normally?"

[michaelaye]

@Carreau I think this should be in a FAQ! With pandas, it's extremely easy to accidentally create a plot over 200 columns with 130,000 data points... and if that plot is inline it 'molasses' the browser. ;)

[michaelaye]

Hm, nbconvert --to= option does not mention .ipynb as a potential output and just using it by --to=ipynb fails.

[Carreau]

nbconvert --help-all extract:

--to=<CaselessStrEnum> (NbConvertApp.export_format)
    Default: 'html'
    Choices: ['custom', 'html', 'latex', 'markdown', 'notebook', 'pdf', 'python', 'rst', 'slides']

I suppose you mean notebook ? Should we alias ipynb to notebook ?

[dsblank]

I see this a lot with my students (and me). How to break ipython in 13 characters:

def f():
    f()
f()

1000 stack listings creates a frozen notebook in Chrome. But then one can't open a new notebook to run those conversion commands, because the dashboard and notebook are locked up... 5 minutes later... released! Now I try to clear output from cell... 5 more minutes... even closing the notebook tab doesn't help.

[michaelaye]

What version? I have 2.3.1 last official release and it has this:

--to=<CaselessStrEnum> (NbConvertApp.export_format)
    Default: 'html'
    Choices: ['custom', 'html', 'latex', 'markdown', 'python', 'rst', 'slides']
    The export format to be used.

I'm using ipython nbconvert, is there an extra application??

[Carreau]

2.x cannot do ipynb to ipynb

[dsblank]

I think the main issue here is the time taken to render large amounts of input in the browser (although it would take some time just to transfer 4.6 MB from the kernel to the browser). Another issue here suggests that a proposed CSS change may help this problem.

On a related point, showing tracebacks that have more than, say 20, function calls in the stack trace is not useful. Consider having a expand/collapse hierarchical tree object that the stacktrace could use. Or maybe for any large output there should be a "Show more..." link to make sure that notebooks can render. (Large outputs don't seem to be a problem in the static nbviewer. I accidentally printed a student's 500 page nbviewer-rendered notebook with stacktrace the other day...)

The problem of large outputs making a notebook unloadable can happen in any Jupyter kernel, so isn't limited to just IPython.

[cancan101]

I am running the kernel locally so I should hope that transferring 4.6MB data does not take that long.
My hunch is that there is some large fixed cost to interpreting / rendering one of those stdout lines. ie what is the time to render 1 standard out with K lines vs K standard outs with 1 line each.

[ellisonbg]

We have talked before about putting in some safe guards to prevent large amounts of output from even reaching the notebook (or other) frontends. The way we handle large outputs right now is very problematic. Some points:

• The kernel itself should manage this - it should refuse to even send output beyond a certain point.
• The kernel should save large output somewhere (probably to disk, but that may even be a problem beyond a certain point).
• The the kernel should send something that indicates that large output was generated and provide the user with some way to view it, or at least tell the user how much output was generated and where it was put.
• All of this should be integrated with the output scroll/collapse in an intelligent manner.

I have changed the title of this issue to reflect this more general issue.

[dsblank]

Ok, but it seems like the frontend knows better what is "too much". If the kernel decides, then it would be the same for all frontends. What is too much for the console is different than that for the notebook.

[ellisonbg]

I mostly agree with you. It probably makes sense to have a tiered approach
where initially this handled by the frontend. But for really large data,
you know that there isn't any frontend that can handle it. But the most
important thing is that beyond a certain point, no human can reasonably
look at the output. It isn't just about performance, it is about user
experience. I think we can be fairly aggressive in telling the user "you
just tried to generate more output than you could possibly look like"
regardless of the performance issue.

On Mon, Jan 12, 2015 at 10:38 AM, Doug Blank notifications@github.com
wrote:

Ok, but it seems like the frontend knows better what is "too much". If the
kernel decides, then it would be the same for all frontends. What is too
much for the console is different than that for the notebook.

—
Reply to this email directly or view it on GitHub
#6771 (comment).

Brian E. Granger
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
bgranger@calpoly.edu and ellisonbg@gmail.com

[juhasch]

Inspired by a comment from @Carreau I made this notebook extensions:
https://github.com/ipython-contrib/IPython-notebook-extensions/wiki/limit-output

Works well for me to prevent the browser from crashing in endless print loops.

[Carreau]

Agreed with @juhasch most of browser crashing happend because of DOM element.
and the 'limit' highly depend on the type of output. Displaying the same amount of data as PNG or text is completly different in term of what browser could handle.

[dsblank]

@juhasch I think I may have to enable your extension by default for IPython 3, otherwise stack overflows make the notebook unloadable.

BTW, what is it that allows you to go to /nbextensions/ and click on Activate for an extension... I want that too!

[juhasch]

The server extension for /nbextensions/ is in this pull request:
ipython-contrib/jupyter_contrib_nbextensions#164

[Carreau]

Closing as this issue is not in IPython itself and if still problematic and relevant should be opened on the right repository. This will allow to keep the number of opened issue on the IPython repo under control.

Feel free to keep commenting or reopen if necessary.

Thanks.