Support for jupyter-cache?

[jeromekelleher]

Is support for jupyter-cache something that is planned/feasible in jupyter-sphinx? It is very useful to be able to avoid reexecuting code blocks over and over when working on a page.

[akhmerov]

Hi @jeromekelleher!

There are several aspects to your proposal that I'd like to bring up in order to make an informed decision.

• First of all, execution of the notebooks is extremely localized in the codebase, and therefore it would take a limited amount of work to make this more general and include jupyter-cache.
• Bringing in another execution framework adds complexity in configuration (e.g. where jupyter-cache should store its cache), and I think limiting this complexity is a good idea.
• Sphinx allows document-level parallelization via multiprocessing. I am not sure whether jupyter-cache would play nicely with that (see also Add parallel execution implementation executablebooks/jupyter-cache#37)
• Likewise sphinx has its own caching machinery. jupyter-sphinx uses it to not reexecute files that didn't change at all. It may be possible to rely on that, although I'm unsure whether it's a reasonable approach.
• Multiple cache layers are frequently not transparent to the users and lead to subtle problems when it isn't clear which cache should be invalidated.

So overall I am open to incorporate integration with jupyter-cache, and it seems like a good idea, but planning this would require addressing the above considerations.

[jeromekelleher]

Thanks for the detailed response @akhmerov, this is very helpful! I agree it would be quite a lot of extra complexity and its debatable whether it's worth it.

[akhmerov]

I didn't mean to say that it's not worth it. It's likely worth, however the specific implementation requires evaluating and answering the above questions.