ipypublish and jupytext

[parmentelat]

Is your feature request related to a problem? Please describe.

my use case is pretty simple
I write notebooks for teaching, mostly rise-powered slideshows
I also use ipypublish to produce handouts on readthedocs
recently I have decided to go for jupytext, it is so much simpler for me to store my contents under git as either .py or .md files
however I have not been able to keep on using ipypublish after moving to jupytext

truth be said :

• I have not touched the few repos where that were working fine, I was trying to apply the same recipe that works for me for example from this git repo to this readthedocs result
• so I don't know for sure if that is even supported, nor that I've done everything right, or if I just misconfigured something somewhere

Describe the solution you'd like

first off I'd like to know if that is something that is supposed to be easy to get to work, or not

Describe alternatives you've considered

I have given a shot at nbsphinx, with which I was more successful wrt jupytext (I guess I found the recipe for binding .py extension) but am running into problems of other kinds (images for example seem to be a struggle; also I had created a custom thingy for having the slide notes showup as a custom class); it's be much easier for me to be able to stick to one solution for this need; it's rather crucial, but otoh the whole point would be ideally to not have to worry about this sort of things to much ;-)

Additional context
I have not kept track of my early attempts, that I scratched when going to nbspinx if relevant I can put that back in shape and provide more details

as an aside, and more generally, I have to say that I am quite confused by the wide spectrum of possibilities for doing this sort of things (ipypublish, nbsphinx, jupyter-book ?, this new myst-nb thing I've [come across recently from here]), that are intimidating b/c of number of options, the pace of change, etc.. if anyone has the time and knowledge to write a short overview on all these within a bit of context, it would honestly be tremendously useful IMHO

thanks in any case for the great tool :)

[chrisjsewell]

as an aside, and more generally, I have to say that I am quite confused by the wide spectrum of possibilities for doing this sort of things (ipypublish, nbsphinx, jupyter-book ?, this new myst-nb thing I've [come across recently from here])

Thanks for the feedback @parmentelat. I'm glad you mention this, if you have a look at the contributors to MyST-NB, you may note that the mains ones are myself and @choldgraf (the maintainer of jupyter-book) 😁

The Executable Book Project is essentially a collaboration between ipypublish, jupyter-book and jupinx to take the best bits of our packages and combine them. This project is only a few months old, but we are looking to make a beta release soon,
with the main entry point being jupyter-book 2.0: https://beta.jupyterbook.org/intro.html

Note with the new package(s) we are specifically collaborating with jupytext and considering the use of .md files:

• https://jupytext.readthedocs.io/en/latest/formats.html#myst-markdown
• https://beta.jupyterbook.org/use/myst-notebooks.html

See spatialaudio/nbsphinx#420 for some outline/debate about the differences with nbsphinx, which has some overlapping but somewhat different goals.

Also note at some point, once I'm happy that all the relevant functionality from ipypublish is available in this new package, I imagine I will probably look to "deprecate" ipypublish in some fashion.

So yes it would be great if you have the time to check that out and give any feedback on any improvements you would desire to meet your use case 😄

[choldgraf]

hey @parmentelat - echoing what Chris mentioned - our goal is to (somewhat) reduce the forking-paths problem by unifying ipypublish/jupyter-book/myst-nb (with jupyter-book being the most user-facing of the three, and myst-nb being used "under the hood" or also used in "regular" Sphinx documentation.

Check out beta.jupyterbook.org, and please do let us know if you have questions and such!

[mgeier]

@parmentelat Author of nbsphinx here!

[...] but am running into problems of other kinds (images for example seem to be a struggle; also I had created a custom thingy for having the slide notes showup as a custom class)

It would be great if you could report your problems at https://github.com/spatialaudio/nbsphinx/issues!

[parmentelat]

hiya everyone

thanks for the feedback; I'll be looking into this new option, this looks promising; would anyone care to confirm what I think I gathered after this exchange

• ipypublish is currently no longer actively supported/maintained so I'd be better off picking something else on mid term
• jupyter-book currently has some provision to read jupytext-based notebooks but at first sight it looks limited to one specific subflavour of .md, is that right ?
• if so, but this is just me being curious, at first it felt like supporting jupytext - any jupytext format - was simple enough, but somehow at this point it no longer looks that way; why is that ? I'm asking because I had just transformed all my input material from .ipynb to .py, if only I'd known ;)

thanks again for any additional tip; I'll report here when I have more hands-on feedback with jupyter-book

[choldgraf]

Quick response from me:

ipypublish is currently no longer actively supported/maintained so I'd be better off picking something else on mid term

That's @chrisjsewell's call, but from my perspective, many of the ideas in ipypublish have now made their way into the new MyST-NB/MyST-parser/jupyter-book stack, and I think that stack will get more development attention moving forward.

jupyter-book currently has some provision to read jupytext-based notebooks but at first sight it looks limited to one specific subflavour of .md, is that right ?

Right now that is the case...some functionality was reduced because of the new build system but we could extend it to support some other use-cases, just want to do it intentionally!

if so, but this is just me being curious, at first it felt like supporting jupytext - any jupytext format - was simple enough, but somehow at this point it no longer looks that way; why is that ? I'm asking because I had just transformed all my input material from .ipynb to .py, if only I'd known ;)

Wanna open an issue about this in MyST-NB? To me it seems reasonable to support jupytext .py files as well as .md files, but it may take some discussion from the community.

[parmentelat]

Wanna open an issue about this in MyST-NB? To me it seems reasonable to support jupytext .py files as well as .md files, but it may take some discussion from the community.

right; I'll do that, thank you

[chrisjsewell]

ipypublish is currently no longer actively supported/maintained so I'd be better off picking something else on mid term

I'm still happy to accept/merge PRs and look into "minor" bugfixes. But obviously there is only finite time in the day lol, so I don't intend to do anything more substantial with ipypublish.

at first it felt like supporting jupytext - any jupytext format - was simple enough

Yeh its something we can discuss further on MyST-NB. In lieu of direct support is this not just a matter of calling jupytext --from py:percent --to myst *.py though?

[choldgraf]

Good point @chrisjsewell - as a stopgap, I added a section on converting files with jupytext here: https://beta.jupyterbook.org/content-types/jupytext.html

[parmentelat]

Yeh its something we can discuss further on MyST-NB. In lieu of direct support is this not just a matter of calling jupytext --from py:percent --to myst *.py though?

well, one could make the same answer for executing the notebooks with nbconvert ;)
my point is, when publishing on readthedocs, the deal is to just drop the files in github and let the whole magic happen for you without the need to fuss with any kind of preprocessing, and this truly has real value indeed :)

[choldgraf]

@parmentelat ideally we'd like this to be possible with many different build engines, not just readthedocs. e.g., see https://github.com/ExecutableBookProject/cli/issues/44 and what will hopefully be a growing set of subsections here: https://beta.jupyterbook.org/start/publish.html

[phaustin]

Related to #133 (comment) -- we just converted stack of jupytext py files to myst, and the one gotcha is that the "language_info" metadata field is inferred when you're using py:percent, but if it's missing in a myst:md file sphinx raises a key_error. So I needed to go through and add the language_info to each of the converted myst notebooks.

[chrisjsewell]

Perhaps you could make an issue for this, with the actual error raised. Because what you said is not exactly correct. When the notebook is run, the kernel adds the language_info to it. So the only reason it would not be present is if the file had not been executed.
Plus the only point at which this is needed in sphinx, is for jupyter_sphinx, to know what extension to save the script file with (technically this should be optional)

[phaustin]

thanks for that hint: I had added
jupyter_execute_notebooks = "off"
because a couple of the notebooks prompted for input, so the notebooks weren't being executed.

[parmentelat]

@chrisjsewell

I'm still happy to accept/merge PRs and look into "minor" bugfixes. But obviously there is only finite time in the day lol, so I don't intend to do anything more substantial with ipypublish.

I've just created #134; it's probably not something you'll accept as-is but I wanted to start the discussion as to how the config system could allow me to achieve this effect

[chrisjsewell]

Hey @parmentelat thanks I will try and check it out over the weekend.
But actually I also have another question: I know you have been trying out myst-nb and jupyter-book, and it appears that you are still using ipypublish. So what do you think that myst-nb/jupyter-book lack currently that ipypublish provides?
Same to @phaustin if you see anything that could be ported over 😄

[phaustin]

I came over from nbsphinx, and the two things I missed at the start were parsing <img> and bare latex environments -- these now in or in-process I think. Not sure about cross-referencing between notebooks -- is something like the nbsphinx :doc: directive possible in myst_nb? i.e.: https://nbsphinx.readthedocs.io/en/0.7.1/a-normal-rst-file.html#links-to-notebooks-and-other-sphinx-source-files

[parmentelat]

@chrisjsewell

I'm preparing for the courses that start in september; I have several courses that were formerly ipynb and I want to migrate to some text format because the json format is a pain wrt git, both for teachers and for propagating course changes to students

with the previous workflow, some of the contents were ipypublish-ready, and ended up on readthedocs
ideally I'd go for jupyter-book, but my progress so far has been very slow for many reasons, both on my end (poor structure) and with the toolset

• the outcome of jupyter-book at this point still has quite a few flaws for me; I have started to report some as issues, but the process is time-consuming as it involves scaling down the input to isolate the problem
• in addition I still have no idea how I'd publish the outcome of jupyter-book as I would rather stay on readthedocs
• on a smaller scale, I had some custom extensions (typically custom rendering of slideshow notes) that I still have no idea how to handle within jupyter-book

so for mitigating the risk I am seeing ipypublish as a fallback; which if you will allows me to split the move into 2 smaller steps: moving to myst/ipypublish, and moving to jupyter-book later on; at this point in time I'll soon have to accept that I won't be able to swallow both steps in time..

you've made your position very clear regarding ipypublish, that's why I made the PR as small as possible, please let me know how you feel about it

[chrisjsewell]

Oh yeh that’s, I should be able to look at the PR over the weekend. It’s not to say you should be using Jupyter-book over ipypublish, just interested to know what your perspective was