How to embrace literate programming with Livebook?

[fredguth]

Livebook appears to be an ideal platform for literate programming, seamlessly integrating code and explanations.

While I'm new to Elixir, I have experience in ML development using Jupyter notebooks and nbdev. Despite encountering challenges with Jupyter like experiment replication difficulties (because of its global variable space and runtime model), subpar autocompletion, and the absence of my preferred keybindings, I still favor this method because it allows me to articulate my thoughts effectively.

Particularly with nbdev, I can work within notebooks and, via a CLI command, generate a clean code library and an accompanying documentation site that I can easily customize.

Livebook seems to support these capabilities and more. So, my question is, why isn't Livebook itself developed using Livebook?

[jonatanklosko]

Developing whole libs and apps in Livebook alone is not our objective. Doing so means all notebooks need to be run/compiled altogether, which doesn't match the sequential nature of the notebooks. Also making Livebook an effective environment for that purpose basically means developing another IDE with notebooks being project files, and that's not something we aim at either. Elixir already has great tooling for documentation and code examples (doctests). However, Livebook enables you to hook into an existing project, interact with all the modules and write new code, either to have a notebook with domain-specific examples, or to prototype new modules to be incorporated into the codebase later. Notebooks themselves can also included in library documentation, including outputs. In practice, I think being an addition to existing tooling is much more useful than trying to replace all said tooling :)

[fredguth]

I understand your point. But please bear with me for a moment. Imagine you just want to create educational content and you want to build a complex library, step by step, explaining what you are doing (and why).

Doesn't Livebook seem like the perfect environment for such a journey?

Regarding your specific topics:

"Also making Livebook an effective environment for that purpose basically means developing another IDE with notebooks being project files."

• I don't understand this point. Livebook already has features like IntelliSense, keybindings, etc., and, more than most IDEs, it offers visualization tools like Kino. What exactly should Livebook avoid having that IDEs have?

"Elixir already has great tooling for documentation and code examples (doctests)."

• As I am new to this ecosystem, maybe I just haven't come across these examples yet. I would really appreciate it if you could point me to some Livebooks or other resources that demonstrate these kinds of references to HexDocs and doctests.

Still, I don't see a way I could just start defining a module and insert markdown, kino, mermaid in the middle of the "flow" and then come back to the module code. This is something that in jupyter and python people do using decorators (ex. @patch).

[jonatanklosko]

Livebook already has features like IntelliSense, keybindings

Yes, but it is all focused on a single notebook, rather than multiple notebooks. If we were to treat multiple notebooks as a single project, then people would expect module-aware intellisense across notebooks, and a tighter integration across notebooks in general. But then again, the main thing is that our notebooks are sequential by design.

As I am new to this ecosystem, maybe I just haven't come across these examples yet.

See Axon docs for example, the guide pages are notebooks that the user can run, and you can see the notebook outputs and graphs rendered there.

Doctests are an Elixir feature, where you have code examples in the function docs and you can automatically run these examples as part of your test suite to make sure they stay up to date (example). Sidenote, if you define a module with doctests in Livebook, they are automatically run and reported too.

and then come back to the module code

There is no way to patch a module that has been defined, we also explicitly disallow modules to be redefined by subsequent cells, because redefining may lead to reproducability issues. If you want to do module breakdown it's better to either define the module and discuss individual pieces, or write module parts as markdown snippets, discuss them individually and then have the full module definition (that means there's duplicated code, but that's the tradeoff).