Next steps in improving the documentation system.

[mmatera]

Now that mathics-core works again without issues in the CI, I continued working on the documentation system.

Among the pending tasks

• (a.) Fix some broken links in the documentation.
• (b.) DRY code in mathics.doc and mathics.docpipeline.
• (c.) Start to split the code in mathics.doc.common_doc in a part related to the organization of the code, and a part
  related to parsing the documentation entries and organizing the doctests.
• (d.) More comments, documentation, and tests for the documentation system.
• (e.) avoiding go over to the full documentation system to access a specific set of tests.
• (f.) Allow pymathics modules to use the docpipeline system to tests their symbols.
• (g.) restore the proper ordering of chapters and sections.
• (h.) review the structure of the documentation.
• (i.) check that both mathics-django and latex documentation looks as it should look.

... (add other items that you think must be present in this list )

• (y.) Split the documentation system as a separate package.
• (z.) From the homebrew documentation system, produce some more standard documentation like sphinx or at least markdown documentation.

Today I think made some progress in the first of these points

#1021 addresses a. to d.
#1022 addresses partially e. and f.
#1025 goes further in c., g., h. and i.

In order to continue with this task, I think we need to discuss some details about the structure of the documentation. I will try to start the discussion here.

Guide Sections and builtin packages

Several of the "FIXME" comments inside the code of the documentation system are related to the way in which we are handiling packages (subfolders) inside mathics.builtin and pymathics modules. In the current code, each file and folder inside mathics.builtin and pymathics modules are mapped to "chapters" inside a part ("Reference of Built-in Symbols" and "Mathics3 Modules" respectively).

For files inside these modules, the module docstring is used as the introduction to the chapter, which is followed by a Chapter index and a set of sections corresponding to each builtin symbol defined in the module.

For packages inside the top-level modules, the docstring in the __init__.py file is used to generate the introductory text of the chapter, which contains
a "guide section" and "sections". The "guide section" starts with a documentation chunk containing exactly the same than the chapter introduction,
and "subsections", which maps the list of sub-modules. A the level of the chapter, after the "guide section" we find (regular) "sections", with are in correspondece with the submodules. The (regular) sections under a chapter have an initial text, built from the sub-module docstring, and "subsections" comming from the symbols defined inside the corresponding submodule.

To make this galimatias more clear, let's see two examples.

• From mathics.builtin.procedure (a single file module) we produce

   Chapter "Procedural":
      introduction text (from ``mathics.builtin.procedure.__doc__``)
      chapter index
      section "Abort" (from the docstring of the class  `mathics.builtin.procedure.Abort`)
      section "Break" (from the docstring of the class  `mathics.builtin.procedure.Break`)
      ...

• From mathics.builtin.colors

   Chapter "Colors":
       introduction text (from ``mathics.builtin.colors.__init__.__doc__``)
       chapter index
       Guide section `Color`
            introduction text (again from ``mathics.builtin.colors.__init__.__doc__``)
	    subsection "Color Directives" (from ``mathics.builtin.colors.color_directives.__doc__``) (type section)
	    subsection "Named colors" (from ``mathics.builtin.colors.named_colors.__doc__``)         (type section)

       Section "Color Directives" (from ``mathics.builtin.colors.color_directives.__doc__``)
            subsection "CMYKColor" (docstring from the class ``mathics.builtin.colors.color_directives.CMYKColor``)  (type subsection)
	    subsection "ColorDistance" (docstring from the class ``mathics.builtin.colors.color_directives.ColorDistance``) (type subsection)
	    ...

       Section "Named colors" (from ``mathics.builtin.colors.named_colors.__doc__``)
            subsection "Black" (docstring from the class ``mathics.builtin.colors.named_colors.Black``) (type subsection)
	    subsection "Blue" (docstring from the class ``mathics.builtin.colors.named_colors.Blue``)   (type subsection)
	    ...

Notice that with the current organization,

• subsections of a Guide section are of class "DocSection", while in Sections are "DocSubsections".
• In docpipeline "Section" is used as an equivalent of "Symbol".
• In the LaTeX documentation, "Sections" and "Subsections" are shown different. But then, the symbol Black and symbol Break are shown in different formats.
• subsections in a Guide Section actually are pointers to the other sections in the chapter. Then, at the time, a subsection of a guide section has subsections.
• In the docpipeline, since the same documentation entry appears in two places (as a subsection of a subsection in the guide, and as a subsection of the chapter, and the documentation entry is also repeated as the chapter introduction and the Guide Section introduction), we have to actively avoid to process the same test two times.
• If a submodule has a submodule, "init.doc" is loaded as another section, but its subsections are not loaded.

These inconsistencies both in the structure and in the formatting of the documentation must be removed in order to make the code simpler, and the documentation more readable and compatible with other documentation systems.

[mmatera]

Guides in WR

WR is a quite sophisticated documentation structure, which has different kinds of entries. Some of the main kinds of entries are:

• Functions (Symbol entries): contain the documentation about a symbol.
• Guides: A description of a certain set of related symbols, and a list of links to the corresponding symbol entries. The links at the time are organized in subgroups by specific tasks, having their own guides.
• Technotes, Tutorials, etc, etc.

The format is hypermedia and not hierarchic: two guides can include links to the same symbols, and there is no organization in chapters/ sections/ subsections. This scheme is OK for an online documentation system, but probably not for a book.

Proposal to translate this to the Mathics Documentation System

A way to translate this to our documentation system would be

• Each module inside builtin/pymathics, at any level -> a DocChapter.
• If the module is a single file, it contains sections, one for each symbol inside.
• If the module is a package, the chapter includes a guide for each submodule, and a section for each symbol on each submodule.
• Each Part can contain a list of the top-level Modules/chapters to generate the entry point in the online documentation.

With this pattern, each DocEntry appears exactly once. Thoughts?

[rocky]

Maybe the problem here is we are trying to develop a documentation system that describes at the same time a hypermedia document (like WR) and a regular "linear" book, as well as a native "doctest" system. Let me explain what I understand looking at WR, and my reasons for proposing the new documentation structure organization.

Yes. There were very specific goals when I started undertaking this in preparation for a release, and the scope expanded to something totally different. Basically I wanted the documentation to be just like before, but with the ability to be in a position to

• support existing autoloading built-in functions. As with the import mechanism, the doc makes one pass gathering everything and has no support for incremental update after a builtin or set of builtins are autoloaded
• treat the handling of Mathics module the same way as a builtin. So a Mathics module starts off as a chapter like mathics.builtin modules do. (The module path of course is different, but that has to be). Right now, the Mathics3 Module system is organized differently (and less good) than the builtin system which allows for "guide sections".

Side goals that I had hoped would fall out of this would be to be able to produce LaTeX documentation for a smaller section of the book so it can be debugged easier. Also I didn't want to give up on the current behavior in Django where a dosctring can be updated and its effect is seen immediately. The batch and monolithic behavior that we have largely inherited make working on things more tedious and slower and as a result, there are more doc mistakes and review due to the tediousness of building docs.

Adding more URL tags like an image tag, doesn't feel important. I don't think we are in great need of adding static images. If I am wrong, let me know what images we need. Obviously I did not feel that way about adding URLs for cross referencing the document or to link to WMA or Wikipedia. Overall, I think that has been a big win, at least for me.

Replicating the features of WR is also out of scope. It may be good to keep in mind where things might end up. But for now, there are far more important things to do. If I could just get those two items fixed (and it is not clear that right now they are), then I'd be happy.

The coe module structure and books are l not hyperlinked . How one picks a tree structure from what is essentially a directed acyclic graph that WR has is arbitrary. That's just the way it is.

• The same Built-in symbol references entry can be associated with several guides.

This is easily addressed by adding a class variable which lists all of the other guides. In terms of the code the symbol lives in one of guides and sections and subsections.

• Built-in symbol references should be at the same level even if belong to a shallow module.

I don't agree at all, as far as the code goes. I am not sure I agree in a book either. In Hyperlinked documentation, it is hard to tell whether something is at the same level. In Django right now, I don't think anyone is bothered by the fact that from some menus may require another level to get to the symbol than some other symbols. One can get directly to a particular symbol by searching on the name of the symbol. And that's what I do when I want information about a symbol.

• In docpipeline, we associate Symbols with sections when referring to a set of tests.

This doesn't bother me in the slightest. Just as with the Django doc another option can be added distinct from chapter or section to refer to a Symbol.

• In WR, guides and references look very similar

This might be a nice long-term thing to have but it wasn't on the short list of things to do before a release. Feel free to add this as a long-term goal.

• The scheme allows mapping each docstring into the documentation structure.
• gives a clear way to deal with nested packages.

I am not sure I understand exactly what these mean, but I don't think it matters. I'd like to defer all of this until after we get the next release out. None of this was in our roadmap.

[mmatera]

Ok, in that case, I will try to rewrite mathics.doc.gather to reproduce the documentation just as it was in the last release. Then we can continue the discussion and iterate over it.

[rocky]

If we could get the Mathics modules to work like builtins where they can have modules underneath that would be great too. So the Mathics3 Graphs Module can have Sections analogous to https://reference.wolfram.com/language/guide/GraphModifications.html

Also good would be cover the other bullet item where the documentation for a given module can be added lazily as will come up soon in autoloading.

Thanks.

[mmatera]

OK, #1022 now contains changes in #1025, but produces documentation with the format used in 6.0.

Also handles pymathics modules in a similar fashion as regular builtins:

Each pymathics module like Graph is associated to a chapter. Each submodule produces a guide, containing the corresponding symbols as subsections.

Now the question is, what happens if pymathics.graph.centralities had subpackages? Should be subsubsections? we should "flatten" the packages, in a way that a subpackage produces a new guide?

[mmatera]

Notice also that mathics.doc.gather.gather_reference_part allows to process any list of modules to produce a "DocPart", so it would be possible to generate documentation from a Pymathics module, in a way that each module inside the package could produce a chapter.

[mmatera]

Sphinx and readthedocs compatibility

Another possible improvement to the documentation system would be to use more standard syntax like CommonMarkdown or reStructuredText. Using a more standard notation would be helpful to replace the homebrew parser, based on Python regex, with another existing parser library. My first impulse was to consider Markdown, because of its simpler syntax, but @rocky has a point when he proposes using RsT, which has a more precise syntax and plays better with Sphinx, and seems to be the current standard.

#1026 provides support for some basic notation like links, images, and code blocks in common Markdown. When I have another chunk of time, I will try to do the same to support RsT.
Then, the long-term plan would be

• to support the most suitable syntax for using an existing documentation library
• Progressively replace the old syntax with the new one.
• Once the documentation can be handled by a standard parser, remove the homebrew, regex-based parser.