A better renderer and page system

[NiklasRosenstein]

Pydoc-Markdown needs a better system to manage which API object is documented in which file. The new system should overcome all of the current shortcomings:

• Current renderers either expose a pages property, which does gives the user some degree of flexibility of specifying which API object content to render where, but the selection is limited to glob patterns which does not work well when you want to select the contents of a package but not it's submodules (e.g. MkDocs and Hugo renderers). Or the renderer just doesn't give you the flexibility and places everything into pre-determined files according to the module structure (Docusaurus renderer).
• The CrossrefProcessor does not currently understand the concept of multiple pages and thus can only generate links to API objects in the same Markdown file. Even if it did, we would need to track a unique mapping of which API object is mapped to which file, and furthermore, we must ask the renderer to generate a relative hyperlink to that API object in that page.
• While the MarkdownRenderer has a ton of options, it's customizability is still limited to that set of options. Users cannot mix plain documentation and generated markdown from API objects.

Thus I propose the following changes and additions:

1) Internal representation of pages

A page is a Markdown file identified by it's path relative to the Pydoc-Markdown generated files directory (important: This is not the same as the final build directories that some renderers might generate HTML files in). We store a set of pages that are to be generated in a Pages container object. Page paths in this object are treated like POSIX paths independent of the current operating systems.

For each page, we store a list docspec.ApiObjects for which content is going to be generated in the page. The hierarchy of API objects in this listed is ignored, only the immediate elements contained in the page matter. The order of the objects in the page may also describe the order in which the documentation is generated in the page by the renderer.

class Page:
  relative_path: str
  contents: List[docspec.ApiObject]

class Pages(Dict[str, Page]):
  ...

2) Mako template renderer

The Mako template engine provides greater flexibility than Jinja2 because actual Python code can be written in templates more easily. Including complex logic in a template without too much overhead will make authoring customizable default templates for generated Markdown content much easier. By leveraging template partials, the MarkdownRenderer can be implemented entirely as a Mako template.

The advantage to using template partials is that users can then start overriding parts of the template by providing their own partial alongside the Pydoc-Markdown configuration, giving them ultimate control over each individual aspect of how Markdown code is generated from API docs.

\#\# ${cls.name} Objects

${cls.docstring}

Assuming we now have a Mako template and functions to render individual API objects to Markdown using that template, we can use it to render the Page.contents list into a Markdown file. Consequently the MarkdownRenderer would be deprecated.

3) Avoid masking static site generator interfaces

Currently Pydoc-Markdown largely masks the interface for MkDocs or Hugo by expecting that you configure those renderers mostly through pydoc-markdown.yml instead. This is nice on the one hand because you can get started really quickly without having to understand much about the static site generator and just have a rough idea about what Pydoc-Markdown does. However it also makes the end-to-end process (from Pydoc-Markdown config to generated static site) opaque and thus hard to follow, and particularly harder to debug.

Example from the Pydoc-Markdown configuration itself:

pydoc-markdown/pydoc-markdown.yaml

Lines 28 to 48 in c3b9713

	pages:
	- title: Home
	name: index
	source: README.md
	- title: Changelog
	name: changelog
	source: CHANGELOG.md
	- title: Getting Started
	source: docs/getting-started.md
	- title: Read the Docs
	source: docs/readthedocs.md
	- title: Configuration
	source: docs/configuration.md
	- title: Migrate from 3.x to 4.x
	name: migrate-from-3x-to-4x
	source: docs/migrate-from-3x-to-4x.md
	- title: API Documentation
	children:
	- title: CLI
	contents:
	- pydoc_markdown.main.cli

I propose that in the future, Pydoc-Markdown should avoid wrapping static site generator, and instead focus on being used primarily as a generator of Markdown files. We can still provide helpers to kick off the the static site generator from the Pydoc-Markdown CLI after markdown files have been generated, but that's just a nice to have.

Consequently the MkDocs, Hugo and Docusaurus renderer would be deprecated as well, but a library of Mako templates may be provided if there are substantial differences in the Markdown that needs to be generated on the target static site generator.

4) Preprocess Markdown files

If we have Pydoc-Markdown act as a pre-processor for Mardown files, and not just a pure generator, we can allow users to place API documentation inline with static documentation or make linking to API objects from static documentation easier. Example:

# Welcome to Pydoc-Markdown!

Have you checked out the {@pylink pydoc_markdown.PydocMarkdown} class?

## API Documentation

Here is everything you need to know.

@pydoc pydoc_markdown.PydocMarkdown

By pre-processing this Markdown file and replacing the {@pylink } and @pydoc tags respectively, users gain a lot of freedom on how to structure their API documentation and mix static documentation content with dynamically generated content. From the tags in the file we can construct the Page.contents and register it in the Pages container such that the CrossrefProcessor can construct a link to the API object's placed inline into such pages.

The main challenge here is probably going to be to figure out how we can best implement the pre-processing without replacing the un-processed source files with processed ones.

5) Add markdown dialects

The MarkdownDialect will be an interface to provide the hyperlink to an API object in one page relative to another page. The dialect will need to be selected according to the static site generator that the generated Markdown content will be consumed by (e.g. header IDs in MkDocs and Hugo are different).

class MarkdownDialect:
  def get_hyperlink(self, target: docspec.ApiObject, on_page: Page, relative_to: Page) -> str:
    ...

[melMass]

Very interesting! Sorry I had too much work to focus on the image-support feature.
One thing I realized while getting familiar with the codebase is that processors seams to run on source files only, while the pages only go through the renderer? This might change in the new design if I understand the goal properly.
I personally only needed image support for pages so I'm not sure of the right path to take, to keep it lean.

Do you have an ETA for this new system?

Cheers

[NiklasRosenstein]

Hey @melMass , I'm working on the new system right now. It ended up quite a bit different than described here, but I think it will be very powerful.

You can track the progress in this PR: #245

For your particular use case, with the new system, I think you wouldn't need to copy the image file into the build directory because no longer does Pydoc-Markdown generate new files for you, but instead your existing Markdown files are preprocessed. Thus the image file should still sit relative to the file in which you originally inserted the image reference.

If something like this is still needed, it would need to be implemented as part of Novella, which backs the new system. There's already an issue (NiklasRosenstein/novella#2) for implementing this when using the Novella @cat tag, but I don't think I'll come around to implementing that soon.

[melMass]

I had a quick look, this is a great idea and makes a lot of sense!
Novella could actually be used for other doc systems it seems.
I'll dig in this week end! Thanks for the poke ❤️