Question: how do I replicate old pydoc-markdown setup with Novella?

[mcarans]

It looks like you're making some major changes. How do I replicate my simple setup with the old pydoc-markdown of a main.md and the generated API documentation from the TOML description? (In readthedocs, it looks like this.) Can I simply add tags at the bottom of main.md for the various API pages like "Configuration" that you can see below so that they appear under "API Documentation" in the menu on the left as now, or do I have to make one markdown page for each API page (I hope not as that would be a lot of tiny repetitive md files)? Where does the configuration go like loaders, renderer and source_linker or is that not needed any more?

My pyproject.toml looks like this:

[[tool.pydoc-markdown.loaders]]
type = "python"
search_path = ["src"]
packages = ["hdx.api", "hdx.data", "hdx.facades"]

[tool.pydoc-markdown.renderer]
type = "mkdocs"
output_directory = "docs/build"

    [tool.pydoc-markdown.renderer.mkdocs_config]
    site_name = "HDX Python API"
    theme = "readthedocs"
    repo_url = "https://github.com/OCHA-DAP/hdx-python-api"

    [tool.pydoc-markdown.renderer.markdown.source_linker]
    type = "github"
    repo = "OCHA-DAP/hdx-python-api"

    [[tool.pydoc-markdown.renderer.pages]]
    title = "Home"
    name = "index"
    source ="docs/main.md"

    [[tool.pydoc-markdown.renderer.pages]]
    title = "API Documentation"

        [[tool.pydoc-markdown.renderer.pages.children]]
        title = "Configuration"
        contents = ["hdx.api.configuration.*"]

        [[tool.pydoc-markdown.renderer.pages.children]]
        title = "Locations"
        contents = ["hdx.api.locations.*"]
...

[NiklasRosenstein]

Hey @mcarans!

Can I simply add tags at the bottom of main.md for the various API pages like "Configuration" that you can see below so that they appear under "API Documentation" in the menu on the left as now, or do I have to make one markdown page for each API page (I hope not as that would be a lot of tiny repetitive md files)?

Good question! I've been expecting this one to come up and been thinking about it for a while. I'm not entirely sure yet about the extend to which these files should be generated for you automatically, i.e. where is the line between power & control vs. ease of use.

For now, I've made a change to Novella 0.1.14 that allows you to create a custom action with your own logic that can be used to generate the files before the Markdown preprocessor runs (which includes Pydoc-Markdown).

I've added an example to the Novella documentation here: https://niklasrosenstein.github.io/novella/concepts/#actions

template "mkdocs"

def api_pages = {
  "Configuration": "hdx.api.configuration",
  "Locations": "hdx.api.locations",
  # ...
}

action "preprocess-markdown" {
  use "pydoc" {
    loader().packages = api_pages.values()
  }
}

do
  name: "generate-api-pages"
  closure: {
    precedes "preprocess-markdown"
  }
  action: {
    for title, package in api_pages.items():
      def filename = directory / 'content' / 'api' / (package + '.md')
      filename.parent.mkdir(parents=True, exist_ok=True)
      filename.write_text('---\ntitle: {}\n---\n@pydoc {}\n'.format(title, package))
  }

Currently, the @pydoc tag does not understand wildcards but only full object identifiers, so in the example above I put the module name hdx.api.configuration instead of hdx.api.configuration.* into the file.

Let me know what you think, absolutely open for input here.

Where does the configuration go like loaders, renderer and source_linker or is that not needed any more?

The answer here is mostly "no". The PydocTagProcessor that you enable with the use "pydoc" statement in the Novella build script is a bit more opinionated and applies some heuristics compared to the old-style Pydoc-Markdown YAML configuration.

• The loader is a PythonLoader by default that is initialized with sensible default search paths
• The processors are currently hardcoded and no public API exists to modify them. It uses the same set of processors that the old-style configuration used if no processors were explicitly configured, except for a slight change to the cross-ref processor which must produce Novella {@link} tags
• The renderer is a MarkdownRenderer with some parameters changed by default
• It also tries to automatically detect an applicable source linker

pydoc-markdown/src/pydoc_markdown/novella/preprocessor.py

Lines 56 to 76 in 02713da

	def __post_init__(self) -> None:
	# Heuristic to provide a sensible default configuration of the plugin.
	if Path.cwd().name.lower() in ('docs', 'documentation'):
	search_path = ['../src', '..']
	else:
	search_path = ['src', '.']
	source_linker = autodetect_source_linker()
	self._loader = PythonLoader(search_path=search_path)
	self._processors = [
	FilterProcessor(),
	SmartProcessor(),
	# We return the entire link formatted as a Novella {@link} tag in #resolve_ref().
	CrossrefProcessor(resolver_v2=MarkdownReferenceResolver(global_=True)),
	]
	self._renderer = MarkdownRenderer(
	source_linker=source_linker,
	render_novella_anchors=True,
	render_module_header=False,
	descriptive_class_title='Class ',
	)

pydoc-markdown/src/pydoc_markdown/novella/preprocessor.py

Lines 26 to 42 in 02713da

	def autodetect_source_linker() -> t.Optional[SourceLinker]:
	repo = detect_repository(Path.cwd())
	if not repo or repo.type != RepositoryType.GIT:
	return None
	if 'github.com' in repo.url:
	name = assure(re.search(r'github.com/([^/]+/[^/]+)', repo.url)).group(1)
	return git_source_linkers.GithubSourceLinker(root=repo.root, repo=name, use_branch=True)
	elif 'gitlab.com' in repo.url:
	name = assure(re.search(r'gitlab.com/(.*)', repo.url)).group(1)
	return git_source_linkers.GitlabSourceLinker(root=repo.root, repo=name, use_branch=True)
	elif 'gitea.com' in repo.url:
	name = assure(re.search(r'gitea.com/([^/]+/[^/]+)', repo.url)).group(1)
	return git_source_linkers.GiteaSourceLinker(root=repo.root, repo=name, use_branch=True)
	# TODO (@NiklasRosenstein): BitBucket
	return None

---

As a side note: You probably want to place an mkdocs.yml file next to the Novella build script; for example to make sure that it uses the readthedocs theme. Note that the Novella MkDocs template puts some defaults into the config unless you disable it. You can read more about that here: https://niklasrosenstein.github.io/novella/components/templates_/mkdocs/#class-mkdocsupdateconfigaction

To have Novella not touch your mkdocs.yml:

template "mkdocs"

action "mkdocs-update-config" {
  apply_defaults = False
}

[mcarans]

I would guess that my use case is quite common - I want to use markdown and readthedocs. Since I have several projects all using pydoc-markdown, I have made a markdown readthedocs setup that involves as few separate configuration files as possible which meant putting as much as I could into pyproject.toml.

BTW, you may be interested to know that the configuration of pydoc-markdown has been used in discussions about improving nesting support in TOML which are making progress at last.

In the old setup, I could get away with pyproject.toml, .readthedocs.yml and .readthedocs-requirements.txt (along with my main.md file). Ideally I'd like the new Novella setup to minimise the need for multiple configuration files because keeping all these files up to date across multiple projects is a lot of work and it's easy to miss something the more configuration files there are. This could be achieved to some extent by having sensible defaults but also, for example, by having the faciliy for all configuration to be in pyproject.toml rather than separate configuration files.

In terms of how to do something akin to what I was doing in the old setup with the new approach, I could envisage adding to my main.md tags that get expanded into separate markdown pages, maybe something like:

...The static metadata for ACLED is held in human readable files so 
if it needs to be modified, it is straightforward. This is another feature 
of the HDX Python library that makes putting data programmatically 
into HDX a breeze.

@pydoc :start_section API Documentation
@pydoc :apigen ["hdx.api.configuration.*"] :title Configuration
@pydoc :apigen ["hdx.api.locations.*"] :title Locations
...
@pydoc :end_section API Documentation

[NiklasRosenstein]

I totally get the preference to keep the number of configuration files at a minimum. It won't be so easy to make Novella configurable in pyproject.toml because it's configuration file is actually a script; a Python superset. It would be possible to read the script code from the TOML for the sole purpose of not having to have a build.novella file, although I don't feel very positive on this being a good solution.

# As of the time of writing this, this is not actually recognized by Novella
[tool.novella]
script = """
  template "mkdocs"
  action "preprocess-markdown" {
    use "pydoc"
  }
  """

Personally, I kind of like the docs/build.novella file because it keeps everything related to building the docs (aside from the requirements; which are configured in pyproject.toml) inside of the docs/ directory.

But if I understand you correctly, your concern is also with the additional Markdown files that would be needed to describe the pages for which the content should be automatically generated by Pydoc Markdown. That decision was made very consciously, for three reasons:

• it makes it easier to reason about the structure of the final generated docs (the pages in the old-style YAML configuration can easily become difficult to process in ones head, overall the project structure is much closer to what the static site generator being used expects, which also makes it easier and more easy to understand when configuring the nav in MkDocs for example);
• each page in the generated docs has an actual counterpart in the repository (making the "Edit Page" URLs generated by MkDocs, if configured correctly. which Novella makes a fair attempt to do automatically, work properly instead of pointing to a 404 page);
• it allows you to place static content in between automatically generated API documentation

These benefits are in the case of Markdown files that are automatically generated at build time. For the many ways I can think of that could be implemented to describe additional pages to be created containing automatically generated content, I like what you are proposing (although I am struggling a bit with your specific example to understand exactly the expected output).

PS: Thanks for the link, I wasn't aware. Reading the configuration from pyproject.toml was a community feature request and something I never really used personally because of how cumbersome the long table names are. Unless the JSON-like syntax (or something similarly convenient) is going to be added to the language, I don't think I'll start writing highly nested configurations in TOML anytime soon. :^)

[mcarans]

The script = """...""" approach is essentially the one taken by the author(s) of tox and hence I think a reasonable solution that minimises the effort in supporting two sources of configuration: docs/build.novella or pyproject.toml.

[tool.tox]
legacy_tox_ini = """
[tox]
isolated_build = true
envlist = py310, lint
...

I think the approach you have taken with novella will allow much more fine grained control of the output than possible now. Combined with a means to generate new pages from within a markdown file, it should cover all bases.

The example I gave which I'll refer to later was:

...The static metadata for ACLED is held in human readable files so 
if it needs to be modified, it is straightforward. This is another feature 
of the HDX Python library that makes putting data programmatically 
into HDX a breeze.

@pydoc :start_section API Documentation
@pydoc :apigen ["hdx.api.configuration.*"] :title Configuration
@pydoc :apigen ["hdx.api.locations.*"] :title Locations
...
@pydoc :end_section API Documentation

For the above main.md file, what is generated would be a Home page with the content above the @pydoc tags. That page would have a menu on the left with headings. At the bottom of that menu would be "API Documentation" with "Configuration" and "Locations" underneath. Clicking "Configuration" in the menu would take you to a new page with the API documentation generated from hdx.api.configuration.*.

In other words, the output would be similar to the output from the 4.5.1 version of pydoc-markdown which you can see here, where instead of having to write a separate markdown file for each menu item under "API Documentation" on the left (which would be 11 tiny markdown files), there would just be a section of pydoc tags at the bottom of the main.md that generate those pages.

If I later decided that I need more control over the API documentation, then I could remove those tags at the bottom of main.md and instead write 11 individual markdown pages (but I can't see myself needing that level of control at the moment).

[NiklasRosenstein]

I've added a feature to read from pyproject.toml in NiklasRosenstein/novella@61f20a3 (needs to be specified explicitly via the -c,--config-file option), available in Novella 0.2.0.

I'll think a bit more about the tags to generate new files!

[mcarans]

Hi @NiklasRosenstein . I'm just wondering if you've had any more thoughts on the above - i.e., how to develop Novella for very simple use cases like one main markdown page + API documentation which the legacy approach made concise and easy.

[mcarans]

Closing since the YAML/TOML setup is no longer deprecated