mkdocs serve does not watch for changes in mkdocstrings

[cs01]

Usually when I view mkdocs-generated documenation during development, the server will watch for changes and the page will auto-reload when changes are detected. Using mkdocstrings, I have not seen this happening, and to view changes I have to kill the mkdocs serve process and restart it.

[pawamoy]

Yes I noticed it as well while implementing the watch feature. It's not a trivial thing to do. The way I implemented it is a complete hack, see the code at https://github.com/pawamoy/mkdocstrings/blob/master/src/mkdocstrings/plugin.py#L74-L90

When the plugin is instantiated, we store the set of modules currently loaded in a private var, and when a file changes in the watched directories, we get the diff of modules (current ones minus stored ones) and unload them. This was the only way I got it working, but only for other directories than mkdocstrings itself.

I can think of other solutions but they are not trivial either. One of them is using a subprocess to import the Python objects and prebuild their documentation, return it as JSON to be able to pick it up in the parent process. This way we don't interfere with the current Python env and we do not need the black magic unloading trickery at all.

Another solution which would completely change how mkdocstrings work would be to parse the code (just like what we do for attributes and their docstrings) instead of importing objects. But retrieving info with ast is kinda hard, and other libs like lib2to3 have other quirks.

[jaimergp]

Hey, I noticed one bug due to an obscure Numpy bug.

If numpy is being imported by any of the modules being documented (or any of their dependencies), it will throw a RuntimeError exception:

[E 200304 13:33:06 ioloop:909] Exception in callback <bound method LiveReloadHandler.poll_tasks of <class 'livereload.handlers.LiveReloadHandler'>>
    Traceback (most recent call last):
      File "/home/jaime/.conda/envs/kinoml/lib/python3.7/site-packages/tornado/ioloop.py", line 907, in _run
        return self.callback()
      File "/home/jaime/.conda/envs/kinoml/lib/python3.7/site-packages/livereload/handlers.py", line 69, in poll_tasks
        filepath, delay = cls.watcher.examine()
      File "/home/jaime/.conda/envs/kinoml/lib/python3.7/site-packages/livereload/watcher.py", line 105, in examine
        func()
      File "/home/jaime/.conda/envs/kinoml/lib/python3.7/site-packages/mkdocstrings/plugin.py", line 85, in unload_and_rebuild
        builder()
      File "/home/jaime/.conda/envs/kinoml/lib/python3.7/site-packages/mkdocs/commands/serve.py", line 114, in builder
        build(config, live_server=live_server, dirty=dirty)
      File "/home/jaime/.conda/envs/kinoml/lib/python3.7/site-packages/mkdocs/commands/build.py", line 270, in build
        nav = config['plugins'].run_event('nav', nav, config=config, files=files)
      File "/home/jaime/.conda/envs/kinoml/lib/python3.7/site-packages/mkdocs/plugins.py", line 94, in run_event
        result = method(item, **kwargs)
      File "/home/jaime/.conda/envs/kinoml/lib/python3.7/site-packages/mkdocstrings/plugin.py", line 110, in on_nav
        root_object = self.documenter.get_object_documentation(import_string)
      File "/home/jaime/.conda/envs/kinoml/lib/python3.7/site-packages/mkdocstrings/documenter.py", line 310, in get_object_documentation
        module, obj = import_object(import_string)
      File "/home/jaime/.conda/envs/kinoml/lib/python3.7/site-packages/mkdocstrings/documenter.py", line 585, in import_object
        parent_module = importlib.import_module(parent_module_path)
      File "/home/jaime/.conda/envs/kinoml/lib/python3.7/importlib/__init__.py", line 127, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
      File "<frozen importlib._bootstrap>", line 983, in _find_and_load
      File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 728, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/home/jaime/devel/py/openkinome/kinoml/kinoml/core/complex.py", line 2, in <module>
        from .ligand import Ligand
      File "/home/jaime/devel/py/openkinome/kinoml/kinoml/core/ligand.py", line 3, in <module>
        from openforcefield.topology import Molecule
      File "/home/jaime/.conda/envs/kinoml/lib/python3.7/site-packages/openforcefield/topology/__init__.py", line 1, in <module>
        from openforcefield.topology.molecule import (
      File "/home/jaime/.conda/envs/kinoml/lib/python3.7/site-packages/openforcefield/topology/molecule.py", line 36, in <module>
        import numpy as np
      File "/home/jaime/.conda/envs/kinoml/lib/python3.7/site-packages/numpy/__init__.py", line 142, in <module>
        from . import core
      File "/home/jaime/.conda/envs/kinoml/lib/python3.7/site-packages/numpy/core/__init__.py", line 17, in <module>
        from . import multiarray
      File "/home/jaime/.conda/envs/kinoml/lib/python3.7/site-packages/numpy/core/multiarray.py", line 14, in <module>
        from . import overrides
      File "/home/jaime/.conda/envs/kinoml/lib/python3.7/site-packages/numpy/core/overrides.py", line 47, in <module>
        """)
    RuntimeError: implement_array_function method already has a docstring

I think this is due to the fact you reload all modules upon save to obtain the potentially new changes. Is there any way to whitelist numpy from this process?

[jaimergp]

Hm, I patched this in and seems to work. plugin.py at line 74:

    def on_serve(self, server, config, **kwargs):
        builder = list(server.watcher._tasks.values())[0]["func"]

        def unload_and_rebuild():
            diff = set(sys.modules.keys()) - self._sys
            log.info(
                f"mkdocstrings: Unloading modules loaded after mkdocstrings plugin was instantiated ({len(diff)} modules)"
            )
            for module in diff:
                if "numpy" in module:  # <<<<<<<<< add this
                    continue  # <<<<<<<<< add this
                del sys.modules[module]
            self.clear()
            builder()

        for element in self.config["watch"]:
            log.info(f"mkdocstrings: Adding directory '{element}' to watcher")
            server.watch(element, unload_and_rebuild)
        return server

[pawamoy]

Ultimately we won't need this module reloading hack (see #28 (comment)), but until then I can accept this patch if you want to send a PR 🙂

[jaimergp]

Submitted! :)

[pawamoy]

New architecture is almost ready. The data collection is done in a subprocess, with pytkdocs, and so this issue should be resolved, therefore I'm closing it. We'll open a new one if similar problem appear.