mip-installable version of microdot

[jimmo]

Thanks @miguelgrinberg for this fantastic library. I have lost count of how many times I have recommended it in the MicroPython forums and Discord!

The MicroPython project has recently implemented a long-standing TODO which is to upgrade our package manager and downloadable package index (formerly upip, now mip). See https://docs.micropython.org/en/latest/reference/packages.html

It would be great to be able to say to people "just run >>> mip.install("microdot") (or more usefully, per-implementation targets, e.g. "microdot-async", "microdot-thread" etc). And similarly mpremote mip install microdot for installation from a PC.

In the short term, it would be straightforward to take advantage of the "install-from-github" functionality of mip. See https://github.com/jimmo/micropython-mlx90640 for an example. You could provide a .json file for each of the implementations. The disadvantage of this approach is that the files would not be bytecode compiled, but I am happy to send a PR for this if you're interested.

I suspect that moving microdot into micropython-lib probably isn't what you want, but just in case that does work for you, then that would be an easy solution to just getting it into the index and providing bytecode-compiled versions. This would also immediately enable it to work via require() in a frozen manifest.

We can also investigate a way of adding third-party projects into the index... I'm not super keen on setting a precedent of submodules, but maybe we can come up with something. I'm not sure mirroring the project there is a good idea, but would also work. Maybe we can have a way to have a microdot package that just contains a manifest.py that forwards to here (and make require() able to handle this).

[dpgeorge]

Maybe we can have a way to have a microdot package that just contains a manifest.py that forwards to here (and make require() able to handle this).

I like this idea. Kind of like a symlink.

[jimmo]

Maybe we can have a way to have a microdot package that just contains a manifest.py that forwards to here (and make require() able to handle this).

I like this idea. Kind of like a symlink.

FWIW, this is exactly what this TODO is about. https://github.com/micropython/micropython/blob/master/tools/manifestfile.py#L61

So question for @miguelgrinberg -- would you be willing to add some manifest.py files to this repo if I sent a PR for it? This would allow people right now (unrelated to mip) to depend on microdot from their frozen manifest via include("path/to/microdot/manifest-async.py") (assuming they have microdot cloned somewhere, perhaps as a submodule of their own project's repo).

And then, soon, we can forward to them from micropython-lib, allowing both mip.install("microdot-async") and require("microdot-async") to work.

FYI also, this might avoid the need to have the vendored copies of the micropython-lib packages in this repo. (the manifest.py would be able to require() them directly from micropython-lib).

All of this is pretty new and there are lots of details here. If it's easier to explain over chat happy to talk in Discord/video/etc.

[miguelgrinberg]

I will look into this in the coming days, but if I have to be honest, the idea of using micropython-lib as the package repository does not really feel like a great way to foster an ecosystem of third-party packages, but maybe I need to play with this some to start seeing the benefits of doing this.

[miguelgrinberg]

@jimmo It seems right now the best way to use mip would be through the github links:

>>> mip.install('github:miguelgrinberg/microdot/src/microdot.py')
Downloading github:miguelgrinberg/microdot/src/microdot.py to /root/.micropython/lib
Copying: /root/.micropython/lib/microdot.py
Done
>>> mip.install('github:miguelgrinberg/microdot/src/microdot_asyncio.py')
Downloading github:miguelgrinberg/microdot/src/microdot_asyncio.py to /root/.micropython/lib
Copying: /root/.micropython/lib/microdot_asyncio.py
Done

Adding json files does not really seem like it provides much of a benefit, since the files will still be installed in source form. So I'm thinking I'm not going to add json files, and instead will show how to install as above, since the json path seems more of a workaround than a long term solution.

I think there should be a way for me to use a tool that submits versions of my package to https://micropython.org/pi/v2, so that then they can be downloaded in compiled form (something like a twine or similar). Until then, having the manifest files might help some people who are brave enough to compile their firmwares, but considering that neither microdot.py or microdot_asyncio.py have any dependencies and by large these are the only two files that most people use, I'm not really sure there is a huge benefit in being an early adopter. I'm happy to revisit this in the future if manifest files allow me to submit the compiled files to the package index though.

[miguelgrinberg]

Maybe we can have a way to have a microdot package that just contains a manifest.py that forwards to here (and make require() able to handle this).

I like this idea. Kind of like a symlink.

This does not look like a good solution to me. With this I would be losing the ability to publish updates to my package whenever I need/want to. Instead you would have control to publish updates to my package through the placeholder package you create in micropython-lib.

I think now that you have created your own package index you will have to provide developers a tool or API to submit that is independent from micropython-lib. I really don't see any other viable way.

[mirko]

Even if not going the full mip-road, I'd really welcome a respective manifest.py file in order to be able to bake it into micropython images as frozen module.

[miguelgrinberg]

@mirko this could my own misunderstanding, but I thought a custom manifest file applies to a custom firmware built, not to each dependency you want to freeze in that build.

For example, if you wanted to include microdot in your custom firmware, along with its asyncio version, you would have a custom manifest.py such as:

include("$(BOARD_DIR)/manifest.py")

# dependencies
module("microdot.py")
module("microdot_asyncio.py")

If I include a manifest.py file with this project, you will still need to create your own master manifest file and include mine, so this does not seem to be a great solution if you consider that there are many ways to freeze microdot (with or without asyncio, with or without websocket, with or without sessions, etc.)

[mirko]

As you pointed out, I can already just freeze respective files into my project, like:

freeze("$(MPY_DIR)/lib/microdot/src", "microdot.py")
freeze("$(MPY_DIR)/lib/microdot/src", "microdot_asyncio.py")

However, code using that, would not be portable, as I'd have to directly refer to the files rather than a module:
import microdot_asyncio vs from microdot import microdot_asyncio.
It is fair practice, that projects provide their own manifest.py-file, in order to be referenced by the port's/project's upper manifest.py-file. See usyncio as example, which by the port's/project's manifest-file is referenced like:
https://github.com/micropython/micropython/blob/master/ports/esp32/boards/manifest.py#L2
where uasyncio has its own manifest.py file describing how the module is supposed to be structured and used:
https://github.com/micropython/micropython/blob/master/extmod/uasyncio/manifest.py.

So microdot providing a manifest file would make microdot being usable as a module in a consistent way, out of the box, ensuring a certain structure on how to import and use it.

[miguelgrinberg]

@mirko you are confusing two things that are independent. First of all let me clarify two definitions, to make sure you don't misunderstand me. In Python, a "module" is a single file with code in it, while a "package" is a directory with modules inside.

You are saying that you'd like to have the microdot modules included in a package, which is a valid request. Microdot was initially one module, so it did not seem necessary to use a package. Then I added the asyncio version and it become two modules, and from there it took off with a few more modules. At this point I do agree that having a package called microdot with all the modules inside would be better, but this is a breaking change, so I have to be careful about implementing it and documenting it.

Now, this has nothing to do with a manifest file which is not where the structure of the Python package is defined. After all, Microdot runs also on standard Python, so a manifest file cannot be how this is done. What needs to happen is that the files need to be reorganized as a package in this repository, and the documentation needs to be updated to show the new imports, which will all start with from microdot import .... Once this is done, a manifest file such as the following can be used to freeze the entire package:

include("$(BOARD_DIR)/manifest.py")
package("microdot")

And now we are back to the beginning. I can include a manifest file in this repository for convenience (and I think I should), but this isn't really saving you any work, as you will still need to maintain your own master manifest and import mine, so instead of having this line in your manifest:

package("microdot")

you'll have this other line:

include("/path/to/microdot/manifest.py")

So you'll still have a line in your manifest.

[mirko]

Thanks for clarifying on the package / module confusion - my bad.
The advantage of a provided manifest I see is:

• I just have to to include this file without first having to look at the structure and how to best/correctly freeze it in
• Everybody is encouraged to use it this very same way

[miguelgrinberg]

And the disadvantage is that you lose the ability to freeze only the modules of microdot that you need. For standard Python this isn't a problem. In MicroPython, where resources such as disk space are scarce this can be a problem.

But in any case I'm not disagreeing, I can still provide a manifest file and anybody who wants a custom freezing can just ignore it.

[jimmo]

@mirko Adding some extra notes from the conversation in Discord:

It doesn't really matter if microdot provides a manifest.py. It certainly makes it easier because then your board manifest just has to do a single include().

But there are many different configurations of microdot, so there isn't really a one-size-fits-all way to do this. We do technically support passing options to include() but I don't think this is the right approach here.

Anyway, the "recommended" way for now is as @miguelgrinberg points out -- your board manifest should just include the modules from microdot that you need. This is what the module() function is for. e.g.

MICRODOT_PATH = "path/to/microdot/repo/src"
module("microdot.py", base_path=MICRODOT_PATH)
module("microdot_asyncio.py", base_path=MICRODOT_PATH)

(We have also struggled with the terminology here... module vs package vs package-the-other-kind, etc... I tried to write a summary here: https://github.com/micropython/micropython-lib#notes-on-terminology )

[miguelgrinberg]

We do technically support passing options to include() but I don't think this is the right approach here.

@jimmo I was thinking that maybe using environment variables to control what is defined in the manifest would work. Like for example, if MICRODOT_ASYNCIO is defined, then the asyncio module is included, else it is skipped. This seems like an appropriate pattern when building the firmware, since many other things are also configured from the environment.