Managing persistent state and configuration

[EternityForest]

One problem with modularity is you lose the nice integration where everything can be managed through one central Web UI.

I think a modular system has these desirable qualities:

• It should be possible for a driver to give the main server a definition for a vendor-specific UI
• It should be possible to make a fully standalone vendor specific server with it's own UI and independent operation, that can optionally integrate with the larger system
• If changes are made via the independent server, they should be reflected on the main server and vice versa
• There should never be more than a bare minimum of config file editing needed.
• Devices that have their own internal MQTT server should have the possibility to be entirely config free, just point the central server at them.

A solution to this is to specify a way to represent and manage configuration, allowing individual modules to have no persistent state at all.

The simplest and easiest way to do this would just be to define a JSON based file format that encapsulated the entire state of a node as one file. The problem here is that if an integration. manages thousands of devices, it will mean gigabytes needed to add devices one a time, and it would be almost impossible to use large media files or anything of the sort in the state.

For simplicity those cases could be handled by further extensions.

We would also define a config topic. Whenever either side of a connection updates state, it posts it to the MQTT broker. If we get a statefile that is different from what we have, we update our own internal state and can then save it to a file.

This should be entirely sufficient for something like a smart bulb that has its own MQTT and MDNS.

Standardizing UI around it is harder, but still important. There's not many truly cross platform and language UI descriptions. The two I know of are JSON schema and HTML.

In a truly modular system, we do not want to reinvent UI all the time for every automation server. We also don't want to force one particular UI style. But we also don't want to separately manage authentication on everything.

One option is that we totally leave config UI up to individual modules. We only specify stste sync and storage.

Another rather hacky option that could be excellent from a UI perspective, is that we let the device send the host an HTML form template that the host then renders, which returns any POST data back to the device. This is basically what I'm currently doing for some internal tools, just in a lot less modular of a way. You could also give the form Websocket access to a single MQTT shared topic broadcast topic to be used for realtime stuff.

In this way you could implement anything you want, but the host/primary server could still apply it's own accees controls. It would also free up the device from having to understand HTTP in addition to MQTT, and allow the host to apply it's own theming and extra boilerplate to the management page.