Home Assistant MQTT Auto Discovery

[Elliot-Ford]

Proposal

Home Assistant's MQTT listener can auto create devices and sensors for topics published on MQTT by having a device publish the appropriate JSON(s). I've hacked this a bit on my camera to create a camera instance on my home assistant. By publishing { "name": "AllSky Camera", "unique_id": "allskycam", "topic": "indi-allsky/latest" } to the topic /homeassistant/camera/AllSkyCam/config. However this is pretty hacky and I'd like to improve on it and define the other fields as sensors in HA.

Resources

https://www.home-assistant.io/integrations/mqtt/#mqtt-discovery

[Elliot-Ford]

I also wouldn't mind being assigned to this as it's something I've been actively tooling around with

[aaronwmorris]

Sounds interesting. I have started setting up my own home sensor network, but I have not made it to Home Assistant yet. I am interested to see what you come up with.

[vastmeridian]

I've managed to set up MQTT dashboard on Android and I will be integrating this into my edition of Home Assistant. Happy to help with testing, if this is of use.

[Elliot-Ford]

Thanks for the offer, I'll be sure to update here when I've got something worthy of testing with. Been slow going as I had to redesign my camera weatherproofing after a seriously heavy rain storm took it out last month. Finally got it back up and running so hopefully I'll have proof of concept soonish.

[WarrenSchultz]

Add me as another who'd be willing to help with this. I just started tinkering around with how to make it happen, then realized I should check the issues list. :D

[maphilli14]

I too am keen on this one! indi-allsky is publishing fine to mosquito mqtt but doesn't get 'discovered' in a working mqtt to homeassistnat integration. Anything you can advise?

[maphilli14]

Could be this? https://www.home-assistant.io/integrations/mqtt/#mqtt-discovery

[aaronwmorris]

That link is very helpful.

Edit: I also see the same link was in the original post, too.

[aaronwmorris]

Merged #1096 with a script to deploy the auto discovery topics for Home Assistant.

This is my first pass at this. I have done basic validation that this works in a VM lab, but I do not currently have any of my systems online to test some of the SBC temperature data.

I am not comfortable with putting the auto discovery stuff in the core code, so it is just a separate script for now. The script only takes a few seconds to run, and you should be all set. You should only need to run this once.

source virtualenv/indi-allsky/bin/activate

./misc/home_assistant_auto_discovery.py

[vastmeridian]

I'll try and give that a test run, Aaron.
I've happily added a couple of MQTT entries to my HA instance by editing the configuration.yaml directly and these work fine. In these dark and cloudy days I'm planning on (ab)using your Sky Quality readings to decide if the light level outside is too low and turning on internal lights automatically

It would, however, be good to have the values autodiscovered.

[vastmeridian]

Hmm.

Had this error cropping up when running setup.sh after updating from git:

**** Backup DB prior to migration ****
Error: While importing 'app', an ImportError was raised:

After the traceback I got this:
Usage: flask [OPTIONS] COMMAND [ARGS]...
Try 'flask --help' for help.

Error: No such command 'db'.

[aaronwmorris]

This has been happening to a few people. Try upgrading the following module:

source virtualenv/indi-allsky/bin/activate

pip install flask-migrate --upgrade

If that does not fix it, just upgrade all of your modules.

source virtualenv/indi-allsky/bin/activate

pip install -r requirements/requirements_latest.txt --upgrade

[vastmeridian]

The first suggestion acheived the following:

Successfully uninstalled Flask-Migrate-4.0.4
Successfully installed flask-migrate-4.0.5

but didn't fix it. The second option upgraded to

Flask-SQLAlchemy-3.1.1 
Flask-WTF-1.2.1 
Pillow-10.2.0 
astroalign-2.5.1 
astropy-6.0.0 
stropy-iers-data-0.2024.1.15.0.30.14 
bcrypt-4.1.2 
ccdproc-2.4.1 
cython-3.0.8 
ephem-4.1.5 
imageio-2.33.1 
imageio-ffmpeg-0.4.9 
lazy_loader-0.3 
lxml-5.1.0 
numpy-1.26.3 
paramiko-3.4.0 
prettytable-3.9.0 
psutil-5.9.7 
rawpy-0.19.0 
scikit-image-0.22.0 
setuptools-rust-1.8.1 
shapely-2.0.2 
sqlalchemy-2.0.25 
tomli-2.0.1 
typing-extensions-4.9.0

and this did the trick. I hope the list of updated modules helps.

[aaronwmorris]

I am starting to suspect it is not any specific module causing this, just the action of upgrading of some of these modules fixes the problem.

[maphilli14]

I think most of my systems are up to date but the topics only show in mosquito and not in homeassitant.... Script ran fine!

[aaronwmorris]

Find the MQTT integration and click on the vertical ellipsis ⋮ and try clicking Reload. You may have to run the script again. The elements would automatically populate on the Overview in my lab environments.

[maphilli14]

I tried several reloads and script reruns, what HomeAssistant and MQTT versions are you running? It shows up in MQTT, just not seen in HA

[aaronwmorris]

I have tested the latest version from docker.
Core 2024.1.2
Frontend 20240104.0

Can you verify your base discovery topic is homeassistant?

[maphilli14]

Was

Didn't work

Changed to...

no change, do I need to start listening?

[maphilli14]

Upgraded docker image to latest instead of stable, reran and it shows now...

TYVM!

[maphilli14]

square: false
type: grid
cards:
  - type: entity
    entity: sensor.stars
    icon: mdi:creation
  - type: entity
    entity: sensor.sqm
    icon: mdi:chart-histogram
  - type: entity
    entity: sensor.smoke_rating
    icon: mdi:smoke
  - type: entity
    entity: sensor.moon_phase_2
    icon: mdi:moon-waxing-crescent
  - show_state: true
    show_name: true
    camera_view: auto
    type: picture-entity
    entity: image.indi_allsky_camera
    image: https://demo.home-assistant.io/stub_config/bedroom.png
columns: 1

[maphilli14]

Any ideas on doing an animation other than publishing to external site?
Looking great otherwise, now I can get an alert if the stars drop to 0 or less than normal etc... w00t!

[maphilli14]

Other than Grafana I'd love to see sqm on a chart but it might need stat tracking - https://www.home-assistant.io/more-info/statistics/

[aaronwmorris]

The SQM value is published to MQTT and should be a data point in home assistant. Once its there, it does graph the value. You may want to re-run the auto discovery script again.

By the way, you can add the indi-allsky data to Grafana. There is a JSON file generated in /var/lib/indi-allsky/indi_allsky_status.json for every image. This data can be imported via telegraf into influxdb. There is a telegraf config example in examples/telegraf/ along with a Grafana dashboard template.

[vastmeridian]

I already track the SQM on my HA instance. The sensor definition is fairly simple (but I do scale it to something more manageable): It graphs readily with just the standard history graph.

mqtt:
  sensor:
    - state_topic: "indi-allsky/sqm"
      value_template: "{{ (value | float / 100000) | round(2) }}"
      unit_of_measurement: "%"
      name: "Sky_Quality"

And in the dashboard this is what I get:

I wanted the value to determine if it was getting 'dark' and to use this to turn lights on in the house.

[vastmeridian]

I've not tried playing with the auto-discovery yet. That's for when I get a moment to myself (whoever said that you're not busy when you retire got things very, very wrong ... )

[maphilli14]

I tried what @vastmeridian suggested and it messed up the rest of my MQTT integrations and now I can't get the image to re'render! :(

[aaronwmorris]

I may need to integrate the auto-discovery publishing into the main program so that it is registered every time indi-allsky starts.

[maphilli14]

I tried to rerun and one every few minutes it will flash a valid photo then back to broken jpg status.

[vastmeridian]

I tried what @vastmeridian suggested and it messed up the rest of my MQTT integrations and now I can't get the image to re'render! :(

Sorry about that. I don't pull the image into HA. I only use the text/numerical entities.

Have you tried image MQTT?
https://www.home-assistant.io/integrations/image.mqtt/

[vastmeridian]

Just added this to my configuration.yaml:

mqtt:
  image:
    - image_topic: "indi-allsky/latest"
      name: "AllSkyLatest"
      content_type: "image/jpg"

and this is the result:

[aaronwmorris]

Nevermind what I said before about publishing. My mosquitto docker container did not have persistence configured. I have corrected my mosquitto docker config (rebuild your container if you are using this). #1104

The picture of the data above is a little different from what I see in my lab instance of home assistant. The image and data values are all automatically populated on the overview page. There are no gaps between the data values like in your screenshot.

You should not have to manually configure anything other than the MQTT integration for all of these values to automatically populate.

[vastmeridian]

You should not have to manually configure anything other than the MQTT integration for all of these values to automatically populate.

Presumably you're setting indi-allsky to have the homeassistant/ base topic so it can be autodiscovered?

[aaronwmorris]

Not exactly. The way the auto-discovery works is the ./misc/home_assistant_auto_discovery.py script creates the discovery topics under homeassistant/ for all of the data values. The auto-discovery topics have a reference to the indi-allsky data topics under indi-allsky/ (or whatever you have configured in indi-allsky).

[maphilli14]

I do like the idea of the stat graph for trends and history! I'm doing the manaul thing for now until we can get the stats saved autmatcailly and it's nice, but I need help with stars as a string or qty?

```

mqtt:
sensor:
- state_topic: "indi-allsky/sqm"
value_template: "{{ (value | float / 100000) | round(2) }}"
unit_of_measurement: "%"
name: "Sky_Quality"
- state_topic: "indi-allsky/stars"
value_template: "{{ (value | float }}"
name: "Star_Count"
image:
- image_topic: "indi-allsky/latest"
name: "AllSkyLatest"
content_type: "image/jpg"

[vastmeridian]

Not exactly. The way the auto-discovery works is the ./misc/home_assistant_auto_discovery.py script creates the discovery topics under homeassistant/ for all of the data values. The auto-discovery topics have a reference to the indi-allsky data topics under indi-allsky/ (or whatever you have configured in indi-allsky).

Ah. OK. I'll try that, but as I said above, time is short at the moment. For what I need the manual edit is just fine (as I have other entities from different systems defined similarly, so incorporating autodiscover would need a systematic edit of a number of platforms).

[vastmeridian]

I do like the idea of the stat graph for trends and history! I'm doing the manaul thing for now until we can get the stats saved autmatcailly and it's nice, but I need help with stars as a string or qty?

This should help with the definitions:
https://github.com/aaronwmorris/indi-allsky/wiki/MQTT-Broker-Publishing

[aaronwmorris]

Ah. OK. I'll try that, but as I said above, time is short at the moment. For what I need the manual edit is just fine (as I have other entities from different systems defined similarly, so incorporating autodiscover would need a systematic edit of a number of platforms).

If you use a different autodiscovery topic in Home Assistant, you can change that in the ./misc/home_assistant_auto_discovery.py script (in a single place) before you run it.

class HADiscovery(object):


    discovery_base_topic = 'homeassistant'
    unique_id_base = '001'

[maphilli14]

I'm still confused on the sensor config for the stat graph?! https://developers.home-assistant.io/docs/core/entity/sensor/#long-term-statistics

All of this works however, no long term istiry is avail in the drop down for statistics graph

mqtt:
  sensor:
    - state_topic: "indi-allsky/sqm"
      value_template: "{{ (value | float / 100000) | round(2) }}"
      unit_of_measurement: "%"
      name: "Sky_Quality"
    - state_topic: "indi-allsky/stars"
      name: "allskystars"
    - state_topic: "indi-allsky/smoke_rating"
      name: "allskysmoke_rating"
    - state_topic: "indi-allsky/moonphase"
      name: "allskymoonphase"
  image:
    - image_topic: "indi-allsky/latest"
      name: "AllSkyLatest"
      content_type: "image/jpg"

EDIT:

mqtt:
  sensor:
    - state_topic: "indi-allsky/sqm"
      value_template: "{{ (value | float / 100000) | round(2) }}"
      unit_of_measurement: "%"
      state_class: "measurement"
      name: "Sky_Quality"
    - state_topic: "indi-allsky/stars"
      value_template: "{{ (value | float ) }}"
      name: "allskystars"
      state_class: "measurement"
    - state_topic: "indi-allsky/smoke_rating"
      name: "allskysmoke_rating"
    - state_topic: "indi-allsky/moonphase"
      name: "allskymoonphase"
  image:
    - image_topic: "indi-allsky/latest"
      name: "AllSkyLatest"
      content_type: "image/jpg"

Nets me a blank graph via this yaml

[vastmeridian]

I'm still confused on the sensor config for the stat graph?! https://developers.home-assistant.io/docs/core/entity/sensor/#long-term-statistics

I've only just started playing with statistics so I'm equally puzzled. I'll play a bit with the definitions and see how I get on.

Building this into the autodiscovered entities may prove more of a challenge.

[vastmeridian]

Success, I think...

Code (from Lovelace UI)

type: horizontal-stack
cards:
  - show_state: true
    show_name: true
    camera_view: auto
    type: picture-entity
    entity: image.allskylatest
    image: https://demo.home-assistant.io/stub_config/bedroom.png
  - type: vertical-stack
    cards:
      - chart_type: line
        period: 5minute
        type: statistics-graph
        entities:
          - sensor.starcount
        stat_types:
          - mean
          - min
          - max
        title: Star Count
      - chart_type: line
        period: 5minute
        type: statistics-graph
        entities:
          - sensor.sky_quality
        stat_types:
          - mean
          - min
          - max
        title: Sky Quality

You might want to play with the period: setting if you're running on a Pi with an SD card for storage

[maphilli14]

Success, I think...

Code (from Lovelace UI)

type: horizontal-stack
cards:
  - show_state: true
    show_name: true
    camera_view: auto
    type: picture-entity
    entity: image.allskylatest
    image: https://demo.home-assistant.io/stub_config/bedroom.png
  - type: vertical-stack
    cards:
      - chart_type: line
        period: 5minute
        type: statistics-graph
        entities:
          - sensor.starcount
        stat_types:
          - mean
          - min
          - max
        title: Star Count
      - chart_type: line
        period: 5minute
        type: statistics-graph
        entities:
          - sensor.sky_quality
        stat_types:
          - mean
          - min
          - max
        title: Sky Quality

You might want to play with the period: setting if you're running on a Pi with an SD card for storage

This is the stuff, perfect, thanks!!!

[ispybadguys]

How do I run the script? I get this:
kurt@indi-allsky:~/indi-allsky $ ./misc/home_assistant_auto_discovery.py
Traceback (most recent call last):
File "/home/kurt/indi-allsky/./misc/home_assistant_auto_discovery.py", line 11, in
import paho.mqtt.publish as publish
ModuleNotFoundError: No module named 'paho'

[aaronwmorris]

@ispybadguys you just need to activate the virtualenv before running the script.

source virtualenv/indi-allsky/bin/activate

./misc/home_assistant_auto_discovery.py