Integration for ADE7880 (3-phase power monitor, used in Shelly 3EM)

[michaelpiron]

Describe the problem you have/What new integration you would like

I have a Shelly 3PM 3-phase energy monitor: https://shelly.cloud/products/shelly-3em-smart-home-automation-energy-meter/
Inside the product is an ESP8266 SoC and a ADE7880 power monitoring chip.
I’d like to put ESPhome firmware on my Shelly 3PM for two reasons:

• I like the ESPhome/HA ecosystem and the possibility to make tailer-made firmware
• The Shelly standard firmware doesn’t accomodate my use case: the Shelly firmware assumes that the user has a 3x400V grid connection (star-connection, with neutral), whereas my home grid connection is 3x230V (delta-connection, without neutral). The ADE7800 supports both use cases by changing a specific register of the ADe7880 (which the Shelly firmware doesn’t support).

Therefore I see an added value in having an ADE7880 component in ESPhome, next to the existing ADE7953 (used in the single-phase Shelly PM) -> https://esphome.io/components/sensor/ade7953.html

Please describe your use case for this integration and alternatives you've tried:

use case: being able to do 3-phase energy monitoring

Additional context

I started developing a ADE7880 component, based on the existing ADE7953 component and taking into account the ADE7880 datasheet specifications. The majority of the work is done and is available in a fork of the esphome repository:
https://github.com/michaelpiron/esphome/tree/dev/esphome/components/ade7880
However, some outstanding issues remain at this moment, for which I would need some help:

• Which are the connections between ESP and ADE chip in the Shelly 3PM product? I have to know which GPIO pins of the ESP are connected to the IRQ’s of the ADE. I can’t see this visually on the PCBs without desoldering the top and bottom PCB, and even then I’m not sure if it can be deduced from the PCB tracks.
• There are quite some calibration constants to figure out. Anyone who is familiar with this or who could reverse engineer it from the Shelly 3PM firmware binary?

furthermore: in the Tasmota Github community, people also raised the idea of having a Tasmota component for the ADE7880: arendst/Tasmota#13515
So there are also other people interested in having an ADE7880 integration.

[michaelpiron]

Setting the GPIO linked to the ADE’s IRQ in a correct way is important, as wrong configuration can lead to heating of the device and potential fire hazard! Refering to this related issue: arendst/Tasmota#7991 (comment)

[michaelpiron]

A Tasmota component for the Shelly 3EM is being built as people started contributing photos of the PCB's and investigated code dumps. All information is in this discussion: arendst/Tasmota#13515.

As I'm not that proficient with coding in Python and C (yet), would someone be willing to complete my draft code for an ADE7880 component in ESPHome?
Link to my fork: https://github.com/michaelpiron/esphome/tree/dev/esphome/components/ade7880

[kroimon]

Hi Michael!
Did you ever get to the finishing touches of your component? I would love to run ESPhome on my Shelly 3EM :⁠-⁠)
If not, I could probably pick up where you left. What were your plans with the component going forward?

[michaelpiron]

Hi @kroimon , I didn’t progress further on this, unfortunately. My fork of esphome is still the most recent one. In the meantime, I see quite some activity in the Tasmota discussion (see link in my previous post) with interesting info on how to download the device calibration parameters from the original firmware, before putting new firmware on it.
If you can help out, that would be very welcome. I won’t have time in the next months, due to other priorities.
Feel free to take the code from my fork. If you can get it into the esphome codebase, mentioning my name as co-author would be kindly appreciated.

[majuss]

It's standard and known for a long time, if you're not downloading the calibration data for your 3EM and flash it, then it's worthless electronic waste.

See: https://templates.blakadder.com/shelly_3EM.html

[stich86]

@michaelpiron what's the current status on support 3-phases system without neutral?
Shelly OS doesn't support this type of installation...

thx!

[kpfleming]

Since I have four 3EMs and a strong desire to run ESPHome on them, I've begun working on this component. Hopefully I'll get some early progress posted in the next day or two.

[kpfleming]

Development is happening here.

Current status:

• Config schema and code generation built and tested.
• Chip reset and IRQ handling built and tested.

Tasks left (at least):

• Chip register initialization
• Sensor value publication
• Calibration data in config schema and applied to chip
• Documentation of component
• Documentation of fully-worked example for Shelly 3EM device

[kpfleming]

Major progress... I now have most of the sensors reporting as planned, and the numbers match very closely to the Shelly firmware. Tomorrow I'll start working on docs!

The image below has the top three channels from a 3EM running Shelly firmware, and the bottom three channels from a 3EM running ESPHome, with the current clamps on the same wires.

[kpfleming]

The PRs to merge this into ESPHome are up for review now.

If anyone would like to help test this before it is merged, the docs are here. You can add the component to your configuration like this:

external_components:
  - source: github://kpfleming/esphome/ade7880
  - components: [ ade7880 ]

I'll be opening a PR later today for the esphome-devices repo with a full configuration example for the Shelly 3EM.

[kpfleming]

Example config for Shelly 3EM.

[dinamitemic]

The PRs to merge this into ESPHome are up for review now.

If anyone would like to help test this before it is merged, the docs are here. You can add the component to your configuration like this:

external_components:
  - source: github://kpfleming/esphome/ade7880
  - components: [ ade7880 ]

I'll be opening a PR later today for the esphome-devices repo with a full configuration example for the Shelly 3EM.

Hi guys, my Shelly 3EM is on its way so I started to prepare everything for the firmware conversion.
First of all thank you for the custom component.
I tried to follow your infos for the manual integration of the external component but I had to make some adjustements.
Using ESPHome 2023.7.1

• At the beginning of the firmware file put:

external_components:
  - source:
      type: git
      url: https://github.com/kpfleming/esphome
      ref: ade7880
    components: [ ade7880 ]
    #refresh: never

• Try to compile once and wait for the custom components to be downloaded.
  It will fail with errors finding some constants in const.py

• Manual edit the file sensor.py in

\<your_ha_ip>\config\esphome.esphome\external_components\c4a2a214\esphome\components\ade7880

(Please note that c4a2a214 can be different, be sure to edit the right one)
Re-adding the following lines:

CONF_PHASE_A = "phase_a"	
CONF_PHASE_B = "phase_b"	
CONF_PHASE_C = "phase_c"`

Put them just above CONF_NEUTRAL = "neutral"

• Go back in esphome device file and remove the comment on the line # refresh: never
• Click install button and now you should be able to compile

INFO ESPHome 2023.7.1
INFO Reading configuration /config/esphome/sh-3em-1.yaml...
INFO Generating C++ source...
INFO Compiling app...
Processing sh-3em-1 (board: esp01_1m; framework: arduino; platform: platformio/espressif8266@3.2.0)
--------------------------------------------------------------------------------
HARDWARE: ESP8266 80MHz, 80KB RAM, 1MB Flash
Dependency Graph
|-- ESPAsyncTCP-esphome @ 1.2.3
|-- ESPAsyncWebServer-esphome @ 2.1.0
|-- DNSServer @ 1.1.1
|-- ESP8266WiFi @ 1.0
|-- ESP8266mDNS @ 1.2
|-- noise-c @ 0.1.4
|-- Wire @ 1.0
Compiling /data/sh-3em-1/.pioenvs/sh-3em-1/src/main.cpp.o
Linking /data/sh-3em-1/.pioenvs/sh-3em-1/firmware.elf
RAM:   [====      ]  44.2% (used 36212 bytes from 81920 bytes)
Flash: [=====     ]  51.3% (used 525701 bytes from 1023984 bytes)
Building /data/sh-3em-1/.pioenvs/sh-3em-1/firmware.bin
esp8266_copy_factory_bin(["/data/sh-3em-1/.pioenvs/sh-3em-1/firmware.bin"], ["/data/sh-3em-1/.pioenvs/sh-3em-1/firmware.elf"])
========================= [SUCCESS] Took 8.18 seconds =========================
INFO Successfully compiled program.

[kpfleming]

• Try to compile once and wait for the custom components to be downloaded.
  It will fail with errors finding some constants in const.py

Ahh, you are correct, thanks for the details. This branch won't work completely standalone as an external component because it contains a patch to const.py. Your workaround is reasonable until the component has been merged.

[dinamitemic]

Ciao @kpfleming, I can confirm you that everything is working correctly.
I just want to report 2 infos:

1. Esphome logs are reporting a high execution time for the ade component:

[17:47:07][D][ade7880:159]: update took 180 ms
[17:47:07][W][component:204]: Component ade7880.sensor took a long time for an operation (0.18 s).
[17:47:07][W][component:205]: Components should block for at most 20-30ms.

2. In my calibration data I got "current_s": 266881794 that is strange and cannot be accepted since the maximum is 8388607.
   Have you any idea abou it?

[kpfleming]

1. Esphome logs are reporting a high execution time for the ade component

Which ESP board are you using, and how many of the ade7880 sensors do you have enabled?

2. In my calibration data I got "current_s": 266881794 that is strange and cannot be accepted since the maximum is 8388607.
Have you any idea abou it?

Unfortunately I do not; I had the same situation on my units but since I don't have clamps installed on the neutral conductors I didn't care about it. Someone will eventually need to talk to Shelly about those calibration constants to find out what the correct way to use current_n and current_s are.

[dinamitemic]

Which ESP board are you using, and how many of the ade7880 sensors do you have enabled?

It is a Shelly 3EM, I took your config example and just added some minor changes. If you want to check I can attach the full config.

Someone will eventually need to talk to Shelly about those calibration constants to find out what the correct way to use current_n and current_s are.

Hmm, interesting, I will try to find out more and eventually I'll write to the shelly team.

[kpfleming]

It is a Shelly 3EM, I took your config example and just added some minor changes.

Yes, of course... it was a silly question I asked :-)

Most likely the cause of those warnings is that the logger level is set to debug (either in your config, or because it is the default). When that is the case, the sensor core code emits a "Sending state..." message for every sensor value change, and it takes 3-6ms for that message to be emitted. As a result, if a single component has more than 7 or 8 sensors enabled, the total time required to update them all will exceed 50ms and the warning will be triggered.

You can ignore the warning, as there is no actual problem. You can also set the log level to info and the warnings will probably go away.

[dinamitemic]

Hi @kpfleming this morning I switched to Tasmota in order to make some comparison.
With ESPHome I noticed some wrong calculation on the energy side, let me add some details:

• The unit is reported to be Wh but the value are soooo small.
• It is not only a matter of a wrong unit (like a x1000 missing), it had a non correct correlation with the actual power usage.

Let me provide you an example with some numbers:
From 17:58 to 18:58 I had an average power consumption of around 240W, I would expect a computed energy of 240Wh.
But if I look at the forward energy channel it reports the following data
17:58 --> 0,02Wh
18:58 --> 0,07Wh

Have you experienced something similar? Maybe I'm missing something.
With Tasmota and (of course) the same calibration parameters I can find a correct correlation between power and energy computation.

[kpfleming]

Thanks for the report! I have not seen this sort of problem on my own devices; I ran two 3EMs in parallel (one with the Shelly firmware and one with ESPHome) for about a week to confirm that the reported numbers were close enough to be acceptable. There were minor differences, but I assumed those were caused by CT clamp and/or calibration differences.

This component does gather the Wh data differently than the Shelly firmware and Tasmota do; they both read the Wh registers every second, and then report the sum periodically. This component reads the registers at each update_interval, under the assumption that the ADE7880 itself won't overflow the registers during that time. What is your update_interval set to?

[kpfleming]

Also, are only the Wh sensors incorrect (meaning the other sensors are correct)?

[dinamitemic]

What is your update_interval set to?

I have not specified it so should be 60s as default. I can confirm that also in the logs the data were retrieved every minute.

Also, are only the Wh sensors incorrect (meaning the other sensors are correct)?
I cannot test both fw in parallel like you (since I have only one device) but the data that I've seen were the same.

You're idea about the register overflow and the update time can be the point. Are you updating the sensors faster?

[kpfleming]

No, I'm also using the default update interval of 60s.

[kpfleming]

In order to avoid spamming everyone else subscribed to this issue, please open one in https://github.com/kpfleming/esphome and we'll work through this there. I'll report back results to this issue once we've reached a conclusion.

[kpfleming]

@dinamitemic I've just pushed an update to the PR branch. It resolves the issue with not being able to provide the current_s calibration value... Shelly reports those differently than the voltage calibration values (negative values are reported as large positive values). I don't expect that this will address your Wh reporting problem, but there's a chance it could.

I've also removed the commit which moved around the CONF_PHASE_A/B/C constants, so now the branch can be used as a normal 'external component' with no patching required, and no need to suppress refreshes.

[kpfleming]

Since the weirdness with the current_s values is specific to the Shelly 3EM and not to the ADE7880 in general, I've reverted those code changes and replaced them with a note in the 'Devices' guide for the Shelly 3EM documenting the procedure for converting the value to the necessary format.

[ghost]

Is this component dependent on having all three phases connected? I'm testing it with just neutral and A wired - and I'm not getting the expected readings.

[12:24:41][D][sensor:093]: 'Neutral Current': Sending state 0.01175 A with 2 decimals of accuracy
[12:24:41][D][sensor:093]: 'Phase A Current': Sending state 0.01353 A with 2 decimals of accuracy
[12:24:41][D][sensor:093]: 'Phase A Voltage': Sending state 0.11360 V with 1 decimals of accuracy
[12:24:41][D][ade7880:159]: update took 21 ms

Using the https://deploy-preview-499--esphome-devices.netlify.app/devices/Shelly-3EM yaml with my calib.dat entries merged (and a lot of the other entities commented out, same result with the full config for that matter).

[kpfleming]

No, the phases (all four) are independently read and reported. Would you be willing to test with Tasmota to see if it reports the same readings (or correct readings) using the same calibration values?

[ghost]

Would you be willing to test with Tasmota

When pasting the template it ignores the ADE7880 IRQ, adding both back after and running the calib rule in the console gives me all 0 values.

Restoring the stock firmware from backup gives me a good voltage reading.

Guess its time to see if they've made a new revision or something. Will compare available pictures to my hardware.
Edit: hardware looks identical to pictures from tasmota thread, except different batches of the ADE7880. Will poke it some more when I have more time.

[r10r]

@kpfleming I have issues compiling a firmware for the shelly3em because my calibration data does not validate.
I followed the instructions on https://deploy-preview-499--esphome-devices.netlify.app/devices/Shelly-3EM#calibration-data

The power values are to low and return the error value must be at least -8388608 on compilation.
The phase_angle values are negative and return the error value must be at least 0 on compilation.

This is my calibration data:

{
  "state": 0,
  "rms": {
    "current_a": 3115853,
    "current_b": 3145899,
    "current_c": 3118537,
    "current_n": 55507790,
    "current_s": 3683735,
    "voltage_a": -754486,
    "voltage_b": -745905,
    "voltage_c": -768669
  },
  "angles": {
    "angle0": -10091,
    "angle1": -10096,
    "angle2": -10099
  },
  "powers": {
    "totactive": {
      "a": -15427176,
      "b": -15431191,
      "c": -15434854
    },
    "apparent": {
      "a": 1383297,
      "b": 1383213,
      "c": 1383263
    }
  },
  "energies": {
    "totactive": {
      "a": -56297,
      "b": -56294,
      "c": -56297
    },
    "apparent": {
      "a": 146044,
      "b": 146123,
      "c": 146229
    }
  }
}

This is my esphome sensor configuration:

sensor:
  - id: ade7880_device
    platform: ade7880
    irq0_pin:
      number: GPIO13
    irq1_pin:
      number: GPIO5
    reset_pin:
      number: GPIO16
    frequency: 60Hz
    phase_a:
      name: Phase A
      voltage: Voltage
      current: Current
      active_power: Active Power
      power_factor: Power Factor
      forward_active_energy: Forward Active Energy
      reverse_active_energy: Reverse Active Energy
      calibration:
        current: 3115853
        voltage: -754486
        power: -15427176
        phase_angle: -10091
    phase_b:
      name: Phase B
      voltage: Voltage
      current: Current
      active_power: Active Power
      power_factor: Power Factor
      forward_active_energy: Forward Active Energy
      reverse_active_energy: Reverse Active Energy
      calibration:
        current: 3145899
        voltage: -745905
        power: -15431191
        phase_angle: -10096
    phase_c:
      name: Phase C
      voltage: Voltage
      current: Current
      active_power: Active Power
      power_factor: Power Factor
      forward_active_energy: Forward Active Energy
      reverse_active_energy: Reverse Active Energy
      calibration:
        current: 3118537
        voltage: -745905
        power: -15434854
        phase_angle: -10099
    neutral:
      name: Neutral
      current: Current
      calibration:
        current: 55507790

[kpfleming]

Thanks for the report! I'm investigating this now, it appears that the 3EM calibration data may be the 'raw' values from the I2C bus and not the documented values from the datasheet. I'd already dealt with that for the current sensor calibration, but I'll check it for the others too.

[r10r]

I‘ve already disabled the checks and now I‘m getting correct power values. So it seems the calibration values are valid and must not be converted. 😀

…

Am 21.10.2023 um 14:11 schrieb Kevin P. Fleming ***@***.***>:  Thanks for the report! I'm investigating this now, it appears that the 3EM calibration data may be the 'raw' values from the I2C bus and not the documented values from the datasheet. I'd already dealt with that for the current sensor calibration, but I'll check it for the others too. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.

[kpfleming]

Yes, that's what I suspected. It's not clear how the Shelly firmware is deriving these calibration values as they do not align with what the ADE7880 datasheet indicates, but since the Tasmota firmware blindly accepts them and writes them to the chip (and that produces the expected results), this component could do the same. I'll rework the code to not care about the interpretation of the calibration data at all.

[kpfleming]

All done; the three PRs have all been updated to accept any integer value for the calibration registers.

[Alibloke]

@kpfleming I get a compilation error for the neutral channel:

INFO ESPHome 2023.12.8
INFO Reading configuration /config/heatpumpmonitor.yaml...
INFO Detected timezone 'Etc/UTC'
INFO Generating C++ source...
INFO Compiling app...
Processing heatpumpmonitor (board: esp01_1m; framework: arduino; platform: platformio/espressif8266@3.2.0)
--------------------------------------------------------------------------------
HARDWARE: ESP8266 80MHz, 80KB RAM, 1MB Flash
Dependency Graph
|-- ESPAsyncTCP-esphome @ 2.0.0
|-- ESPAsyncWebServer-esphome @ 3.1.0
|-- DNSServer @ 1.1.1
|-- ESP8266WiFi @ 1.0
|-- ESP8266mDNS @ 1.2
|-- noise-c @ 0.1.4
|-- Wire @ 1.0
|-- ArduinoJson @ 6.18.5
Compiling .pioenvs/heatpumpmonitor/src/main.cpp.o
src/main.cpp: In function 'void setup()':
src/main.cpp:833:27: error: 'struct esphome::ade7880::NeutralChannel' has no member named 'set_current_calibration'; did you mean 'set_current_gain_calibration'?
  833 |   ade7880_neutralchannel->set_current_calibration(44329829);
      |                           ^~~~~~~~~~~~~~~~~~~~~~~
      |                           set_current_gain_calibration
*** [.pioenvs/heatpumpmonitor/src/main.cpp.o] Error 1
========================= [FAILED] Took 10.79 seconds =========================

If I comment out the neutral: block it compiles and installs fine, though voltage is 0.

edit: voltage is fine, I got my phases mixed up

[kpfleming]

The calibration constant config entries recently changed; current is now called current_gain. I'm not sure why your build didn't report a failure when parsing the config, as it should have failed validation if it still has current in the calibration section. Can you post the ade7880 section of your YAML config?

[Alibloke]

sensor:
  - id: ade7880_device
    platform: ade7880
    irq0_pin:
      number: GPIO13
    irq1_pin:
      number: GPIO5
    reset_pin:
      number: GPIO16
    frequency: 50Hz
    phase_a:
      name: Phase A
      voltage: Voltage
      current: Current
      active_power: Active Power
      power_factor: Power Factor
      forward_active_energy: Forward Active Energy
      reverse_active_energy: Reverse Active Energy
      calibration:
        current_gain: 3164337
        voltage_gain: -789668
        power_gain: -1310795
        phase_angle: 188
    phase_b:
      name: Phase B
      voltage: Voltage
      current: Current
      active_power: Active Power
      power_factor: Power Factor
      forward_active_energy: Forward Active Energy
      reverse_active_energy: Reverse Active Energy
      calibration:
        current_gain: 3177488
        voltage_gain: -813346
        power_gain: -1305703
        phase_angle: 204
    phase_c:
      name: Phase C
      voltage: Voltage
      current: Current
      active_power: Active Power
      power_factor: Power Factor
      forward_active_energy: Forward Active Energy
      reverse_active_energy: Reverse Active Energy
      calibration:
        current_gain: 3072438
        voltage_gain: -789750
        power_gain: -1304045
        phase_angle: 136
#    neutral:
#      name: Neutral
#      current: Current
#      calibration:
#        current_gain: 44329829

[kpfleming]

Well, that certainly looks correct :-) Can you run an esphome clean for that YAML config before compiling again? I want to make sure there aren't any old build files left. I've also just rebased the branch onto current dev so your next build should re-clone the branch.

[kpfleming]

Ahh... never mind. I just reproduced with the updated PR branch. I'll fix it now.

[kpfleming]

The PR branch has been fixed and CI tests now pass, your config should compile properly. Thanks for the report, I didn't catch this locally as I don't use the 'neutral' phase monitoring.