Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

i2cdetect -y xx doesn't seem to clear buffer after scanning for i2c devices on MULTIPLE MUX #6039

Open
Trilife opened this issue Mar 14, 2024 · 33 comments

Comments

@Trilife
Copy link

Trilife commented Mar 14, 2024

Describe the bug

I have 2 MUX installed, as follows:
dtoverlay=i2c-mux,pca9548,addr=0x74,base=40 dtoverlay=i2c-mux,pca9548,addr=0x70,base=30
(note base=xx is a beta provided by PhilE, but the problem was there before this).

When I type i2cdetect -y 3x the first time, it correctly looks for all the available devices on bus 3x (MUX1). But, when I then type i2cdetect -y 4y to show devices on bus 4y (MUX2), it superimposes what it found on bus 3x to what it finds on 4y. If my interpretation is correct, it holds onto the buffer from MUX1 when displaying MUX2.

It doesn't do this, when you stick to MUX1 addresses. Only superimposes MUX1 onto MUX2. I tested the reverse, with MUX2 first and MUX1 second. Same behavior.

Example:

Helios@HeliosRPi4:~ $ i2cdetect -y 30
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- 57 -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- 68 -- -- -- -- -- -- --
70: UU -- -- -- UU -- -- --
Helios@HeliosRPi4:~ $ i2cdetect -y 40
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- 28 -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- 57 -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- 68 -- -- -- -- -- -- --
70: UU -- -- -- UU -- -- --
Helios@HeliosRPi4:~ $

57 and 68 do not exist on bus 40...

Steps to reproduce the behaviour

1- install 2 identical MUX via dtoverlay:

dtoverlay=i2c-mux,pca9548,addr=0x74
dtoverlay=i2c-mux,pca9548,addr=0x70

2- install a few i2c devices on each MUX, ideally different I2C addresses
3- reboot
4- type 'i2cdetect -y xx' (MUX1) and note the devices shown on monitor
5- type 'i2cdetect -y yy' (MUX2) and note that the devices shown from the previous step (port XX) are still visible, in addition to the actual devices on port YY.

Device (s)

Raspberry Pi 4 Mod. B

System

Helios@HeliosRPi4:~ $ cat /etc/rpi-issue Raspberry Pi reference 2023-12-05 Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 70cd6f2a1e34d07f5cba7047aea5b92457372e05, stage4
Helios@HeliosRPi4:~ $ vcgencmd version Oct 17 2023 15:39:16 Copyright (c) 2012 Broadcom version 30f0c5e4d076da3ab4f341d88e7d505760b93ad7 (clean) (release) (start)
Helios@HeliosRPi4:~ $ uname -a Linux HeliosRPi4 6.6.21-v8+ #1 SMP PREEMPT Thu Mar 14 10:53:16 UTC 2024 aarch64 GNU/Linux

Logs

No response

Additional context

No response

@pelwell
Copy link
Contributor

pelwell commented Mar 14, 2024

I think it's worse than you describe, because aren't the UUs in 70: UU -- -- -- UU -- -- -- the PCF9548s themselves?

@6by9
Copy link
Contributor

6by9 commented Mar 14, 2024

I have 2 MUX installed, as follows:

dtoverlay=i2c-mux,pca9548,addr=0x74,base=40
dtoverlay=i2c-mux,pca9548,addr=0x70,base=30 

(note base=xx is a beta provided by PhilE, but the problem was there before this).

When I type i2cdetect -y 3x the first time, it correctly looks for all the available devices on bus 3x (MUX1). But, when I then type i2cdetect -y 4y to show devices on bus 4y (MUX2), it superimposes what it found on bus 3x to what it finds on 4y. If my interpretation is correct, it holds onto the buffer from MUX1 when displaying MUX2.

It doesn't do this, when you stick to MUX1 addresses. Only superimposes MUX1 onto MUX2. I tested the reverse, with MUX2 first and MUX1 second. Same behavior.

MUX1 knows nothing about MUX2.
By default the mux is left selected to the last used output in the hope that it saves reselecting the same output for the next transaction.
So your observation is correct - activating a 3x output and then a 4x output will see the combination of the two mux output buses.

Reading the bindings, there is the i2c-mux-idle-disconnect or idle-state properties that allow control of the idle state of any mux.
Adding either

i2c-mux-idle-disconnect;

or

idle-state = <MUX_IDLE_DISCONNECT>;

to the pca95x nodes should cause it to always disconnect when idle, at the expense of an extra I2C write to the mux after every transaction (this will add up for something like an i2cdetect).

I think it's worse than you describe, because aren't the UUs in 70: UU -- -- -- UU -- -- -- the PCF9548s themselves?

Addresses used on the parent bus will also be reserved on all child buses.

@Trilife
Copy link
Author

Trilife commented Mar 14, 2024 via email

@pelwell
Copy link
Contributor

pelwell commented Mar 14, 2024

Addresses used on the parent bus will also be reserved on all child buses.

I guess that is required, otherwise there would be no way to say "I want to talk to the parent bus now".

BTW, is it possible that the mod you made to add base= to the i2c-mux broke
the GPIO edge detection:

I get this error, after I installed it:

GPIO.add_event_detect(self.input_pin, GPIO.BOTH,
callback=self._rising_or_falling, bouncetime=200)
RuntimeError: Failed to add edge detection.

That's already covered by #6037

@6by9
Copy link
Contributor

6by9 commented Mar 14, 2024

#6040 adds an override disconnect_on_idle that adds idle-state = <MUX_IDLE_DISCONNECT>; to the relevant nodes.

@pelwell
Copy link
Contributor

pelwell commented Mar 14, 2024

I've merged #6040 and rebased #6038 on top of it. As usual, it will take about 45 minutes for the auto-builds to complete, then sudo rpi-update pulls/6038 again. (If you installed pulls/6040 instead you would lose the base= parameter support)

@Trilife
Copy link
Author

Trilife commented Mar 14, 2024 via email

@6by9
Copy link
Contributor

6by9 commented Mar 14, 2024

What are your complete dtoverlay lines now?
Have you added the disconnect_on_idle override to both of them?

xxd /proc/device-tree/soc/i2c@7e804000/mux@70/idle-state should return

00000000: ffff fffe

(aka -2) if correctly configured.

@pelwell
Copy link
Contributor

pelwell commented Mar 14, 2024

In other words:

dtoverlay=i2c-mux,pca9548,addr=0x74,base=40,disconnect_on_idle
dtoverlay=i2c-mux,pca9548,addr=0x70,base=30,disconnect_on_idle

@Trilife
Copy link
Author

Trilife commented Mar 14, 2024 via email

@6by9
Copy link
Contributor

6by9 commented Mar 14, 2024

I can't find my PCA9548 at present, but I've done a quick hack to the driver to ignore I2C errors, and used I2C event logging (see https://riptutorial.com/linux-kernel/example/11983/tracing-i2c-events for how)

My log (plus annotations) gives me this as a fragment from an i2cdetect -y 29 (8th bus on the mux as I haven't got the base_nr patches).

Write 0 bytes to address 0x72 on bus 29
       i2cdetect-2334    [001] ..... 12005.787266: i2c_write: i2c-29 #0 a=072 f=0000 l=0 []

Write 1 byte to set the mux to port 8
       i2cdetect-2334    [001] ..... 12005.787268: i2c_write: i2c-1 #0 a=070 f=0000 l=1 [80]
       i2cdetect-2334    [001] ..... 12005.787409: i2c_result: i2c-1 n=1 ret=-5

Write 0 bytes to address 0x72 on bus 1 (parent of bus 29)
       i2cdetect-2334    [001] ..... 12005.787411: i2c_write: i2c-1 #0 a=072 f=0000 l=0 []
       i2cdetect-2334    [001] ..... 12005.787550: i2c_result: i2c-1 n=1 ret=-121

Set the mux to no output
       i2cdetect-2334    [001] ..... 12005.787552: i2c_write: i2c-1 #0 a=070 f=0000 l=1 [00]
       i2cdetect-2334    [001] ..... 12005.787690: i2c_result: i2c-1 n=1 ret=-5

Overall write failed
       i2cdetect-2334    [001] ..... 12005.787692: i2c_result: i2c-29 n=1 ret=-121


Write 0 bytes to address 0x73 on bus 29
       i2cdetect-2334    [001] ..... 12005.787725: i2c_write: i2c-29 #0 a=073 f=0000 l=0 []

Set the mux to port 8
       i2cdetect-2334    [001] ..... 12005.787727: i2c_write: i2c-1 #0 a=070 f=0000 l=1 [80]
       i2cdetect-2334    [001] ..... 12005.787867: i2c_result: i2c-1 n=1 ret=-5

Write 0 bytes to address 0x73 on bus 1 (parent of bus 29)
       i2cdetect-2334    [001] ..... 12005.787870: i2c_write: i2c-1 #0 a=073 f=0000 l=0 []
       i2cdetect-2334    [001] ..... 12005.788009: i2c_result: i2c-1 n=1 ret=-121

Set the mux to no output
       i2cdetect-2334    [001] ..... 12005.788011: i2c_write: i2c-1 #0 a=070 f=0000 l=1 [00]
       i2cdetect-2334    [001] ..... 12005.788151: i2c_result: i2c-1 n=1 ret=-5

Overall write failed
       i2cdetect-2334    [001] ..... 12005.788153: i2c_result: i2c-29 n=1 ret=-121

So that is doing exactly what I'd expect it to.

rpi-update normally caches the build that it last installed. I don't know whether it does so with pulls, but it's possible that it didn't update to the new build.
If that has happened, then sudo vclog -m will report the fact it has been ignored, eg from dtoverlay=i2c-mux,pca9548,wibble

005904.876: Loaded overlay 'i2c-mux'
005904.891: dtparam: pca9548=true
005905.568: dtparam: wibble=true
005910.874: Unknown dtparam 'wibble' - ignored

Memory says it is delete /boot/firmware/.firmware_revision to force rpi-update to download the files again.

@Trilife
Copy link
Author

Trilife commented Mar 22, 2024 via email

@popcornmix
Copy link
Collaborator

And, in the meantime is there a fix around the upgrade breaking my setup at
every iteration?

The fix is merged into source tree and is in current rpi-update kernel.
It will be in subsequent updates to apt kernel.
You won't get overwritten unless there is an update (or you request install --reinstall).

So it shouldn't be an issue going forward.

@Trilife
Copy link
Author

Trilife commented Apr 5, 2024 via email

@Trilife
Copy link
Author

Trilife commented Apr 5, 2024 via email

@Trilife
Copy link
Author

Trilife commented Apr 6, 2024 via email

@pelwell
Copy link
Contributor

pelwell commented Apr 8, 2024

If you can access either mux successfully after a reboot but not then use the other then the deselection is not working (unless the problem is power related).

@6by9 has given you some debug tools that allow you to confirm that the latest driver and overlay are installed:

$ xxd /proc/device-tree/soc/i2c@7e804000/mux@70/idle-state
$ sudo vclog -m

The active idle_state value should also be available somewhere in sysfs - try:

$ find /sys -name idle_state

@pelwell
Copy link
Contributor

pelwell commented Apr 8, 2024

By the way, it may be possible (as a diagnostic test only) to force the muxes to report or disable the attached buses:

$ i2cget -f -y 1 0x70
$ i2cset -f -y 1 0x70 0

where the -f flag forces the accesses, even though a kernel driver is using the address.

@Trilife
Copy link
Author

Trilife commented Apr 8, 2024 via email

@Trilife
Copy link
Author

Trilife commented Apr 8, 2024 via email

@pelwell
Copy link
Contributor

pelwell commented Apr 8, 2024

I think your troubleshooting is good, but the results are showing that the deselection/disconnection is not working. Try accessing both muxes, but with i2cset -f -y 1 0x70 0 and/or i2cset -f -y 1 0x74 0 in between.

hexdump is probably closer to xxd, or use od:

$ od -An -td4 --endian=big /proc/device-tree/soc/i2c@7e804000/mux@70/idle-state

@Trilife
Copy link
Author

Trilife commented Apr 8, 2024 via email

@pelwell
Copy link
Contributor

pelwell commented Apr 8, 2024

Something strange is happening when the firmware is applying the overlay - the idle-state value is coming through as -1 instead of -2. For now, try applying the overlays at runtime:

  1. Comment out the config.txt dtoverlay=i2c-mux lines.
  2. Sometime after booting but before using the I2C muxes, run this:
sudo dtoverlay i2c-mux pca9548 addr=0x70 base=30 disconnect_on_idle
sudo dtoverlay i2c-mux pca9548 addr=0x74 base=40 disconnect_on_idle

@pelwell
Copy link
Contributor

pelwell commented Apr 8, 2024

Alternatively, download a modified version of the i2c-mux overlay from here: https://drive.google.com/file/d/1L8eY0GQVYd6R4926R13td9rcz9igQOj6/view?usp=sharing
It uses a different way of encoding the same changes that avoids the problem (which I think is due to a difference in the C library used in the firmware).

@Trilife
Copy link
Author

Trilife commented Apr 8, 2024 via email

@Trilife
Copy link
Author

Trilife commented Apr 9, 2024 via email

@Trilife
Copy link
Author

Trilife commented Apr 13, 2024 via email

@Trilife
Copy link
Author

Trilife commented Apr 13, 2024 via email

pelwell added a commit to raspberrypi/utils that referenced this issue Apr 15, 2024
Integer values passed as literal cell values are converted to strings
along the way. The result is passed to strtoull, which has a right to
object to being given a negative input, therefore use %u in the
format string rather than %d.

See: raspberrypi/linux#6039

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
pull bot pushed a commit to Leuca/utils that referenced this issue Apr 16, 2024
Integer values passed as literal cell values are converted to strings
along the way. The result is passed to strtoull, which has a right to
object to being given a negative input, therefore use %u in the
format string rather than %d.

See: raspberrypi/linux#6039

Signed-off-by: Phil Elwell <phil@raspberrypi.com>
timg236 added a commit to timg236/rpi-eeprom that referenced this issue Apr 17, 2024
…same branch

* Switch to building the Pi4 firmware from the common Pi4/Pi5
  mainline release. This doesn't change the Pi4 features
  but should make it quicker to release bug fixes in common code.
* Fix issue that caused the TRYBOOT flag to be lost in secure-boot mode.
* dtoverlay: Use %u when converting u32s to strings
   See: raspberrypi/linux#6039
* Improved debug messages for secure-boot.
* Generate the bootloader diagnostics qrcode at run time.
timg236 added a commit to raspberrypi/rpi-eeprom that referenced this issue Apr 17, 2024
…same branch

* Switch to building the Pi4 firmware from the common Pi4/Pi5
  mainline release. This doesn't change the Pi4 features
  but should make it quicker to release bug fixes in common code.
* Fix issue that caused the TRYBOOT flag to be lost in secure-boot mode.
* dtoverlay: Use %u when converting u32s to strings
   See: raspberrypi/linux#6039
* Improved debug messages for secure-boot.
* Generate the bootloader diagnostics qrcode at run time.
popcornmix added a commit to raspberrypi/rpi-firmware that referenced this issue Apr 17, 2024
See: raspberrypi/linux#6062

firmware: arm_dt: Improve power HAT+ support

firmware: arm_loader: Add user otp read and write functions
See: raspberrypi/linux#6014

firmware: dtoverlay: Use %u when converting u32s to strings
See: raspberrypi/linux#6039

firmware: video_decode: CONFIGCHANGED not wanted with lack of aspect ratio in new frame
See: https://forum.libreelec.tv/thread/28391-cvideoplayeraudio-process-stream-stalled/?postID=190597#post190597
popcornmix added a commit to raspberrypi/firmware that referenced this issue Apr 17, 2024
See: raspberrypi/linux#6062

firmware: arm_dt: Improve power HAT+ support

firmware: arm_loader: Add user otp read and write functions
See: raspberrypi/linux#6014

firmware: dtoverlay: Use %u when converting u32s to strings
See: raspberrypi/linux#6039

firmware: video_decode: CONFIGCHANGED not wanted with lack of aspect ratio in new frame
See: https://forum.libreelec.tv/thread/28391-cvideoplayeraudio-process-stream-stalled/?postID=190597#post190597
@Trilife
Copy link
Author

Trilife commented Apr 23, 2024 via email

@pelwell
Copy link
Contributor

pelwell commented Apr 23, 2024

You shouldn't need to install it any more - just install the latest stable firmware: sudo rpi-update stable

@Trilife
Copy link
Author

Trilife commented Apr 24, 2024 via email

@pelwell
Copy link
Contributor

pelwell commented Apr 24, 2024

Hmm, that's a new failure - I'll try and find out what's going wrong. You should be OK with updating to the latest instead - sudo rpi-update.

@pelwell
Copy link
Contributor

pelwell commented Apr 24, 2024

My advice was showing its age - the preferred syntax is now sudo rpi-update stable, which works. The BRANCH=stable version is equivalent to sudo rpi-update stable stable, which means "stable software, stable eeprom", and the eeprom has no "stable" branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants