i2c transfer timed out

[f18m]

Hi all,
I'm using this HAT: https://sequentmicrosystems.com/products/16-universal-inputs-card-for-raspberry-pi
reading the input very frequently (1Hz, forever) and my application works fine for several hours.
However, in a random way apparently, I get in my Python application the following exception:

Mar 02 10:40:32 raspberrypi python3[1298]: | Traceback (most recent call last):
Mar 02 10:40:32 raspberrypi python3[1298]: | File "/root/ha-alarm-raspy2mqtt-venv/lib/python3.11/site-packages/lib16inpind/__init__.py", line 35, in readAll
Mar 02 10:40:32 raspberrypi python3[1298]: | val = bus.read_word_data(hw_add, INPUTS16_INPORT_REG_ADD)
Mar 02 10:40:32 raspberrypi python3[1298]: | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 02 10:40:32 raspberrypi python3[1298]: | File "/root/ha-alarm-raspy2mqtt-venv/lib/python3.11/site-packages/smbus2/smbus2.py", line 474, in read_word_data
Mar 02 10:40:32 raspberrypi python3[1298]: | ioctl(self.fd, I2C_SMBUS, msg)
Mar 02 10:40:32 raspberrypi python3[1298]: | TimeoutError: [Errno 110] Connection timed out
Mar 02 10:40:32 raspberrypi python3[1298]: |
Mar 02 10:40:32 raspberrypi python3[1298]: | During handling of the above exception, another exception occurred:
Mar 02 10:40:32 raspberrypi python3[1298]: |
Mar 02 10:40:32 raspberrypi python3[1298]: | Traceback (most recent call last):
Mar 02 10:40:32 raspberrypi python3[1298]: | File "/root/ha-alarm-raspy2mqtt/raspy2mqtt/raspy2mqtt.py", line 415, in sample_and_publish_optoisolated_inputs
Mar 02 10:40:32 raspberrypi python3[1298]: | sampled_values_as_int = lib16inpind.readAll(SEQMICRO_INPUTHAT_STACK_LEVEL)
Mar 02 10:40:32 raspberrypi python3[1298]: | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mar 02 10:40:32 raspberrypi python3[1298]: | File "/root/ha-alarm-raspy2mqtt-venv/lib/python3.11/site-packages/lib16inpind/__init__.py", line 38, in readAll
Mar 02 10:40:32 raspberrypi python3[1298]: | raise Exception(e)
Mar 02 10:40:32 raspberrypi python3[1298]: | Exception: [Errno 110] Connection timed out

and in the journalctl I can see:

Mar 02 10:40:32 raspberrypi kernel: i2c-bcm2835 20804000.i2c: i2c transfer timed out

When this happens... the Raspberry is totally unable to even restore I2C communications with the HAT.
Any attempt to read data through I2C results in the same error in journalctl:

Mar 02 23:35:04 raspberrypi kernel: i2c-bcm2835 20804000.i2c: i2c transfer timed out
Mar 02 23:35:05 raspberrypi kernel: i2c-bcm2835 20804000.i2c: i2c transfer timed out
Mar 02 23:35:06 raspberrypi kernel: i2c-bcm2835 20804000.i2c: i2c transfer timed out
Mar 02 23:35:07 raspberrypi kernel: i2c-bcm2835 20804000.i2c: i2c transfer timed out

Even a reboot of the Raspberry PI does not work :-(
I'm really lost on how to fix this.
Last time it happened I think I did a complete power cycle (unplugged the power adapter and plugged it back).

A few more notes:
* I didn't connect anything to GPIO lines related to SDA and SCL, beyond the SequentMicrossytem HAT
* I'm using a Raspberry Model B+ from 2014
* I'm using Raspbian 12 (bookworm), with kernel

Linux raspberrypi 6.1.0-rpi7-rpi-v6 #1 Raspbian 1:6.1.63-1+rpt1 (2023-11-24) armv6l GNU/Linux

 

[f18m]

I forgot to mention that I have a Raspberry with the 16LV digital inputs HAT currently stuck in this state where every I2C transaction is "transfer timed out".
Before I power-cycle, is there any debug information I can retrieve to help understand the issue?
thanks!

 

[alexburcea]

It looks like the I2C port hung completely, never so something like this.
I do not know a method to log this but you can try instead of using python library to use the command line driver "16inpind" even runed as a program from python or from a shell script. If the problem persist I suggest you to reinstall the raspbian or to try a different version of it.

 

[f18m]

It looks like the I2C port hung completely, never so something like this.
I do not know a method to log this but you can try instead of using python library to use the command line driver "16inpind" even runed as a program from python or from a shell script. If the problem persist I suggest you to reinstall the raspbian or to try a different version of it.

Ok thanks I will try that!
Generally speaking, do you think it should be fine to poll at 1hz for several hours the I2c bus?
In my experience it works for several hours (like 12-24) but then it enters this "hung" condition and requires a power cycle to restore :-(

 

[f18m]

Btw I found by chance the following thread: https://forum.sequentmicrosystems.com/index.php?threads/polling-inputs.58/

My understanding is that there is a special firmware for the 16 LV inputs board that provides a very short pulse on gpio26 (Same of the push button) to signal a change in any of the 16inputs right? That would make it possible for me to stop polling at 1hz and instead read the inputs only when they change...is that correct? Is that firmware stable ( for production use)?
Thanks!!!

 

[alexburcea]

Normally there is no problem to poll at 10 or even 100Hz...
The feature you so there is apply to a different card but your card is made with an IOExpander chip and has this functionality by default enabled. You can poll or set an interrupt for GPIO11 (connector pin 23) to see if any input has changed. Make sure you enable the Raspberry pin pull-up resistor because the chip output is open-collector.

 

[f18m]

Normally there is no problem to poll at 10 or even 100Hz...
The feature you so there is apply to a different card but your card is made with an IOExpander chip and has this functionality by default enabled. You can poll or set an interrupt for GPIO11 (connector pin 23) to see if any input has changed. Make sure you enable the Raspberry pin pull-up resistor because the chip output is open-collector.

Sorry I'm a bit confused. I downloaded the schematic (v3.0) from https://sequentmicrosystems.com/pages/sixteen-inputs-downloads but I cannot see anything connected between the chip U2 (it would be nice to know the part number, btw) and the GPIO11 a.k.a connector pin23 a.k.a. SPICLK... what am I missing? Is it the IO expander chip that drives down somehow the GPIO11/connector-pin23 ?
thanks!

 

[alexburcea]

Hi,

You have the V2.0 card, please look at the corresponding schematics. The V3.0 is still in the development.

 

[f18m]

Thanks, that helps a lot, my bad I didn't realize which version I bought
Btw one suggestion: the availability of the interrupt line is not mentioned in the docs..it would be useful to mention that
Finally I found some references to situations where I2C bus might lock up, see e.g. https://pebblebay.com/i2c-lock-up-prevention-and-recovery/
I also found references of this issue specifically happening on raspberry PIs: https://github.com/PiSupply/PiJuice/issues/634

This is something to consider very well when using I2C on raspberry PIs... I wonder if the SPI bus is better/more stable (any hat with IO expanders using SPI coming soon perhaps? )
Anyway I'm going to test a few different approaches and will update this thread.
Thanks for the fast support!

 

[alexburcea]

Thanks for the suggestion about the manual.
I2C is an addressable bus, we choose it because we can attach multiple devices on the same Raspberry Pi.
Thank you for testing this and keep us informed.

 

[f18m]

Short update on this: the I2C bus lockup happened also today.
I don't know how it's possible but even after several power-cycles, the I2C remains locked... at reboot the Raspberry keeps complaining:

[Wed Mar 6 00:25:49 2024] i2c-bcm2835 20804000.i2c: i2c transfer timed out
[Wed Mar 6 00:26:06 2024] i2c-bcm2835 20804000.i2c: i2c transfer timed out
[Wed Mar 6 00:26:18 2024] i2c-bcm2835 20804000.i2c: i2c transfer timed out
[Wed Mar 6 00:26:30 2024] i2c-bcm2835 20804000.i2c: i2c transfer timed out
[Wed Mar 6 00:26:41 2024] i2c-bcm2835 20804000.i2c: i2c transfer timed out

I'm not sure how's that possible. Nothing is wired on the pins of the I2C appearing on the GPIO connector. No change in wiring has happened between yesterday (during which the software polled the I2C bus at 1Hz for the whole day just fine) and today (when the I2C bus lockup happened).
And how's possible that after a power cycle where I cut the power off, watch all LEDs turn off, and then provide again the power, the I2C bus remains locked up? I have no idea.

I do not know a method to log this but you can try instead of using python library to use the command line driver "16inpind" even runed as a program from python or from a shell script. If the problem persist I suggest you to reinstall the raspbian or to try a different version of it.

The 16pind compiled utility (compiled from C) does not work either. The i2cdetect utility has timeouts.
Nothing seems to work; every attempt to use I2C results in another line in the dmesg log:

[Wed Mar 6 00:26:41 2024] i2c-bcm2835 20804000.i2c: i2c transfer timed out

I start to believe that Raspberry Model B+ I2C is seriously bugged. Lots of people on the web have found the "I2C clock stretching" bug, best described by https://www.advamation.com/knowhow/raspberrypi/rpi-i2c-bug.html
I'm not sure if the clock stretching is done by the IO expander used by the hat. Is the hat v2.0 using the AWINIC AW9523BTQR as I/O expander?
I looked in the datasheet and clock stretching is not mentioned.

Anyway I'm considering replacing that old raspberry with a newer Raspberry Model 4... at this page: https://hackaday.com/2022/02/01/did-you-know-that-the-raspberry-pi-4-has-more-spi-i2c-uart-ports/ they mention that at least the clock-stretching bug has been fixed and generally speaking the I2C kernel driver is probably a bit different.

I also believe that this hat has probably been used mostly with recent raspberry models.... which model of Raspberry do you use to test/QA/validate the hats like the 16LV digital inputs HAT ?

thanks

 

[alexburcea]

We validate with all Raspberry models but use constantly Zero, 4, and CM4. We have one 5 from one week but we did not test all the cards with it.
I personally installed one RPI3 with a Home Automation and 4 x 8-relays cards and is working for a few years, keeping heating and cooling, of a house, controlling also the irrigation. It is possible that you have a faulty Raspberry Pi 3 and if you replace it is probably better to get a 4 or 5.

 

[f18m]

It is possible that you have a faulty Raspberry Pi 3 and if you replace it is probably better to get a 4 or 5.

Actually I was using a Raspberry Pi 1 (model B+) from 2014. I bought a Raspberry Pi4 and will install it soon... I really hope to get I2C communication stable with Raspberry Pi4... I will update. thanks for support.

 

[f18m]

hi @alexburcea ,
I wanted to report back after several days a good news: after I received a Raspberry Pi4 and I started using it, I never found the I2C hangup happening again.
I believe it was an HW issue after all.
I'm also wondering if it makes sense to change the software to use the interrupt-based approach, instead of the continouus 1Hz polling... since now the polling works just fine

Thanks for all the support!! I think the issue is finally "resolved"

 

[alexburcea]

Hi,
Thank you for letting me know. Please rate our products.