Issue with Pico W when using a W5500 and WiFi at the same time.

[rpineau]

Fisrt some background :
I have a system where a "rotator" Arduino Due controls an observatory dome and a "shutter" Arduino Due which control said dome shutter.
The "rotator" arduino due use a W5500 to connect to the local network and both Due communicate with each other using XBee modules:

Local network <----> [W5500/Rotator Due/XBee] <-----> [Xbee/Shutter Due]

I'm now working on replacing these with Raspberry Pi Pico W and want to use the WiFi connection to replace the XBees.
I ported the code to the Pico W. Core 0 is for communications (Ethernet, WiFi and serial) and Core 1 takes care of the motion (it drives
stepper motor using the AccelStepper library) :

Local network <----> [W5500/Pico W/WiFi] <-----> [WiFi/Pico W]

The problem(s):
When disabling WiFi on the "rotator", Ethernet works fine and all is good.
But as soon as I want to add WiFi problems starts and the WiFi connections between the 2 Pico W keeps failing.
It also seems to create some timeout on the W5500 side (pinging the "rotator" Pico W from a machine starts failing).
To try to figure out the issue I made 2 small test firmware that only use the ping function.
As far as I can tell the issue comes from the Pico W that also has the W5500 . After a bit of time it locks up in the wifi ping function.
I also try with just WiFi between the 2 Pico W. This is more stable but I see a lot of drops at the beginning and then latter on (not repeatable timing as far as I can see).
So the question is .. Am I doing something wrong or is this a "bug" and the Pico W can't do Ethernet and WiFi at the same time ?
The test code can be found there :

https://rti-zone-files.s3.amazonaws.com/test-rp2040.tgz

Thanks for any insight.
Rodolphe

[earlephilhower]

Moved to an issue. This should work, because LWIP handles the routing tables just fine, but there is most likely an IRQ protection missing between the LWIP_Ethernet and cyw43_driver and we're getting parallel access to LWIP which is a very bad thing...

[rpineau]

I ran more test by just connection the "shutter" Pico W to my local wifi.. same code, just changed the SSID and password.
It connects and disconnects a few time, then the ping works... for a few minutes and then the RP2040 crashes.
So there is potentially 2 issues :

• Interrupt conflict between the W5500 and CYW43drivers
• Instability in the CYW43drivers

To double check that it was not something in my (very simple and short ) code, I did the same test with an ESP32 (WROOM-32), it never disconnected from my wifi and only missed 4 pings over the 25 hours test (yes 25 hours...) .
The Pico W can't stay connected for more than a few minutes and crash after a few disconnect/reconnect.

Regarding interrupt protection, is there a thread lock or some atomic variables ? (I use std::atomic for variables access across cores and interrupts).

[earlephilhower]

Thanks, that's still pretty odd. It's actually extra work to filter down the filter out the file/line for each stack frame and I don't think they have added any value by filtering like that. In any case, looks like prior memory corruption which gets caught here by a sanity check. So this trace seems OK but there's something more subtle afoot.

If it's easy to re-run, can you remove the IRQ from the W5500 constructor (remove the parameter and everything should still "just work," albeit a little slower)? If that works then I think it will really shrink the places to look for badness...

[rpineau]

Yes, I can rerun this without the interrupt pin.

[earlephilhower]

Err, I think something else is up there. That's right at the beginning of the initialization of the core. I see you're calling something which is trying to add the W5500::begin inside of loop(). In the sketch I see you are calling configureEthernet inside setup() as well, so that's definitely not legal. You can only do begin() once per app...

[rpineau]

Without the interrupt pin the ping are not working (even with the default 5000ms timeout).
Stack trace :

_exit@0x1000a61c
panic@0x1000a6b2
raw_input@0x1000f684
ip4_input@0x1001597e
ethernet_input@0x10015e7a
__wrap_cyw43_cb_process_ethernet@0x1001e47a
cyw43_ll_process_packets@0x10016914
cyw43_poll_func@0x1000ce4c
cyw43_poll_func@0x1000ce4c
cyw43_do_poll@0x1000d4d2
async_context_base_execute_once@0x1000de2e
process_under_lock@0x1000da00
low_priority_irq_handler@0x1000db62
<signal handler called>@0xfffffff9
__wfe@0x10008b8a
sleep_until@0x10008b8a
sleep_us@0x10008bb8
sleep_ms@0x10008bd2
delay@0x10006812
LwipIntfDev<Wiznet5500>::ping@0x100037e8

rotator.ino229:0
    PingEthernetGateway@0x10003c0e
rotator.ino137:0
    loop@0x100040f2

main@0x1000707e

[earlephilhower]

Ah, good info. It does at least show it's not unique to the IRQ path. Oh well, I'm going to have to wire some stuff up, it looks like!

[earlephilhower]

This has nothing to do with multiple Ethernet interfaces, I believe. I can reproduce it with a single W5500 or a CYW43. It's related to the ping operation and the raw_recv which is enabled while a ping response is outstanding. The simple sketch below on a W5500 only device can repro the issue:

#include <W5500lwIP.h>

const char* host = "djxmmx.net";
const uint16_t port = 17;
Wiznet5500lwIP eth(1, SPI, 4);

void setup() {
  // Set up SPI pinout to match your HW
  SPI.setRX(0);
  SPI.setCS(1);
  SPI.setSCK(2);
  SPI.setTX(3);

  Serial.begin(115200);
  delay(5000);
  Serial.println();
  Serial.println();
  Serial.println("Starting Ethernet port");

  // Start the Ethernet port
  if (!eth.begin()) {
    Serial.println("No wired Ethernet hardware detected. Check pinouts, wiring.");
    while (1) {
      delay(1000);
    }
  }

  while (!eth.connected()) {
    Serial.print(".");
    delay(500);
  }

  Serial.println("");
  Serial.println("Ethernet connected");
  Serial.println("IP address: ");
  Serial.println(eth.localIP());
}

void loop() {
  Serial.printf("+ping %lu\n", millis());
  Serial.flush();
  eth.ping(IPAddress(192,168,1,8), 32);
  //eth.ping(IPAddress(192,168,99,1), 32);
  Serial.printf("-ping\n");
  Serial.flush();
  delay(30);
}

Have a PC on the network do ping <pico.io> and eventually it will hang If you change the address the eth is pinging to something that won't respond (i.e. it will wait 5s to timeout with the raw_recv callback active) then you can get the issue immediately.

Still looking into it, but after adding logging in the :LwipIntfDev::_pingCB I see that as soon as I get a single packet into it that's not a ping response (but is, instead, a ping query from the PC) things go south fast.

[rpineau]

Interesting finding.
I still suspect there is more than just the ping function having issues.
If you look at my first post, I had the issue with standard TCP connection.
When not using WiFi the ethernet side was working find, no data loss, no errors on the ethernet side (2 "WiFiServer" on the ethernet side , one use for some ascii commands over TCP, the other one is a REST server). I also have a UDP service running on the Ethernet. No WiFi enable = no problems.
As soon as WiFi is enable and I have a connection (Also a TCP WiFiServer running there) , I start seeing errors on the Ethernet side (command send from my computer but no response and the debug log on the Pico W do not show the received command) and communication with the WiFi client drops very quickly... all of this followed by a crash.
The "ping" test was written to try to reproduce the issue in a simple setup ... now if there is an issue with the ping... that sort of void that "simple" test :)
So it might be more general that just the ping issue when receiving a ping request, other packet type (TCP) might also be triggering some similar issue.
The fact that you can reproduce with both the W5500 and CYW43 probably mean the issue is in a code path used by both instance .

[earlephilhower]

When #2159 merges, give it a try and see how things go from there. Right now that was the only thing I was able to find. Easy to see the bug and trivial to reproduce the failure and then a pass with the 2-line PR.

[earlephilhower]

FWIW, I did start out w/a WiFiClient (doing the MOTD loop every second), a WiFiServer (loopback of large chunks of data) and even a WebServer (wildcard from the BasicWebServer) in my test but I couldn't get a crash until I threw in some pings from the Pico (even w/pings from the PC). As soon as I added the eth.ping() or WiFi.ping() calls I could get crashes with your same trace and just started deleting bits until only the pings were there, still showing the same stack trace (LWIP complaining that I didn't handle a raw packet but I munged up the payload data even so....which I was doing, actually).