Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RP2040: USB data stopping #4190

Closed
DynamoBen opened this issue Feb 12, 2021 · 53 comments · Fixed by #4538, hathach/tinyusb#1020 or #5172
Closed

RP2040: USB data stopping #4190

DynamoBen opened this issue Feb 12, 2021 · 53 comments · Fixed by #4538, hathach/tinyusb#1020 or #5172
Assignees
Labels
needs retest rp2 Raspberry Pi RP2 Micros usb
Milestone

Comments

@DynamoBen
Copy link

DynamoBen commented Feb 12, 2021

I created a test program that loops turning a MIDI note on and off. After running for a while (minutes not hours) I lose USB communications (both USB-MIDI and serial debug). I have the LED flashing to debug and it appears the loop is continuing it's just USB comms that stops inexplicably.

This is only reproducible on one of my Wins10 PCs (Dell XPS desktop) and the USB driver is still active when data isn't flowing. Further if I switch to a Teensy 4.0 running the same test (written in C++) USB comms continues indefinitely.

See forum thread for troubleshooting details: https://forums.adafruit.com/viewtopic.php?f=60&t=175112&p=854548 ]

[edit by dhalbert: on RPi Pico]

@dhalbert dhalbert added rp2 Raspberry Pi RP2 Micros usb labels Feb 12, 2021
@dhalbert dhalbert changed the title USB data stopping RP2040: USB data stopping Feb 12, 2021
@dhalbert dhalbert added this to the 6.x.x - Bug Fixes milestone Feb 12, 2021
@hathach
Copy link
Member

hathach commented Feb 13, 2021

There is tinyusb PR that fix this issue recently, please try to manually update tinyusb to see if that fix your issue

hathach/tinyusb#618

@dhalbert
Copy link
Collaborator

@DynamoBen Here is a build for the Pico that updates tinyusb to the tip of its master branch as of now. The commit hash is in the filename:
pico-with-tinyusb-at-09868434c.uf2.zip

Could you test with this and see if it improves things? Thanks.

I only tested this enough to see that it presents the REPL and CIRCUITPY.

@DynamoBen
Copy link
Author

@dhalbert Some progress, now my test app is locking up instead of USB comms stopping.

@DynamoBen
Copy link
Author

Grabbed latest (adafruit-circuitpython-raspberry_pi_pico-en_US-20210223-cca6cfe) this morning and haven't see any problems. I'll check it again in a few hours and will post an update if it fails.

@DynamoBen
Copy link
Author

Sadly USB comms are still stopping. I will continue to try future builds I guess.

@hathach
Copy link
Member

hathach commented Feb 24, 2021

did you call midi read from your firmware application. The data is stored in midi fifo and need to be retrieved else it will be full and prevent further data transfer. It is mentioned in the hathach/tinyusb#618 discussion

@DynamoBen
Copy link
Author

My tests thus far have been limited to transmitting MIDI. However I did update my test code to include a read but it made no difference.

@tannewt
Copy link
Member

tannewt commented Feb 24, 2021

Note we have a TinyUSB update in #4245

@DynamoBen
Copy link
Author

I've been trying the latest build the last several days and the symptom now is the application not running ("locked up").

@hathach
Copy link
Member

hathach commented Mar 2, 2021

can you post your updated script to your first post, so that other could reproduce as well when they try to take a look

@DynamoBen
Copy link
Author

DynamoBen commented Mar 2, 2021

Test code: code.zip

@hathach
Copy link
Member

hathach commented Mar 4, 2021

There is several update for rp2040 lately. I will try to update & test with this next. Though I am using Linux, not sure if it has any differences.

@DynamoBen
Copy link
Author

I keep trying the dailies and not much progress thus far. Normally I'd blame this on the computer but this is a stock Dell XPS desktop and I'm concerned that this isn't an isolated issue, albeit not widespread.

@dhalbert
Copy link
Collaborator

@DynamoBen #4517 (now merged, so you can try "Absolute Newest") might be yet another reason for this problem, so if you wouldn't mind testing again, that would be great.

@dhalbert dhalbert modified the milestones: 6.x.x - Bug Fixes, 7.0.0 Mar 31, 2021
@DynamoBen
Copy link
Author

Grabbed latest just now, no change in behavior. Runs for a little while and then stops.

@hathach
Copy link
Member

hathach commented Apr 2, 2021

@DynamoBen I make an PR #4538 that should fix this issue, I have run your code on pico for an hour, and it still blink as normal. also attached the uf2.zip here firmware.zip for quick testing.

@DynamoBen
Copy link
Author

DynamoBen commented Apr 2, 2021

@hathach it does keep blinking for a while but eventually disconnects. When the code is running MIDI eventually stops being sent.

@hathach
Copy link
Member

hathach commented Apr 2, 2021

I am still seeing message fine with my pico, tested on ubuntu 20.04

image

@DynamoBen
Copy link
Author

@hathach OK but I'm on Win10

@hathach
Copy link
Member

hathach commented Apr 2, 2021

Thanks for testing, seem like this is another the issue than the one is fixed in the #4538. Will reopen this

@hathach
Copy link
Member

hathach commented Aug 2, 2021

@hathach I have just noticed USB disconnects randomly while trying to debug some audio playback on RP2040. I am not doing MIDI. USB becomes disconnected, but when I reconnect, everything is fine (no crash). I will try to narrow this down later. Perhaps the USB interrupt handling is being starved or disabled due to some other operations. I am doing some DMA (audio and display), though at the time of the disconnect, there is no audio playing, and the display is not changing.

It could probably an different issue, could you provide detail to reproduce the issue, I will try to see if I could figure out something.

@dhalbert
Copy link
Collaborator

@hathach I can try testing with your UART debug output build on several different Windows machines. Would you want a Beagle trace if it does hang up?

@hathach
Copy link
Member

hathach commented Aug 11, 2021

@dhalbert A beagle trace would be great, since it capture what actually happens. UART log does show how the stacks interpret the bus event. The more info the better

PS: I am also trying to troubleshoot this, and will compile an new debug build for testing as well.

@hathach
Copy link
Member

hathach commented Aug 12, 2021

@DynamoBen @dhalbert I tried my best to update tinyusb to address this issue. However, since all my PC cannot reproduce the issue. Would you mind trying this out again. zip contains uf2 for for pico and macropad with multiple debug level (0, 2, 3) . 0 is no debug just for verifying (in case debug log accidentally resolve it by slowing down cpu)
Note: pico uart tx is GPIO0, macropad, uart tx is GPIO20 (SDA in qtstemma connector). Looking forward to your feedback.

midi-usb-connection.zip

@dhalbert
Copy link
Collaborator

I tried to reproduce on a Dell Optiplex 3020 last night with Windows 10 and failed. I will try an Intel machine with an older motherboard.

Do you have any idea what this might be related to: is it Windows driver related, or might it be more USB controller related? Do you know if USB2 vs USB3 might make a difference? I can try various permutations, but I have many choices right now

@hathach
Copy link
Member

hathach commented Aug 12, 2021

I tried to reproduce on a Dell Optiplex 3020 last night with Windows 10 and failed. I will try an Intel machine with an older motherboard.

By failed, you mean you couldn't reproduce the issue and everything works well right !?

Do you have any idea what this might be related to: is it Windows driver related, or might it be more USB controller related? Do you know if USB2 vs USB3 might make a difference? I can try various permutations, but I have many choices right now

I couldn't reproduce the issue myself, and just update code where I think may be the issue. It is hard to tell for sure, If I am right, it is kind of race condition when host issue a bus reset e.g reset duration etc .. and cause the stack (with rp2040 port) to lock up. Theoretically, there shouldn't be any differences between using usb3 or usb2 host, though we all know it is hardly true in practical, there is always some diff, which cause trigger issue especially race condition.

@dhalbert
Copy link
Collaborator

By failed, you mean you couldn't reproduce the issue and everything works well right !?

Yes, I failed to find a failure 😃 .

@DynamoBen
Copy link
Author

@DynamoBen @dhalbert I tried my best to update tinyusb to address this issue. However, since all my PC cannot reproduce the issue. Would you mind trying this out again. zip contains uf2 for for pico and macropad with multiple debug level (0, 2, 3) . 0 is no debug just for verifying (in case debug log accidentally resolve it by slowing down cpu)
Note: pico uart tx is GPIO0, macropad, uart tx is GPIO20 (SDA in qtstemma connector). Looking forward to your feedback.

midi-usb-connection.zip

I'm still seeing the issue, most interesting parts below. I used Debug level 2, because level 3 caused the USB driver to not be recognized.

`USBD Xfer Complete on EP 85 with 4 bytes
MIDI xfer callback
USBD Bus Reset : Full Speed

USBD Setup Received 80 06 00 01 00 00 40 00
Get Descriptor Device
Queue EP 80 with 18 bytes ...
USBD Xfer Complete on EP 80 with 18 bytes
Queue EP 00 with 0 bytes ...
USBD Xfer Complete on EP 00 with 0 bytes
USBD Bus Reset : Full Speed

USBD Setup Received 00 05 17 00 00 00 00 00
Set Address
USBD Xfer Complete on EP 80 with 0 bytes

USBD Setup Received 80 06 00 01 00 00 12 00
Get Descriptor Device
Queue EP 80 with 18 bytes ...
USBD Xfer Complete on EP 80 with 18 bytes
Queue EP 00 with 0 bytes ...
USBD Xfer Complete on EP 00 with 0 bytes

USBD Setup Received 00 09 01 00 00 00 00 00
Set Configuration
Open EP 81 with Size = 64
Alloced 64 bytes at offset 0x180 (0x50100180)
Open EP 02 with Size = 64
Alloced 128 bytes at offset 0x1c0 (0x501001c0)
Open EP 82 with Size = 64
Alloced 128 bytes at offset 0x240 (0x50100240)
CDC opened
Open EP 83 with Size = 64
Alloced 128 bytes at offset 0x2c0 (0x501002c0)
Open EP 03 with Size = 64
Alloced 128 bytes at offset 0x340 (0x50100340)
Queue EP 03 with 31 bytes ...
MSC opened
Open EP 84 with Size = 64
Alloced 64 bytes at offset 0x3c0 (0x501003c0)
Open EP 04 with Size = 64
Alloced 64 bytes at offset 0x400 (0x50100400)
Queue EP 04 with 64 bytes ...
HID opened
Open EP 05 with Size = 64
Alloced 128 bytes at offset 0x440 (0x50100440)
Open EP 85 with Size = 64
Alloced 128 bytes at offset 0x4c0 (0x501004c0)
MIDI opened
Queue EP 80 with 0 bytes ...
USBD Xfer Complete on EP 80 with 0 bytes
USBD Xfer Complete on EP 03 with 31 bytes
MSC xfer callback
SCSI Command: Test Unit Ready
Queue EP 83 with 13 bytes ...
USBD Xfer Complete on EP 83 with 13 bytes
MSC xfer callback
SCSI Status: 0
Queue EP 03 with 31 bytes ...
QUSBD Xfer Complete on EP 03 with 31 bytes
MSC xfer callback
SCSI Command: Read Capacity10
Queue EP 83 with 8 bytes ...
USBD Xfer Complete on EP 83 with 8 bytes
MSC xfer callback
SCSI Data
Queue EP 83 with 13 bytes ...
`

@hathach
Copy link
Member

hathach commented Aug 13, 2021

@DynamoBen the usb log look pretty normal, I should add timestamp to the log (will do it next), do you know how long does it take until the USBD Bus Reset : Full Speed occurs.

  1. Does the usb comm both msc and terminal is both freeze ?
  2. Can you still connect / disconnect terminal to cdc
  3. Is pico still recognized by windows (even msc/cdc not responding) , you could check it using the usbtreeview software https://www.uwe-sieber.de/usbtreeview_e.html#download

Look like the issue is different than what I thought it is. Somehow the windows issue bus reset, and it is not related to the physical bus like suspend/disconnect. Probably windows driver think there is an issue with cpy and try to recover by reset it, however, the issue still exist (to windows), and it probably gave up afterwards. Maybe looking at host syslog would help, though I am not sure how to do equivalent dmesg on windows.

@DynamoBen
Copy link
Author

@hathach It usually fails within 20 min, I can get some more exact timing if needed. Terminal continues after MIDI has stopped. I didn't try reconnecting the terminal. Pico is recognized still and no driver errors.

It's been mentioned by others but I can use other USB MIDI devices without failure and I can run similar test code with a Teensy with no failure. It doesn't help with this specific issue but does mean there isn't a systemic USB issue with newer Dell computers (which seems to be the common thread).

@hathach
Copy link
Member

hathach commented Aug 14, 2021

@DynamoBen device log look normal to me, I think we should look more at the host log/app. Can you tell me more about what does not work when the issue occurs. Original post said both MIDI and serial not working. Can you re-test it.

  1. Check if REPL still work
  2. Check if MSC still work
  3. What is your MIDI app for testing, maybe either try to close and relaunch it (probably a bus reset cause it to loose connection and could;t reconnect with midi port)
  4. Try to test with other MIDI app if possible.

@DynamoBen
Copy link
Author

@hathach I see this issue was closed, do you still need feedback from me? I see #1020 does that fix this issue? If so I assume I can pull latest and test it? I see a mention of some HW changes, but nothing specific to the RP2040, anything needed before testing?

@hathach
Copy link
Member

hathach commented Aug 27, 2021

@DynamoBen sorry, it is my bad, the comment on the PR close this automatically, we merged since it contains other useful code even though it still hasn't fixed this one. Please provide your feedback as mentioned in above comment for further troubleshooting.

@hathach hathach reopened this Aug 27, 2021
@tannewt
Copy link
Member

tannewt commented Aug 31, 2021

Moving this to post-7.0.0. @DynamoBen we'd like your feedback on #4190 (comment)

@tannewt tannewt modified the milestones: 7.0.0, 7.x.x Aug 31, 2021
@LibertyBeta
Copy link

LibertyBeta commented Aug 31, 2021 via email

@tannewt
Copy link
Member

tannewt commented Aug 31, 2021

I am hoping to merge #5268 for the next release.

@DynamoBen
Copy link
Author

I'm testing with latest right now, so far it's looking better. I should know tomorrow if the issues I've been seeing are resolved. I plan to do extended testing on two different Dell systems which previously exhibited the failure.

@DynamoBen
Copy link
Author

Been running continuously for days, not sure what changed but things seem to be working now.

@hathach
Copy link
Member

hathach commented Sep 12, 2021

@DynamoBen thank you for your feedback, lots of usb driver updated recently to make it compliance to USB test suite hathach/tinyusb#1059 , which help to review several corner cases bugs. It probably fixes this one as well (hopefully). When you are confident that the bug is fixed, let us know so that we could close this one.

@tannewt tannewt closed this as completed Sep 13, 2021
@DynamoBen
Copy link
Author

@DynamoBen thank you for your feedback, lots of usb driver updated recently to make it compliance to USB test suite hathach/tinyusb#1059 , which help to review several corner cases bugs. It probably fixes this one as well (hopefully). When you are confident that the bug is fixed, let us know so that we could close this one.

Looking good, I'm fine with closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment