Hardware in The Loop (HITL) setup

[perigoso]

I think this project could really benefit from a Hardware In The Loop setup (think of it like a workflow to test the code, but with actual hardware)

There's a lot that goes into this, just leaving this here for documentation and future discussion

[hathach]

I thought about this long time ago, but couldn't manage time to do so. We can write a python script to run on a pi4 to deploy and test much like production tester

• to flash example and test if new firwmare run OK like Serial,MSC is enumerated correctly.
• USB Compliance Verification Test Suite (USBCV) #1059 USBCV auto test would be great, unfortunately, the suite only runs on windows which is hard to automate and script, but also cost more in electricity bill.

Currently tinyusb supports too many platforms, we probably only pick up a few common mcu like pico which can be flashed using msc uf2 without jlink.

[perigoso]

This is something I would love to play with, there's a personal project that could benefit from a similar system, unfortuantely I don't have that much time either, but this is something we can gradually build.

It's unfortunate USBCV is windows only, didn't realise it, it would be one of the more useful things we could use this for. maybe making a similar test tool might be feasible, provided someone finds time to do it.

[hathach]

more update (before I forgot): hardware in the loop may be possible with github self-hosted

• https://docs.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners
• https://docs.github.com/en/actions/hosting-your-own-runners/using-self-hosted-runners-in-a-workflow

basically, we will have to setup an local machine and install an github self-host app to connect to github. It will run as an ci instance, we could attach physical device then check the enumeration of new firmware e.g dual cdc example should have both /dev/ttyACM0 & ttyACM1 appeared etc .. USBCV may be possible as well with self-host is a windows with decent script to launch usbcv and get its result.

[pigrew]

I think it would be a great thing to do. I have a python script that I run with my USBTMC class driver that verifies a number of conditions (but certainly could be expanded). Using libusb probably would help, as we could more easily test various corner cases, like aborts in the middle of transmissions. (In fact, I lately found a few bugs using my test script...).

It may also be useful to develop a special class for testing the USB peripheral driver and USBD core. A custom class likely is needed if we want to test all four endpoint types, setting configurations, the various control transfers, etc.

[pigrew]

Self Hosted runners

Based on the documentation, this sounds like a questionable idea. The docs suggest only using this for private repositories... I think our "workflow" currently runs when anyone creates a PR, so that could overload the private system, or folks even could run malicious code on the microcontroller... I don't know if there is a way to restrict the workflow to be only when maintainers approve?

Even if it's not automatically run, the hardware-rich testing is a good idea.

[hathach]

we can limit the condition to run the self-host e.g only run when master is updated and/or when committed by contributor (with write permission etc..), the rest of the flow can just run as it is.

[hathach]

part of the HITL is implemented in #1724, It works with an RPI4 + pico running several device examples. The script does try to interact with device image correctly e.g dual cdc, cat msc's readme, run dfu-util with dfu examples etc... It is neither perfect nor complete, but is a good start.

Note: due to limited resources on Pi4, currently only code committed by me (or after merged to master) could run the self-hosted instance

https://github.com/hathach/tinyusb/blob/master/.github/workflows/build_arm.yml#L166

[HiFiPhile]

@hathach I can provide a VM of my home server with a dedicated USB card if we go further :)
Performance won't be an issue as I've a EPYC 7232 with 128GB RAM.
I have bunch of dev boards with on-board debugger like LPCXpresso43xx or STM32F3-Discovery which can be used.

[hathach]

@HiFiPhile Thank you, that is a great computing power !! I will email you the token and setup steps to register an self-hosted instance. For dev boards, feel free to attach any boards that are redundant and/or one with same MCUs that your product is using. That will at least help to verify latest code with the mcu you are interested with.

PS: you could terminate the VM anytime you like, just let me know to disable the workflow 😉

[HiFiPhile]

@hathach

I'd like to add a weak callback in tud_descriptor_string_cb() to allow MCUs to use Chip UID as serial number if available, which makes identify multiple devices easier, what do you think ?

[hathach]

@HiFiPhile the callback is already existed, did you mean you want to implement it.

tinyusb/examples/device/cdc_msc/src/usb_descriptors.c

Line 254 in 081e4ee

uint16_t const* tud_descriptor_string_cb(uint8_t index, uint16_t langid)

Yeah, I am planning to add and board_get_uid()/board_get_usb_serial() similar to this implementation of tinyuf2 here, but always got too busy with other works.

https://github.com/adafruit/tinyuf2/blob/4fb0d86fe82659d599b034864f8b1aee1656de75/src/usb_descriptors.c#L190

[HiFiPhile]

Yeah, I am planning to add and board_get_uid()/board_get_usb_serial() similar to this implementation of tinyuf2 here, but always got too busy with other works.

Yes it's what I'm talking about :)

[hathach]

hardware already in the loop with rp2040 pico and stm32l4 board running with self-hosted ci. Due to limited of hardware power, we cannot test every mcu. This issue is therefore is closed as complete