[Feature Request]: Hardware in the loop testing

[jkeilson]

Platform

NRF52, ESP32

Description

It seems like it would be a good idea to do something around hardware-in-the-loop testing, to enable new releases to have some amount of testing on real hardware before they go out into the wild. I'm setting out a plan for how I might tackle this, but I'm open to feedback and modification. It may be worth breaking down into multiple issues. It's also worth noting that in the long term, this will need some community support, in the form of volunteers with different boards and configurations willing to run a github local runner.

If you don't believe this adds value, or you don't think the community is willing and able to support this, please feel free to close this.

• Read/understand the current CI/CD structure.
• Figure out how to run a github local runner that has access to hardware.
• Create a test in github CI that can call the local runner and flash firmware to a board.
• Create a method to automate first-time configuration.
• Parameterize appropriately so that it can scale out to different runners with different boards.
• Document the heck out of this so that new people can follow a script to add new boards to the fleet.
• Onboard additional volunteers based on the above.
• Investigate extending the tests to cover more functionality. For example, a really useful test would be to flash two boards with new firmware, start the mesh, and send a message from each and confirm it was received by the other.