Detailed PX4 CI documentation #2284

junwoo091400 · 2023-02-17T16:22:13Z

About

More rigorous documentation actually detailing what goes behind the curtain on PX4 CI system is added.

Motivation

This was one of the most magical things I learned when I was working on Frames metadata support in QGC: mavlink/qgroundcontrol#10472

In hopes of having this mysterious CI system getting better documented, I created this PR!

en/test_and_ci/continous_integration.md

junwoo091400 · 2023-02-17T16:27:10Z

en/test_and_ci/continous_integration.md

+- `mavros_mission_tests.yml`: Executes Mission plans using MAVROS. It can check for functional compatibility between MAVROS and PX4
+- `checks.yml`: Runs various make targets (using `make` command). The commands for each targets are documented in the [Makefile](https://github.com/PX4/PX4-Autopilot/blob/main/Makefile)
+  - `check_format`: Checks the coding style format using Clang Tidy ?


@dagar Is this different from the clang-tidy.yml part below?

This one uses astyle

@MaEtUgR This section might be affected by linter changes

@hamishwillee Certainly, thanks for the heads-up. We won't change it over night. Style is currently exclusively checked by astyle, see https://github.com/PX4/PX4-Autopilot/blob/3b2d76657312d47e6495736dfd4d694b546497d5/Tools/astyle/fix_code_style.sh#L8-L11

OK, so this is good for now. Thanks!

en/test_and_ci/continous_integration.md

hamishwillee · 2023-02-22T01:47:09Z

en/test_and_ci/continous_integration.md

+## Other
+
+Jenkins also updates the PX4 ROS messages repository [PX4/px4_msgs](https://github.com/PX4/px4_msgs) and uploads the metadata to the [Amazon S3 bucket server](https://px4-travis.s3.amazonaws.com/).


"uploads the metadata to the Amazon S3 bucket server." - what metadata? YOu already covered a lot here, is this metadata something associated with px4_msgs?

Ooh I think so, I didn't cover it specifically but can add that info as well

If it is metadata associated with ROS 2 messages then "probably not". The only dependency now from PX4 in ROS2 is the PX4/px4_msgs. There used to be more stuff, but XRCE-DDS means it isn't needed any more.

hamishwillee · 2023-02-22T01:49:44Z

@junwoo091400 I've restructured this a bit. All the stuff specific to QGC and PX4 docs should go under those sections. So you can talk about what PX4 does with actions and Jenkins in those folders, and there might end up being some duplication in the QGC and Docs sections.

It is not clear to me how Jenkins is kicked off? Is this magic or is it done by deploy_all action? In other words what is the relationship/handover, if any between the systems?

I can't review this for technical accuracy. Who can?

hamishwillee · 2023-02-22T01:52:02Z

PS The Jenkins stuff doesn't happen by default - it has been broken for months and months. I kick it off manually every week or so. No one seems to care.

junwoo091400 · 2023-02-22T06:12:10Z

It is not clear to me how Jenkins is kicked off? Is this magic or is it done by deploy_all action?

This I have no clue for now. I can ask @dagar

Jenkins stuff doesn't happen by default - it has been broken for months and months

I'm sorry that this has been happening. I was aware & wanted to at least figure out for myself what was going wrong, which was partly the motivation for this PR.

I will ask about Jenkins situation in the maintainers chat 🙏. I will help getting this resolved!

en/test_and_ci/continous_integration.md

hamishwillee · 2023-03-01T00:28:17Z

I will ask about Jenkins situation in the maintainers chat 🙏. I will help getting this resolved!

Thanks. @dagar is probably the only one who can fix it.

hamishwillee · 2023-03-15T04:58:20Z

@junwoo091400 Is this waiting on review? In that case, perhaps we could ask @bkueng for a sanity check.

junwoo091400 · 2023-03-15T06:04:52Z

Yes that makes sense!

en/test_and_ci/continous_integration.md

bkueng · 2023-03-16T06:53:19Z

en/test_and_ci/continous_integration.md

+- `mavros_mission_tests.yml`: Executes Mission plans using MAVROS. It can check for functional compatibility between MAVROS and PX4
+- `checks.yml`: Runs various make targets (using `make` command). The commands for each targets are documented in the [Makefile](https://github.com/PX4/PX4-Autopilot/blob/main/Makefile)
+  - `check_format`: Checks the coding style format using Clang Tidy ?


This one uses astyle

en/test_and_ci/continous_integration.md

bkueng · 2023-03-16T06:56:55Z

en/test_and_ci/continous_integration.md

+- `ekf_update_change_indicator.yml`: Check if there's any functionality change in EKF ?
+- `failsafe_sim.yml`: Builds [failsafe simulator](../config/safety_simulation.md)
+- `clang-tidy.yml`: Runs Clang Tidy to check if coding style is consistent


This also runs static analysis to detect problematic code patterns (e.g. dead code).

bkueng · 2023-03-16T06:59:46Z

en/test_and_ci/continous_integration.md

+- Events metadata (`events/*.xz`)
+- Actuators metadata (`actuators.json*`)
+- Target binary (`*.px4`)


It's only metadata. compile_nuttx uploads the binaries, but I'm not sure how they get to s3.

Suggested change

- Target binary (`*.px4`)

Turns out, they are uploaded via the "PX4_misc/Firmware_s3_deploy_*" pipelines that seems to be triggered manually! (right? @dagar)

https://github.com/PX4/PX4-user_guide/pull/2284/files#diff-7a54d71845d3b204f4ac49e9f2f65213d70376270e2856b33257405ca349cd7bR110-R112

en/test_and_ci/continous_integration.md

junwoo091400 · 2023-03-24T13:53:42Z

en/test_and_ci/continous_integration.md

+- [.github/workflows/](https://github.com/PX4/PX4-Autopilot/tree/main/.github/workflows).
+
+The Jenkins build status can be viewed in [ci.px4.io:8080/job/PX4/](http://ci.px4.io:8080/job/PX4/)


I just discovered that there's 2 Jenkins Server:

http://ci.px4.io/

http://px4-jenkins.dagar.ca:8080/blue/organizations/jenkins/PX4-Autopilot/activity

@dagar What is the difference between them? And why do we need two separate Jenkins server?

It's just an optimization because Jenkins transfers quite a lot of data between the server and runners and it's also a bit more reliable overall. For the hardware test rack each board is Jenkins slave, but it's just a container all on the same physical system with the Jenkins server.

Ah so you mean your Jenkins is the hardware rack testing & px4 Jenkins (CI.px4.io) is for everything else? (Metadata deployment, world magnetic model update, etc)

I think more information about test rack and how it runs can be added here. If it is mentioned anywhere in docs a link might suffice

junwoo091400 · 2023-04-14T06:35:01Z

en/test_and_ci/continous_integration.md

+- Target binary (`*.px4`)
+
+These files are uploaded to the [Amazon S3 bucket server](https://px4-travis.s3.amazonaws.com/).


Note, part where QGC fetches .px4 files from the S3:

https://github.com/mavlink/qgroundcontrol/blob/f9f04ecbbdabb7d2a160f7ab9152fbe45ef4d5cb/src/VehicleSetup/FirmwareUpgradeController.cc#L356-L362

Example link: https://px4-travis.s3.amazonaws.com/Firmware/stable/px4_fmu-v5x_default.px4

Also, the mechanism how QGC determines which version is stable, and beta is via this:

https://github.com/mavlink/qgroundcontrol/blob/f9f04ecbbdabb7d2a160f7ab9152fbe45ef4d5cb/src/VehicleSetup/FirmwareUpgradeController.cc#L654-L671

Basically thru this link: https://api.github.com/repos/PX4/Firmware/releases

It returns a JSON file that includes the versioning info

hamishwillee · 2023-04-20T03:36:34Z

Note, I am not tracking this. @junwoo091400 when you have resolved issues with @bkueng et al, come back to me and I will subedit.

en/test_and_ci/continous_integration.md

junwoo091400 · 2023-11-16T09:03:34Z

Updated the PR. Two remaining notes:

@dagar Seems like test rack CI is not running recently (jenkins/continuous-integration one that used to fail a lot). Was that disabled?
As I reported in PX4 Metadata upload to S3 isn't being separated to appropriate releases PX4-Autopilot#22383, currently metadata isn't being uploaded into appropriate directories correctly so I didn't go in detail regarding that mechanism in the docs, since it's not fixed yet (still docs is true to code's status)

junwoo091400 · 2023-11-18T09:51:56Z

en/test_and_ci/continous_integration.md

+
+Although referred to as "Jenkins", PX4 has two Jenkins servers, and multiple pipelines running in parallel. Here are the summary of most pipelines:
+
+### Non-hardware Jenkins
+
+A Webhook is configured to [ci.px4.io/](http://ci.px4.io/) for every event of Pushes (new commits, branches) and Pull Requests in PX4-Autopilot repository.
+
+| Pipeline Name | Description | Trigger | Script |
+|---|---|---|---|
+| [PX4_misc/*](http://ci.px4.io/job/PX4_misc/) | These are helper pipelines for once-in-a-while, miscellaneous tasks needed for PX4 ecosystem, and the scripts are not documented within the PX4-Autopilot repository. | NA | NA |
+| [PX4_misc/Firmware-compile](http://ci.px4.io/job/PX4_misc/job/Firmware-compile/) (On Github shown as **Jenkins Compile All Boards**) | Builds specific list of board targets to check if they all compile. | Pushes & Pull Requests (all Webhook events) | [.ci/Jenkinsfile-compile](https://github.com/PX4/PX4-Autopilot/blob/main/.ci/Jenkinsfile-compile) |
+| [PX4_misc/Firmware_update-world_magnetic_model](http://ci.px4.io/job/PX4_misc/job/Firmware_update-world_magnetic_model/) | Create PR updating world magnetic model data | Manual | In Jenkins |
+| [PX4_misc/Firmware_update_nuttx_kconfigs](http://ci.px4.io/job/PX4_misc/job/Firmware_update_nuttx_kconfigs/) | Create PR updating inconsistent NuttX KConfig files | Manual | In Jenkins |
+| [PX4_misc/Firmware_update_submodules](http://ci.px4.io/job/PX4_misc/job/Firmware_update_submodules/) | Create PRs updating the submodules to latest | Manual | In Jenkins |
+| [PX4/*](http://ci.px4.io/job/PX4/) | Main CI pipelines for PX4 ecosystem | NA | NA |
+| [PX4/PX4-Autopilot](http://ci.px4.io/job/PX4/job/PX4-Autopilot/) (On Github shown as **continuous-integration/jenkins/branch or pr-head**) | Updates the `master` metadata files in the S3 bucket | Push to `main` | [Jenkinsfile](https://github.com/PX4/PX4-Autopilot/blob/main/Jenkinsfile) |
+| [PX4/PX4-user_guide](http://ci.px4.io/job/PX4/job/PX4-user_guide/) | Builds the Html version of the user guide from the markdown format and deploys to the web | Updates in `main` (latest) or `v1.*` (past releases) branches | [PX4-user_guide/Jenkinsfile](https://github.com/PX4/PX4-user_guide/blob/main/Jenkinsfile) |
+
+For the full view of all different pipelines, visit [ci.px4.io/](http://ci.px4.io/).
+
+### Hardware testing Jenkins
+
+A Webhook is configured to [px4-jenkins.dagar.ca](http://px4-jenkins.dagar.ca:8080/) for every event of Pushes (new commits, branches) and Pull Requests in PX4-Autopilot repository.
+
+| Pipeline Name | Description | Trigger | Script |
+|---|---|---|---|
+| [PX4-Autopilot](http://px4-jenkins.dagar.ca:8080/job/PX4-Autopilot/) (On Github shown as **continuous-integration/jenkins/branch**) | Runs hardware test script | Pushes & Pull Requests (all Webhook events) | [.ci/Jenkinsfile-hardware](https://github.com/PX4/PX4-Autopilot/blob/main/.ci/Jenkinsfile-hardware) |
+
+This CI is crucial for detecting failures in hardware that can't be detected via Software CI tools, such as hardfault, NuttShell, NuttX bugs, etc.
+
+To view all the past runs of the hardware rack CI, visit [px4-jenkins.dagar.ca](http://px4-jenkins.dagar.ca:8080/job/PX4-Autopilot/).


I've added this table because I thought the list view (still kept, but commented out below) was confusing, would like to know your thoughts on this! @hamishwillee

junwoo091400 · 2023-11-18T09:56:09Z

en/test_and_ci/continous_integration.md

+An example commit of parameter metadata can be found [here](https://github.com/mavlink/qgroundcontrol/commit/7f4f3b6253fe80a881e9a91a1f5b6d960ad11834).
+
+Note that QGroundControl normally gets metadata for [Events](../concept/events_interface.md#implementation)and [Parameters](../advanced/parameters_and_configurations.md#publishing-parameter-metadata-to-a-gcs) from the vehicle (it is built into the firmware, or a URL is provided in the firmware from which it can be downloaded).


Not sure if it is true to say that 'URL is provided in the firmware from which metadata can be downloaded'? @bkueng

junwoo091400 · 2023-11-18T10:22:15Z

en/test_and_ci/continous_integration.md

+
+- `cubepilot_cubeorange_test`: Cubepilot CubeOrange
+- `cuav_x7pro_test`: CUAV X7 Pro
+- `px4_fmu-v4_test`: FMU v4
+- `px4_fmu-v4pro_test`: FMU v4 Pro
+- `px4_fmu-v5_debug`: FMU v5 with debug flag, prints out debug information
+- `px4_fmu-v5_stackcheck`: FMU v5 compiled with stackcheck, detects stack overflow, etc.
+- `px4_fmu-v5_test`: FMU v4
+- `nxp_fmuk66-v3_test`: FMU k66 v3
+
+Note that `_test` label for the target means ... (TODO)
+
+It performs the following tests on each build node targets:
+
+- Build: Build bootloader and firmware binary files
+- Flash: Flash the hardware via USB, JLink, and other methods (specific to hardware)
+- Tests: Various tests regarding sensors, commander module, uorb topics, etc.
+- Status: Reboot and check filesystem (/proc, /dev, etc), module, system commands, and do quick IMU calibration
+- Print topics: Print out selected set of uORB topic data (for debugging purposes)
+
+This CI is crucial for detecting failures in hardware that can't be detected via Software CI tools, such as hardfault, NuttShell, NuttX bugs, etc.
+
+For the full test script, visit [.ci/Jenkinsfile-hardware](https://github.com/PX4/PX4-Autopilot/blob/main/.ci/Jenkinsfile-hardware).


Added general information about the hardware rack here. Anything else you want me to add? @mnumanuyar

junwoo091400 · 2023-11-18T10:22:56Z

en/test_and_ci/continous_integration.md

+- `nxp_fmuk66-v3_test`: FMU k66 v3
+
+Note that `_test` label for the target means ... (TODO)


@bkueng I actually don't know how "stackcheck", "test", etc labels get incorporated into the build process. What does '_test' postfix do regarding the built target binary?

It's just different build variants for selecting a certain sets of modules: the label defines which .px4board file is used under the board directory.

Ahaa true, I forgot about that! Thanks for the pointer 👍

More rigorous documentation actually detailing what goes behind the curtain on PX4 CI system. Hope this helps someone!

Co-authored-by: Junwoo Hwang <junwoo091400@gmail.com>

Co-authored-by: Beat Küng <beat-kueng@gmx.net>

Clarify that Firmware and Metadata deployments are handled separately, from different entities (for now): Github Actions and Jenkins Also, add further feedback from the comments from the PR

hamishwillee · 2023-11-23T05:40:29Z

@junwoo091400 Just a reminder to me when you're ready for final review. That should be after Beat and Daniel say they are happy enough for this to be published.

junwoo091400 · 2023-11-23T05:58:01Z

@junwoo091400 Just a reminder to me when you're ready for final review. That should be after Beat and Daniel say they are happy enough for this to be published.

I would hold it a bit for now, as we just had discussion on the overall CI infrastructure and there are some improvements we can have, and change the Docs appropriately.

github-actions · 2023-11-23T09:53:20Z

No flaws found

junwoo091400 commented Feb 17, 2023

View reviewed changes

en/test_and_ci/continous_integration.md Outdated Show resolved Hide resolved

junwoo091400 commented Feb 17, 2023

View reviewed changes

junwoo091400 requested a review from hamishwillee February 17, 2023 16:27

junwoo091400 changed the title ~~Add PX4 CI build documentation~~ Detailed PX4 CI documentation Feb 17, 2023

hamishwillee reviewed Feb 22, 2023

View reviewed changes