Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge #29771: test: Run framework unit tests in parallel
f19f0a2 test: Run framework unit tests in parallel (tdb3) Pull request description: Functional test framework unit tests are currently run prior to all other functional tests. This PR enables execution of the test framework unit tests in parallel with the functional tests, rather than before the functional tests, saving runtime and more efficiently using available cores. This is a follow up to #29470 (comment) ### New behavior: 1) When running all tests, the framework unit tests are run in parallel with the other tests (unless explicitly skipped with `--exclude`). This parallelization introduces marginal time savings when running all tests, depending on the machine used. As an example, a 2-3% time savings (9 seconds) was observed on a machine using `--jobs=18` (with 18 available cores). 2) When running specific functional tests, framework unit tests are now skipped by default. Framework unit tests can be added by including `feature_framework_unit_tests.py` in the list of specific tests being executed. The rationale for skipping by default is that if the tester is running specific functional tests, there is a conscious decision to focus testing, and choosing to run all tests (where unit tests are run by default) would be a next step. 3) The `--skipunit` option is now removed since unit tests are parallelized (they no longer delay other tests). Unit tests are treated equally as functional tests. ### Implementation notes: Since `TextTestRunner` can be noisy (even with verbosity=0, and therefore trigger job failure through the presence of non-failure stderr output), the approach taken was to send output to stdout, and forward test result (as determined by `TestResult` returned). This aligns with the previous check for unit test failure (`if not result.wasSuccessful():`). This approach was tested by inserting `self.assertEquals(True, False)` into test_framework/address.py and seeing specifics of the failure reported. ``` 135/302 - feature_framework_unit_tests.py failed, Duration: 0 s stdout: .F ====================================================================== FAIL: test_bech32_decode (test_framework.address.TestFrameworkScript.test_bech32_decode) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/dev/myrepos/bitcoin/test/functional/test_framework/address.py", line 228, in test_bech32_decode self.assertEqual(True, False) AssertionError: True != False ---------------------------------------------------------------------- Ran 2 tests in 0.003s FAILED (failures=1) stderr: ``` There was an initial thought to parallelize the execution of the unit tests themselves (i.e. run the 12 unit test files in parallel), however, this is not anticipated to further reduce runtime meaningfully and is anticipated to add unnecessary complexity. ACKs for top commit: maflcko: ACK f19f0a2 🌽 achow101: ACK f19f0a2 kevkevinpal: Approach ACK f19f0a2 Tree-SHA512: ab9f82c30371b2242bc7a263ea0e25d35e68e2ddf223d2a55498ad940d1e5b73bba76cce8b264d71e2ed31b753430d8ef8d57efc1e4fd9ced7fb845e27f4f47e
- Loading branch information