[FEATURE] speed up CI tests

[VukW]

Is your feature request related to a problem? Please describe.

Now running all the tests takes around ~30 mins. I understand we run a few training loops (on CPU) during tests; however, maybe it is still possible to speed them up?

Describe the solution you'd like

• Profile tests:
  • Are there any specific tests that takes bigger part of time or do all tests have relatively the same amount of time?
  • Profile slow tests: is it just a training loop that takes time? or do we make any additional transformations or something else?
  • Tune slow tests: reduce dataset size? reduce number of epochs? something else?

Crucial is to keep the same tests coverage while tuning tests.

[sarthakpati]

It is definitely possible to speed the tests up, but we need to be cognizant about the possibility that it could come at a cost (for instance, in terms of code coverage or test readability).

For example, right now, the tests are triaged in terms of data type (rad/histo images), data dimensionality (2d/3d), workload (segmentation/classification/regression). and so on. Some of these can be combined, such as the tests for the schedulers/optimizers can be combined with one of the data types (for e.g., 2d rad), in which case there would be 1 (large) test that would be providing coverage for huge swaths of the code. It would work in terms of code coverage, but would degrade the readability of the testing code.

There is definitely a better solution for our testing, though. There is quite a bit of redundancy (multiple rad/histo tests with 2d/3d) where the same part of the code gets traversed. Will definitely need someone with a good QA bent of mind to tackle this. 😉

[sarthakpati]

One idea could perhaps be to split each of the test "categories" into separate workflows. This won't reduce the test time for local runs, but it will parallelize the CI for PRs.

[github-actions]

Stale issue message