Test Sharding for CI Matrix Purposes

[amerryma]

What are you trying to achieve?

I'd like to be able to use CodeceptJS to run tests on multiple machines in a CI environment without needing to use grep or manually pick which tests run on which machines. You can read more about this in a few other test runners such as Playwright and Codeception .

So if we had 100 tests, and I split into groups of 4, then tell one machine to execute group 1, it should execute 25 tests. One method of determining which test are part of which groups are simply running every 4th test.

We can then use the matrix strategy in Github Actions to execute each shard on a different machine in parallel.

I've spent some time researching and developing a solution that could make this work. Essentially, we would need to update Mocha to support this and then forward through the option from CodeceptJS. I've proposed the feature be added to Mocha (mochajs/mocha#4958) and we would need to wait for them to consider adding the feature before we could do it here.

I would be responsible for building out the feature should we decide this would be a good addition to CodeceptJS.

It is possible that Codecept implements this feature themselves without using Mocha so I wouldn't mind having a conversation about that.

Thank you for your consideration!

[alaz-aura]

Out of curiosity, could you not achieve this with current codeceptjs? You could utilise test suite tagging to split up your run commands across different machines or providing multiple codecept.conf.ts files with each containing different test suite paths to execute.

Just thinking if you could achieve your desired goal albeit by slightly different means?

[amerryma]

@alaz-aura That is a good idea. We were hoping it wouldn't need to be so explicit as to overburden the developers but this can work in the meantime. I have continued my work within the Mocha repository so I'm hoping we can still get something like this in. I'll keep this issue updated with my progress.

[alaz-aura]

Glad I could help @amerryma, interested to see how it turns out. I'm also curious what benefit this would give over running all tests over a single machine. I suppose test suite duration would benefit from this split up. But I probably haven't encountered such a large suite requiring it to be split up in such a way

[amerryma]

Yes, we have over 150 e2e tests that take quite a bit of time to run, they can't run in parallel. We run them on every commit for a PR. They take about 40 minutes to run, but if we split it up to 4 runners, then it takes only 10 minutes total. My compute costs don't change of course.