chore: add e2e tests using cypress

[jooola]

This allow us to write e2e tests for the current legacy project. The future web app should have those tests embed inside the webapp project.

We previously were running the tests by hand on the vagrant install, we should write them to cypress tests and automate the testing process.

How this works:

• Setup a vagrant instance vagrant up bullseye
• Go in the e2e directory and install cypress cd e2e && yarn
• Open the Cypress dashboard yarn start
• Follow instruction on the cypress dashboard to run the tests.

[jooola]

Maybe I should move thoses files to legacy/e2e.

[paddatrapper]

This is great. How does Cypress compare to Selenium?

[jooola]

How does Cypress compare to Selenium?

Cypress is way easier to use. From what I read in the selenium tests you have a lot of waiting code while you don't need it in cypress. If you have experience with Selenium, I suggest you to write a test and you will see how easy it is !

[jooola]

This is blocked because we need a way to run a libretime inside the CI, and this is only doable with the docker setup.

Once we can run LibreTime inside the CI, we can start implementing more tests for the legacy app.

[codecov]

Codecov Report

Merging #1896 (bf78edb) into main (bcddc90) will decrease coverage by 11.33%.
The diff coverage is n/a.

❗ Current head bf78edb differs from pull request most recent head e74b698. Consider uploading reports for the commit e74b698 to get more accurate results

@@             Coverage Diff             @@
##             main    #1896       +/-   ##
===========================================
- Coverage   69.96%   58.64%   -11.33%     
===========================================
  Files         147      186       +39     
  Lines        3972     4686      +714     
===========================================
- Hits         2779     2748       -31     
- Misses       1193     1938      +745     

Flag	Coverage Δ
analyzer	46.72% <ø> (-0.19%)	⬇️
api	93.88% <ø> (+0.19%)	⬆️
api-client	40.80% <ø> (-23.60%)	⬇️
playout	20.20% <ø> (-27.47%)	⬇️
shared	92.68% <ø> (+4.26%)	⬆️
worker	50.54% <ø> (-38.56%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 98 files with indirect coverage changes

[jooola]

Cool, so I have some basic e2e tests running in CI, and it is working nicely !

We currently only test the main branch, meaning that the pull requests only trigger the CI e2e tests if the tests files are changed, not the application code.

I am not sure if we want to test pull requests with the e2e tests. It will take ages every time to first build the containers, save the artifacts, pull them in another workflow and run them. Maybe we should improve the container workflow first and see if we can export the build images for reusing them for e2e and other advanced tests.

[paddatrapper]

Yeah I think we can test PRs at a later stage once we've improved that workflow a bit