Add automated tests?

[HonkingGoose]

What would you like to be able to do?

When I made PR #97, I forgot to check the display of our config presets page. This means we deployed a change that was not ready for production. 😞

It easy to miss bad changes like this when testing manually. It also means that updates to Material for MkDocs are risky.

I would like to have some automated tests that check for visual changes, so that unintended visual regressions are caught by the machine.

I would also like to have some basic tests that check if things like search, sidebar items, previous page/next page button, light/dark mode all work properly.

Did you already have any implementation ideas?

We could probably use Cypress ¹ to test the functionality of our docs pages.

I'm not really familiar with any way to test for visual changes yet.

Footnotes

1. https://www.cypress.io/ ↩

[RahulGautamSingh]

Hey @HonkingGoose, Can I work on implementing automated tests using Cypress on this one? I am fairly new to front-end testing and this seems to be a good task for me to learn more.

[viceice]

Sure, go ahead.

[RahulGautamSingh]

I have added cypress's basic config in a PR and will add new tests for most basic functionality like:

• search
• sidebar items,
• previous page/next page button
• light/dark mode
  Will try to add 2-3 tests in each PR.

@HonkingGoose Do we need to make any updates to docs regarding the cypress testing?

I think we can add this:

To write more tests using cypress, follow these steps:

• Install and start Cypress on your PC as per the docs
• Initiate and use the renovate submodule help
• write new tests or update the old ones

[HonkingGoose]

I have added cypress's basic config in a PR

Cool!

I think we should install Testing Libary as a development dependency as well. It makes writing tests easier. But I don't know how to set up Testing Library so it works with Cypress. 😞

@Belco90 are you willing to help us with Testing Library? If not that's fine as well. 😉

[Belco90]

@Belco90 are you willing to help us with Testing Library? If not that's fine as well. 😉

Sure! It should be a matter of installing Cypress Testing Library and then extending cy commands by importing the corresponding module. You can find its setup instructions here: https://testing-library.com/docs/cypress-testing-library/intro

Feel free to reach out if you struggle to set it up!

[HonkingGoose]

Someone suggested adding Cypress tests to the upstream mkdocs-material project. The maintainer of mkdocs-material says: ¹

Thanks for suggesting! I've thought on and off about adding end-to-end tests to this project, but my conclusion is that the project is so amazingly complex that E2E tests would barely help to secure the quality of this project. The reason is that we have dozens of configuration options that significantly alter the HTML and/or behavior of the resulting documentation page, which we'd need to test in all combinations. For instance, we have support for 58 languages – we would need to define a test case for each of those languages to make sure that at least search works. This is impractical from a maintenance and runtime perspective.

Furthermore, implementations between the community edition and Insiders sometimes differ, which means even more effort would be needed. As much as I want to add tests, I've decided at some point that it's currently not feasible with the time I have to work on this project. Also, when we implement tests, they would need to be maintained for which I don't have the time to do – it's already hard for me to keep up with bug fixing, feature development and community management.

Closing as (currently) not planned.

It sounds to me like adding, or even using, Cypress tests is the wrong way to go for us. Even the maintainer of the project doesn't see how it can work for them. 😄

I'll let @rarkins decide what's best, based on this information.

Footnotes

1. https://github.com/squidfunk/mkdocs-material/issues/4266#issuecomment-1229157966 ↩

[HonkingGoose]

We now also benefit from tests built into mkdocs:

• anchor link checking
• relative link checking
• and we use the strict mode of mkdocs, so we don't publish a bad version of the docs

So I'm thinking of closing this issue as done enough.