End-to-end testing for openHAB 4.0? #1501
Replies: 5 comments
-
|
I'd volunteer except for the "some understanding of JS and CSS selectors". I don't even know what a CSS selector is. I definitely could manage and shepherd but probably not contribute tests in a significant way. My time is also somewhat limited with rewriting my rule templates and soon I need to return to the docs so they are ready for a 4.0 release. The Getting Started Tutorial in particular is showing it's age and the rules docs rewrite needs to be picked back up again too. What are some of the limits of this sort of testing? There are lots of things that have all sorts of dependencies in OH. Obviously stuff that depends on add-ons with hardware requirements (e.g. USB transceiver) won't be covered. Anything else? Can the tests be modularized? For example if I want to test out creating a Blockly rule I'll need to install the JS Scripting add-on which is a test unto itself. Can the Blockly rule test call the install add-on test to get the add-on installed or does each test need to define everything from start to finish? What is the hotlist of things that need to be tested? These sorts of tests could easily dwarf the actual code that makes up OH itself to cover everything. Do we have a short list of tests where errors are most likely to occur or where most of the changes between OH 3 and OH 4 are included? Without some support from the maintainers we will be constantly chasing changes even more so than what we have to do with the docs. Do we envision these remaining up to date forever, or are we mainly looking at these to get us past the 4.0 release and then let them go until OH 5? I really like the idea of automated testing. I'm cautious about their value though if the maintainers are not on board and if we don't have an idea where they will do the most benefit. We won't be able to reach anything like 100% coverage so we need to focus our efforts. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks Rich for your input. I'm not pushing or anything, I realize this takes up everybody's time and nobody is entitled to ask anything of anybody. What I was after with this initiative initially was to ensure a good release, but it kind of evolved into an idea to ensure the community could "keep checks" on what the maintainers, in particular when regressions or breaking changes are involved, are doing. But it could actually be both - like, don't make a release until the e2e tests pass. For instance, I could see a test going "I create this thing, the items and links associated, and I create a UI page and add a card to display this one item, and then go see what the UI card looks like, and it shows 360.0 instead of 360 W which was what I expected and actually got before, so it's a regression."
Yes the stuff with hardware requirements obviously wouldn't be covered by these tests, the goal is to test what can be tested without, like what happens in core, and what impact it has on the REST API, and the UIs (Main UI and Basic UI). Things that could be added without dependencies (HTTP to to well-known website, Astro with a prefedined location, etc.) would be candidates for conducting tests, if they involve Things at all.
Yes! Although going through the setup wizard (https://cloud.cypress.io/projects/quu4in/runs/20/specs/933ac166-4baf-4341-8672-47ca934b1588/video) is part of one those tests, and it involves installing some add-ons (including the JS Scripting add-on: https://github.com/ghys/openhab-e2e/blob/9fbf9d4917de8bc9c5ad7b2f252d0295798817a5/cypress/e2e/setup/1-setup-wizard.cy.js#L76-L84), so you would have this as part of the baseline for the tests. If another add-on is required, we could either add it in the "setup wizard" test case or have some helper function - or even a test which goes through the add-on store, installs the add-on before conducting a test, then uninstall it to clean up after itself (this is an important part).
This is a good and critical question. This is perhaps where the community would be involved. The goal is to make sure 1) a new user would have a good experience, 2) an experienced user would not stumble upon something that wasn't caused by an add-on's implementation but somewhere in the overall core/API/UIs stack.
This doesn't have to be. The e2e tests would be critical only in the buildup to a release, and even, them passing could even be a condition to said release if we have an understanding. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @ghys, I very much like the idea and in my projects at work we were using Cypress for tests with very good results (i.e. stability of the application). I am not convinced that we should start with a "tests only need to be green before a release" (i.e. twice a year) approach. From my experience, the tests need continuous maintenance and neglecting them will lead to a mess before the release - nobody really knows, what is broken and what has been changed intentionally. |
Beta Was this translation helpful? Give feedback.
-
Yes, I'm convinced too; I'm familiar with collaboration between testing and engineering teams in a professional environment because everyone's paid to work together. How it could translate to a project where everyone contributes what they wish, and nobody is entitled to ask anything of anybody, that I don't know. My idea was to let the community take complete charge of these tests, if they're interested, i.e. it would be a separate project and there wouldn't be as much constraints for contributing as for contributing actual project code. Perhaps an even more lax license than the EPL and no DCO required...
I can see that so maybe the e2e test results would be just indicative at first, until we're sure they can be relied upon. |
Beta Was this translation helpful? Give feedback.
-
|
I think the e2e test can be very useful too. 👍 If they keep being maintained they can help with quickly finding regressions! With an agile approach it should turn into something nice eventually. Seeing the Docker container being used for this is nice too as it may also help with finding issues in the container build. 😉 |
Beta Was this translation helpful? Give feedback.
-
Dear @openhab/architecture-council,
I would like to recap the discussion I have been having with the maintainers on the forum about having some sort of e2e testing, but I realize it might not have reached all potential interesting parties.
Maybe this could be best as an announcement in the forum or in its Development category but I wanted to run it through here first.
We're only a few months away from the 4.0 release so I think it might be a good thing to make sure nothing goes wrong in the buildup to this release. There are a number of incoming and recent breaking changes that would benefit to be tested end-to-end, so we won't have to invalidate a major 4.0.0 release and make a patch release immediately after if something obvious goes wrong - in other words we don't want to have to fix things we could have identified in the first place.
I'm proposing to have some automation to this end-to-end testing.
I have prepared a repository with GitHub Actions and a Cypress Project to make it happen.
The current tests spin up a fresh openHAB instance using Docker, go through the initial setup steps in the Chrome browser, then run a test suite in both Chrome and Firefox. It's just the beginning of what could be tested, and you can easily check out how everything went (https://cloud.cypress.io/projects/quu4in/runs/19/specs) and even review previous runs including viewing videos and eventual screenshots when tests failed.
The maintainers have not expressed an interest in keeping this up and living, it's understandable, they have enough on their plate already.
So I'm wondering if we could ask the community to take charge.
The code doesn’t need to be very academic (one example: openhab-e2e/create-nibe-thing.cy.js at main · ghys/openhab-e2e · GitHub 1) so “gatekeeping”, which is probably a major deterrent to contribution, would be kept to a minimum - if you have some understanding of JS and CSS selectors to write some e2e tests and make sure things are properly checked before we release (it might be even 'fun' to see your code played on your browser and in video), then any contribution would be welcome - it might be a good way to empower the community to actively participate in the project.
Would there be a community stalwart taking charge of that? 😉
Just let me know what you think, and if it's an idea worth pursuing.
Cheers
Yannick
Beta Was this translation helpful? Give feedback.
All reactions