Some integration tests are broken, and github actions actually tests zero of them so this is not detected #5376
Comments
|
Yeah, since travis-ci.org shut us out the tests appear to have all fallen over. I tried over the weekend to get them running again but didnt work. Im looking at moving them over to browserstacks since it has better intergration with nightwatch. For the moment in the pipelines intergration tests have been disabled, and I couldnt get them to run locally either. |
|
Related discussion in PR comments: #5371 |
|
Turns out I didnt disable it in pipeline.. so going to do it now. |
|
Thanks, then it's not just me having problems. I am manly interested in getting the integration tests running locally. Web development is annoying in that things get outdated so fast. I started on Fedora 35 and I did not get the tests to run there, so thinking it possibly might be a bit on the too new side, I also installed an VM with ubuntu 21.04 (which has nodejs version 12.22.8 which is just one version too old to be supported by some dependency, so I downloaded 12.22.9 to override with). However this environment also fails. Do you have any recommendation on what you would expect to have highest chance of success? As new as possible with everything (like using the newer nightwatch 2 beta), stay as old as possible or somewhere in between? |
|
All of the intergration tests requires RES to be active on the browser, so with the grep above the reason x-post isnt seen as its a clean pull of the page. While I was trying to get it working I did notice that firefox had issues loading RES into the browser, whereas chrome didnt have an issue. Were you trying to get them running in Firefox?
Even then, Chrome was throwing some strange errors. I can try have a look this weekend to see if its possible to give them some life with a quick fix rather than having to rebuild them. |
|
Ah, that makes sense. The tests failed, and then when I opened the url in the plain firefox inside the newly created VM (I have not even installed noscript and ublock origin) the text was missing and I did not connect the dots. The text is there on my main browser with RES installed. I managed to get nightwatch 2.0.0-beta.2 to launch tests individually with running This is running against I do not know if there is something output missing from the above? According to this issue marionette supports loading addons with some |
This started by trying to run the integration tests locally, however despite whatever problems I might have setting up a working testing environment, I found that there must be some inherent problems with at least one of the tests. Picking the smallest and simplest file,
tests/xPostLinks.js, I cannot see how this possibly can pass.The test was added in 2016 and it tests that some element on the page contains the phrase "x-posted". I am sure that the page did contain the text the tests expect back then, but today the page does not in any way contain text that resembles the text that the test is expecting:
$ head -8 tests/xPostLinks.js module.exports = { 'basic functionality': browser => { browser .url('https://en.reddit.com/r/RESIntegrationTests/comments/5lc3yw/test_xpost_links_xpost_renhancement/') .waitForElementVisible('#thing_t3_5lc3yw') .assert.containsText('#thing_t3_5lc3yw .tagline', 'x-posted from /r/Enhancement') .end(); }, $ wget https://en.reddit.com/r/RESIntegrationTests/comments/5lc3yw/test_xpost_links_xpost_renhancement/ -O en.html --2022-01-17 16:55:55-- https://en.reddit.com/r/RESIntegrationTests/comments/5lc3yw/test_xpost_links_xpost_renhancement/ Resolving en.reddit.com (en.reddit.com)... 151.101.121.140 Connecting to en.reddit.com (en.reddit.com)|151.101.121.140|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 53091 (52K) [text/html] Saving to: ‘en.html’ en.html 100%[======================================================>] 51,85K --.-KB/s in 0,08s 2022-01-17 16:55:56 (666 KB/s) - ‘en.html’ saved [53091/53091] $ sed -i 's/>/>\n/g' en.html $ grep x-posted en.html $ sed -n '/thing_t3_5lc3yw/,$p' en.html | sed -n '/tagline/,$p' | head <p class="tagline "> submitted <time title="Sat Dec 31 22:38:53 2016 UTC" datetime="2016-12-31T22:38:53+00:00" class="live-timestamp"> 5 years ago</time>  by <a href="https://en.reddit.com/user/erikdesjardins" class="author may-blank id-t2_9288u" > erikdesjardins</a> <span class="userattrs"> </span> <span class="awardings-bar" data-subredditpath="/r/RESIntegrationTests/" > </span> </p> $So am I missing anything?
Looking into the latest github action run as of writing this the
testjob is green, however looking into the "Run yarn integration chrome --retries 2 && yarn integration-only firefox --retries 2" steps it contains at the bottom after the long (~800 lines) build section the following:so that test job both fails to run any of the integration tests and also fails to detect this as a failure. This is obviously a different problem than integration tests expecting missing text to be present, but it masks detecting them.
@BenMcGarry you have been doing a lot of work with the pipelines recently, I assume you are capable and interested in looking into this?
(I found no other existing issue related to this. The code is the latest, one commit after v5.22.8)
The text was updated successfully, but these errors were encountered: