CI test results in false positive

[shawnbot]

This staging build is a false positive. A couple of things are going on here:

1. Something in our test/ci.sh script is causing the server to stop when the browser tests fail, which means that the a11y tests will also fail regardless.
2. In this particular case, the a11y tests failed but the runner didn't exit with a non-zero status code, which Circle treats as a success.
3. Strangely, Circle sees failures in the XUnit results from the browser test and lists them, but still treats the build as a success.

The a11y tests also aren't outputting XUnit results for Circle (unlike the unit and browser tests), which makes it more difficult to find those specific errors.

Parellelizing the builds might help, but doing so may prove tricky with the whole Sauce Labs/Sauce Connect setup.

[shawnbot]

Next steps:

• Look at previous builds and make sure that this is an isolated issue.
• Investigate removing the site build and serve steps from our CI script and run the tests against Federalist, as in this example.

[shawnbot]

Parallelizing builds also becomes much easier when we remove Sauce Connect (and the related site build + serve steps) from the CI test runner.

[shawnbot]

So yeah, we have other false positives, including the latest dev build. What's really annoying about these is that Circle actually lists the failing tests, but still marks the whole build as successful:

This particular failure is another pernicious timeout that I haven't been able to reproduce locally, but should probably try with Sauce Labs again. What I'm doing right now is refactoring the test/ci.sh script so that all it does is build the site and start the server, then moving the browser and a11y tests commands to the circle.yml, where they can fail independently of the server starting and stopping.

[shawnbot]

That appears to have worked, because the latest build fails. Now, to fix the failing tests and see how to get it passing again...

[shawnbot]

Good news! This build failed in a slightly more useful way: I added the /data/ URL to the a11y test list, and that's the only thing that failed. Next step is to comment that out and run the tests again, and hope that the whole thing passes.

[shawnbot]

Okay, so as of this build the tests are running with Jekyll in the background, then the wget command that blocks until the server starts up and the browser and a11y tests are running from the circle.yml instead of the test/ci.sh script (which is now test/server.sh, and just sets the API base URL then runs Jekyll, and doesn't barf access logs on stdio). The build fails because of the /data/ page, so my next push is going to comment that out and deal with those accessibility issues in #1475.

[shawnbot]

Fixed! 🎉

[barkimedes]

SGTM 🇺🇸