Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove local build+serve step from CI tests #1474

Closed
shawnbot opened this issue Feb 5, 2016 · 7 comments
Closed

Remove local build+serve step from CI tests #1474

shawnbot opened this issue Feb 5, 2016 · 7 comments

Comments

@shawnbot
Copy link
Contributor

shawnbot commented Feb 5, 2016

This will go a long way in addressing #1468 and possibly #1471. Currently, our CI test script builds the site and serves it locally, then establishes a secure tunnel with Sauce Labs so that it can test the site from localhost. This not only takes more time per build (setting up the Ruby environment, building the site, and waiting for the Sauce Connect tunnel), but also introduces a level of complexity that makes CI failures (and, apparently, false positives) difficult to debug.

Removing the build and serve steps, and removing Sauce Connect from the equation, dramatically simplifies our build process. I've already figured out how to coordinate the timing between Circle CI and Federalist builds, and this approach should be easy to integrate.

@shawnbot
Copy link
Contributor Author

There's one snag in this that I realized while trying to figure out why this build keeps failing is that Circle doesn't enable our custom environment variables for pull requests from a fork. (This would be a security risk because a malicious forker could modify the test script to just echo $SAUCE_ACCESS_KEY and get the keys to our Sauce Labs account.)

Fork PRs will need to be exempt from automated browser tests because, even if we remove Sauce Connect from the equation, Federalist still doesn't build a preview for forks, so there wouldn't be a public URL to test. In the case of PRs from open source contributors, we can either:

  1. Merge to dev as-is and let the tests run afterward; or
  2. Close fork PRs in favor of a branch off the main repo that incorporates the fork's commits.

Either way, we can detect a fork PR in our test script with something like:

if [ "${CIRCLE_PROJECT_USERNAME}" != "${CIRCLE_PR_USERNAME}" ]; then
  exit 1
fi

Thoughts, @ultrasaurus or @yozlet?

@ultrasaurus
Copy link
Contributor

I like the idea of running unit tests always, then limiting browser tests to branches, then always test locally or push a branch before merging... @yozlet ?

@ultrasaurus
Copy link
Contributor

main thing is that I want to make sure we never have a false positive: #1471

and simplicity of the tests is a big plus

@yozlet
Copy link
Contributor

yozlet commented Feb 16, 2016

👍, plus add an echo to say why browser tests aren't being run.

@shawnbot
Copy link
Contributor Author

@barkimedes this is ready to go. You can see all of the gory details in #1496.

@barkimedes
Copy link
Contributor

I'm taking your word for this one. If you say this is done and the tests are passing, then I'm cool with closing the issue whenever you're cool with closing it. 🇺🇸

@shawnbot
Copy link
Contributor Author

Sweet, closing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants