try creating a system test and running in Travis #4888

jywarren · 2019-02-27T19:06:41Z

https://guides.rubyonrails.org/testing.html#implementing-a-system-test

https://chriskottom.com/blog/2017/04/full-stack-testing-with-rails-system-tests/

plotsbot · 2019-02-27T19:19:09Z

	1 Message
📖	@jywarren Thank you for your pull request! I’m here to help with some tips and recommendations. Please take a look at the list provided and help us review and accept your contribution! And don’t be discouraged if you see errors – we’re here to help.

Generated by 🚫 Danger

jywarren · 2019-02-28T20:02:11Z

Needs chromedriver! https://dev.to/aergonaut/running-rails-5-system-tests-on-travis-ci-with-chromedriver-4nm7

jywarren · 2019-02-28T20:08:59Z

https://stackoverflow.com/questions/20364674/chromedriver-on-travis-ci

jywarren · 2019-02-28T20:36:57Z

https://stackoverflow.com/questions/42210295/travis-unable-to-find-chromedriver?rq=1

jywarren · 2019-02-28T23:10:58Z

Selenium::WebDriver::Error::WebDriverError: Selenium::WebDriver::Error::WebDriverError: Unable to find chromedriver. Please download the server from http://chromedriver.storage.googleapis.com/index.html and place it somewhere on your PATH. More info at https://github.com/SeleniumHQ/selenium/wiki/ChromeDriver.

jywarren · 2019-02-28T23:32:58Z

Hmm, @siaw23 have you ever done this? I think this could dramatically stabilize our testing of front-end features!

siaw23-retired · 2019-03-02T14:26:49Z

I took a shot at this. System tests have been fixed but only locally(on development). I think we can't run system tests on Travis if we're using Passenger because Passenger runs multiple process which won't work with Capybara. So we can either change from Puma to Passenger on production (which I recommend) or run system tests only on development with Puma 😬...

jywarren · 2019-03-02T16:48:25Z

Well, system tests should run in test mode, not development mode, right? Sorry, I'm new to this! Right now tests don't run on passenger at all, they are lower level. And we don't run them in production mode at all. So don't we not lose anything in running system tests on Puma?

Thanks!!!

jywarren · 2019-03-02T17:38:47Z

@publiclab/plots2-reviewers anyone want to take a crack at getting these to run in Travis? The above posts seem to show it's possible.

siaw23-retired · 2019-03-02T17:48:02Z

Oh I'll have to check this again! I left this like this thinking the usual tests (non system tests) are running.

siaw23-retired · 2019-03-03T18:39:04Z

OK so I had to re-read what you wrote:

Right now tests don't run on passenger at all, they are lower level.

They do. 884/884: [===============================] 100% Time: 00:03:01, Time: 00:03:01

And we don't run them in production mode at all.

We run the normal tests like usual. The only test we don't run now are system tests and that's because of Passenger. So in this PR we're running normal tests + system tests in development and test modes. Then on production, we're just running the normal tests. Though ideally we'd like to run system tests on production too.

So don't we not lose anything in running system tests on Puma?

I'm not sure what this means but I don't see how we'd lose anything on running system tests on puma, it's only to test UI right?

jywarren

Hmm, maybe I am misunderstanding as well; it's my understanding that prior to system tests in Rails, all tests run in "test" mode, not development or production. And that this is true in Travis as well.

jywarren · 2019-03-04T21:18:25Z

.travis.yml

-  - docker-compose exec web bash -c "CI=TRUE GENERATE_REPORT=true rails test:system"
+  # Capybara can't work with Passenger on Travis, hence disabling system test on
+  # production until we change over to a Rack server.
+  # - docker-compose exec web bash -c "CI=TRUE GENERATE_REPORT=true rails test:system"


So I think we really need to re-enable this line, and to follow the guidance in the links I provided to try to get system tests to run in Travis.

jywarren · 2019-03-04T21:19:41Z

README.md

@@ -93,7 +93,7 @@ For information on how to install for use with the cloud environment, please see
 9. By default, start rails with `passenger start` from the Rails root and open http://localhost:3000 in a web browser.
 (for local SSL work, see [SSL](#ssl+in+development) below)
 10. Wheeeee! You're up and running! Log in with test usernames "user", "moderator", or "admin", and password "password".
-11. Run `rails test -d` to confirm that your install is working properly.
+11. Run `rails test` to confirm that your install is working properly. Or `rails test:system` for system tests.


what was the -d originally?

-d = Output test failures and errors after the test run

jywarren · 2019-03-04T21:24:00Z

Anyways, my biggest interest in this PR is to get system tests running in Travis, so we can use them as part of our CI. Where would we set it to use Puma?

https://dev.to/aergonaut/running-rails-5-system-tests-on-travis-ci-with-chromedriver-4nm7 seems to show it working in Travis.

Actually the errors I had seen were about not being able to find ChromeDriver, not related to Capybara. But maybe you got past those errors and saw a Capybara/Passenger error? If so, could you point me at the commit where you saw that?

sashadev-sky · 2019-03-07T12:44:34Z

@jywarren @siaw23 I am looking through this right now. To try to run this I can just fetch your PR locally and then push to this branch?

siaw23-retired · 2019-03-07T13:06:36Z

Yes you can @sashadev-sky

sashadev-sky · 2019-03-07T13:18:28Z

@jywarren @siaw23 Ok I am new to a bunch of the things mentioned here so my PRs are probably just going to be trial and error at first so feel free to ignore them

sashadev-sky · 2019-03-07T17:09:46Z

@jywarren @siaw23 I changed some setup configurations and I have it working locally fine now. Is the issue just getting them into Travis? I think we can just make a rake task for that?

jywarren · 2019-03-07T17:44:24Z

Yes, i think that's right - and in Travis it has to be using the configured chromedriver and chrome from either apt or the premade Travis versions. The examples near the top (blog posts and such) seem to show that it has worked at least for some people, although perhaps we're missing something about Capybara/Puma or something in the setup as @siaw23 has pointed out.

My guess is that if there's a way we can configure Travis to run them either as they are or with Puma (if that's the missing part) then that would be best.

This will hopefully be really great for JS testing. The hidden treat is also that it may be possible to get it to generate a set of screenshots of different pages automatically 💥

siaw23-retired · 2019-03-07T17:47:24Z

@jywarren I have seen those posts you referred to. I know system tests are possible on Travis. What the articles you referred to don't mention is what server they are using. They only showed how it works on Travis without making mention of which server they are using. With Thin, Webrick, Puma that article is relevant, with Passenger (which is not a Rack server...I think, it doesn't work).

jywarren · 2019-03-07T18:31:50Z

Hi @siaw23 -- so sorry, i know you know! Sincere apologies, didn't mean to run through the same info for you, just trying to catch Sasha up a bit. Thank you, that's an excellent point on their omission of the server in those posts.

What your observation makes me wonder is whether we can override the Passenger server config with a Rack server solely in the Travis context, for the purposes of the Travis CI. Do you think that'd be a reasonable way to achieve system tests in Travis, or do you think we should consider another path? Thank you! ❤️

sashadev-sky · 2019-03-07T18:45:22Z

@jywarren yes I have been playing with the screenshots, it's really great. teaspoon mocha sinon with rails fixtures had my head spinning so i'm excited about getting this to work.

https://www.rubydoc.info/github/jnicklas/capybara#The_DSL this page is the capybara docs and it has every possible way you can configure it and pros and cons, etc.

Like for ex., I got it to run with poltergeist driver because poltergeist relies on phantomjs, but the issue with that is it gets weird about file loading order and could also have discrepancies with Travis. And then you need to go through and see all of the configuration options its offering to try to fix that. (And also phantomjs is not maintained by anyone anymore)

So instead I tried RSpec and got it running really easily, but I realized I think the results might only work with static data (or I am not writing my tests correctly, TBD) and they have their own recommendations about configurations and what drivers to add that we would need to go through...

So yeah I agree out of this list of options there could definitely be a solution that doesn't involve infrastructure changes but we would need to dig a bit.

siaw23-retired · 2019-03-07T18:51:09Z

I have no experience with running system tests + Passenger + Travis. I saw a comment on Passenger or Travis' repo once stating how this may not be possible. Couldn't find it unfortunately when I tried to search for it. If we used something like Puma we could avoid a lot of headaches for contributors.

So for this particular issue and with my commits, we can run system tests locally until we switch to a Rack server to have them run on Travis. It'll be ideal to have system test running on CI. I honestly don't know how to do it with Passenger + Travis.

sashadev-sky · 2019-03-07T19:08:33Z

@siaw23 Yeah sorry me too I didn't see your comment until after I had posted my really long one 🙈I will be so happy the day travis CI makes perfect sense to me

jywarren · 2019-03-30T22:43:20Z

Ooh, a more descriptive error:

[/usr/local/bundle/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/remote/response.rb:32]: Selenium::WebDriver::Error::UnknownError: unknown error: Chrome failed to start: exited abnormally (unknown error: DevToolsActivePort file doesn’t exist) (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.) (Driver info: chromedriver=73.0.3683.68 (47787ec04b6e38e22703e856e101e840b65afe72),platform=Linux 4.4.0-101-generic x86_64) test/system/search_test.rb:6:in ‘block in '

jywarren · 2019-03-30T22:47:30Z

test/application_system_test_case.rb

+require "test_helper"
+
+class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
+  caps = Selenium::WebDriver::Remote::Capabilities.chrome("chromeOptions" => {"args" => %w(--headless)})


Maybe we need to set chrome flags here instead of in .travis.yml?

jywarren · 2019-03-30T23:18:32Z

uhhhhhhhh.......

jywarren · 2019-03-30T23:20:36Z

🎉 🎉 🎉 ✔️

Capybara starting Puma...
* Version 3.12.1 , codename: Llamas in Pajamas
* Min threads: 0, max threads: 4
* Listening on tcp://127.0.0.1:38566
1 items added from related_and_hyphenated_terms.dict.txt
1 items added from related_and_hyphenated_terms.dict.txt
  1/1: [===================================] 100% Time: 00:00:08, Time: 00:00:08
Finished in 8.53907s
1 tests, 1 assertions, 0 failures, 0 errors, 0 skips

jywarren · 2019-03-30T23:20:58Z

Now I guess we should see what we can remove and still let this run... to simplify!

jywarren · 2019-03-31T00:05:45Z

Anything else we can remove?

jywarren · 2019-03-31T00:24:10Z

.travis.yml

+  - docker-compose exec web bash -c "wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb"
+  - docker-compose exec web bash -c "dpkg -i google-chrome-stable_current_amd64.deb; apt-get -fy install"
+  - docker-compose exec web bash -c "nohup Xvfb -ac :9 -screen 0 1280x1024x16 &"
+  - docker-compose exec web bash -c "export DISPLAY=:99.0"


Let's try removing these and also add links for where to update dependencies

jywarren · 2019-03-31T00:24:43Z

Gemfile

@@ -36,7 +36,6 @@ gem 'omniauth-github', '~> 1.1', '>= 1.1.2'
 gem 'omniauth-google-oauth2'
 gem 'omniauth-twitter'
 gem "paperclip", "~> 6.1.0"
-gem 'passenger'


Oh let's try re adding this?

jywarren · 2019-03-31T02:47:24Z

OK this is pretty awesome. I've refined it down to what I believe is the minimum changes needed and cleaned up comments. This should be ready for a merge and then we can start building:

more system tests for critical systems
(potentially) a screenshot-generating test :-)

jywarren · 2019-03-31T02:57:19Z

This is ready to merge if it passes tests. Opening next steps here: #5316

jywarren · 2019-03-31T03:04:06Z

🎉 🎉 🎉

* try creating a system test * Create application_system_test_case.rb * Update .travis.yml * Update Gemfile * Update Gemfile * Update .travis.yml * Update application_system_test_case.rb * Update .travis.yml * Update .travis.yml * Update .travis.yml * Fix system test * Disable system test on production for now * Comment system test related configs from .travis.yml * Update Gemfile * Update Gemfile.lock * Update application_system_test_case.rb * Update .travis.yml * update gemfile.lock * Update .travis.yml * Update .travis.yml * Update .travis.yml * Update .travis.yml * Update .travis.yml * Update .travis.yml * Update .travis.yml * Update .travis.yml * Update application_system_test_case.rb * apt-get install google-chrome-stable chromium-chromedriver * wget/dpkg install of chromium and chromedriver * Update .travis.yml * Update .travis.yml * gemfile.lock update * Gemfile.lock update * change curl to wget * Chromedriver 2.35 and removed finds missing * Turn off binary option * Add command to start Chrome * update selenium-webdriver to 3.141.0 * --no-sandbox * export DISPLAY=:99.0 * Xvfb : nohup Xvfb -ac :9 -screen 0 1280x1024x16 & * Remove prefix * Chromedriver 73.0.3683.68 * Add chrome flags to ruby initializer * comment out first start of chrome, remove extraneous comments * commenting out `xvfb` and `DISPLAY` * move passenger to general gems; move back if it fails * cleaning up commented lines * Update Gemfile * Update Gemfile * Update Gemfile * Update application_system_test_case.rb

This was referenced Feb 27, 2019

Likes Fix (button mashing and initial negative likes) #4852

Merged

system tests (rails 5.1+ feature) #3683

Closed

siaw23-retired force-pushed the system-test branch from fcdfd41 to 150cdc4 Compare March 2, 2019 14:18

siaw23-retired added the ready label Mar 2, 2019

siaw23-retired removed the ready label Mar 2, 2019

jywarren commented Mar 4, 2019

View reviewed changes

jywarren mentioned this pull request Mar 21, 2019

Reply to Comment emoji placement - move them above responses? #5226

Open

jywarren added 4 commits March 30, 2019 17:07

export DISPLAY=:99.0

ab06162

Xvfb : nohup Xvfb -ac :9 -screen 0 1280x1024x16 &

dee5a44

Remove prefix

ce3ea24

Chromedriver 73.0.3683.68

b9b7458

jywarren commented Mar 30, 2019

View reviewed changes

Add chrome flags to ruby initializer

d7cd5f6

comment out first start of chrome, remove extraneous comments

24951c7

jywarren commented Mar 31, 2019

View reviewed changes

jywarren added 7 commits March 30, 2019 21:22

commenting out xvfb and DISPLAY

afe955b

move passenger to general gems; move back if it fails

2d610f0

cleaning up commented lines

6a18408

Update Gemfile

4ef3106

Update Gemfile

d610768

Update Gemfile

ad57126

Update application_system_test_case.rb

e8cfcd4

jywarren mentioned this pull request Mar 31, 2019

Brainstorm a list of critical full-stack system tests we should add #5316

Closed

13 tasks

jywarren merged commit a95ba1f into master Mar 31, 2019

kaustubh-nair mentioned this pull request May 15, 2019

Configure Travis for system tests publiclab/mapknitter#607

Closed

cesswairimu deleted the system-test branch May 4, 2021 14:05

try creating a system test and running in Travis #4888

try creating a system test and running in Travis #4888

Conversation

jywarren commented Feb 27, 2019 • edited by siaw23-retired

plotsbot commented Feb 27, 2019 • edited

jywarren commented Feb 28, 2019

jywarren commented Feb 28, 2019

jywarren commented Feb 28, 2019

jywarren commented Feb 28, 2019

jywarren commented Feb 28, 2019

siaw23-retired commented Mar 2, 2019

jywarren commented Mar 2, 2019

jywarren commented Mar 2, 2019

siaw23-retired commented Mar 2, 2019

siaw23-retired commented Mar 3, 2019 • edited

jywarren left a comment

Choose a reason for hiding this comment

jywarren Mar 4, 2019

Choose a reason for hiding this comment

jywarren Mar 4, 2019

Choose a reason for hiding this comment

siaw23-retired Mar 5, 2019

Choose a reason for hiding this comment

jywarren commented Mar 4, 2019

sashadev-sky commented Mar 7, 2019

siaw23-retired commented Mar 7, 2019

sashadev-sky commented Mar 7, 2019

sashadev-sky commented Mar 7, 2019

jywarren commented Mar 7, 2019

siaw23-retired commented Mar 7, 2019 • edited

jywarren commented Mar 7, 2019

sashadev-sky commented Mar 7, 2019

siaw23-retired commented Mar 7, 2019

sashadev-sky commented Mar 7, 2019

jywarren commented Mar 30, 2019

jywarren Mar 30, 2019

Choose a reason for hiding this comment

jywarren commented Mar 30, 2019

jywarren commented Mar 30, 2019

jywarren commented Mar 30, 2019

jywarren commented Mar 31, 2019

jywarren Mar 31, 2019

Choose a reason for hiding this comment

jywarren Mar 31, 2019

Choose a reason for hiding this comment

jywarren commented Mar 31, 2019

jywarren commented Mar 31, 2019

jywarren commented Mar 31, 2019

jywarren commented Feb 27, 2019 •

edited by siaw23-retired

plotsbot commented Feb 27, 2019 •

edited

siaw23-retired commented Mar 3, 2019 •

edited

siaw23-retired commented Mar 7, 2019 •

edited