Skip to content

Commit

Permalink
Merge pull request #847 from neuroscout/cypress
Browse files Browse the repository at this point in the history
Add end-to-end frontend testing using Cypress
  • Loading branch information
adelavega committed Jan 20, 2021
2 parents 796f12d + 570551d commit f630dad
Show file tree
Hide file tree
Showing 23 changed files with 21,878 additions and 27 deletions.
42 changes: 34 additions & 8 deletions .github/workflows/tests.yml
Expand Up @@ -3,18 +3,18 @@ name: Neuroscout tests
on: [push, pull_request]

jobs:
test:
name: pytest
tests:
name: neuroscout tests
runs-on: ubuntu-latest
env:
APP_SETTINGS: neuroscout.config.app.GHIConfig
PYTHONHASHSEED: 0
strategy:
matrix:
type: ['frontend', 'backend']
services:
postgres:
image: postgres:12
env:
POSTGRES_PASSWORD: password
POSTGRES_DB: travis_ci_test
POSTGRES_DB: ci_test
ports:
- 5432:5432
options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
Expand All @@ -28,9 +28,8 @@ jobs:
run: |
bash <(wget -q -O- http://neuro.debian.net/_files/neurodebian-travis.sh)
sudo apt-get -qq update
sudo apt-get install -yq git-annex-standalone tesseract-ocr libavformat-dev libavfilter-dev libavdevice-dev ffmpeg
sudo apt-get install -yq git-annex-standalone tesseract-ocr libavformat-dev libavfilter-dev libavdevice-dev ffmpeg libgconf-2-4
cp neuroscout/config/example_app.py neuroscout/config/app.py
cp neuroscout/frontend/src/config.ts.example neuroscout/frontend/src/config.ts
- uses: actions/cache@v2
with:
path: ~/.cache/pip
Expand All @@ -42,7 +41,34 @@ jobs:
python -m pip install -r neuroscout/requirements.txt
python -m pip install -e git+https://github.com/PsychoinformaticsLab/pliers.git#egg=pliers
python -m pliers.support.download
- name: Test with pytest
env:
APP_SETTINGS: neuroscout.config.app.GHIConfigBackend
PYTHONHASHSEED: 0
run: |
python -m pytest neuroscout/tests/ --cov=./neuroscout --cov-report xml
if: matrix.type == 'backend'
- uses: codecov/codecov-action@v1
if: matrix.type == 'backend'

- uses: actions/setup-node@v2
if: matrix.type == 'frontend'
- uses: actions/cache@v2
with:
path: 'neuroscout/frontend/node_modules'
key: modules-${{ hashFiles('**/yarn.lock') }}
if: matrix.type == 'frontend'
- name: Install frontend dependencies
run: |
cd neuroscout/frontend
yarn install
if: matrix.type == 'frontend'
- name: Run frontend tests with backend
env:
APP_SETTINGS: neuroscout.config.app.GHIConfig
run: |
bash setup_frontend_tests.sh
sleep 15
cd neuroscout/frontend && ./node_modules/.bin/cypress run --record --key ${{ secrets.CYPRESS_RECORD_KEY }}
if: matrix.type == 'frontend'
6 changes: 2 additions & 4 deletions .gitignore
Expand Up @@ -110,7 +110,6 @@ jest_0
.my-env
.pliersenv
celery_worker/src
<<<<<<< Updated upstream
.env
*.dump
*.sql
Expand All @@ -122,6 +121,5 @@ nginx/certs
nginx/certs-data

neuroscout/frontend/src/config.ts
=======
neuoscout/
>>>>>>> Stashed changes

cypress/videos/
23 changes: 22 additions & 1 deletion README.md
Expand Up @@ -131,7 +131,7 @@ You can then insert this token into the header to authorize API requests:
Note that in order to use any protected routes, you must confirm the email on your account. Confusingly, you can get a valid token without confirming your account, but protected routes will not function until confirmation.


## Running tests
## Running backend tests
To run tests, after starting services, create a test database:

docker-compose exec postgres psql -h postgres -U postgres -c "create database scout_test"
Expand All @@ -148,6 +148,27 @@ To run frontend tests run:

docker-compose run --rm -w /neuroscout/neuroscout/frontend neuroscout npm test

## Running frontened tests
To run frontend tests, have Cypress 6.0 or greater installed locally.
First, ensure neurscout is running:

docker-compose -f docker-compose.yml -f docker-compose.dev.yml up -d

Next, set up the test environment:

docker-compose exec neuroscout bash
export APP_SETTINGS=neuroscout.config.app.DockerTestConfig
bash setup_frontend_tests.sh

In a separate window, you can run cypress:

cd neuroscout/frontend
cypress open

Once done, kill the first command, and run the following to tear down the test db

docker-compose exec -e APP_SETTINGS=neuroscout.config.app.DockerTestConfig neuroscout python manage.py teardown_test_db

## Renewing HTTPS certs
Ideally, this is take care of by a cronjob

Expand Down
3 changes: 3 additions & 0 deletions docker-compose.dev.yml
Expand Up @@ -11,3 +11,6 @@ services:
neuroscout:
command: /usr/local/bin/gunicorn -w 2 -b :8000 neuroscout.core:app --log-level debug --timeout 120 --reload
restart: "no"
ports:
- "4000:4000"
- "3000:3000"
47 changes: 47 additions & 0 deletions manage.py
Expand Up @@ -14,6 +14,7 @@
from neuroscout.core import app, db
from neuroscout.models import user_datastore
from neuroscout import models
from neuroscout.tests import conftest

app.config.from_object(os.environ['APP_SETTINGS'])
migrate = Migrate(app, db, directory=app.config['MIGRATIONS_DIR'])
Expand Down Expand Up @@ -92,5 +93,51 @@ def ingest_from_json(config_file, update_features=False, reingest=False):
reingest=reingest)


@manager.command
def setup_test_db():
# Only run if in setup mode
if not app.config['TESTING']:
raise Exception("This fixture can only be run in test mode")

# Init db
db.init_app(app)
db.create_all()

# Create test users
users = [('user@example.com', 'string', 'testuser'),
('test2@gmail.com', 'test2', 'testuser2')]

for email, password, name in users:
user_datastore.create_user(
email=email, password=encrypt_password(password),
user_name=name, confirmed_at=datetime.datetime.now())
db.session.commit()

id_1 = user_datastore.find_user(email=users[0][0]).id

dataset_id = populate.add_task(
'bidstest', local_path=conftest.DATASET_PATH)

predictor_id = populate.extract_features(
conftest.EXTRACTORS, 'Test Dataset', 'bidstest')

analysis_id = conftest.add_analysis_abstract(db.session, id_1, dataset_id)

pred = models.Predictor(dataset_id=dataset_id, name="RT")

db.session.add(pred)
db.session.commit()


@manager.command
def teardown_test_db():
# Only run if in setup mode
if not app.config['TESTING']:
raise Exception("This fixture can only be run in test mode")

db.session.remove()
db.drop_all()


if __name__ == '__main__':
manager.run()
2 changes: 1 addition & 1 deletion neuroscout/Dockerfile
Expand Up @@ -5,7 +5,7 @@ RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

RUN apt-get -qq update
RUN apt-get install -yq ffmpeg tesseract-ocr apt-transport-https
RUN apt-get install -yq ffmpeg tesseract-ocr apt-transport-https libnss3 xvfb
RUN pip install -e git+https://github.com/PsychoinformaticsLab/pliers.git#egg=pliers
RUN pip install clarifai duecredit google-api-python-client IndicoIo librosa>=0.6.3 pysrt pytesseract spacy rev_ai

Expand Down
10 changes: 7 additions & 3 deletions neuroscout/config/example_app.py
Expand Up @@ -7,8 +7,7 @@


class Config(object):
SERVER_NAME = 'localhost' # Set to external server name in production

# SERVER_NAME = localhost
GOOGLE_CLIENT_ID = 'clientid' # Must set this for frontend to build
SECRET_KEY = 'A_SECRET!'
HASH_SALT = 'dfdfdf'
Expand Down Expand Up @@ -50,6 +49,7 @@ class ProductionConfig(Config):


class DevelopmentConfig(Config):
SERVER_NAME = 'localhost'
ENV = 'development'


Expand All @@ -64,5 +64,9 @@ class DockerTestConfig(TestingConfig):


class GHIConfig(TestingConfig):
SQLALCHEMY_DATABASE_URI = "postgresql://postgres:password@localhost/travis_ci_test"
SQLALCHEMY_DATABASE_URI = "postgresql://postgres:password@localhost/ci_test"
FILE_DIR = Path('./tmp/file-data').absolute()


class GHIConfigBackend(GHIConfig):
SERVER_NAME = 'localhost'
4 changes: 4 additions & 0 deletions neuroscout/frontend/cypress-test.json
@@ -0,0 +1,4 @@
{
"baseUrl": "http://localhost:3000",
"projectId": "jfpnqv"
}
1 change: 1 addition & 0 deletions neuroscout/frontend/cypress/fixtures/datasets.json

Large diffs are not rendered by default.

Large diffs are not rendered by default.

111 changes: 111 additions & 0 deletions neuroscout/frontend/cypress/integration/analysis_builder.spec.js
@@ -0,0 +1,111 @@
describe('Analysis Builder', () => {
beforeEach(() => {
cy.login('user@example.com', 'string')
cy.get('.newAnalysis > a')
cy.visit('/builder')
})

let name = 'dataset_name';
let predCount = 3;
it('analysis builder', () => {
/* Overview Tab */
cy.get('.builderAnalysisNameInput').type(name)
cy.get(`.ant-col > .ant-input[value=${name}]`)
cy.get('.builderAnalysisDescriptionInput').type(name)
cy.get('.ant-form-item-children > .ant-input').contains(name)

cy.get('.selectDataset')
cy.get('td').contains('Test Dataset').parent().within(() => {
cy.get('input[type=radio]').click()

})
cy.get('.builderAnalysisTaskSelect')
cy.get('.ant-collapse-item > .ant-collapse-header').contains('Runs: All selected').click()
cy.get('.builderAnalysisRunsSelect')
cy.get('.ant-collapse-item-active > .ant-collapse-content > .ant-collapse-content-box').find('tr').its('length').should('be.gte', 1)
cy.contains('Next').click()

/* Predictor Tab - select first 3 predictors */
let count = 0
cy.contains('Select Predictors').parent().within(() => {
cy.contains('Brightness')
cy.get('.ant-table-body').find('input[type=checkbox]').each(($el, index, $list) => {
if (count < predCount) {
$el.click()
count++
}
})
})
cy.get('.ant-tag').its('length').should('be.gte', predCount)
cy.get('button:visible').contains('Next').parent().click()

/* Transformation Tab - create a scale xform with all predictors */
cy.get('button').contains('Add Transformation').parent().click()
cy.get('.ant-select').click()
cy.get('li').contains('Scale').click()
cy.get('.ant-table-body:visible').find('input[type=checkbox]').each(($el, index, $list) => {
$el.click()
})
cy.get('button:visible').contains('OK').parent().click()
cy.get('button:visible').contains('Next').parent().click()
let xformCount = 2

/* HRF Tab - select all non counfound predictors, check length is correct */
cy.get('button').contains('Select All Non-Confound').parent().click()
cy.get('.ant-tag:visible').its('length').should('be.eq', predCount + 1)
cy.get('button:visible').contains('Next').parent().click()

/* Contrast Tab - generate dummy contrasts, then delete one of them */
cy.get('button').contains('Generate Dummy').parent().click()
cy.get('.ant-list:visible').find('.ant-btn-danger').its('length').should('be.eq', predCount)
cy.get('.ant-list:visible').find('.ant-btn-danger').first().click()
cy.get('.ant-list:visible').find('.ant-btn-danger').its('length').should('be.eq', predCount - 1)
cy.get('button:visible').contains('Next').parent().click()

/* Review Tab
- Ensure the number of contrasts in summary section matches what we expect.
- Intercept call to generate design matrix, reply with known good response.
- ensure that vega embed is displayed
*/
cy.get('.ant-collapse-item').contains('Contrasts').click()
cy.get('.ant-collapse-item').contains('Contrasts').parent().find('.ant-collapse-content-box > div').children().should('have.length', predCount - 1)
cy.get('.ant-collapse-item').contains('Transformations').click()
cy.get('.ant-collapse-item').contains('Transformations').parent().find('.ant-collapse-content-box > div').children().should('have.length', xformCount)
cy.get('.ant-spin-spinning:visible')
let dmResp;
cy.fixture('design_matrix_response.json').then(dmBody => {
dmResp = dmBody
});
cy.intercept('GET', '**/report**', (req) => {
req.reply(dmResp)
})
cy.get('.vega-embed')
cy.get('button:visible').contains('Next').parent().click()

/* Status Tab
- ensure that generate button disabled
- toggle TOS checkbox
- generate button should now be enabled, hit it
- confirm in popup modal
- ensure transition to pending status after we confirm submit
*/
cy.get('.statusTOS').get('button:disabled')
cy.get('span').contains('terms of service').parent().find('input').check()
cy.get('.statusTOS').get('span:visible').contains('Generate').parent().click()
cy.intercept('GET', '**/report**', (req) => {
req.reply(dmResp)
})
cy.intercept('POST', '**/compile**', {
statusCode: 200,
body: {'status': 'SUBMITTING', 'traceback': ''}
})
cy.intercept('GET', '**/compile', {
statusCode: 200,
body: {'status': 'PENDING', 'traceback': ''}
})

cy.get('.ant-modal-body').contains('submit the analysis').get('button').contains('Yes').parent().click()

cy.contains('Analysis Pending Generation')
});
});

0 comments on commit f630dad

Please sign in to comment.