pytest-mip: Pytest and Selenium -based integration testing and reporting of the up-and-running MIP federations

This open-source project is developed for testing and reporting the status of the up-and-running federations of the MIP.

It provides integration tests, written in Python, based on selenium, pytest, and pytest-html, and encapsulated in a Docker software container image for easy deployment.

The process of building the container image, running the tests, generating and deploying the report is fully automated through a GitHub Action workflow that is run periodically at 6:55am UTC every day. Updated overall status of the tests is indicated by the badge above.

The report, deployed as a Github Page website can be viewed @ https://hbpmedical.github.io/pytest-mip/.

Content

• Under the Hood
• Installation instructions
• How to run pytest-mip using the Docker image
• Want to add tests for a new MIP federation?
• Funding

Under the Hood

This project contains the following files and folders:

• requirements.txt: List all the Python dependencies for the pytest-mip project.

• Dockerfile: Contain all the commands to assemble the Docker container image of pytest-mip.

• entrypoint.sh: Entrypoint script of the Docker image which runs pytest with the options --html, --self-contained-html, --css, and --junitxml. Arguments given to the Docker image are passed to pytest.

• report/: Store HTML / CSS / JavaScript files related to the HTML report.

  • index.html: Main reporting web page that contains an history of the tests (run up to the present moment) in the form of a table with links to their corresponding pytest-html reports.

  • js/reports-parser.js: Define functions in Javascript to dynamically parse the pytest-html report in JUnitXML format and build the table of the reporting web page (index.html).

  • js/table-sorter.js: Define functions in Javascript to sort by date the table of tests of the reporting web page (index.html).

  • css/style.css: Define CSS for the main reporting web page (index.html).

  • logo_small.png: Ebrains logo displayed at the left-hand side of the title of the reporting web page (index.html).

• tests/: Store all files necessary for the tests.

  • pytest.ini: Define configuration options for pytest.

  • pytest_report.css: Define CSS for pytest the HTML report generated by pytest-html.

  • conftest.py: Define fixtures that are shared between all pytest tests, such as selenium_driver which connects with a selenium Google Chrome web driver, or pytest_runtest_makereport which is run as a post hook and allows the insertion of screenshot when a test fails in the report table.

  • basetest.py: Define a base class BaseMIPFederationTest for the pytest tests.

  • test_<FEDERATION_NAME>_federation.py: Pytest tests for the MIP federation <FEDERATION_NAME>.

  • test_federation.py.template: Pytest tests template to be used for a new federation.

• .gitignore: Specify intentionally untracked files that Git should ignore such as the file .DS_Store generated by Mac OSX computers or the folders __pycache__.

• .zenodo.json: Metadata for publication of pytest-mip to Zenodo, which includes among others the list of authors.

• .github/workflows/build-test-report.yml: Describe the GitHub Action workflow that builds the Docker container image, runs the tests, generates and deploys the report to GitHub Page. It consists of several "so-called" stages that are presented in the diagram below.

  graph TB
  A["Checkout"] ~~~|"Checkout the GitHub repository"| A
  B["Set up Docker Buildx"] ~~~|"Use GitHub Action docker/setup-buildx-action<br>to set up Docker Buildx"| B
  C[Install JQ] ~~~|"Install JQ, a package to manipulate JSON files"| C
  D["Setup project parameters"] ~~~|"Set the variables UserID and UserPWD<br>from the project environment variables<br>$EBRAINSUSERID and $EBRAINSUSERPWD"| D
  E[Build and push] ~~~|"Use GitHub Action docker/build-push-action<br>to build the Docker image"| E
  F[Get current date] ~~~|"Get the current date to use<br>in the generation of folder and file names<br>to store the reports"| F
  G[Run integration tests of<br>the up-and-running MIP federations] ~~~|"Run the tests with the built Docker image"| G
  H{Success?}
  I["Update Reports JSON (SUCCESS)"]
  J["Update Reports JSON (FAILURE)"] ~~~|"Add item to the reports.json<br>describing the test date, link to its report,<br>and different test status"| J
  K[Deploy report to GitHub Page] ~~~|"Deploy the reporting website<br>with updated content to<br>hbpmedical.github.io/pytest-mip"| K
  A --> B --> C --> D --> E --> F --> G --> H
  H -->|YES| I --> K
  H -->|NO| J --> K
  

Installation instructions

Prerequisites

This tool is intended to be run using its Docker image and so Docker is required to be installed.

How to build the Docker image

1. Clone the repository

2. Go to the clone directory

3. Edit the project_parameters.py.template and set the variables UserID and UserPWD with your own EBRAIN credentials for login to the different MIP federations.

4. Rename project_parameters.py.template to project_parameters.py

5. Build the Docker image with the following command:

   $ docker build -t pytest_mip .

How to run pytest-mip using the Docker image

Test all MIP federations

Once pytest_mip is built, you can execute the tests for all federations as follows:

$ docker run (-v /local/path/to/report:/app/report) -t pytest_mip .

where -v /local/path/to/report:/app/report can be used to make the pytest-html report generated by the tests available outside the container in your /local/path/to/report folder.

Once done, you can check the generated report by (1) going to the /local/path/to/report folder, and (2) opening the ìndex.html file in your favorite browser.

Test a specific MIP federation

You can run the following command to test a specific federation:

$ docker run (-v /local/path/to/report:/app/report) -t pytest-mip test_<fed_name>.py

where <fed_name> designs a specific federation. Note that a report can similarly be generated in your folder of wish (such as /local/path/to/report) by using the -v /local/path/to/report:/app/report option in the docker run command.

Here is a list of <fed_name> / filename pairs for the different federations that are now available:

• qa_federation : test_qa_federation.py
• public_mip : test_public_mip.py
• stroke_federation : test_stroke_federation.py
• ...

Want to add tests for a new MIP federation?

To facilitate this process, the template file test_federation.py.template has been created. This can be achieved by following the steps below.

1. Start by creating a dedicated branch feat/add-<NEW_FEDERATION_NAME>-federation from the main branch:

   $ git checkout main
   $ git pull origin
   $ git checkout -b <NAME_OF_YOUR_BRANCH>

2. Copy the template file to tests/test_<NEW_FEDERATION_NAME>_federation.py, where <NEW_FEDERATION_NAME> is the name of your new federation (e.g. qa), and customize it as follows:

   • Update the FEDERATION_URL variable below with the URL of the federation to test.
   • Update the class name to TestMIP<NEW_FEDERATION_NAME>Federation (e.g. TestMIPQAFederation)
   • Replace all occurrences of <NEW_FEDERATION_NAME> with the name of the federation to test.
   • Re-implement the test_data() method to test the data of the federation. The original test_data() method of the parent class can be found in tests/basetest.py and be used an example.
   • Edit this docstring, introduce the test of the federation, replace <NEW_FEDERATION_NAME> with the name of the federation to test, replace information about authorship and creation / modification dates.
   • Remove the notes from the docstring, which provides you with the same detailed guidelines.

3. Test the test locally. This can be achieved as follows in the tests/ folder of the cloned repository:

$ cd tests/
$ sh ../entrypoint.sh test_<NEW_FEDERATION_NAME>_federation.py

4. When the tests passed and you are satisfied of the results, put the file on stage (e.g. git add tests/test_<NEW_FEDERATION_NAME>_federation.py), and make a commit with a message compliant to the Angular conventional commits definition (Please check here) such as feat: add new <NEW_FEDERATION_NAME> test.
5. Update the list of tested federations in the README. Put the file on stage (e.g. git add README) and make commit with message similar to docs(README): update list of federation with the new <NEW_FEDERATION> federation.
6. Update the FEDERATION_URL_NAME_MAPPING object in report/js/reports-parser.js with a new entry corresponding to the federation. Put the file on stage (e.g. git add report/js/reports-parser.js) and make commit with a message similar to feat(reports-parser.js): update mapping with new entry for the <NEW_FEDERATION> federation.
7. Push the branch to GitHub and open a Pull Request (target: main branch).
8. Review a last time the changes summarized in the PR Changes tab.
9. Ask for review.
10. Once PR is approved, merge and close the PR.
11. Make a new release following the previous release description as example.

Additional notes

• It is recommended to use Chrome with the SelectorHubs extension to determine absolute XPATHs that are used by Selenium to get the different web page elements.

• It is possible to test a specific federation test locally. This can be achieved as follows in the repository's directory:

  $ cd tests/
  $ sh ../entrypoint.sh test_<NEW_FEDERATION_NAME>.py

Funding

This project received funding from the European Union's H2020 Framework Programme for Research and Innovation under the Specific Grant Agreement No. 945539 (Human Brain Project SGA3, as part the Medical Informatics Platform (MIP)).