QA and testing

Actual status

Integration tests

Priority: high

Integration tests are part of the fabric8-analytics-common repository

They are based on the Behave project

Documentation

• Documentation is part of the fabric8-analytics-common repository, see README.md

• Contains outdated informations

• Needs to be updated

• Test steps description are now generated and presented on page fabric8-analytics.github.io

Issues/TODO

• Update information about Integration Tests for Bayesian Core API

• Document all test steps implemented in the common.py module

Blockers

• The link to API examples does not work

• README.md: Link to the RAML file is invalid

• README.md: Link to the JSON schema is invalid

Done

• Document all test steps implemented in the common.py module

---

End-to-end tests

Priority: high

Documentation

POC for EE tests : https://github.com/naina-verma/poc-tests-analytics/blob/master/README.md All EE tests should go to: https://github.com/fabric8io/fabric8-test

Issues/TODO

Blockers

Due to Build pipeline cannot proceed with automating UI stack reports.

---

Unit tests

Priority: normal

Unit tests are part of sources for all components

Documentation

N/A (see Issues below)

Issues/TODO

• RFE: FAQ for unit test developers

• RFE: figure out how to measure code coverage for unit test

• run test coverage reports periodically

• publish code coverage on https://codecov.io/gh/fabric8-analytics/

• setup a git hook?

Blockers

---

Performance tests

Priority: normal

• Performance tests are run against Core server (almighty-core)

• Performance tests are run against Jobs server (flow-analysis and so)

• CRUD tests for workitems

• Contact information: Pavel Macik

Test results

• https://github.com/fabric8-analytics/fabric8-analytics.github.io/tree/master/perf-tests

Documentation

Issues/TODO

Blockers

---

Code quality tests

Priority: low

N/A at this moment

Documentation

Issues/TODO

• RFE: implement some sort of code quality tests (lints)

• setup a git hook?

Blockers

---

Regression tests

Priority: low

N/A at this moment

Documentation

N/A

Issues/TODO

• TBD

Blockers

• TBD

---

Test plans

Overall

Move this document from wiki to the new repo? fabric8-qa or something like this?

End-to-end tests

• [×] Make sure that all tests are on https://github.com/fabric8io/fabric8-test

• [×] Dockerize the tests

• [×] Further cooperate with Leonard Dimaggio on plans for EE tests

• [×] Specify test scenarios

Integration tests

• [✓] Provide documentation of test steps

• [-] Display link to test results

• [-] Cooperate with devels to update API documentation

• [✓] Update https://github.com/fabric8-analytics/fabric8-analytics-common/blob/master/integration-tests/README.md

• [✓] Create smoketests

• [-] Create tests for all server API calls

• [-] Create tests for all jobs API calls

• [×] Create API test coverage report

Unit tests

• [-] Cooperate with devels to integrate tests into MR workflow

• [?] Possibly use GIT hooks for this task

• [-] https://github.com/fabric8-analytics/fabric8-analytics-server/issues/28

• [-] https://github.com/fabric8-analytics/fabric8-analytics-server/issues/29

Performance tests

• [-] Cooperate with Pavel Macik who is responsible for perftests for all components

• [-] Define stories for perftests for analytics modules

Code quality tests

• [-] The process needs to be refined

• [-] https://github.com/fabric8-analytics/fabric8-analytics-server/issues/30

• [-] Cooperate with devels to integrate tests into MR workflow

• [?] Possibly use GIT hooks for this task

Regression tests

• [-] Define the process

• [-] Cooperate with devels to follow the process

• [-] Update/create Jenkins jobs to perform regression tests

---

Useful links

• Introductions to Python Testing Frameworks

• Behave framework

• unittest — Unit testing framework

• Requests: HTTP for Humans

• Regression testing

• Red Hat Developer Performance Tests