Skip to content

Demo of multi-layer test automation in a two service Node web app with third-party data depencies, including Consumer Driven Contract Testing

Notifications You must be signed in to change notification settings

jimCresswell/general-JS-test-automation-demo

Repository files navigation

General JS Test Automation Demo for a Multi-Service Web App

This is a general functional test automation demo for JavaScript digital services. The project has a provider service, which manages data about pollinator supporting plants (with a focus on the UK/northern Europe), and a consumer service which takes that data, enhances it with more data from Wikipedia, and renders the result in a responsive web GUI. The services communicate over HTTP and serve their results on localhost.

The automated tests operate a different levels, from unit to individual service e2e test to whole-of-product tests, with different intents and for different audiences. The project includes a consumer driven contract testing (CDCT) demo using Pact JS. See the testing section below for more information.


Effectively the product takes this

{
  "id": 2,
  "common_name": "Marsh Marigold",
  "species": "Caltha palustris",
  "perennial": true,
  "wikilink": "https://en.wikipedia.org/wiki/Caltha_palustris",
  "notes": "Flowers for most of the year, does well in very wet or flooded sites.",
  "supports": [
    {
      "common_name": "Marsh-Marigold Moth",
      "species": "Micropterix calthella",
      "wikilink": "https://en.wikipedia.org/wiki/Micropterix_calthella"
    },
    {
      "common_name": "Hoverflies",
      "species": "Syrphidae (family)",
      "wikilink": "https://en.wikipedia.org/wiki/Hoverfly"
    }
  ]
},

Decorates it with additional information from Wikipedia using the provided URLs, and turns it into this

An image of the consumer web GUI showing the Marsh Marigold section

Project Structure

The Consumer

The consumer is an Express server-side web app. It takes data from the provider app, combines it with additional data from the Wikipedia REST API, and renders it using Handlebars templates into a responsive web GUI built with Bootstrap and a small amount of custom jQuery logic (in line with the Bootstrap logic). HTTP calls within the app use the Axios library. The app includes unit and end-to-end tests and consumer driven contract tests built with Pact, see below for more information.

The Provider

The provider is an Express based CRUD app in front of a data model and in-memory data store. The app includes unit and end-to-end tests and code to verify compliance with consumer driven Pact contracts, see below for more information.

The Product Level Tests

Product level specifications verified against the system as a whole using Cucumber and WebDriver. See the relevant README for more details.


Usage

Everything is controlled from scripts in the package.json. For script execution yarn can be replaced with npm run. The scripts are described in the relevant sections below, here is a summary.

script result
test:unit:consumer unit test the consumer
test:unit:provider unit test the provider
test:unit:debug unit test the consumer and provider with Node listening for a debugging client
test:e2e:consumer consumer end-to-end tests
test:e2e:provider provider end-to-end tests
test:contract:consumer run consumer contract tests and create contracts (pact files)
test:contract:provider verify the contracts (pact files) against the provider
publish_pacts publish the contracts (pact files) to a broker service
test:product end-to-end tests of the product as a whole
start start the services together
start:consumer start the consumer and serve on localhost
start:provider start the provider and serve on localhost

Running

All processes are started with Nodemon and are configured to restart the relevant server file change.

  • yarn start Starts the whole system using Concurrently.
  • yarn start:consumer Starts just the consumer.
  • yarn start:provider Starts just the provider.

Testing

The primary purpose of automated testing is to rapidly and reproducibly measure when work is complete (when the system behaviour meets expectations). Work happens at many levels, from whole-of-product behaviour design to single component technical implementation to performance and security concerns, so automated testing happens at multiple levels of granularity, with different intent, different tools and different audiences for the feedback it provides on different timescales (milliseconds to tens of minutes). In general, the earlier in the development process a test can be run the faster its feedback can be used to correct mistakes before rework becomes necessary.

This also means that the tests often have different authors. Unit, integration and technical API tests should sit with engineers, who are also the main audience for those tests. Higher level tests such as product behaviour validation, can be a collaboration between most stakeholders, and communication of results should happen in formats and through channels appropriate to that broad audience.

As a secondary effect, the automated tests also check that the system does not unintentionally drift from expected behaviour.

Writing automated tests after development is "complete" misses most of the benefits of testing and leads to avoidable rework. Quality cannot be tested into a product, it needs to be built in from conception to realisation. For some testing, e.g. product level GUI tests, it is very hard to create the tests before implementation work begins, but it can and should be done in parallel and the work not considered complete until the automated validation is successful.

A complete automated test suite, covering the different levels and aspects of a digital product, is incredibly powerful and done well will hugely increase delivery velocity as part of a continuous integration process. However the majority of the digital products we create are ultimately for human use and so we will always need humans, sometimes engineers sometimes users, to validate the subject human experience that defines our products.

Non-functional concerns, such as product level performance and security, are specialist areas. Automation can be set up to catch some issues, but experts should be brought in to advise or audit. For product level performance, automated tests are only an approximation and focus should be given to log design and analysis to gain real world statistical insights.

A Note on Test Coverage

Test coverage is a very useful statistic at lower levels of testing, and less useful at higher levels of testing designed to validate the services a product provides rather than technical implementation. See here for an example of test coverage driven by unit tests.

Unit Testing

Unit tests provide the fastest feedback (alongside automated syntax and style checking) when run locally on file change during development. This gives developers technical feedback while they are working on the problem and avoids the high mental cost of task switching.

They are also typically run as part of continuous integration, this can be especially important after code merges which have the potential to introduce syntax errors (and also unexpected emergent behaviour, but that is checked by higher-level test approaches).

yarn test:unit Provider and Consumer unit tests with Mocha and Chai, asynchronous assertions are demonstrated in a variety of ways including Chai-as-promised. Where a network interaction is tested HTTP calls and responses are mocked with Axios Mock Adapter.

The unit tests live with modules they test e.g. my_module.js will have a my_module.test.js in the same directory.

Consumer Driven Contract Testing (with Pact JS)

Consumer Side
  • yarn test:contract:consumer Generate the consumer contracts, this is where the actual tests exist.
  • yarn publish_pacts Publish the consumer contracts to a remote Pact broker. See here for the code.
Provider Side

End to End API Testing of Isolated Services

These have a different intent from the contract tests. On the consumer side they test the whole consumer service rather than just the module that talks to the provider service. On the provider side they test all provider service end-points whereas the contract tests by design only test the interactions and returned payload data required by the consumer service.

Consumer

yarn test:e2e:consumer Consumer end-to-end API sanity test with supertest. Where a network interaction is tested HTTP calls and responses are mocked with Axios Mock Adapter. Returned HTML is parsed with Cheerio. These tests are in the consumer/e2e_test directory.

Provider

yarn test:e2e:provider Provider end-to-end API sanity test with supertest. These tests are in the provider/e2e_test directory.

Product Level Testing

yarn test:product see details here.


Potential Future Improvements

  • Configure a CI platform to hierarchically run the tests, fastest first bail on suite failure, and report to Github. See here for an existing example with CI configuration.
  • Add various status badges to the main README.
  • Deploy the services to cloud hosting and put a link in the Github page.