Skip to content

Testing strategy and how to test

Jump to bottom
Jose Alcérreca edited this page Mar 27, 2024 · 1 revision

Types of tests in NiA

From smallest to biggest:

Local tests

Instrumented tests

Types of tests that we don't use:

  • Robolectric non-UI tests, using shadows as test doubles
  • UI Automator tests for general UI testing

General guidelines

As a general rule we adhere to the idea behind shift-left testing, trying to find bugs as early in the process as possible, and using bigger tests for catching regressions.

Visual elements are verified with screenshot tests. APIs that check colors, fonts etc. should not be used.

When a component correctly depends on platform types, such as Context or isSystemInDarkTheme, it can be run as an instrumentation test.

If something can be tested in a reasonable way, it must be tested. There are some classes and functions that are not (and should not be) covered by tests, such as Compose previews, data classes, Hilt modules, etc.

Code coverage

We use code coverage figures to detect gaps in testing, but a minimum coverage is not enforced, as it depends a lot on the type of feature.

Our CI system shows a coverage report for information only. The author and reviewers must decide if the changed files are conveniently tested.

Screenshot from 2024-03-27 10-41-05

Test doubles (official docs)

Don't use mocking frameworks. Instead, use fakes. Fakes can live in the same file where tests are used, in their own file inside the test source set or, if shared across multiple modules, in a shared module like :core:data-test or :core:testing.

Naming

Generally, classes are named as {SubjectUnderTest}Test. They must contain a link in their kdoc to the subject under test.

Test names should try to follow the following naming convention: given_when_then.

  • given - a subject or scenario
  • when - a trigger, situation or event occurs
  • then - a result is produced

Other rules for consistency:

  • Do not add the subject under test in test names, it should be already present in the class name.
  • Do not add "given", "when" or "then".
  • Use lowerCamelCase separated by underscores.

Example:

/**
 * UI tests for [BookmarksScreen] composable.
 */
class BookmarksScreenTest {
    @Test
    fun feed_hasNoBookmarks_showsEmptyState() { ... }
}

Variants

Logic present in different variants must be tested independently. Place tests in their corresponding source sets. For example: testDemo or androidTestDemo for tests that only apply to the demo product flavor.

How to test: Features

Screenshot tests

The visual aspects of the UI must be verified by screenshot tests.

You can use composeTestRule.captureMultiTheme to create multiple versions of a UI in the different themes. You should test the different relevant states of the UI.

If the component behaves differently when placed on different size screens or parents, include tests to verify this.

UI tests

If your UI contains logic, it should be tested using the Compose Test APIs. Place these tests in the test/ source set so that they're run with Robolectric, but do not use any of the Robolectric shadows or any APIs that would prevent this test from running on a device, as a regular instrumented test.

Configuration changes should be covered by tests when state restoration is involved. Use StateRestorationTester for this.

Unit tests

If the feature contains a ViewModel, it must be unit tested with JVM tests targeting 100% of coverage. You can use Turbine to verify complex flows, but most should be tested with standard tools.

UseCases and other classes with no platform dependencies must also be unit tested with JVM tests.

How to test: Design system

Every component in :core:designsystem must be covered by screenshot tests in every combination of themes and state.

When they display text they must contain a screenshot test with a font scale of 2.

How to test: Layouts

Layouts must be covered by screenshot tests on the different COMPACT, MEDIUM and EXPANDED widths (and heights, if relevant). See NiaAppScreenSizesScreenshotTests for an example.

How to work with screenshot tests

A screenshot test takes a screenshot of a screen or a UI component within the app, and compares it with a previously recorded screenshot which is known to be rendered correctly.

For example, Now in Android has screenshot tests to verify that the navigation is displayed correctly on different screen sizes (reference images).

Now In Android uses Roborazzi to run screenshot tests of certain screens and UI components. When working with screenshot tests the following Gradle tasks are useful:

  • verifyRoborazziDemoDebug run all screenshot tests, verifying the screenshots against the known correct screenshots.
  • recordRoborazziDemoDebug record new "known correct" screenshots. Use this command when you have made changes to the UI and manually verified that they are rendered correctly. Screenshots will be stored in modulename/src/test/screenshots.
  • compareRoborazziDemoDebug create comparison images between failed tests and the known correct images. These can also be found in modulename/src/test/screenshots.

Note: The known correct screenshots stored in this repository are recorded on CI using Linux. Other platforms may (and probably will) generate slightly different images, making the screenshot tests fail. When working on a non-Linux platform, a workaround to this is to run recordRoborazziDemoDebug on the main branch before starting work. After making changes, verifyRoborazziDemoDebug will identify only legitimate changes. You don't have to check in the reference (or golden) images as they're generated automatically by Github Actions.

For more information about screenshot testing check out this talk.

Clone this wiki locally