Skip to content

Latest commit

 

History

History
254 lines (194 loc) · 12.4 KB

ONBOARDING.md

File metadata and controls

254 lines (194 loc) · 12.4 KB
Raw

So you want to write Complement tests

Complement is a black box integration testing framework for Matrix homeservers.

This document will outline how Complement works and how you can add efficient tests and best practices for Complement itself. If you haven't run Complement tests yet, please see the README and start there!

Terminology

  • Blueprint: a human-readable outline for what should be done prior to a test (such as creating users, rooms, etc).
  • Deployment: controls the lifetime of a Docker container (built from a Blueprint). It has functions on it for creating deployment-scoped structs such as Client-Server API clients for interacting with specific homeservers in the deployment.

Architecture

Each Complement test runs one or more Docker containers for the homeserver(s) involved in the test. These containers are snapshots of the target homeserver at a particular state. The state of each container is determined by the Blueprint used. Client-Server and Server-Server API calls can then be made with assertions against the results.

In order to actually write a test, Complement needs to:

  • Create Deployments, this is done by calling deployment := Deploy(...) (and can subsequently be killed via deployment.Destroy(...)).
  • (Potentially) make API calls against the Deployments.
  • Make assertions, this can be done via the standard Go testing mechanisms (e.g. t.Fatalf), but Complement also provides some helpers in the must and match packages.

For testing outbound federation, Complement implements a bare-bones Federation server for homeservers to talk to. Each test must explicitly declare the functionality of the homeserver. This is done using functional options and looks something like:

// A federation server which handles serving up its own keys when requested,
// automatically accepts make_join and send_join requests and deals with
// room alias lookups.
srv := federation.NewServer(t, deployment,
    federation.HandleKeyRequests(),
    federation.HandleMakeSendJoinRequests(),
    federation.HandleDirectoryLookups(),
)
// begin listening on a goroutine
cancel := srv.Listen()

// Shutdown the server at test end
defer cancel()

Network topology for all forms of communication look like this:

+------+                  Outbound federation           +-----------+             Network: one per blueprint                +-----------+
|      | :12345 <------ host.docker.internal:12345 ---- |           |                                                       |           |
| Host |                                                | Container | ------- SS API https://hs2/_matrix/federation/... --> | Container |
|      | -------CS API http://localhost:54321 --------> |   (hs1)   |                                                       |   (hs2)   |
|      | ------SS API https://localhost:55555 --------> |           |                                                       |           |
+------+                                                +-----------+                                                       +-----------+
                                             docker -p 54321:8008 -p 55555:8448

The high numbered ports are randomly chosen, and are for illustrative purposes only.

The mapping of hs1 to localhost:port combinations can be done automatically using a docker.RoundTripper.

How do I...

Get a Client-Server API client:

// the user and homeserver name are from the blueprint
// automatically maps localhost:12345 to the right container
alice := deployment.Client(t, "hs1", "@alice:hs1")

Make a Federation server:

srv := federation.NewServer(t, deployment,
    federation.HandleKeyRequests(),
    federation.HandleMakeSendJoinRequests(),
)
cancel := srv.Listen()
defer cancel()

Get a Federation client:

// Federation servers sign their requests, so you need a server before
// you can make a client.
// automatically accepts self-signed TLS certs
// automatically maps localhost:12345 to the right container
// automatically signs requests
// srv == federation.Server
fedClient := srv.FederationClient(deployment)

FAQ

How should I name the test files / test functions?

Test files have to have _test.go else Go won't run the tests in that file. Other than that, there are no restrictions or naming convention. If you are converting a sytest be sure to add a comment anywhere in the source code which has the form:

// sytest: $test_name

e.g:

// Test that a server can receive /keys requests:
// https://matrix.org/docs/spec/server_server/latest#get-matrix-key-v2-server-keyid
// sytest: Federation key API allows unsigned requests for keys
func TestInboundFederationKeys(t *testing.T) {
    ...
}

Adding // sytest: ... means sytest_coverage.go will know the test is converted and automatically update the list when run! Use go run sytest_coverage.go -v to see the exact string to use, as they may be different to the one produced by an actual sytest run due to parameterised tests.

Where should I put new tests?

If the test only has CS API calls, then put it in /tests/csapi. If the test involves both CS API and Federation, or just Federation, put it in /tests. This will change in the future once we have decided how to split tests by category.

Should I always make a new blueprint for a test?

Probably not. Blueprints are costly, and they should only be made if there is a strong case for plenty of reuse among tests. In the same way that we don't always add fixtures to sytest, we should be sparing with adding blueprints.

How should I assert JSON objects?

Use one of the matchers in the match package (which uses gjson) rather than json.Unmarshal(...) into a struct. There's a few reasons for this:

  • Removes the temptation to use gomatrixserverlib structs.
  • Forces exact key matching (without JSON tags, json.Unmarshal will case-insensitively match fields by default).
  • Clearer syntax: match.JSONKeyEqual("earliest_events", []interface{}{latestEvent.EventID()}), is clearer than unmarshalling into a slice then doing assertions on the first element.

If you want to extract data from objects, just use gjson directly.

How should I assert HTTP requests/responses?

Use the corresponding matcher in the match package. This allows you to be as specific or as lax as you like on your checks, and allows you to add JSON matchers on the HTTP body.

I want to run a bunch of tests in parallel, how do I do this?

This is done using the standard Go testing mechanisms. Add t.Parallel() to all tests which you want to run in parallel. For a good example of this, see registration_test.go which does:

// This will block until the 2 subtests have completed
t.Run("parallel", func(t *testing.T) {
    // run a subtest
    t.Run("POST {} returns a set of flows", func(t *testing.T) {
        t.Parallel()
        ...
    })
    // run another subtest
    t.Run("POST /register can create a user", func(t *testing.T) {
        t.Parallel()
        ...
    })
})

Tests in a directory will run in parallel with tests in other directories by default. You can disable this by invoking go test -p 1 which will force a parallelisation factor of 1 (no parallelisation).

How should I do comments in the test?

Add long prose to the start of the function to outline what it is you're testing (and why if it is unclear). For example:

// Test that a server can receive /keys requests:
// https://matrix.org/docs/spec/server_server/latest#get-matrix-key-v2-server-keyid
func TestInboundFederationKeys(t *testing.T) {
    ...
}

I think Complement is doing something weird, can I get more logs?

You can pass COMPLEMENT_DEBUG=1 to add lots of debug logging. You can also do this via os.Setenv("COMPLEMENT_DEBUG", "1") before you make a deployment. This will add trace logging to the clients which logs full HTTP request/responses, amongst other debug info.

How do I set up a bunch of stuff before the tests, e.g before each?

There is no syntactically pleasing way to do this. Create a separate function which returns a function. See https://stackoverflow.com/questions/42310088/setup-and-teardown-for-each-test-using-std-testing-package?rq=1

How do I log messages in tests?

This is done using standard Go testing mechanisms, use t.Logf(...) which will be logged only if the test fails or if -v is set. Note that you will not need to log HTTP requests performed using one of the built in deployment clients as they are already wrapped in loggers. For full HTTP logs, use COMPLEMENT_DEBUG=1.

How do I show the server logs even when the tests pass?

Normally, server logs are only printed when one of the tests fail. To override that behavior to always show server logs, you can use COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1.

How do I skip a test?

To conditionally skip a single test based on the homeserver being run, add a single line at the start of the test:

runtime.SkipIf(t, runtime.Dendrite)

To conditionally skip an entire file based on the homeserver being run, add a build tag at the top of the file which will skip execution of all the tests in this file if Complement is run with this flag:

// +build !dendrite_blacklist

You can also do this based on features for MSC tests (which means you must run Complement with this tag for these tests to run):

// +build msc_2836

See GH Actions for an example of how this is used for different homeservers in practice.

Why do we use t.Errorf sometimes and t.Fatalf other times?

Error will fail the test but continue execution, where Fatal will fail the test and quit. Use Fatal when continuing to run the test will result in programming errors (e.g nil exceptions).

How do I run tests inside my IDE?

Make sure you have first built a compatible complement image, such as complement-dendrite:latest (see README.md "Running against Dendrite"), which will be used in this section. (If you're using a different server, replace any instance of complement-dendrite:latest with your own tag)

For VSCode, add to settings.json:

"go.testEnvVars": {
    "COMPLEMENT_BASE_IMAGE": "complement-dendrite:latest"
}

For Goland:

  • Under "Run"->"Edit Configurations..."->"Edit Configuration Templates..."->"Go Test", and add COMPLEMENT_BASE_IMAGE=complement-dendrite:latest to "Environment"
  • Then you can right-click on any test file or test case and "Run ".

How do I make the linter checks pass?

Use goimports to sort imports and format in the style of gofmt.

Set this up to run on save in VSCode as follows:

  • File -> Preferences -> Settings.
    • Search for "Format On Save" and enable it.
    • Search for go: format tool and choose goimports.

How do I hook up a Matrix client like Element to the homeservers spun up by Complement after a test runs?

It can be useful to view the output of a test in Element to better debug something going wrong or just make sure your test is doing what you expect before you try to assert everything.

  1. In your test comment out defer deployment.Destroy(t) and replace with defer time.Sleep(2 * time.Hour) to keep the homeserver running after the tests complete
  2. Start the Complement tests
  3. Save the Element config as ~/Downloads/riot-complement-config.json and replace the port according to the output from docker ps (docker ps -f name=complement_ to just filter to the Complement containers) 
    {
  "default_server_config": {
    "m.homeserver": {
      "base_url": "http://localhost:55449",
      "server_name": "my.complement.host"
    }
  },
  "brand": "Element"
}
  4. Start up Element (your matrix client) 
    docker run -it --rm \
    --publish 7080:80 \
    --volume ~/Downloads/riot-complement-config.json:/app/config.json \
    --name riot-complement \
    vectorim/riot-web:v1.7.8
  5. Now you can visit http://localhost:7080/ and register a new user and explore the rooms from the test output

What do I need to know if I'm coming from sytest?

Unlike Sytest, each test must opt-in to attaching core functionality to the test federation server so the reader can clearly see what is and is not being handled automatically.