HTTP testing with vcr

Resources

Not many resources exist for this topic. Most important is the excellent HTTP testing in R book by rOpenSci. I recommend the Introduction (here, here, and here) for some background and the Whole Game vcr. To understand more about vcr specifically, the later chapters are good.

The problem

Our package, kbtbr is an API wrapper. This means that its main functionality is providing functions that make it easer to interact with the APIs of KoBoToolbox. For the (interactive) development of our functions, we use the CorrelAid KoBoToolbox server. Which is good because we can directly test out whether / how our functions work in the R console.

However, we should also put those 'informal' tests into formal tests (on why we test in general, read here). For 'normal' functions, we can write simple unit tests using just the testthat package. We could do the same for our API calling functions but this poses some problems:

1. our tests are dependent on an internet connection and the server being alive
2. we need to keep the test environment stable, i.e. we can never delete our test surveys/assets, let alone the whole api_user account.
3. it is hard to trigger certain things for testing, e.g. an internal server error.
4. tests rely on authentication credentials being present in the developer environment -> new contributors can't run the tests

You can read more about those challenges here

Especially the first two points are an important reason to consider an HTTP testing package. So what does it do and how does it work?

vcr

The vcr package is one of the solutions presented in the HTTR testing with R book. It is based on a Ruby gem (Ruby is a programming language) of the same name - which makes googling sometimes challenging.

vcr helps us with the challenges above in that it allows us to:

• bring the test environment local by recording the responses that we get from the server in yml files. This way we don't have to make 'real' requests when we test, we just use the local files. We can find the yml files in the tests/fixtures/ folder. This addresses challenges 1, 2 and 4 (if we edit out the credentials, which vcr does for us:)).
• we can manually edit those files to mock / fake situations which are time-consuming to manually construct on the user interface. (3)

When should we use vcr?

We should use vcr in our tests whenever a request is being made against the server.

For example:

• koboclient$get()
• koboclient$post()
• kobo$get_assets()

we don't have to use vcr when we test things like class initialization, utility functions, data cleaning functions etc.

How do we use vcr? What does it do exactly?

Short description from the PR

How does it work?

1. when we wrap a test with vcr (see how to do that here) and run it for the first time, vcr serializes the response from the server into a yml and puts it into the tests/fixtures folder with the filename that we specified as argument to use_cassette.
2. Later on, when we re-run the test, code is run as usual (the url is built, etc) until the private make_request function of HttpClient checks that vcr is enabled -> instead of making the request to the server, the cassette (=the yml) is loaded and the request that matches the request that was about to be made is replayed instead. The matching is based on the URI (e.g. "https://kobo.correlaid.org/api/v2/assets?format=json") and the HTTP method (get, post etc)
3. If you change the request in the test, make sure to delete the yml file so that vcr can record a new cassette for the changed test.

Long version

We use vcr by including a vcr::use_cassette(filename) call around our test. There are two options where to put the statement, see here. Most tests so far just wrap put it inside the test right around the code that makes the API call:

test_that("Kobo can fetch assets", {
  # the use_cassette command looks into the fixtures directory and checks
  # whether a "cassette" with the given name already exists. if yes, it loads it. if no, the
  # code is run and the response is saved as a cassette.
  vcr::use_cassette("kobo-get-assets", {
    kobo <- Kobo$new(base_url_v2 = BASE_URL, kobo_token = Sys.getenv("KBTBR_TOKEN"))
    assets <- kobo$get_assets()
  })
  expect_setequal(names(assets), c("count", "next", "previous", "results"))
  expect_true(all(c("url", "owner", "kind", "name", "asset_type") %in% colnames(assets$results)))
  expect_equal(nrow(assets$results), 8)
  expect_equal(assets$count, 8)
})

This will have the following effect: when we run the test, the following will happen:

1. the test will first run normally, i.e. our class will prepare the request, collecting all arguments etc. -> let's call this the prepared request/the request that is about to be made.
2. shortly before the API call is actually made, vcr "hooks" itself in (it detects that it is enabled in the test) and will:

• look into the tests/fixtures directory to look for a file with the name that we specified as argument to vcr::use_cassette() exists. in our example, it will look for kobo-get-assets.yml (cassettes are stored as yml so the extension is added automatically).
• if the file does not exist, it will just run the prepared request normally and then will write ("record") the request and response to the tests/fixtures folder as a yml file. In our example, vcr would create kobo-get-assets.yml with the recorded request and response (cf. the file on GitHub). This includes both information about the prepared request as well as the response from the server (i.e. the json with the assets). Business as usual, the request is made as if you would run it in the R console, but the response is recorded to a file for later.
• if the file does exist, vcr will load the recorded request and compare it to the prepared request to see whether they match with regards to the HTTP method (GET, ...) and the request URL (https://kobo.correlaid.org/api/v2/assets, https://kobo.correlaid.org/api/v2/assets/...). (see more here)
  • if those two things are not the same for the prepared request and the recorded request, vcr will throw an error saying that it did not find a matching request for the prepared request in the vcr file that it should be in (based on the argument of use_cassette()). This typically happens when we have run the test before but changed the way the request is made, e.g. different query parameters etc. In this case, delete the file and re-record the test (see here)
  • if those two things are the same for the prepared request and the recorded request, vcr will load the response and the prepared request is not made.

Challenges with vcr / Notes / Limitations

When writing those tests, a couple of things can be annoying/challenging:

• there is a lot of additional 'chatter' by devtools::test()
• the matching thing can be a trap, especially because Kobo tends to change things server-side (see my PR)

I haven't played around with mocking / editing the files manually to create server-side-error situations.