If a test collector is not available for your test framework, you can upload tests results directly to the Test Analytics API or write your own test collector. You can upload JSON-formatted test results (described in this page) or JUnit XML.
It's possible to import JSON (or JUnit files) to Buildkite Test Analytics with or without the help of a plugin.
To import JSON-formatted test results to Test Analytics using Test Collector plugin from a build step:
steps:
- label: "🔨 Test"
command: "make test"
plugins:
- test-collector#v1.0.0:
files: "test-data-*.json"
format: "json"{: codeblock-file="pipeline.yml"}
See more configuration information in the Test Collector plugin README.
Using the plugin is the recommended way as it allows for a better debugging process in case of an issue.
If for some reason you cannot or do not want to use the Test Collector plugin, or if you are looking to implement your own integration, another approach is possible.
To import JSON-formatted test results in Buildkite, make a POST request to https://analytics-api.buildkite.com/v1/uploads with a multipart/form-data.
For example, to import the contents of a JSON-formatted test results (test-results.json):
-
Securely set the Test Analytics token environment variable (
BUILDKITE_ANALYTICS_TOKEN). -
Run the following
curlcommand:curl \ -X POST \ -H "Authorization: Token token=\"$BUILDKITE_ANALYTICS_TOKEN\"" \ -F "data=@test-results.json" \ -F "format=json" \ -F "run_env[CI]=buildkite" \ -F "run_env[key]=$BUILDKITE_BUILD_ID" \ -F "run_env[url]=$BUILDKITE_BUILD_URL" \ -F "run_env[branch]=$BUILDKITE_BRANCH" \ -F "run_env[commit_sha]=$BUILDKITE_COMMIT" \ -F "run_env[number]=$BUILDKITE_BUILD_NUMBER" \ -F "run_env[job_id]=$BUILDKITE_JOB_ID" \ -F "run_env[message]=$BUILDKITE_MESSAGE" \ https://analytics-api.buildkite.com/v1/uploads
To learn more about passing through environment variables to run_env-prefixed fields, see CI environments.
A single file can have a maximum of 5000 test results, and if that limit is exceeded then the upload request will fail. To upload more than 5000 test results for a single run upload multiple smaller files with the same run_env[key].
To import JSON-formatted test results, make a POST request to https://analytics-api.buildkite.com/v1/uploads with a multipart/form-data body, including as many of the following fields as possible in the request body:
For example, to import the contents of a test-results.json file in a CircleCI pipeline:
-
Securely set the Test Analytics token environment variable (
BUILDKITE_ANALYTICS_TOKEN). -
Run the following
curlcommand:curl \ -X POST \ -H "Authorization: Token token=\"$BUILDKITE_ANALYTICS_TOKEN\"" \ -F "data=@test-results.json" \ -F "format=json" \ -F "run_env[CI]=circleci" \ -F "run_env[key]=$CIRCLE_WORKFLOW_ID-$CIRCLE_BUILD_NUM" \ -F "run_env[number]=$CIRCLE_BUILD_NUM" \ -F "run_env[branch]=$CIRCLE_BRANCH" \ -F "run_env[commit_sha]=$CIRCLE_SHA1" \ -F "run_env[url]=$CIRCLE_BUILD_URL" \ https://analytics-api.buildkite.com/v1/uploads
To learn more about passing through environment variables to run_env-prefixed fields, see CI environments.
A single file can have a maximum of 5000 test results, and if that limit is exceeded then the upload request will fail. To upload more than 5000 test results for a single run upload multiple smaller files with the same run_env[key].
To import JSON-formatted test results, make a POST request to https://analytics-api.buildkite.com/v1/uploads with a multipart/form-data body, including as many of the following fields as possible in the request body:
For example, to import the contents of a test-results.json file in a GitHub Actions pipeline run:
-
Securely set the Test Analytics token environment variable (
BUILDKITE_ANALYTICS_TOKEN). -
Run the following
curlcommand:curl \ -X POST \ -H "Authorization: Token token=\"$BUILDKITE_ANALYTICS_TOKEN\"" \ -F "data=@test-results.json" \ -F "format=json" \ -F "run_env[CI]=github_actions" \ -F "run_env[key]=$GITHUB_ACTION-$GITHUB_RUN_NUMBER-$GITHUB_RUN_ATTEMPT" \ -F "run_env[number]=$GITHUB_RUN_NUMBER" \ -F "run_env[branch]=$GITHUB_REF" \ -F "run_env[commit_sha]=$GITHUB_SHA" \ -F "run_env[url]=https://github.com/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID" \ https://analytics-api.buildkite.com/v1/uploads
To learn more about passing through environment variables to run_env-prefixed fields, see CI environments.
A single file can have a maximum of 5000 test results, and if that limit is exceeded then the upload request will fail. To upload more than 5000 test results for a single run upload multiple smaller files with the same run_env[key].
JSON test results data is made up of an array of one or more test result objects. A test result object contains an overall result and metadata. It also contains a history object, which is a summary of the duration of the test run. Within the history object, detailed span objects record the highest resolution details of the test run.
Schematically, the JSON test results data is like this:
Or in a simplified code view:
[
{
/* Test result object */
"history": {
/* history object */
"children": [
/* span objects */
]
}
},
{ /* Test result object */ },
]A test result represents a single test run.
<% def render_enumerated_values(values) if values.nil? or values.length == 1 return '' end
if values.length == 1
return "(#{values[0]})"
end
if values.length == 2
return "(#{values[0]} or #{values[1]})"
end
return '(' + values[0..-2].map{|value| "#{value}"}.join(', ') + " or #{values[-1]})"
end
%>
| Key | Type | Description | Examples |
|---|---|---|---|
<%= field['name'] %> <%= '(required)' if field['required'] %> |
<%= field['type'] %> <%= render_enumerated_values(field['enumerated_values']) %> | <%= render_markdown(text: field['desc']) %> |
<%= field['examples'].map{|example| "#{render_markdown(text: example)}"}.join(', ') unless field['examples'].nil? %>
|
Example:
{
"id": "95f7e024-9e0a-450f-bc64-9edb62d43fa9",
"scope": "Analytics::Upload associations",
"name": "fails",
"location": "./spec/models/analytics/upload_spec.rb:24",
"file_name": "./spec/models/analytics/upload_spec.rb",
"result": "failed",
"failure_reason": "Failure/Error: expect(true).to eq false",
"failure_expanded": [
/* failure_expanded object */
],
"history": {
/* history object */
}
}A failure expanded array contains extra details about the failed test.
<% TEST_ANALYTICS_JSON_FIELDS_FAILURE_EXPANDED['fields'].each do |field| -%> <% end -%>| Key | Type | Description |
|---|---|---|
<%= field['name'] %> <%= '(required)' if field['required'] %> |
<%= field['type'] %> <%= render_enumerated_values(field['enumerated_values']) %> | <%= render_markdown(text: field['desc']) %> |
Example:
{
"expanded": [
" expected: false",
" got: true",
"",
" (compared using ==)",
"",
" Diff:",
" @@ -1 +1 @@",
" -false"," +true"
],
"backtrace": [
"./spec/models/analytics/upload_spec.rb:25:in `block (3 levels) in <top (required)>'","./spec/support/log.rb:17:in `run'",
"./spec/support/log.rb:66:in `block (2 levels) in <top (required)>'",
"./spec/support/database.rb:19:in `block (2 levels) in <top (required)>'",
"/Users/abc/Documents/rspec-buildkite-analytics/lib/rspec/buildkite/analytics/uploader.rb:153:in `block (2 levels) in configure'",
"-e:1:in `<main>'"
]
}A history object represents the overall duration of the test run and contains detailed span data, more finely recording the test run.
<% TEST_ANALYTICS_JSON_FIELDS_HISTORY['fields'].each do |field| -%> <% end -%>| Key | Type | Description |
|---|---|---|
<%= field['name'] %> <%= '(required)' if field['required'] %> |
<%= field['type'] %> <%= render_enumerated_values(field['enumerated_values']) %> | <%= render_markdown(text: field['desc']) %> |
Example:
{
"start_at": 347611.724809,
"end_at": 347612.451041,
"duration": 0.726232000044547,
"children": [
/* span objects */
]
}Span objects represent the finest duration resolution of a test run. It represents, for example, the duration of an individual database query within a test.
<% TEST_ANALYTICS_JSON_FIELDS_SPAN['fields'].each do |field| -%> <% end -%>| Key | Type | Description |
|---|---|---|
<%= field['name'] %> <%= '(required)' if field['required'] %> |
<%= field['type'] %> <%= render_enumerated_values(field['enumerated_values']) %> | <%= render_markdown(text: field['desc']) %> |
Example:
{
"section": "sql",
"start_at": 347611.734956,
"end_at": 347611.735647,
"duration": 0.0006910000229254365
"detail": {
...
}
}Detail objects contains additional information about the span.
<% TEST_ANALYTICS_JSON_FIELDS_DETAIL['fields'].each do |field| -%> <% end -%>| Key | Type | Description |
|---|---|---|
<%= field['name'] %> <%= '(required)' if field['required'] %> |
<%= field['type'] %> <%= render_enumerated_values(field['enumerated_values']) %> | <%= render_markdown(text: field['desc']) %> |
HTTP Example:
{
"detail": {
method: "POST",
url: "https://example.com",
lib: "curl"
}
}SQL Example:
{
"detail": {
query: "SELECT * FROM ..."
}
}