Giant Swarm Testing Infrastructure

Getting started

Interacting with our test-infra setup is currently done exclusively through the releases repository. On this repository you can trigger new test runs through comments on Pull Requests.

Please be aware that all interactions with test-infra are designed with the idea in mind that a pull request will either modify or add a single release in the releases repository! Commands which are currently available are as follows:

• /test cncf runs the cncf suite against the release.
• /test cis runs the cis suite against the release.
• /test aws runs awscnfm against the release - this only works for AWS.

There are multiple dashboards which can be useful to track the progress of tests:

• prow gives a rough overview of the running tests.
• tekton gives a more in depth overview of the test progress,
• kibana gives full insight of the logs emitted during your test.

Each test will generally perform the same flow currently which can be summarized as this:

1. Determine release CR from PR diff.
2. Create release CR on provider CP (gaia,godsmack,gorgoth).
3. Create tenant cluster with created release on target CP.
4. Run tests against created tenant cluster.
5. Report test results.
6. Delete tenant cluster.
7. Delete release on CP.

Feel free to reach out to team ludacris if you have further questions.

Developing test-infra

You can find guides for the creation of new tests in the docs subfolder.

Local development

A lot of functionality of tekton is available locally - the biggest exceptions being anything requiring an Ingress of webhooks.

Tools required for local development:

• KIND
• Kustomize
• Kubectl
• Giant Swarm internal tooling for secrets

To setup tekton in a KIND cluster you can simply do the following:

1. Create a cluster with kind create cluster
2. Apply all necessary files with ./tekton-kind.sh

Hack away!

In our ops cluster

Deploying test-infra is currently a bit cumbersome as it is running as an app inside a tenant cluster. The tenant cluster is rfjh2 on gorilla. To update test-infra you will need to manually edit the app CR on the gorilla CP in the namespace rfjh2. There is currently no automatic rollout by merging to master!

The roles and serviceaccounts on the target CPs are currently created manually. You can find the resources here.

Kubeconfigs to the target CPs are currently generated with a script using opsctl and then manually updated to rfjh2. You can find the necessary scripts here.

Included components

This repository contains manifests for our individual test pipelines as well as the components we run to build up our testing infrastructure. The two main open source projects used are Prow and Tekton

Prow

Prow is a CI/CD system running on Kubernetes. We utilize prow to trigger tests and have a nice Github integration to run high cost conformance tests.

Components

This is a list of components which we run:

• Deck - UI for prow jobs.
• Sinker - Clean up stale prow jobs.
• Hook - Handle GitHub events dispatching them to plugins.
• Horologium - Start periodic jobs.
• Plank - Start prow jobs.
• ghProxy - Designed to reduce Github API token usage.
• Tide - Automatically retest PRs & merge them if tests go green.
• Crier - Reports your prowjob status changes.
• Pipelines - Automatically creates Tekton pipelines for each Prow job.

Tekton

Tekton is a CI/CD system which is highly integrated with Kubernetes. We utilize it to actually run tests and to structure our test stages.

Components

This is a list of components which we run:

• Pipelines - This enables the meat of functionalities by Tekton.
• Dashboard - Gives and overview of active Tekton tasks and pipelines.
• prow-log-aggregator - Collects logs from Tekton tasks and makes them available to Deck.

Reference

• How to write ProwJobs