Working with Prow

Prow provides CI features as well as a set of tools to improve developers productivity. Prow was developed by the Kubernetes community and is deployed in Google Container Engine, but could be deployed in any Kubernetes cluster.

Prow runs pre-submit, post-submit and periodic jobs, but also comes with a set of productivity tools:

• Tide: Automatically merges PRs
• hold: Hold a PR such that it is not merged
• branch-protection: Update github branch protection based on configuration and prow jobs config.
• needs-rebase: Alert used that PR needs rebase

Prow is available at https://prow.istio.io and more details about the plugins can be found on Prow Command Help section.

Prow Configuration

There are 2 main configurations:

• config.yaml: Define jobs, general settings
• plugins.yaml: Plugins configuration

All deployment information is stored here.

Prow configuration can be updated via PR on istio/test-infra. Once the PR is merged, Prow will update its configuration.

Branch Protection

Istio uses GitHub Branch Protection to control use of individual branches. Prow helps Org Admins manage this, by setting multiple repo/branch protections at once and keeping changes in git history. It enables contributors to add or remove required checks and it enable all of us to be able to audit.

Each update is done via PR. Everybody can create a PR, but only a certain set of people can approve them (as determined by the OWNERS file). All changes are stored within Git history.

In Branch Protection, prow jobs are protected by default and do not need to be specified. In order to make a prow job optional, just add optional: true to its job definition.

A daily cronjob will push branch protection changes to GitHub, meaning that manual changes in GitHub will be overridden.

More updated information about branch-protection.

Tide (Merge Bot)

Tide will merge PRs that have been approved and have all required tests passing. If a PR needs to be rebased, a comment will be added to the PR to alert the user.

When merging a PR, Tide will check to ensure that the PR has been tested with the latest changes. This ensures that two conflicting PRs do not get merged. If a PR is approved, but other changes have been merged since the tests were ran, Tide will queue the PR for retesting. Because this process can make it hard to get a large volume of changes merged, Tide will also batch changes together. For example, if 10 PRs were approved at once, Tide would trigger a test with all of these PRs merged together, into a "Batch". If the tests pass, the whole batch will be merged at once.

Tide configuration is stored in config.yaml in the tide: section. Tide works by querying GitHub for PRs (with existing or missing labels, GitHub approval, etc...) and attempting to merge them.

More updated information about tide.

Tide's status can be found on the status page.

Why isn't my PR merging?

Tide will post a status explaining why a PR is not ready to merge, indicating the PR needs an approval or a test has failed for example.

Occasionally, every check will pass, but the PR is not merged. This is likely because Tide is waiting for a batch to complete, or the PR is queued for a retest. In these cases, the tide status will indicate the PR is in the merge pool, with a link to the status page with more info.

Boskos

Boskos is the system that manages resources for Prow. When our E2E tests run, they will acquire a dedicated cluster to run tests on. Occasionally, tests will fail with errors "failed to acquire resource from Boskos". This may happen when resources are fully utilized. Resource usage can be found on the boskos dashboard.

We plan to decrease our usage of Boskos in favor of kind.

kind

kind is a tool to run a full Kubernetes cluster inside a docker container, with startup times under a minute. This allows lightweight tests that depend on Kubernetes. Kind can easily handle complex Kubernetes setups, such as multiple nodes, different Kubernetes versions, etc. Kind also allows developers to run the same tests that are run in prow locally.

Currently, our integration tests utilize kind. These can be run locally, by executing the same test script run on Prow. For example, to run the Pilot integration tests, run prow/integ-suite-kind.sh test.integration.pilot.kube. This will build all docker images locally, create a cluster, and load the images directly into the cluster (no docker registry required), before running the test. On subsequent runs you can set --skip-setup --skip-build to skip creating a cluster and building docker images

Writing Tests

Configuration for all tests can be found in here. See this doc for instructions on creating new jobs. After creating a job or making modifications, run make gen to update the autogenerated files.

Testing changes locally

There are a few ways to test prow jobs locally.

For the most part, tests should be able to run completely independently from prow, so to run the unit tests you can just run make test and get the same behavior. However, some tests are more complicated, performing tasks like checking out multiple repos or using docker in docker for example.

To create a ProwJob, which is a CRD that defines a job, you can use mkpj. Since the deprecation of go get to install standalone binaries, you have to install mkpj from source.

git clone --depth 1 git@github.com:kubernetes/test-infra.git
cd test-infra/prow/cmd/mkpj
go install

​ To use, run a command like:

mkpj --github-token-path=./path/to/oauth/token --config-path prow/config.yaml --job-config-path prow/cluster/jobs/ --job istio-unit-tests-master > prowjob.yaml

(this assumes your working directory is https://github.com/istio/test-infra). For presubmit jobs, you will need to enter a PR number to run against, and for postsubmit jobs a branch. --github-token-path is a path to a file containing a GitHub personal access token with repo scope.

Next, the job can be run locally using phaino, which can be installed from source in a similar way as mkpj.

To run the job we just created, you can run:

phaino --privileged prowjob.yaml

Additionally, all test results will have a link to a ProwJob YAML. This can be used to directly rerun a test, for example phaino https://prow.istio.io/prowjob?prowjob=my-test-id-here.

Alternatively, the test can be run in a Kubernetes cluster, just like the are in Prow, by converting the ProwJob into a Pod definition. This can be done with mkpodwhich can be in a similar way as mkpj.

To create a Pod from the ProwJob we created above, run:

mkpod --local --prow-job prowjob.yaml > pod.yaml

This can be applied into any cluster. Note: some tests rely on secrets mounted in the cluster to do things like upload to GCR, or interact with Boskos. If you do not have these secrets, the tests may fail. If you do, please use caution. Further, pods may have node selectors and require service accounts. ​ The resulting artifacts will be in /tmp of the node that ran the job. To access the artifacts, run ​

kubectl debug -it --image=ubuntu node/<node that ran the earlier pod>
$ cd /host/tmp/<job output directory>