What quality gates do you have?

[justinabrahms]

I'm curious what quality gates you have in your pipelines. If I sat and thought about it, I suspect my first pass would only capture about 40% of the interesting things I've seen people do. As a way to help others, let's document what we use for quality gates so people can copy all of our good ideas to their own teams. :)

[justinabrahms]

eBay uses (to varying degrees of ubiquity)..

Build:

• Checking CVEs in SBOM
• Validate unit tests pass
• Run consumer-driven contract tests
• quality checks (e.g. sonar)

After first environment deploy

• run e2e tests

After prod deploy:

• canary rollout, monitoring a few key metrics for the latest build

[oleg-nenashev]

FTR Keptn is reviewing its terminology at the moment, and terms like "quality gate" are under review. We will follow-up with the SIG and with the CD Events Terminology track once we have news. CC @StackScribe

[fdegir]

To add another dimension to the discussion, various activities could be run based on when things happen per PR/MR/change/commit/artifact/loop basis.

For example

• Commit Gating: These are generally used for gating PRs/MRs/changes to make sure they don't break things. Various activities that are run as part of these gates are
  • Commit message validation (pre-push hooks could be used as well but I've seen this being part of some communities)
  • CLA checks
  • Static analysis
  • FOSS Checks
  • Build & Packaging
  • Deployment on virtual environments
  • Contract Tests
  • Healthcheck
• Project Gating: Is a mechanism where CI takes over the ability to merge code into the repo. Instead it first re-runs the Commit Gating checks against the merge result of a PR/MR/change and only merges if it passes. This is an in-between gate between Commit Gating and Post-merge.
• Post-merge: These are used once PRs/MRs/changes are merged to verify the sanity of the main development branch and depending on the setup, the resulting feedback could be used to determine whether the commit should be tagged and artifacts produced should be promoted for further testing such as on staging environments.
  • Build & Packaging
  • SBOM Generation
  • Deployment on different target environments
  • Smoke Test
  • Performance Benchmarking
• Periodic: These are similar to post-merge however they run on a periodic basis (e.g., daily, weekly) due to different reasons such as not having sufficient resources or having long running tests that may take hours or even days. These take the latest promoted artifacts from earlier "loops" which could be post-merge loops as well as shorter periodic loops. (daily -> weekly)
  • Stability Tests
  • Robustness Tests
  • Characteristics Tests
• Production: These are the usual things that run in production based on different deployment strategies such as canary.
• Release: These are generally run for validating release readiness.
  • Generating Customer Documentation/Release Notes
  • FOSS Checks

Please note that the above list is combination of things observed/developed/used under different communities.

[zxiiro]

To add to your list there's also what Zuul CI calls Project Gating where the Commit Gating checks are run in a Post-merge state but before the merge actually happens in repo to make sure that the PR still passes before pushing the merge result while blocking the PR if it fails. I see it as a distinctive in-between state of Commit Gating and Post-merge gating

[fdegir]

Exactly! I don't know how I've forgotten that Thanh.
I'm not sure if it is possible to edit it directly but if it is, please feel free to do that.
Otherwise, I can add it.

[zxiiro]

Done. Kind of surprised it let me edit your comment.

[anderseknert]

From my (Open Policy Agent) perspective, we see policy used for many of the aforementioned activities and in many of the stages, altough with the shift-left strategy prevalent these days, more and more checks are being moved from deployment or post-deployment to pre-commit or PR checks. Adding to the combined list of @justinabrahms and @fdegir (good to see another face from Stockholm, btw!), some additional rules commonly enforced by policy:

• Application misconfiguration - make sure number of replicas are within an allowed range, that containers don't run as root, nothing is mounted from the host, and so on.

• Infrastructure policy - don't expose storage publicly, ensure particular instance types, security configurtion (Terraform, Cloud Formation)...

• Deployment windows - don't push production changes unless people are on call, "black Friday"

• Organizational requirements - ensure any deployed resource is labeled with an owner, cost centre, department...

• Kubernetes admission control (I guess this one is pretty much included in the other points, but still) and compliance and drift control, i.e. making sure that the state of the system is in an expected state. Not all state might be caught in deployments, i.e. superadmins could make runtime changes, internal systems sometimes bypass deployment checks, etc.

• Impact analysis - what would the estimated cost of the change be (e.g. infrastructure), how many objects would be affected by a namespace deletion (i.e. "blast radius")

• With regards to build and deployment pipelines, theres' also an interesting "meta" category of policies, where policy is used to enforce rules on the build and deployment configurations or tasks themselves, i.e, do not allow any build configuration to be deployed without a unit test or linter step, don't allow production deployment configurations to change without a sign-off from at least 2 colleagues, and so on. This is useful when you want to allow teams to set up their own build and deployment pipelines, but keep some level of organizational control / requirements enforced.

[amfred]

We're building a compliance pipeline with these tasks in it:
- name: setup-init
- name: source-clone-repository
- name: build-configure-build
- name: secret-detection-detect-secrets
- name: policy-code-reviews-check
- name: policy-branch-protection-check
- name: build-source-build-step
- name: discovery-create-build-graph
- name: bill-of-materials-create-build-bom
- name: test-unit-test
- name: coverage-unit-test-coverage
- name: scan-lint
- name: scan-dependency-vulnerability-scan
- name: scan-static-security-test
- name: scan-code-smells
- name: policy-provenance-check
- name: policy-cis-checks
- name: policy-license-check
- name: package-build-container
- name: discovery-create-output-graph
- name: bill-of-materials-create-output-bom
- name: sign-artifacts-build
- name: publish-push-to-image-registry
- name: scan-twistlock
- name: provision-provision-dev
- name: deploy-deploy-in-dev
- name: verify-deployment-verify-dev-deployment
- name: test-acceptance-test
- name: scan-dynamic-security-test
- name: cleanup-deprovision-dev
- name: record-results-show-summary
- name: message-slack-summary

[amfred]

In reality, often-times people split these up into a few pipelines, but these are the general stages that happen.

[amfred]

Some ideas for a very basic level of certification, purely from a DevSecOps perspective:
Peer code reviews happen - we might even validate the repo configuration to make sure that's enforced
Containers / container manifests are signed
We’re only using approved base images
Repos are scanned for known vulnerabilities
Images are scanned with Clair or something along those lines after the build
Ensure no embargoed issues are getting released before the embargo is lifted.

[mukteshkrmishra]

For us, we have categorized into 13 checks which go from all along from code, build, test and Opex lenses. It covers the entire pipeline and release-telemetry data. We use it for both proactive/on demand analysis and also reactive analysis.

The gates are around source code (code reviews, size, Static analysis, coverage), build(dependency, license, versioning), Test(Types, coverage) and opEx (Golden flows, Deployment model, etc.).

[amfred]

Something to add from my experience. The more gates you have, the more important it is to have a way for the pipeline to continue when some of the checks fail. That allows the developers to fix multiple failures in parallel.

A typical developer might be able to get code to pass 5 checks within a couple of days, but passing 30 checks might take months.

You can help people make progress with things like blocking and non-blocking gates, optional steps that can be enabled/disabled, or processes that collect results from the various steps into an evidence locker, and require that specific results were "good enough" (according to your policies) when promoting from one environment to another.