Skip to content
This repository has been archived by the owner on Dec 21, 2023. It is now read-only.

Release Checklist

Moritz Wiesinger edited this page Nov 4, 2021 · 58 revisions

This document describes the process currently (status: Kept 0.8.0-alpha and newer) required on how to create a new Keptn release. If you have any questions, ideas, improvements, etc... for this process, please raise it via a GitHub Issue.

AUTOMATION WANTED - we want to automate this process as much as possible. If you want to help us, please get in touch with us via Slack!

Creating a new Keptn release consists of multiple steps.

  1. An appropriate version of the keptn spec needs to linked to the release (might require a new release of keptn/spec)
  2. Keptn examples need to be released (same version as Keptn)
  3. Keptn kubernetes-utils and go-utils need to be released (same version as Keptn)
  4. (relevant) keptn-contrib and keptn-sandbox services need to be updated accordingly
  5. A release-branch is created on the Keptn repo, which triggers inofficial builds tagged with the release version (not publicily announced)
  6. Inofficial build and keptn-contrib/keptn-sandbox builds are tested according to Keptn tutorials
  7. If everything works, the build is made an official release

Important: Most steps in this process require at least WRITE access to the respective GitHub repositories (some of them even ADMIN access).

Preparations

Versioning scheme for Keptn

The projects versioning scheme is related to the semantic versioning structure and implemented for Keptn as follows:

  • Major: currently 0 as the spec is actively developed and not yet finalized
  • Minor: is incremented for new features (including Keptn Enhancement Proposals) and if a breaking change in the Keptn spec occurs
  • Patch: contains patches and bug fixes

Select semantic versions

  • Choose a semantic version for each release (Keptn, spec, keptn-contrib/keptn-sandbox services) and verify that it is semantic (using https://jubianchi.github.io/semver-check/#/version/0.1.2-alpha1 )
    • for keptn/keptn, keptn/examples, keptn/go-utils, keptn/kubernetes-utils: $KEPTN_VERSION
    • for keptn/spec (if needed): $KEPTN_SPEC_VERSION
    • Note: For pre-releases, e.g., alpha/beta/rc versions, please make sure to carry the same pre-release suffix for the above repositories
    • Optional: Define versions for keptn-contrib/keptn-sandbox services, if needed
  • Replace every occurance of $KEPTN_VERSION within this checklist with the actual version
  • Replace every occurance of $KEPTN_SPEC_VERSION within this checklist with the actual version

GitHub Issues, Milestone, etc...

  • Create a new issue called Release $KEPTN_VERSION in keptn/keptn using this template
  • ⌛ Set a release date in the future (e.g., 3 weeks from now)
  • 🔧 Finish all stories, issues, etc. defined for the release/milestone
  • 👍 Define a release manager 🤔 (responsible for driving/leading the release process) and assign the Release issue to them

Final steps

Please keep in mind

  • Pipeline needs to be green ✔️ and tests need to be successful (Exception: pre-releases, e.g., alpha, beta, rc).
  • Bugs 🐛 first policy is strict - if a bug is found, it needs to be fixed.
  • If during testing a problem is found, it should be fixed with a Pull Request straight away 💪. It is not required to create a new GitHub issue about it, but feel free to do so (especially if you can't fix it yourself or a discussion is required).
  • Some parts of this release issue contain the word TODO with a task behind it, e.g., TODO: Insert link

Feel free to delete everything above the following line when creating the GH release issue


Keptn $KEPTN_VERSION

  • Link to milestone(s): Milestone $KEPTN_VERSION (TODO: Insert link to milestone)
  • Release Date: <= Month XX th, 2021 (TODO: Specify release date)
  • Release Managers: Firstname Lastname (Insert GitHub Handle here) (TODO: Specify two release managers)

Prerequisites

Step 1 - Release of keptn/spec [OPTIONAL]

In case no new keptn/spec is released:

  • This version of Keptn does not require a new version of keptn/spec.

else:

  • Artifact: GitHub Release keptn/spec:$KEPTN_SPEC_VERSION (also referred as spec.keptn.sh/$KEPTN_SPEC_VERSION)
  • Assignee: Firstname Lastname (insert GitHub handle here) (TODO: Assign this step to someone)
  • Repo: https://github.com/keptn/spec
  • Note: Not every Keptn release requires a new Keptn spec.

Tasks:

  • All release-related PRs/discussions are resolved and are merged into master (please check back with the Issue/PR owner)
  • keptn/go-utils is in sync with spec/cloudevents (e.g., using keptn generate cloud-events)
  • Cleanup (Requires ADMIN role): Inactive branches are deleted
  • Draft a new release $KEPTN_SPEC_VERSION https://github.com/keptn/spec/releases with release notes (collect them from PR/discussion titles) on master branch - again, ensure a semantic version is used!
  • Ask for feedback about release notes via Slack (link to the draft release can only be seen by users with write access to the repo)
  • If you're happy with everything and the release notes have been approved/accepted, pull the trigger and release the spec
  • Notify Developers on Slack that a new release of keptn/spec has been created and link it (AUTOMATION WANTED)

Step 2 - Release of keptn/examples

  • Artifact: GitHub Release keptn/examples:$KEPTN_VERSION
  • Assignee: Firstname Lastname (insert GitHub handle here) (TODO: Assign this step to someone)
  • Repo: https://github.com/keptn/examples
  • Note: Use the same version as for keptn/keptn (every Keptn release requires a new release of Keptn examples)
  • All release-related feature/bugfix/hotfix branches are merged into master (please check back with the PR owner)
  • Cleanup (Requires ADMIN role): Inactive branches are deleted
  • If this is a fix/patch release and a maintenance branch (0.N.x) isn't there yet for the current feature/minor version, create the according branch. (for release of next bug version after 0.8.6 use the branch 0.8.x) If this is an arbitrary release with other features, then the release should not happen on a maintenance branch but rather from master directly to automatically create a new feature/minor version.
  • If you're happy with everything, pull the trigger and release keptn/examples, use the 'Release' GitHub action and trigger a manual run with the branch that you want to release from. (❗️ CAUTION: this will create the release commit and tag! ❗️)
  • The GitHub release was created in draft mode. Double check the content of the new release and then publish it if everything is correct.
  • If the release was done from a maintenance branch, create a PR from the released branch to master and request reviews (review of release notes is mandatory)
  • If the PR is mergeable and approved, get it squash & merged into master (don't delete the branch if it was a maintenance release, restore the branch if GitHub automatically deleted it!)
  • Notify Developers on Slack that a new release has been created (AUTOMATION WANTED)

Step 3 - Release of Keptn kubernetes-utils

  • Artifact: GitHub Release of keptn/kubernetes-utils:v$KEPTN_VERSION
  • Assignee: Firstname Lastname (insert GitHub handle here) (TODO: Assign this step to someone)
  • Repo: https://github.com/keptn/kubernetes-utils
  • Note: The v in the version, which can be used in GoLang using go get github.com/keptn/kubernetes-utils@v$KEPTN_VERSION
  • All release-related feature/bugfix/hotfix branches are merged into master (please check back with the PR owner)
  • Cleanup (Requires ADMIN role): Inactive branches are deleted
  • If this is a fix/patch release and a maintenance branch (0.N.x) isn't there yet for the current feature/minor version, create the according branch. (for release of next bug version after v0.8.6 use the branch 0.8.x) If this is an arbitrary release with other features, then the release should not happen on a maintenance branch but rather from master directly to automatically create a new feature/minor version.
  • If you're happy with everything, pull the trigger and release keptn/kubernetes-utils, use the 'Release' GitHub action and trigger a manual run with the branch that you want to release from. (❗️ CAUTION: this will create the release commit and tag! ❗️)
  • The GitHub release was created in draft mode. Double check the content of the new release and then publish it if everything is correct.
  • If the release was done from a maintenance branch, create a PR from the released branch to master and request reviews (review of release notes is mandatory)
  • If the PR is mergeable and approved, get it squash & merged into master (don't delete the branch if it was a maintenance release, restore the branch if GitHub automatically deleted it!)
  • Notify Developers on Slack that a new release has been created (AUTOMATION WANTED)

Step 4 - Release of Keptn go-utils

  • Artifact: GitHub Release of keptn/go-utils:v$KEPTN_VERSION
  • Assignee: Firstname Lastname (insert GitHub handle here) (TODO: Assign this step to someone)
  • Repo: https://github.com/keptn/go-utils
  • Note: The v in the version, which can be used in GoLang using go get github.com/keptn/go-utils@v$KEPTN_VERSION
  • All release-related feature/bugfix/hotfix branches are merged into master (please check back with the PR owner)
  • Cleanup (Requires ADMIN role): Inactive branches are deleted
  • If this is a fix/patch release and a maintenance branch (0.N.x) isn't there yet for the current feature/minor version, create the according branch. (for release of next bug version after v0.8.6 use the branch 0.8.x) If this is an arbitrary release with other features, then the release should not happen on a maintenance branch but rather from master directly to automatically create a new feature/minor version.
  • If you're happy with everything, pull the trigger and release keptn/go-utils, use the 'Release' GitHub action and trigger a manual run with the branch that you want to release from. (❗️ CAUTION: this will create the release commit and tag! ❗️)
  • The GitHub release was created in draft mode. Double check the content of the new release and then publish it if everything is correct.
  • If the release was done from a maintenance branch, create a PR from the released branch to master and request reviews (review of release notes is mandatory)
  • If the PR is mergeable and approved, get it squash & merged into master (don't delete the branch if it was a maintenance release, restore the branch if GitHub automatically deleted it!)
  • Notify Developers on Slack that a new release has been created (AUTOMATION WANTED)

Prepare release of associated services

⚠️ WARNING/STOP/DANGERZONE: Do not start with this section unless all the above steps have been completed successfully.


Step 5 - Create Release Branch for keptn-contrib services

Please update version tags of below services accordingly; use semantic versioning

foreach:

  • prometheus-service > prometheus-service:x.x.x

follow https://github.com/keptn/keptn/wiki/Keptn-Contrib-Release-Checklist Step 1

Prepare Release of Keptn


Step 6 - Release of Keptn

  • Update JMeter readme to reference $KEPTN_VERSION
  • keptn upgrade - verify if any changes/migrations/... in the CLI command are needed for this release
  • keptn upgrade - Create a PR at https://github.com/keptn/get.keptn.sh and modify version.json accordingly
  • Upgrade docs available in https://keptn.sh/docs/
  • Were there changes on K8s roles made? If yes, ensure (automatically) that the least privileges are always enforced for Keptn K8s roles: Who has which rights?
  • Ensure that the latest auto-generated PR to update the git submodule spec is merged in keptn/keptn. This PR is there to update the specification submodule when a new version of the spec is released.
  • All relevant feature/bugfix/hotfix branches are merged into master
  • Cleanup (Requires ADMIN role): Inactive branches are deleted
  • Latest go-utils version is used (run: go get github.com/keptn/go-utils@v$KEPTN_VERSION && go get ./... && go mod tidy) in all sub-folders that use it (see this code snippet) and the latest kubernetes-utils version is used (run: go get github.com/keptn/kubernetes-utils@v$KEPTN_VERSION && go get ./... && go mod tidy) in all sub-folders that use it (see this code snippet).
  • Keptn release notes are up-to-date in releasenotes/develop.md and spec version is referenced in release notes
  • Use Create Release Branch Workflow Workflow
  • Verify release-$KEPTN_VERSION and a PR have been created (automated using "Create Release Branch Workflow")
  • Ask stakeholders to review the release notes (Note: Do not use a GitHub DRAFT Release, it does not allow editing from multiple parties at the same time and we will lose changes**)
  • Check GH Workflow CI for release branch
  • Check GH Workflow Integration Test for release branch
  • Check for a draft release with the correct version https://github.com/keptn/keptn/releases (DO NOT RELEASE IT YET)
  • Draft release contains CLI binaries for Linux, Windows and MacOS
  • Draft release contains Keptn installer helm chart
  • Draft release contains Helm Charts for jmeter-service and helm-service

Check on GCP Cloud Storage:

  • keptn-installer $KEPTN_VERSION + index files updated

Notify certain users and stakeholders about the release, such that they can test it and ensure compatibility with their services/docs. Make sure to provide instructions on how to download the CLI and install Keptn. Make sure to let them know that this might not the final release, rather that it is a release candidate.

  • Notify developers, keptn/tutorials CODEOWNERS, keptn-contrib and keptn-sandbox repo owners via Keptn Slack (e.g., keptn-integrations channel) about the upcoming release and provide a preview to the release notes

Step 7 - Final Preparation phase

  • keptn.sh/docs/integrations/ is up-to-date
  • keptn CLI command docs generated and updated on keptn.sh/docs/x.x.x/reference/cli/commands/ (TODO: Update link to docs) A PR with the updated CLI command docs was created after the tag for keptn/keptn was pushed. You will find the PR in the keptn/keptn.github.io repo.
  • keptn.sh/docs/ is up-to-date
  • tutorials.keptn.sh: update of tutorials in this branch/PR: xyz (TODO: Add link to PR/branch)
    • Copy all 00.md files in site/tutorials and rename
    • Change tags and id in the .md files accordingly
    • Change versions 00 and 0.0 in the .md files
    • Copy all 00 folders in site/tutorials and rename
    • Copy site/app/views/keptn00x folder
    • Edit site/app/views/keptn00x/view.json
    • Copy site/tutorials/assets/…/0.0 where needed
    • Run builder.sh

Step 8 - Test phase

Upgrader:

  • Upgrade command tested for full installation (GKE or K3s)
  • Upgrade command tested for quality-gates/control-plane installation (GKE or K3s)
  • Upgrade command tested for full installation (OpenShift 3.11 or MiniShift)
  • Upgrade command tested for quality-gates/control-plane installation (OpenShift 3.11 or MiniShift)

Tests:

Note: The following is partially covered by integration tests. For pre-releases, it can be skipped. However, for GA releases the tests in this checklist need to be executed manually. Focus on usability and how it feels for the end-user!

  • GCP - GKE - Keptn core team
    • installation: keptn install
      • Quality Gates UC completed according to tutorials.keptn.sh with Dynatrace
      • Quality Gates UC completed according to tutorials.keptn.sh with Prometheus
    • Installation: keptn install --use-case=continuous-delivery
      • Tutorials completed with Dynatrace
        • Onboarding a Service
        • Deployments with Quality Gates
        • Up-scale with Dynatrace
        • Self-healing with Feature Flags
        • Are there changes required on the Tutorial? > Please file a PR
      • Tutorials completed with Prometheus
        • Onboarding a Service
        • Deployments with Quality Gates
        • Up-scale with Prometheus
        • Are there changes required on the Tutorial? > Please file a PR
    • Uninstall Keptn from cluster: keptn uninstall
    • Update https://keptn.sh/docs/x.x.x/operate/install/ based on used K8s version
  • Azure - AKS
    • Installation: keptn install --use-case=continuous-delivery
    • Ensure keptn version prints the correct version
    • keptn create project xyz
    • Ensure Keptn Bridge can be accessed and works
    • Uninstall Keptn from cluster: keptn uninstall
    • Update https://keptn.sh/docs/x.x.x/operate/install/ based on used K8s version
  • AWS - EKS
    • All tutorials completed with Dynatrace
      • Onboarding a Service
      • Deployments with Quality Gates
      • Up-scale with Dynatrace
      • Self-healing with Feature Flags
    • Update https://keptn.sh/docs/x.x.x/operate/install/ based on used K8s version
  • OpenShift 3.11 - MiniShift
  • OpenShift 4.x
    • All tutorials completed with Dynatrace
      • Onboarding a Service
      • Deployments with Quality Gates
      • Self-healing with Feature Flags
    • Update https://keptn.sh/docs/x.x.x/operate/install/ based on used OpenShift version
    • Update support matrix on keptn.sh/docs and release notes (key announcement + special thanks to Marc)
  • Kubernetes (e.g., K3s, Rancher)
    • Installation: keptn install --use-case=continuous-delivery
    • Ensure keptn version prints the correct version
    • keptn create project xyz
    • Ensure Keptn Bridge can be accessed and works
    • Uninstall Keptn from cluster: keptn uninstall
    • Update https://keptn.sh/docs/x.x.x/operate/install/ based on used K8s version

Step 9 - Official release of keptn-contrib services

foreach:

  • prometheus-service > prometheus-service:x.x.x

follow https://github.com/keptn/keptn/wiki/Keptn-Contrib-Release-Checklist Step 3


Step 10 - Official release of Keptn

Note: Some steps are only for GA releases, not for alpha, beta, etc.

Check on GCP Cloud Storage:

  • keptn-installer x.x.x

Release Keptn

  • Publish the draft release for Keptn $KEPTN_VERSION on GitHub, select release-$KEPTN_VERSION as a target
  • Verify that installing Keptn CLI works via curl -sL https://get.keptn.sh | KEPTN_VERSION=$KEPTN_VERSION bash and execute keptn version to check the version number
  • Merge PR from release-x.x.x to master (merge only if needed)
  • Created new and empty release notes (releasenotes_develop.md) file on master branch
  • Update Quick Start on keptn.sh with newest release (only for GA, not for pre-releases)
  • Add release notes to: https://keptn.sh/docs/news/release_announcements/ (only for GA, not for pre-releases; AUTOMATION WANTED)
  • Update: https://github.com/keptn/keptn.github.io/blob/master/layouts/partials/announcement.html

Final steps

  • Send out Slack notification
  • Send out Twitter notification
  • Send out Release-Email
  • Make sure any (relevant) changes to the release checklist are also saved in the wiki