docs: werf with ArgoCD experimental CI/CD setup documentation pages

Signed-off-by: Timofey Kirillov <timofey.kirillov@flant.com>

docs/_data/breadcrumbs.yml

@@ -26,6 +26,9 @@ en:
   /advanced/ci_cd/run_in_container: &documentation-advanced-ci-cd-run-in-container
     title: Run in containers
     url: advanced/ci_cd/run_in_container/use_docker_container.html
+  /advanced/ci_cd/werf_with_argocd: &documentation-advanced-ci-cd-werf-with-argocd
+    title: werf with ArgoCD
+    url: advanced/ci_cd/werf_with_argocd/ci_cd_flow_overview.html
   /advanced/helm: &documentation-advanced-helm
     title: Helm
     url: advanced/helm/overview.html

docs/_data/sidebars/_documentation.yml

@@ -129,6 +129,18 @@ entries:
                   - title: How it works
                     url: /advanced/ci_cd/run_in_container/how_it_works.html
 
+              - title: werf with ArgoCD (experimental)
+                f:
+
+                  - title: CI/CD flow overview
+                    url: /advanced/ci_cd/werf_with_argocd/ci_cd_flow_overview.html
+
+                  - title: Prepare Kubernetes cluster
+                    url: /advanced/ci_cd/werf_with_argocd/prepare_kubernetes_cluster.html
+
+                  - title: Configure CI/CD
+                    url: /advanced/ci_cd/werf_with_argocd/configure_ci_cd.html
+
           - title: Building images with stapel
             f:
 

docs/_data/sidebars/documentation.yml

@@ -473,6 +473,18 @@ entries:
                   - title: How it works
                     url: /advanced/ci_cd/run_in_container/how_it_works.html
 
+              - title: werf with ArgoCD (experimental)
+                f:
+
+                  - title: CI/CD flow overview
+                    url: /advanced/ci_cd/werf_with_argocd/ci_cd_flow_overview.html
+
+                  - title: Prepare Kubernetes cluster
+                    url: /advanced/ci_cd/werf_with_argocd/prepare_kubernetes_cluster.html
+
+                  - title: Configure CI/CD
+                    url: /advanced/ci_cd/werf_with_argocd/configure_ci_cd.html
+
           - title: Building images with stapel
             f:
 

docs/pages_en/advanced/ci_cd/werf_with_argocd/ci_cd_flow_overview.md

@@ -0,0 +1,128 @@
+---
+title: CI/CD flow overview
+permalink: advanced/ci_cd/werf_with_argocd/ci_cd_flow_overview.html
+---
+
+In this article we will describe how to implement continuous delivery and continuous deployment for your project using werf with gitops model. Briefly it implies:
+* usage of **werf** to build and publish release artifacts;
+* usage of **argocd gitops tool** to deliver release artifacts into the kubernetes;
+* gluing together werf and argocd with **bundles**.
+
+## Keypoints and principles
+
+In this section we would define keypoints and principles used to implement CI/CD process.
+
+### Single source of truth
+
+Project git repository is the single source of truth, which contains:
+* application source code;
+* instructions to build images;
+* helm chart files.
+
+### Release artifacts
+
+1. Release artifacts published into container registry and consists of: 
+   * built images;
+   * helm chart.
+2. Release artifact could be prepared and published for an arbitrary git commit.
+3. Once published for a commit release artifact should be immutable and reproducible.
+4. Dependening on CI/CD flow:
+   * Release artifact could be published for each commit of the main branch, and then deployed into production.
+   * Release artifact could be published only for git tags, which represent application releases.
+5. Any conventional CI/CD system (like Gitlab CI/CD or GitHub Actions) could be used to prepare release artifacts.
+
+### Deploy into Kubernetes with GitOps
+
+There is a gitops operator deployed into the cluster which implements pull model to deploy project releases:
+* Operator watches for new versions of release artifacts in the container registry.
+* Operator rollout some release artifact version into kubernetes cluster.
+
+Depending on CI/CD flow GitOps operator can rollout release artifact either manually (Continuous Delivery) or automatically (Continous Deployment). 
+
+## Reference CI/CD flow
+
+Reference CI/CD flow will be implemented using 2 main tools: werf and ArgoCD.
+
+In the following scheme we have a flow blocks and specify which tool implement blocks of this flow.   
+
+<a class="google-drawings" href="{{ "images/advanced/ci_cd/werf_with_argocd/ci-cd-flow.svg" | true_relative_url }}" data-featherlight="image">
+<img src="{{ "images/advanced/ci_cd/werf_with_argocd/ci-cd-flow.svg" | true_relative_url }}">
+</a>
+
+### 1. Local dev
+
+Flow starts with local development which is covered by the following werf abilities:
+* All werf commands support `--dev` and `--follow` flags to simplify work with uncommitted changes and new changes introduced into the application source code.
+* werf allows usage of the same build and deploy configuration to deploy an application either locally or into production.
+* Using special environment (passed with the `--env ENV` param) project could have customizations of build and deploy configuration, which are used to deploy an application into the local Kubernetes cluster for local development purposes.
+
+### 2. Commit
+
+Commit stage implies that all werf build and deploy configuration and application source code should be strictly commited into the project git repository.
+
+For example, when your CI/CD system checkout some commit and run some instructions these instructions could not accidentally or by intention generate some dynamic input configuration files for werf build and deploy configuration.
+
+This is default werf behaviour which is controlled by the **giterminism**. Giterminism is a tool which allows users to define how reproducible project's build and deploy configuration is (the most strict rules enabled by default, and could be loosened by the `werf-giterminism.yaml` config).
+
+### 3. Build and publish images
+
+werf supports building and publishing images into the container registry using Dockerfile or stapel builder, enables distributed layers caching, and content based tagging, which optimizes images rebuilds.
+
+werf supports building of images with the `werf build` command. More info [in the configuration article]({{ "advanced/ci_cd/werf_with_argocd/configure_ci_cd.html#build-and-publish-images" | true_relative_url }}).
+
+### 4. Fast commit testing
+
+It is essential part of CI/CD approach to perform so called fast testing of each commit in the git repository. Typically during this stage we run unit tests and linters.
+
+Such tests would run in one-shot containers in built images in the common case. werf supports running such one-shot containers with the following commands:
+* `werf run` — run one-shot command in container in the docker server.
+* `werf kube-run` — run one-shot command container in the Kubernetes cluster. 
+
+More info [in the configuration article]({{ "advanced/ci_cd/werf_with_argocd/configure_ci_cd.html#fast-commit-testing" | true_relative_url }}).
+
+### 5. Prepare release artifacts
+
+At this stage we should prepare [release artifact](#release-artifacts) using so called [werf bundles]({{ "advanced/bundles.html" | true_relative_url }}).
+
+`werf bundle publish` command:
+* ensures that all needed images are build;
+* publishes helm chart files configured to use built images into the container registry — so-called bundle.
+
+Published bundle can be later deployed into the kubernetes cluster by ArgoCD as a regular OCI helm chart.  
+
+More info [in the configuration article]({{ "advanced/ci_cd/werf_with_argocd/configure_ci_cd.html#prepare-release-artifacts" | true_relative_url }}).
+
+### 6. Acceptance testing
+
+It is also essential part of CI/CD to perform so called acceptance testing. Such tests are typically long-running and may require full production-like environment to be available during tests.
+
+Typically acceptance testing should be performed automatically after pull request merge into the main branch.
+
+Pass of acceptance tests do not block Continuous Integration process, but in most cases should block Continuous Delivery of new releases. It means that it is allowed to merge new commits and PRs into main branch without running acceptance tests, but it is required to pass acceptance tests for a release artifact before releasing it into production. 
+
+With werf we could define special test environment (with `--env ENV` param).
+* In the test environment there may be special build and deploy configuration files.
+* In the test environment there will be special helm post install `Job` resource.
+
+werf supports running acceptance tests with the following commands:
+* `werf converge` to install production-like environment into the Kubernetes and run test `Job` as a helm post install hook.
+  * This command will need project git repository.
+* `werf dismiss` to uninstall previously installed production-like test environment.
+* `werf bundle apply` to install production-like environment into the Kubernetes and run test `Job` as a helm post install hook.
+  * This command will not need the git repository, only bundle published into the container registry.
+
+Alternatively acceptance tests could be run by the ArgoCD using published release artifact — werf bundle. In such case ArgoCD would install production-like environment into the Kubernetes and run test `Job` as a helm post install hook.
+
+More info [in the configuration article]({{ "advanced/ci_cd/werf_with_argocd/configure_ci_cd.html#acceptance-testing" | true_relative_url }}).
+
+### 7. Deploy into production
+
+In this step ArgoCD should deploy published release artifact for target application commit into the Kubernetes cluster.
+
+We are using werf bundle as release artifact, which could be consumed as a regular OCI helm chart by the ArgoCD.
+
+ArgoCD could deploy target release artifact by manual user action.
+
+ArgoCD Image Updater with the ["continuous deployment of OCI Helm chart type application" patch](https://github.com/argoproj-labs/argocd-image-updater/pull/405) could be used to enable automatic deploy to the latest version of published release artifact.
+
+More info [in the configuration article]({{ "advanced/ci_cd/werf_with_argocd/configure_ci_cd.html#acceptance-testing" | true_relative_url }}).

docs/pages_en/advanced/ci_cd/werf_with_argocd/configure_ci_cd.md

@@ -0,0 +1,156 @@
+---
+title: Configure CI/CD
+permalink: advanced/ci_cd/werf_with_argocd/configure_ci_cd.html
+---
+
+In this chapter we will describe how to setup following CI/CD steps according to the [reference CI/CD flow overview]({{ "advanced/ci_cd/werf_with_argocd/ci_cd_flow_overview.html#reference-cicd-flow" | true_relative_url }}):
+
+**NOTE** It is important to setup container registry and use single `CONTAINER_REGISTRY` param in all werf commands. 
+
+## Build and publish images
+
+Call following command to build and publish images described in the `werf.yaml`:
+
+```shell
+werf build --repo CONTAINER_REGISTRY
+```
+
+## Fast commit testing
+
+To run fast tests like unit testing or linting we call werf command, which runs one-shot container with the specified command:
+
+```shell
+werf kube-run IMAGE_NAME -- TEST_COMMAND
+```
+
+This command will run specified command in the container using Kubernetes cluster. `IMAGE_NAME` should be defined in the `werf.yaml` — this may be an image built specifically for such tests or regular application image.
+
+## Prepare release artifacts
+
+At this point we need to publish werf bundle into the container registry. `werf bundle publish` command will automatically ensure that all needed images are built and published into container registry and will publish bundle as OCI helm chart by the specified semver tag:
+
+```shell
+werf bundle publish --repo CONTAINER_REGISTRY --tag SEMVER
+```
+
+It is current requirement of helm OCI charts to use semver tags (static tags like `main` / `latest` are not allowed at the moment). Depending on your CI/CD model:
+* If application is already released under semver git tags, then we propose to publish bundle under the same semver tag.
+* If application deployed into production from the main branch (either automatically from HEAD commit or from some commit by user manual action), then we propose to publish bundle under fake tag like `0.0.${CI_PIPELINE_ID}`. `CI_PIPELINE_ID` is some builtin variable of your CI/CD system, which is monotonically increasing number that identifies whole pipeline of CI/CD steps for each git commit.
+
+## Acceptance testing
+
+During acceptance testing step we need to rollout our application into the temporal production-like environment, then run test job and wait for results. This task may be performed in multiple alternative ways.
+
+Typically acceptance testing should run automatically for new commits merged into the main branch. Also typically failed acceptance tests should prevent release artifact from being deployed into production.
+
+So you need to setup CI/CD step which would run test command on new commits of the main integration branch.
+* This could be optimized later to run tests not on every commit of the branch, but only on latest at the moment.
+* Possibly you will need a queue to run only one test acceptance environment (namespace and release) at a time, or run each commit in the separate unique release.
+
+In the following sections we would describe how to run such test
+
+### Run with werf converge
+
+In the case we have defined special test `Job` as a helm `post-install` hook in the `test` environment, we need to run following command in the **project git worktree**:
+
+```shell
+werf converge --repo CONTAINER_REGISTRY --env test
+```
+
+This command will start test by deploying a production-like application into the Kubernetes namespace `PROJECT_NAME-test` release `PROJECT_NAME-test` and run `post-install` Job (or Jobs) with tests. Converge command will automatically get test Job (or Jobs) output and fail whole process in the case of tests failure.
+
+Use dismiss command to terminate test environment:
+
+```shell
+werf dismiss --env test
+```
+
+### Run with release artifact and werf
+
+This way we use `werf bundle apply` command to rollout previously published release artifact. 
+
+In the case we have defined special test `Job` as a helm `post-install` hook in the `test` environment, we need to run following command:
+
+```shell
+werf bundle apply --repo CONTAINER_REGISTRY --env test --namespace TEST_NAMESPACE --release TEST_RELEASE
+```
+
+Note that this command does not need project git worktree (as werf converge does), otherwise it will give the same output and effects as [run with werf converge](#run-with-werf-converge).
+
+`TEST_NAMESPACE` and `TEST_RELEASE` should be specified explicitly and cleaned up afterwards:
+
+```shell
+werf helm uninstall TEST_RELEASE -n TEST_NAMESPACE
+kubectl delete ns TEST_NAMESPACE
+```
+
+### Run with release artifact and ArgoCD
+
+This way we need to configure special ArgoCD Application CRD:
+
+```yaml
+spec:
+  destination:
+    namespace: TEST_NAMESPACE
+  source:
+    chart: PROJECT_NAME
+    repoURL: REGISTRY
+    targetRevision: SEMVER
+```
+
+Perform following command in your CI/CD to start such tests:
+
+```
+argocd app sync TEST_APPNAME --revision SEMVER
+```
+
+— where `SEMVER` could be regular version of fake tag like `0.0.${CI_PIPELINE_ID}` (see [preparing release artifacts](#prepare-release-artifacts)).
+
+## Deploy into production
+
+### Configure ArgoCD Application
+
+When configuring ArgoCD Application CRD the most notable fields are:
+
+```yaml
+spec:
+  destination:
+    namespace: TEST_NAMESPACE
+  source:
+    chart: PROJECT_NAME
+    repoURL: REGISTRY
+    targetRevision: SEMVER
+```
+
+**NOTE** Bundle should be published using `werf bundle publish --repo REGISTRY/PROJECT_NAME --tag SEMVER` format to confirm with the above setting. `REGISTRY/PROJECT_NAME` is the address of repository in the container registry, which contains published werf bundle as helm OCI chart.
+
+Set `targetRevision` to some initial bootstrap version of your application. In the following sections we will describe how to update `targetRevision` when a new release arrives.
+
+### Deploy by manual action
+
+To deploy your application into production in a manual maner, setup the following command in your CI/CD system and trigger this command on manual user action:
+
+```shell
+argocd app sync APPNAME --revision SEMVER
+```
+
+Revision `SEMVER` should be formed the same way as in the [prepare release artifact step](#prepare-release-artifacts). 
+
+### Continuous Deployment
+
+Continuos deployment means that latest release artifact for the `main` branch for example will be automatically deployed into production Kubernetes cluster.
+
+To setup continuous deployment of release artifacts, configure ArgoCD Application CRD with the following annotations:
+
+```
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  annotations:
+    argocd-image-updater.argoproj.io/chart-version: ~ 0.0
+    argocd-image-updater.argoproj.io/pull-secret: pullsecret:NAMESPACE/SECRET
+```
+
+The value of `argocd-image-updater.argoproj.io/chart-version="~ 0.0"` means that operator should automatically rollout chart to the latest patch version within semver range `0.0.*`.
+
+We need to create secret `NAMESPACE/SECRET` (it is recommended to use `type: kubernetes.io/dockerconfigjson` secret) to access `CONTAINER_REGISTRY` which contains published [release artifact](#prepare-release-artifacts). 

docs/pages_en/advanced/ci_cd/werf_with_argocd/prepare_kubernetes_cluster.md

@@ -0,0 +1,47 @@
+---
+title: Prepare Kubernetes cluster
+permalink: advanced/ci_cd/werf_with_argocd/prepare_kubernetes_cluster.html
+---
+
+## 1. Install ArgoCD with werf plugin
+
+[Install ArgoCD](https://argo-cd.readthedocs.io/en/stable/getting_started/#1-install-argo-cd).
+
+Enable werf sidecar plugin:
+
+1. Edit `deploy/argocd-repo-server`:
+    ```shell
+    kubectl -n argocd edit deploy argocd-repo-server
+    ```
+2. Add sidecar container and apparmor annotation:
+    ```yaml
+    ...
+    metadata:
+      annotations:
+        "container.apparmor.security.beta.kubernetes.io/werf-argocd-cmp-sidecar": "unconfined"
+      ...
+    spec:
+      ...
+      template:
+        ...
+        spec:
+          containers:
+          - image: ghcr.io/werf/werf-argocd-cmp-sidecar:1.2-alpha
+            imagePullPolicy: Always
+            name: werf-argocd-cmp-sidecar
+            volumeMounts:
+            - mountPath: /var/run/argocd
+              name: var-files
+            - mountPath: /home/argocd/cmp-server/plugins
+              name: plugins
+            - mountPath: /tmp
+              name: tmp
+    ```
+
+## 2. Install ArgoCD Image Updater
+
+Install ArgoCD Image Updater with the ["continuous deployment of OCI Helm chart type application" patch](https://github.com/argoproj-labs/argocd-image-updater/pull/405):
+
+```shell
+kubectl apply -n argocd -f https://raw.githubusercontent.com/werf/3p-argocd-image-updater/master/manifests/install.yaml
+```