werf new feature: Nelm as new deployment engine (v1.2.234+)

[shurup]

Briefly

A new deployment engine (currently experimental) solves many of the old problems with deploys as well as adds a number of improvements.

This feature became available starting from werf v1.2.234, which is currently in the Alpha release channel.

Background & details

We have developed an alternative implementation of the deployment engine (it will become the default in the future) to solve many of the problems that have long plagued the deployment process and improve the extensibility of the existing code. It is fully backward-compatible with the old engine and Helm.

Here are some of the problems the new engine solves and the improvements it brings:

• The buggy 3-way merge has been replaced with the native Server-Side Apply;
• CRDs in the /crds directories are now applied during release upgrades as well; on top of that, CRDs can also be upgraded if they already exist in the Kubernetes cluster;
• The resources in the cluster that match the desired state will not be updated;
• No more errors like no matches for kind "Deployment" in version "apps/v1beta1" that sometimes occur when upgrading Kubernetes, making it impossible to upgrade the release;
• werf converge now outputs a report at the end, detailing what was actually done during the deployment.

Configuration

How to enable the new engine? For werf v1.2.234 (currently available in Alpha) and newer, set the WERF_EXPERIMENTAL_DEPLOY_ENGINE=1 environment variable and werf converge will use the new engine. No extra steps are required.

Caution: The new engine is at an early stage of development and could theoretically have serious bugs, such as creating/updating/deleting resources that otherwise should not have been modified. It will take time for the engine to become stable. As an option, you can activate the experimental engine for deploying to the dev cluster only, keeping the old engine for the production cluster at this point.

You can track the progress of developing the new engine here: #5639

We would greatly appreciate your feedback as it will help us to make this new werf engine stable even sooner!

[shurup]

The official name for werf's new deployment engine is Nelm. While it's a library internally used by werf now, we also plan to make it available as a standalone CLI tool.

Some other new features implemented in Nelm (and, therefore, introduced in werf) include:

• werf new feature: New `werf plan` command (v1.2.256+) #5817
• werf new feature: improved format for deployment progress, logs and events (v1.2.283+) #5950
• --atomic (--auto-rollback) mode for the werf converge command (v1.2.284)

[shurup]

werf uses Nelm by default in the Early Access release channel

The new deployment engine (Nelm) is now enabled by default in the "ea" release channel. (You can find more details on our release channels here.)

If you experience any issues with Nelm in werf, you may revert to the legacy deployment engine via export WERF_NELM=0

Known issues:

1. failed to create typed patch object — resource validation failed, meaning there is a mistake in the chart resource manifest. Validation passed before because it wasn't as strict with 3-Way Merge as it is now with Server-Side Apply.
2. edge would create a cycle — incorrect resource deployment order in the chart. Here is an example: the Job with helm.sh/hook:pre-upgrade mounts the ConfigMap, but the ConfigMap is deployed as a regular resource, not as a hook. That is, the deployment order is Job > ConfigMap, while it should be the other way around. The solution is to deploy the ConfigMap at the same or earlier "stage" as the Job.

To solve both of these issues, you need to fix your charts.

[shurup]

werf bundles now use Nelm by default

As of werf v1.2.303 (alpha) released today, bundles (werf bundle apply) are deployed via the new deployment engine (Nelm).

Please feel free to report any issues you may encounter!

You can temporarily switch to the old engine via export WERF_NELM=0

[shurup]

Nelm will be default in werf v2.0, not werf v1.2

Briefly

In werf 1.2, the new deployment engine (Nelm) will not be enabled by default. We rolled back its activation in alpha/beta/ea channels as well. You can turn it back on using export WERF_NELM=1. The new engine will be enabled by default in werf 2.0.

Background & details

The reason for that is much more strict validation (primarily because of using Server-Side Apply), which cannot be disabled:

• In werf, which is usually auto-updated via trdl, this results in deployments that suddenly stop working.
• Thus, users are forced to fix mistakes in their charts in all repositories and branches where these charts are used, which is extremely inconvenient.

Therefore, instead of enabling the new engine by default in v1.2, we will enable it in werf v2.0 so that users can migrate in a planned and predictable manner.

Note that export WERF_NELM=1 is usually enough to migrate to a new engine. The new engine may discover some mistakes in the charts, such as unsupported fields in resource manifests. In that case, you should either fix the chart or switch back to the old engine using export WERF_NELM=0.

We plan to release werf v2.0 having Nelm enabled by default in a few weeks.

[shurup]

As werf v2.0 is released, find more details regarding Nelm becoming the default and the only deployment engine in werf here: #6100