v21-1-X.md

Migrate from Orchestrate v2.5.X to v21.1.X (Orchestrate Kubernetes v4.0.X to v5.0.X)

• 1. (Optional) Orchestrate-Kubernetes Helmfile values structure
• 2. Orchestrate Helm Charts
• 3. Hashicorp Vault
• 4. Postgresql Databases
• 5. (Optional) Kafka, Zookeeper, Postgres, Redis values
• 6. Deploy Orchestrate v21.1.X
• 7. Clean migration scripts
• 8. Remove the old Hashicorp Vault kv-v2 secrets

Orchestrate v21.1.X introduces major architectural and deployement changes including:

• New single Helm Chart package deploying an unified Orchestrate API service
• New Hashicorp Vault secret engine using the Orchestrate Plugin
• Unique Postgres Database

If you already installed Orchestrate v2.5.X with Orchestrate-Kubernetes (v4.0.0) please the steps below:

1. (Optional) Orchestrate-Kubernetes Helmfile values structure

This Helmfile repository ease the deployment of Orchestrate and its depedencies thanks to its .Values which are now structured as:

• orchestrate
  • global
  • api
  • keyManager
  • txListener
  • txSender
  • test
• vaultOperator
• vault
• kafka
• redis
• postgresql
• observability

Please see the default values in environments/*.yaml and their usage like in values/orchestrate.yaml.gotmpl We recommend you to follow the same values structure to ease this and future upgrades.

2. Orchestrate Helm Charts

Instead of deploying Orchestrate microservices with multiple releases, Orchestrate Helm deploys all Orchestrate microservices from one release. In helmfile-core.yaml remove core-stack-api (aka. orchestrate-helm-api) and core-stack-worker (aka. orchestrate-helm-worker) which are deprecated and not compatible with Orchestrate v21.1.X.

releases:
- - name: api-transaction-scheduler
-   namespace: {{ requiredEnv "ORCHESTRATE_NAMESPACE" }}
-   chart: helm-pegasys/core-stack-api
-   atomic: true
-   version: 0.5.6
-   values:
-     - values/common.yaml.gotmpl
-     - values/common-api.yaml.gotmpl
-     - values/common-multitenancy.yaml.gotmpl
-     - values/api-transaction-scheduler.yaml.gotmpl
- - name: api-contract-registry
-   namespace: {{ requiredEnv "ORCHESTRATE_NAMESPACE" }}
-   chart: helm-pegasys/core-stack-api
-   atomic: true
-   version: 0.5.6
-   values:
-     - values/common.yaml.gotmpl
-     - values/common-api.yaml.gotmpl
-     - values/common-multitenancy.yaml.gotmpl
-     - values/api-contract-registry.yaml.gotmpl
- - name: api-chain-registry
-   namespace: {{ requiredEnv "ORCHESTRATE_NAMESPACE" }}
-   chart: helm-pegasys/core-stack-api
-   atomic: true
-   version: 0.5.6
-   values:
-     - values/common.yaml.gotmpl
-     - values/common-api.yaml.gotmpl
-     - values/common-multitenancy.yaml.gotmpl
-     - values/api-chain-registry.yaml.gotmpl
- - name: tx-crafter
-   namespace: {{ requiredEnv "ORCHESTRATE_NAMESPACE" }}
-   chart: helm-pegasys/core-stack-worker
-   atomic: true
-   version: 0.5.4
-   values:
-     - values/common.yaml.gotmpl
-     - values/common-worker.yaml.gotmpl
-     - values/common-multitenancy.yaml.gotmpl
-     - values/tx-crafter.yaml.gotmpl
-     - values/e2e.yaml.gotmpl
- - name: tx-signer
-   namespace: {{ requiredEnv "ORCHESTRATE_NAMESPACE" }}
-   chart: helm-pegasys/core-stack-worker
-   atomic: true
-   version: 0.5.4
-   values:
-     - values/common.yaml.gotmpl
-     - values/common-worker.yaml.gotmpl
-     - values/common-multitenancy.yaml.gotmpl
-     - values/tx-signer.yaml.gotmpl
- - name: tx-sender
-   namespace: {{ requiredEnv "ORCHESTRATE_NAMESPACE" }}
-   chart: helm-pegasys/core-stack-worker
-   atomic: true
-   version: 0.5.4
-   values:
-     - values/common.yaml.gotmpl
-     - values/common-worker.yaml.gotmpl
-     - values/common-multitenancy.yaml.gotmpl
-     - values/tx-sender.yaml.gotmpl
- - name: tx-listener
-   namespace: {{ requiredEnv "ORCHESTRATE_NAMESPACE" }}
-   chart: helm-pegasys/core-stack-worker
-   atomic: true
-   version: 0.5.4
-   values:
-     - values/common.yaml.gotmpl
-     - values/common-worker.yaml.gotmpl
-     - values/common-multitenancy.yaml.gotmpl
-     - values/tx-listener.yaml.gotmpl
- - name: orchestrate-service-monitor
-   namespace: {{ requiredEnv "OBSERVABILITY_NAMESPACE" }}
-   chart: incubator/raw
-   version: 0.2.3
-   condition: metrics.enabled 
-   atomic: true
-   values:
-     - orchestrate:
-         namespace: {{ requiredEnv "ORCHESTRATE_NAMESPACE" }}
-     - values/service-monitor.yaml
+ - name: orchestrate
+   namespace: {{ .Values.orchestrate.namespace }}
+   chart: consensys/orchestrate
+   atomic: true
+   version: 1.0.0
+   needs:
+     - {{ .Values.kafka.namespace }}/kafka-{{ .Environment.Name }}
+     - {{ .Values.vault.namespace }}/vault
+     - {{ .Values.redis.namespace }}/redis
+     - {{ .Values.postgresql.namespace }}/postgresql
+   values:
+     - values/orchestrate.yaml.gotmpl

3. Hashicorp Vault

You need to import your keys from the Hashicorp Vault kv-v2 secret used in Orchestrate v2.5.X to the Orchestrate plugin secret engine used in Orchestrate v21.1.X.

Update the Vault Operator with the following chart Make sure you use at least version 1.10.X of the Vault Operator:

releases:
  - name: vault-operator
    chart: banzaicloud-stable/vault-operator
-   version: 1.7.0
+   version: 1.10.1

Update Vault CRD in values/vault.yaml(.gotmpl) to add Orchestrate plugin

For migrating keys from the old KV secret engine (used in Orchestrate v2.5.X) to the Orchestrate Plugin engine (used in Orchestrate v21.1.X), both engine should be mounted in the Vault instance:

  apiVersion: "vault.banzaicloud.com/v1alpha1"
  kind: "Vault"
  metadata:
    name: "vault"
  spec:
    size: 1
-   image: vault:1.3.1
+   image: vault:1.6.1

[...]

    volumes:
+     - name: vault-plugin
+       persistentVolumeClaim:
+         claimName: vault-plugin
      - name: vault-file
        persistentVolumeClaim:
          claimName: vault-file

    volumeMounts:
+     - name: vault-plugin
+       mountPath: /vault/plugins
      - name: vault-file
        mountPath: /vault/file

+   vaultInitContainers:
+     - name: orchestrate-plugin
+       image: alpine:3.12
+       command:
+         - /bin/sh
+         - -c
+         - |
+           apk add --no-cache jq wget
+           echo "Installing orchestrate-hashicorp-vault-plugin version +$PLUGIN_VERSION'"
+           parser=". | map(select(.tag_name == \"$PLUGIN_VERSION\"))[0].+ssets | map(select(.name == \"$PLUGIN_FILE\"))[0].id"
++           echo "Obtaining asset_id of $PLUGIN_FILE..."
+           asset_id=`wget -q --auth-no-challenge --proxy off +-header="Authorization: token $GITHUB_TOKEN" --header="Accept: +pplication/vnd.github.v3.raw" https://api.github.com/repos/+PLUGIN_REPO/releases -O - | jq "$parser"`
+           if [ -z "$asset_id" ]; then
+             echo "ERROR: version $PLUGIN_VERSION not found asset +PLUGIN_FILE"
+             exit 1
+           fi;
++           echo "Downloading $FILE(ID:$asset_id)..."
+           wget -q --auth-no-challenge --proxy off +-header='Accept:application/octet-stream' \
+             https://$GITHUB_TOKEN:@api.github.com/repos/$PLUGIN_REPO/+eleases/assets/$asset_id \
+             -O "${PLUGINS_PATH}/orchestrate"
++           echo "File at ${PLUGINS_PATH}/orchestrate"
+           chmod +x ${PLUGINS_PATH}/orchestrate
+       env: 
+         - name: GITHUB_TOKEN
+           value: {{ .Values.vault.plugin.token }}
+         - name: PLUGIN_VERSION
+           value: {{ .Values.vault.plugin.tag }}
+         - name: PLUGIN_FILE
+           value: orchestrate-hashicorp-vault-plugin
+         - name: PLUGIN_REPO
+           value: ConsenSys/orchestrate-hashicorp-vault-plugin
+         - name: PLUGINS_PATH
+           value: /vault/plugins
+       volumeMounts:
+         - name: vault-plugin
+           mountPath: /vault/plugins
++   vaultContainerSpec:
+     lifecycle:
+       postStart:
+         exec:
+           command: ["/bin/sh", "-c", "setcap cap_ipc_lock=+ep /vault/+lugins/orchestrate"]

[...]

    config:
      storage:
        file:
          path: "${ .Env.VAULT_STORAGE_FILE }" # An example how Vault config environment interpolation can be used
      listener:
        tcp:
          address: "0.0.0.0:8200"
          tls_disable: true
+     plugin_directory: /vault/plugins
+     api_addr: http://localhost:8200

    externalConfig:
      policies:
        - name: allow_secrets
          rules: path "secret/*" {
            capabilities = ["create", "read", "update", "delete", "list"]
            }
        - name: tx_signer_demo
          rules: path "secret/data/{{ .Values.orchestrate.namespace }}/keys/*" {
            capabilities = ["create", "read", "update", "delete", "list"]
            }
+       - name: orchestrate_key_manager
+         rules: path "orchestrate/*" {
+           capabilities = ["create", "read", "update", "list"]
+           }
      auth:
        - type: kubernetes
          roles:
            # Allow every pod in the default namespace to use the secret kv store
            - name: tx-signer
              bound_service_account_names: ["tx-signer", "vault-secrets-webhook", "vault"]
              bound_service_account_namespaces: ["{{ .Values.vaultOperator.namespace }}", "{{ .Values.vault.namespace }}", "{{ .Values.orchestrate.namespace }}"]
              policies: ["allow_secrets", "tx_signer_demo"]
+       - type: kubernetes
+         roles:
+           - name: orchestrate-key-manager
+             bound_service_account_names: ["orchestrate-key-manager", "vault-secrets-webhook", "vault"]
+             bound_service_account_namespaces: ["{{ .Values.vault.namespace }}", "{{ .Values.orchestrate.namespace }}"]
+             policies: ["orchestrate_key_manager", "allow_secrets", "tx_signer_demo"]

      secrets:
        - path: secret
          type: kv
          description: General secrets.
          options:
            version: 2
+       - path: orchestrate
+         type: plugin
+         plugin_name: orchestrate
+         description: Orchestrate key storage gateway
+         config:
+           force_no_cache: true
+           passthrough_request_headers: ["X-Vault-Namespace"]

+     plugins:
+       - plugin_name: orchestrate
+         command: orchestrate
+         sha256: {{ .Values.vault.plugin.sha256 }}
+         type: secret

Add a vault plugin PVC in values/vault.yaml(.gotmpl)

+  apiVersion: v1
+  kind: PersistentVolumeClaim
+  metadata:
+    name: vault-plugin
+  spec:
+    accessModes:
+      - ReadWriteOnce
+    resources:
+      requests:
+        storage: 1Gi

Run Key Manager migration scripts to import keys in values/orchestrate.yaml.gotmpl. If you used Orchestrate-Kubernetes, replace <**v2.5.X orchestrate namespace**> in the secret path of your keys.

[...]

keyManager:
  replicaCount: {{ .Values | get "orchestrate.replicaCount" 1 }}
+ migrate: true
  environment:
-   VAULT_ADDR: http://vault.{{ .Values.vault.namespace }}:8200
+   VAULT_ADDR: PUT HERE THE ENDPOINT OF THE CURRENT VAULT 
    VAULT_CACERT: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    VAULT_SKIP_VERIFY: true
+   VAULT_V2_MOUNT_POINT: secret
+   VAULT_V2_SECRET_PATH: <**v2.5.X orchestrate namespace**>/keys
+   VAULT_V2_TOKEN_FILE: /vault/token/.vault-token
    {{- range $key,$value := .Values.orchestrate.keyManager.environment }}
    {{ $key }}: {{ $value | quote }}
    {{- end }}

[...]

4. Postgresql Databases

You need to copy the databases of the Chain Registry, Contract Registry and Transaction Scheduler of Orchestrate v2.5.X to the new single API of Orchestrate v21.1.X.

Run copy DB script in values/orchestrate.yaml.gotmpl. If you used Orchestrate-Kubernetes, replace <**v2.5.X orchestrate namespace**> in the postgress endpoints.

api:
  replicaCount: {{ .Values | get "orchestrate.replicaCount" 1 }}
  environment:
    KAFKA_URL: {{ $kafkaURL }}
    DB_HOST: "postgresql.{{ .Values.postgresql.namespace }}"
    DB_DATABASE: {{ .Values.postgresql.database | quote }}
    DB_USER: {{ .Values.postgresql.username | quote }}

[...]

+ copyDB:
-   enabled: false
+   enabled: true
+   params:
+     chainRegistry:
+       postgresAddress: postgresql-chain-registry.<**v2.5.X orchestrate namespace**>:5432
+       database: chain-registry
+       username: chain-registry
+       password: such-secret
+     contractRegistry: 
+       postgresAddress: postgresql-contract-registry.<**v2.5.X orchestrate namespace**>:5432
+       database: contract-registry
+       username: contract-registry
+       password: such-secret
+     transactionScheduler: 
+       postgresAddress: postgresql-transaction-scheduler.<**v2.5.X orchestrate namespace**>:5432
+       database: transaction-scheduler
+       username: transaction-scheduler
+       password: such-secret

5. (Optional) Kafka, Zookeeper, Postgres, Redis values

Please have a look at the values passed in Kafka, Postgres and Redis. Some values have been updated for a better fit with Orchestrate usage, see values/kafka.yaml, values/postgresql.yaml, values/redis.yaml

6. Deploy Orchestrate v21.1.X

In order to avoid inconsistent databases and key storage, we recommend you interrupt the service by uninstalling Orchestrate v2.5.X releases before deploying v21.1.X. Checklist:

• Empty Kafka queues
• Postgres of Chain Registry, Contract Registry and Transaction Scheduler up and running
• Hashicorp Vault service running with kv-v2 engine and ready to updated with the new Orchestrate Plugin

You can now deploy Orchestrate v21.1.X with their migration scripts, see Quickstart.

If everything goes well you should get the following:

• Migration of keys from the old kv secret to the Orchestrate Plugin

kubectl logs orchestrate-api-8654f867d9-2zxhk -c migrate-import-secrets

level=info msg="Importing Hashicorp kv-v2 secrets to Vault..."
level=info msg="importing accounts [[\"_0x6009608A02a7A15fd6689D6DaD560C44E9ab61Ff\" \"_0x664895b5fE3ddf049d2Fb508cfA03923859763C6\" \"_0x7E654d251Da770A068413677967F6d3Ea2FeA9E4\" \"_0x93f7274c9059e601be4512F656B57b830e019E41\" \"_0xA8d8DB1d8919665a18212374d623fc7C0dFDa410\" \"_0xbfc7137876d7Ac275019d70434B0f0779824a969\" \"_0xdbb881a51CD4023E4400CEF3ef73046743f08da3\" \"_0xf5956Eb46b377Ae41b41BDa94e6270208d8202bb\" \"_0xfF778b716FC07D98839f48DdB88D8bE583BEB684\" \"_0xffbBa394DEf3Ff1df0941c6429887107f58d4e9b\"]]"
level=info msg="Account was imported successfully" address=0x6009608A02a7A15fd6689D6DaD560C44E9ab61Ff namespace=_
level=info msg="Account was imported successfully" address=0x664895b5fE3ddf049d2Fb508cfA03923859763C6 namespace=_
level=info msg="Account was imported successfully" address=0x7E654d251Da770A068413677967F6d3Ea2FeA9E4 namespace=_
level=info msg="Account was imported successfully" address=0x93f7274c9059e601be4512F656B57b830e019E41 namespace=_
level=info msg="Account was imported successfully" address=0xA8d8DB1d8919665a18212374d623fc7C0dFDa410 namespace=_
level=info msg="Account was imported successfully" address=0xbfc7137876d7Ac275019d70434B0f0779824a969 namespace=_
level=info msg="Account was imported successfully" address=0xdbb881a51CD4023E4400CEF3ef73046743f08da3 namespace=_
level=info msg="Account was imported successfully" address=0xf5956Eb46b377Ae41b41BDa94e6270208d8202bb namespace=_
level=info msg="Account was imported successfully" address=0xfF778b716FC07D98839f48DdB88D8bE583BEB684 namespace=_
level=info msg="Account was imported successfully" address=0xffbBa394DEf3Ff1df0941c6429887107f58d4e9b namespace=_
level=info msg="Accounts have been imported successfully"
level=info msg="main: execution completed"

• Migration of data from the Chain Registry, Contract Registry and Transaction Scheduler postgres to the new API postgres

kubectl get pods

orchestrate-api-copy-db-chain-registry-rsdcb          0/1     Completed   0          59s
orchestrate-api-copy-db-contract-registry-nkhdb       0/1     Completed   0          57s
orchestrate-api-copy-db-transaction-scheduler-72zw7   0/1     Completed   0          55s

kubectl logs orchestrate-api-copy-db-chain-registry-rsdcb

level=info msg="main: execution completed"

Note: those jobs are automatically deleted if they succeded.

You can now safely test Orchestrate v21.1.X.

7. Clean migration scripts

If the upgrade went well you could disable the migration scripts in values/orchestrate.yaml.gotmpl:

keyManager:
+ migrate: false
- migrate: true
  environment:
-   VAULT_V2_MOUNT_POINT: secret
-   VAULT_V2_SECRET_PATH: <**v2.5.X orchestrate namespace**>/keys
-   VAULT_V2_TOKEN_FILE: /vault/token/.vault-token

[...]

api:
+ copyDB:
+   enabled: false
-   enabled: true
-   params:
-     chainRegistry:
-       postgresAddress: postgresql-chain-registry.<**v2.5.X orchestrate namespace**>:5432
-       database: chain-registry
-       username: chain-registry
-       password: such-secret
-     contractRegistry: 
-       postgresAddress: postgresql-contract-registry.<**v2.5.X orchestrate namespace**>:5432
-       database: contract-registry
-       username: contract-registry
-       password: such-secret
-     transactionScheduler: 
-       postgresAddress: postgresql-transaction-scheduler.<**v2.5.X orchestrate namespace**>:5432
-       database: transaction-scheduler
-       username: transaction-scheduler
-       password: such-secret

8. Remove the old Hashicorp Vault kv-v2 secrets

In case the migration of the keys went well you could remove the kv-v2 secret in values/vault.yaml.gotmpl:

    externalConfig:
      policies:
-       - name: allow_secrets
-         rules: path "secret/*" {
-           capabilities = ["create", "read", "update", "delete", "list"]
-           }
-       - name: tx_signer_demo
-         rules: path "secret/data/{{ .Values.orchestrate.namespace }}/keys/*" {
-           capabilities = ["create", "read", "update", "delete", "list"]
-           }
        - name: orchestrate_key_manager
          rules: path "orchestrate/*" {
            capabilities = ["create", "read", "update", "list"]
            }
      auth:
-       - type: kubernetes
-         roles:
-           # Allow every pod in the default namespace to use the secret kv store
-           - name: tx-signer
-             bound_service_account_names: ["tx-signer", "vault-secrets-webhook", "vault"]
-             bound_service_account_namespaces: ["{{ .Values.vaultOperator.namespace }}", "{{ .Values.vault.-amespace }}", "{{ .Values.orchestrate.namespace }}"]
-             policies: ["allow_secrets", "tx_signer_demo"]
        - type: kubernetes
          roles:
            - name: orchestrate-key-manager
              bound_service_account_names: ["orchestrate-key-manager", "vault-secrets-webhook", "vault"]
              bound_service_account_namespaces: ["{{ .Values.vault.namespace }}", "{{ .Values.orchestrate.namespace  }"]
-             policies: ["orchestrate_key_manager", "allow_secrets", "tx_signer_demo"]
+             policies: orchestrate_key_manager