Error on upgrading longhorn-manager from 1.5.3 to 1.6.0

[ctraber]

Describe the bug

lognhorn-manager pods not starting after helm upgrade
helm upgrade longhorn longhorn/longhorn --namespace longhorn-system --version 1.6.0

To Reproduce

run helm upgrade

Expected behavior

longhorn manager updated to 1.6.0 and running

Error message in longhorn-manager pod:

Incorrect Usage: flag provided but not defined: -upgrade-version-check

NAME:
   longhorn-manager daemon - 

USAGE:
   longhorn-manager daemon [command options] [arguments...]

OPTIONS:
   --engine-image value                  Specify Longhorn engine image
   --instance-manager-image value        Specify Longhorn instance manager image
   --share-manager-image value           Specify Longhorn share manager image
   --backing-image-manager-image value   Specify Longhorn backing image manager image
   --support-bundle-manager-image value  Specify Longhorn support bundle manager image
   --manager-image value                 Specify Longhorn manager image
   --service-account value               Specify service account for manager
   --kube-config value                   Specify path to kube config (optional)
   
time="2024-02-02T19:40:51Z" level=fatal msg="Critical error: flag provided but not defined: -upgrade-version-check" func=main.main file="main.go:77"
Incorrect Usage: flag provided but not defined: -upgrade-version-check```

[PhanLe1010]

flag provided but not defined: -upgrade-version-check

Looks like the flag should be --upgrade-version-check instead as specified at link

Can you provide:

1. kubectl get pods -l app=longhorn-manager -n longhorn-system -o yaml
2. The output from helm template command?

@ctraber

[ctraber]

Here the longhorn-manager part.

seems to be ok --upgrade-version-check

---
# Source: longhorn/templates/daemonset-sa.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app.kubernetes.io/name: longhorn
    helm.sh/chart: longhorn-1.6.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/instance: longhorn
    app.kubernetes.io/version: v1.6.0
    app: longhorn-manager
  name: longhorn-manager
  namespace: longhorn-system
spec:
  selector:
    matchLabels:
      app: longhorn-manager
  template:
    metadata:
      labels:
        app.kubernetes.io/name: longhorn
        helm.sh/chart: longhorn-1.6.0
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/instance: longhorn
        app.kubernetes.io/version: v1.6.0
        app: longhorn-manager
    spec:
      containers:
      - name: longhorn-manager
        image: longhornio/longhorn-manager:v1.6.0
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true
        command:
        - longhorn-manager
        - -d
        - daemon
        - --engine-image
        - "longhornio/longhorn-engine:v1.6.0"
        - --instance-manager-image
        - "longhornio/longhorn-instance-manager:v1.6.0"
        - --share-manager-image
        - "longhornio/longhorn-share-manager:v1.6.0"
        - --backing-image-manager-image
        - "longhornio/backing-image-manager:v1.6.0"
        - --support-bundle-manager-image
        - "longhornio/support-bundle-kit:v0.0.33"
        - --manager-image
        - "longhornio/longhorn-manager:v1.6.0"
        - --service-account
        - longhorn-service-account
        - --upgrade-version-check
        ports:
        - containerPort: 9500
          name: manager
        - containerPort: 9501
          name: conversion-wh
        - containerPort: 9502
          name: admission-wh
        - containerPort: 9503
          name: recov-backend
        readinessProbe:
          httpGet:
            path: /v1/healthz
            port: 9501
            scheme: HTTPS
        volumeMounts:
        - name: dev
          mountPath: /host/dev/
        - name: proc
          mountPath: /host/proc/
        - name: longhorn
          mountPath: /var/lib/longhorn/
          mountPropagation: Bidirectional
        - name: longhorn-grpc-tls
          mountPath: /tls-files/
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
      volumes:
      - name: dev
        hostPath:
          path: /dev/
      - name: proc
        hostPath:
          path: /proc/
      - name: longhorn
        hostPath:
          path: /var/lib/longhorn/
      - name: longhorn-grpc-tls
        secret:
          secretName: longhorn-grpc-tls
          optional: true
      priorityClassName: "longhorn-critical"
      serviceAccountName: longhorn-service-account
  updateStrategy:
    rollingUpdate:
      maxUnavailable: "100%"
---

[PhanLe1010]

Can you change imagePullPolicy: IfNotPresent to imagePullPolicy: Always to see if Longhorn manager pod can come up?

kubectl edit daemonsets.apps longhorn-manager -n longhorn-system

[ctraber]

Now I'm confused kubectl edit shows image 1.5.3

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: apps/v1
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: "11"
    meta.helm.sh/release-name: longhorn
    meta.helm.sh/release-namespace: longhorn-system
  creationTimestamp: "2023-12-15T18:40:03Z"
  generation: 11
  labels:
    app: longhorn-manager
    app.kubernetes.io/instance: longhorn
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: longhorn
    app.kubernetes.io/version: v1.6.0
    helm.sh/chart: longhorn-1.6.0
  name: longhorn-manager
  namespace: longhorn-system
  resourceVersion: "20429553"
  uid: 381e5451-e0f3-47a8-a2fe-adce8705545f
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: longhorn-manager
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: longhorn-manager
        app.kubernetes.io/instance: longhorn
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: longhorn
        app.kubernetes.io/version: v1.6.0
        helm.sh/chart: longhorn-1.6.0
    spec:
      containers:
      - command:
        - longhorn-manager
        - -d
        - daemon
        - --engine-image
        - longhornio/longhorn-engine:v1.5.3
        - --instance-manager-image
        - longhornio/longhorn-instance-manager:v1.5.3
        - --share-manager-image
        - longhornio/longhorn-share-manager:v1.5.3
        - --backing-image-manager-image
        - longhornio/backing-image-manager:v1.5.3
        - --support-bundle-manager-image
        - longhornio/support-bundle-kit:v0.0.27
        - --manager-image
        - longhornio/longhorn-manager:v1.5.3
        - --service-account
        - longhorn-service-account
        - --upgrade-version-check
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        image: longhornio/longhorn-manager:v1.5.3
        imagePullPolicy: Always
        name: longhorn-manager
        ports:
        - containerPort: 9500
          name: manager
          protocol: TCP
        - containerPort: 9501
          name: conversion-wh
          protocol: TCP
        - containerPort: 9502
          name: admission-wh
          protocol: TCP
        - containerPort: 9503
          name: recov-backend
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /v1/healthz
            port: 9501
            scheme: HTTPS
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /host/dev/
          name: dev
        - mountPath: /host/proc/
          name: proc
        - mountPath: /var/lib/longhorn/
          mountPropagation: Bidirectional
          name: longhorn
        - mountPath: /tls-files/
          name: longhorn-grpc-tls
      dnsPolicy: ClusterFirst
      nodeSelector:
        longhorn: "true"
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: longhorn-service-account
      serviceAccountName: longhorn-service-account
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/control-plane
        operator: Exists
      volumes:
      - hostPath:
          path: /dev/
          type: ""
        name: dev
      - hostPath:
          path: /proc/
          type: ""
        name: proc
      - hostPath:
          path: /var/lib/longhorn/
          type: ""
        name: longhorn
      - name: longhorn-grpc-tls
        secret:
          defaultMode: 420
          optional: true
          secretName: longhorn-grpc-tls
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 100%
    type: RollingUpdate
status:
  currentNumberScheduled: 3
  desiredNumberScheduled: 3
  numberMisscheduled: 0
  numberReady: 0
  numberUnavailable: 3
  observedGeneration: 11
  updatedNumberScheduled: 3

[PhanLe1010]

What is the Helm value that was used?
helm get values longhorn -n longhorn-system

[ctraber]

I used a value file for setting some tolerations previously, and did not specify a value file on upgrade.
With new value file it works now.

Thanks al lot for your help!