Longhorn Manager Pods CrashLoop after upgrade from 1.4.3 to 1.5.4

[jangrui]

Environment

• Longhorn version: v1.5.4
• Kubernetes version: v1.24.11
• Node config
  • OS type and version: Anolis OS 8.8
  • Kernel version: 4.18.0-477.13.1.0.1.an8.x86_64
  • CPU per node: 16
  • Memory per node: 32
  • Disk type: Virtio block device
• Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): KVM

Additional context

• ENV check

[INFO]  Required dependencies 'kubectl jq mktemp sort printf' are installed.
[INFO]  All nodes have unique hostnames.
[INFO]  Waiting for longhorn-environment-check pods to become ready (0/4)...
[INFO]  All longhorn-environment-check pods are ready (4/4).
[INFO]  MountPropagation is enabled
[INFO]  Checking kernel release...
[WARN]  Node silkdo-dev-06 has outdated kernel release: 4.18.0-477.13.1.0.1.an8.x86_64. Recommending kernel release >= 5.8
[WARN]  Node silkdo-dev-04 has outdated kernel release: 4.18.0-477.13.1.0.1.an8.x86_64. Recommending kernel release >= 5.8
[WARN]  Node silkdo-dev-05 has outdated kernel release: 4.18.0-477.13.1.0.1.an8.x86_64. Recommending kernel release >= 5.8
[WARN]  Node silkdo-dev-07 has outdated kernel release: 4.18.0-477.13.1.0.1.an8.x86_64. Recommending kernel release >= 5.8
[INFO]  Checking iscsid...
[INFO]  Checking multipathd...
[INFO]  Checking packages...
[INFO]  Checking nfs client...
[INFO]  Cleaning up longhorn-environment-check pods...
[INFO]  Cleanup completed.

• Helm upgrade

VERSION=1.5.4

curl -L https://mirror.ghproxy.com/https://github.com/longhorn/charts/releases/download/longhorn-${VERSION}/longhorn-${VERSION}.tgz -o /root/.cache/helm/repository/longhorn-${VERSION}.tgz

cat <<-'EOF' | helm upgrade -i longhorn /root/.cache/helm/repository/longhorn-${VERSION}.tgz --namespace longhorn-system --create-namespace -f - 
global:
  cattle:
    systemDefaultRegistry: uhub.service.ucloud.cn/silkdo
    windowsCluster:
      enabled: true
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      defaultSetting:
        taintToleration: "node-role.kubernetes.io/control-plane:NoSchedule;node-role.kubernetes.io/master:NoSchedule"
        systemManagedComponentsNodeSelector: node.longhorn.io/create-default-disk:true

service:
  ui:
    type: NodePort
    nodePort: 30890

defaultSettings:
  backupTarget: nfs://192.168.21.2:/opt/scback
  # backupTargetCredentialSecret: 
  createDefaultDiskLabeledNodes: true
  # defaultDataLocality: "best-effort"
  replicaAutoBalance: best-effort
  storageOverProvisioningPercentage: 100
  upgradeChecker: false
  defaultLonghornStaticStorageClass: longhorn
  priorityClass: system-cluster-critical
  nodeDownPodDeletionPolicy: delete-both-statefulset-and-deployment-pod
  allowVolumeCreationWithDegradedAvailability: false
  concurrentAutomaticEngineUpgradePerNodeLimit: 5
EOF

• Logs

W0301 06:38:30.392657       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0301 06:38:30.394821       1 shared_informer.go:311] Waiting for caches to sync for longhorn datastore
I0301 06:38:30.995120       1 shared_informer.go:318] Caches are synced for longhorn datastore
time="2024-03-01T06:38:30Z" level=info msg="Starting longhorn conversion webhook server" func=webhook.StartWebhook file="webhook.go:23"
time="2024-03-01T06:38:30Z" level=info msg="Waiting for conversion webhook to become ready" func=webhook.StartWebhook file="webhook.go:42"
time="2024-03-01T06:38:30Z" level=warning msg="Failed to get webhook health endpoint https://localhost:9501/v1/healthz" func=webhook.StartWebhook file="webhook.go:54" error="Get \"https://localhost:9501/v1/healthz\": dial tcp [::1]:9501: connect: connection refused"
time="2024-03-01T06:38:31Z" level=info msg="Listening on :9501" func=server.ListenAndServe.func2 file="server.go:77"
time="2024-03-01T06:38:31Z" level=info msg="certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca,O=dynamiclistener-org: notBefore=2023-05-08 11:19:35 +0000 UTC notAfter=2025-03-01 06:38:31 +0000 UTC" func=factory.NewSignedCert file="cert_utils.go:122"
time="2024-03-01T06:38:31Z" level=info msg="Active TLS secret longhorn-system/longhorn-webhook-tls (ver=442470798) (count 2): map[listener.cattle.io/cn-longhorn-admission-webhook.longhor-59584d:longhorn-admission-webhook.longhorn-system.svc listener.cattle.io/cn-longhorn-conversion-webhook.longho-6a0089:longhorn-conversion-webhook.longhorn-system.svc listener.cattle.io/fingerprint:SHA1=5946290E62AC5F8B9112FCA045533014A56AFDFA]" func="memory.(*memory).Update" file="memory.go:42"
time="2024-03-01T06:38:32Z" level=info msg="Starting apiregistration.k8s.io/v1, Kind=APIService controller" func="controller.(*controller).run" file="controller.go:144"
time="2024-03-01T06:38:32Z" level=info msg="Starting /v1, Kind=Secret controller" func="controller.(*controller).run" file="controller.go:144"
time="2024-03-01T06:38:32Z" level=info msg="Starting apiextensions.k8s.io/v1, Kind=CustomResourceDefinition controller" func="controller.(*controller).run" file="controller.go:144"
time="2024-03-01T06:38:32Z" level=info msg="Building conversion rules..." func="server.(*WebhookServer).runConversionWebhookListenAndServe.func1" file="server.go:195"
time="2024-03-01T06:38:32Z" level=info msg="Updating TLS secret for longhorn-system/longhorn-webhook-tls (count: 2): map[listener.cattle.io/cn-longhorn-admission-webhook.longhor-59584d:longhorn-admission-webhook.longhorn-system.svc listener.cattle.io/cn-longhorn-conversion-webhook.longho-6a0089:longhorn-conversion-webhook.longhorn-system.svc listener.cattle.io/fingerprint:SHA1=5946290E62AC5F8B9112FCA045533014A56AFDFA]" func="kubernetes.(*storage).saveInK8s" file="controller.go:225"
time="2024-03-01T06:38:32Z" level=info msg="Webhook conversion is ready" func=webhook.StartWebhook file="webhook.go:56"
time="2024-03-01T06:38:32Z" level=warning msg="Started longhorn conversion webhook server" func=webhook.StartWebhook file="webhook.go:74"
time="2024-03-01T06:38:32Z" level=info msg="Starting longhorn admission webhook server" func=webhook.StartWebhook file="webhook.go:23"
time="2024-03-01T06:38:32Z" level=info msg="Waiting for admission webhook to become ready" func=webhook.StartWebhook file="webhook.go:42"
time="2024-03-01T06:38:32Z" level=info msg="Add validaton handler for nodes.longhorn.io (Node)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:32Z" level=info msg="Add validaton handler for settings.longhorn.io (Setting)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:32Z" level=info msg="Add validaton handler for recurringjobs.longhorn.io (RecurringJob)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:32Z" level=info msg="Add validaton handler for backingimages.longhorn.io (BackingImage)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:32Z" level=info msg="Add validaton handler for volumes.longhorn.io (Volume)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:32Z" level=info msg="Add validaton handler for orphans.longhorn.io (Orphan)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:32Z" level=info msg="Add validaton handler for snapshots.longhorn.io (Snapshot)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:32Z" level=info msg="Add validaton handler for supportbundles.longhorn.io (SupportBundle)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:32Z" level=info msg="Add validaton handler for systembackups.longhorn.io (SystemBackup)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:32Z" level=info msg="Add validaton handler for systemrestores.longhorn.io (SystemRestore)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:32Z" level=info msg="Add validaton handler for volumeattachments.longhorn.io (VolumeAttachment)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:32Z" level=info msg="Add validaton handler for engines.longhorn.io (Engine)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:32Z" level=info msg="Add validaton handler for replicas.longhorn.io (Replica)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:32Z" level=info msg="Add mutation handler for backups.longhorn.io (Backup)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:32Z" level=info msg="Add mutation handler for backingimages.longhorn.io (BackingImage)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:32Z" level=info msg="Add mutation handler for backingimagemanagers.longhorn.io (BackingImageManager)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:33Z" level=info msg="Add mutation handler for backingimagedatasources.longhorn.io (BackingImageDataSource)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:33Z" level=info msg="Add mutation handler for nodes.longhorn.io (Node)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:33Z" level=info msg="Add mutation handler for volumes.longhorn.io (Volume)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:33Z" level=info msg="Add mutation handler for engines.longhorn.io (Engine)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:33Z" level=info msg="Add mutation handler for recurringjobs.longhorn.io (RecurringJob)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:33Z" level=info msg="Add mutation handler for engineimages.longhorn.io (EngineImage)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:32Z" level=warning msg="Failed to get webhook health endpoint https://localhost:9502/v1/healthz" func=webhook.StartWebhook file="webhook.go:54" error="Get \"https://localhost:9502/v1/healthz\": dial tcp [::1]:9502: connect: connection refused"
time="2024-03-01T06:38:33Z" level=info msg="Add mutation handler for orphans.longhorn.io (Orphan)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:33Z" level=info msg="Add mutation handler for sharemanagers.longhorn.io (ShareManager)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:33Z" level=info msg="Add mutation handler for backupvolumes.longhorn.io (BackupVolume)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:33Z" level=info msg="Add mutation handler for snapshots.longhorn.io (Snapshot)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:33Z" level=info msg="Add mutation handler for replicas.longhorn.io (Replica)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:33Z" level=info msg="Add mutation handler for supportbundles.longhorn.io (SupportBundle)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:33Z" level=info msg="Add mutation handler for systembackups.longhorn.io (SystemBackup)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:33Z" level=info msg="Add mutation handler for volumeattachments.longhorn.io (VolumeAttachment)" func=server.addHandler file="handler.go:17"
time="2024-03-01T06:38:33Z" level=info msg="Building conversion rules..." func="server.(*WebhookServer).runConversionWebhookListenAndServe.func1" file="server.go:195"
time="2024-03-01T06:38:33Z" level=info msg="Active TLS secret longhorn-system/longhorn-webhook-tls (ver=442470798) (count 2): map[listener.cattle.io/cn-longhorn-admission-webhook.longhor-59584d:longhorn-admission-webhook.longhorn-system.svc listener.cattle.io/cn-longhorn-conversion-webhook.longho-6a0089:longhorn-conversion-webhook.longhorn-system.svc listener.cattle.io/fingerprint:SHA1=5946290E62AC5F8B9112FCA045533014A56AFDFA]" func="memory.(*memory).Update" file="memory.go:42"
time="2024-03-01T06:38:33Z" level=info msg="Updating TLS secret for longhorn-system/longhorn-webhook-tls (count: 2): map[listener.cattle.io/cn-longhorn-admission-webhook.longhor-59584d:longhorn-admission-webhook.longhorn-system.svc listener.cattle.io/cn-longhorn-conversion-webhook.longho-6a0089:longhorn-conversion-webhook.longhorn-system.svc listener.cattle.io/fingerprint:SHA1=5946290E62AC5F8B9112FCA045533014A56AFDFA]" func="kubernetes.(*storage).saveInK8s" file="controller.go:225"
time="2024-03-01T06:38:33Z" level=info msg="Updating TLS secret for longhorn-system/longhorn-webhook-tls (count: 2): map[listener.cattle.io/cn-longhorn-admission-webhook.longhor-59584d:longhorn-admission-webhook.longhorn-system.svc listener.cattle.io/cn-longhorn-conversion-webhook.longho-6a0089:longhorn-conversion-webhook.longhorn-system.svc listener.cattle.io/fingerprint:SHA1=5946290E62AC5F8B9112FCA045533014A56AFDFA]" func="kubernetes.(*storage).saveInK8s" file="controller.go:225"
time="2024-03-01T06:38:33Z" level=info msg="Listening on :9502" func=server.ListenAndServe.func2 file="server.go:77"
time="2024-03-01T06:38:33Z" level=info msg="Building validation rules..." func="server.(*WebhookServer).runAdmissionWebhookListenAndServe.func1" file="server.go:113"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:nodes Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0011ea1c0 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:settings Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc00598b320 OperationTypes:[CREATE UPDATE DELETE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:recurringjobs Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc005986a80 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:backingimages Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc005986f00 OperationTypes:[CREATE DELETE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:volumes Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc006426900 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:orphans Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0059d9760 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:snapshots Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0011eb180 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:supportbundles Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc00087a700 OperationTypes:[CREATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:systembackups Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc000d496c0 OperationTypes:[CREATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:systemrestores Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0059d9b80 OperationTypes:[CREATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:volumeattachments Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0059df400 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:engines Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0012c3200 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:replicas Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc006408840 OperationTypes:[CREATE UPDATE DELETE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=info msg="Building mutation rules..." func="server.(*WebhookServer).runAdmissionWebhookListenAndServe.func1" file="server.go:115"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:backups Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc006404780 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:backingimages Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc005987380 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:backingimagemanagers Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc001082540 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:backingimagedatasources Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc00599a800 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:nodes Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc001083a40 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:volumes Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc006427680 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:engines Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0012c3b00 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:recurringjobs Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc005987800 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:engineimages Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc00065b680 OperationTypes:[CREATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:orphans Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0060a8f20 OperationTypes:[CREATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:sharemanagers Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0060a9340 OperationTypes:[CREATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:backupvolumes Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc005dbaa00 OperationTypes:[CREATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:snapshots Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc001491c00 OperationTypes:[CREATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:replicas Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0060b1600 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:supportbundles Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc001294a80 OperationTypes:[CREATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:systembackups Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0060d01c0 OperationTypes:[CREATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:volumeattachments Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0060a3e00 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="DesiredSet - No change(2) admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration /longhorn-webhook-validator for  longhorn-system/longhorn-webhook-ca" func="apply.(*desiredSet).compareObjects" file="desiredset_compare.go:262"
time="2024-03-01T06:38:33Z" level=debug msg="DesiredSet - No change(2) admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration /longhorn-webhook-mutator for  longhorn-system/longhorn-webhook-ca" func="apply.(*desiredSet).compareObjects" file="desiredset_compare.go:262"
time="2024-03-01T06:38:33Z" level=info msg="Building conversion rules..." func="server.(*WebhookServer).runConversionWebhookListenAndServe.func1" file="server.go:195"
time="2024-03-01T06:38:33Z" level=info msg="Updating TLS secret for longhorn-system/longhorn-webhook-tls (count: 2): map[listener.cattle.io/cn-longhorn-admission-webhook.longhor-59584d:longhorn-admission-webhook.longhorn-system.svc listener.cattle.io/cn-longhorn-conversion-webhook.longho-6a0089:longhorn-conversion-webhook.longhorn-system.svc listener.cattle.io/fingerprint:SHA1=5946290E62AC5F8B9112FCA045533014A56AFDFA]" func="kubernetes.(*storage).saveInK8s" file="controller.go:225"
time="2024-03-01T06:38:33Z" level=info msg="Building validation rules..." func="server.(*WebhookServer).runAdmissionWebhookListenAndServe.func1" file="server.go:113"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:nodes Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0011ea1c0 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:settings Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc00598b320 OperationTypes:[CREATE UPDATE DELETE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:recurringjobs Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc005986a80 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:backingimages Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc005986f00 OperationTypes:[CREATE DELETE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:volumes Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc006426900 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:orphans Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0059d9760 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:snapshots Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0011eb180 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:supportbundles Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc00087a700 OperationTypes:[CREATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:systembackups Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc000d496c0 OperationTypes:[CREATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:systemrestores Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0059d9b80 OperationTypes:[CREATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:volumeattachments Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0059df400 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:engines Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0012c3200 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:replicas Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc006408840 OperationTypes:[CREATE UPDATE DELETE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=info msg="Building mutation rules..." func="server.(*WebhookServer).runAdmissionWebhookListenAndServe.func1" file="server.go:115"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:backups Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc006404780 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:backingimages Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc005987380 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:backingimagemanagers Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc001082540 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:backingimagedatasources Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc00599a800 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:nodes Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc001083a40 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:volumes Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc006427680 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:engines Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0012c3b00 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:recurringjobs Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc005987800 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:engineimages Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc00065b680 OperationTypes:[CREATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:orphans Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0060a8f20 OperationTypes:[CREATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:sharemanagers Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0060a9340 OperationTypes:[CREATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:backupvolumes Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc005dbaa00 OperationTypes:[CREATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:snapshots Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc001491c00 OperationTypes:[CREATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:replicas Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0060b1600 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:supportbundles Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc001294a80 OperationTypes:[CREATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:systembackups Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0060d01c0 OperationTypes:[CREATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="Add rule for {Name:volumeattachments Scope:Namespaced APIGroup:longhorn.io APIVersion:v1beta2 ObjectType:0xc0060a3e00 OperationTypes:[CREATE UPDATE]}" func="server.(*WebhookServer).buildRules" file="server.go:249"
time="2024-03-01T06:38:33Z" level=debug msg="DesiredSet - No change(2) admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration /longhorn-webhook-validator for  longhorn-system/longhorn-webhook-ca" func="apply.(*desiredSet).compareObjects" file="desiredset_compare.go:262"
time="2024-03-01T06:38:33Z" level=debug msg="DesiredSet - No change(2) admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration /longhorn-webhook-mutator for  longhorn-system/longhorn-webhook-ca" func="apply.(*desiredSet).compareObjects" file="desiredset_compare.go:262"
time="2024-03-01T06:38:35Z" level=info msg="Webhook admission is ready" func=webhook.StartWebhook file="webhook.go:56"
time="2024-03-01T06:38:35Z" level=warning msg="Started longhorn admission webhook server" func=webhook.StartWebhook file="webhook.go:74"
time="2024-03-01T06:38:35Z" level=info msg="Starting longhorn recovery-backend server" func=recovery_backend.StartRecoveryBackend file="recovery_backend.go:13"
time="2024-03-01T06:38:35Z" level=info msg="Started longhorn recovery-backend server" func=recovery_backend.StartRecoveryBackend file="recovery_backend.go:22"
W0301 06:38:35.004231       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2024-03-01T06:38:35Z" level=info msg="Recovery-backend server is running at :9503" func="server.(*RecoveryBackendServer).ListenAndServe" file="server.go:36"
time="2024-03-01T06:38:35Z" level=info msg="Checking if the upgrade path from v1.4.3 to v1.5.4 is supported" func=util.CheckUpgradePathSupported file="util.go:198"
time="2024-03-01T06:38:35Z" level=info msg="Waiting for old Longhorn manager pods to be fully removed" func=upgrade.waitForOldLonghornManagersToBeFullyRemoved file="upgrade.go:276"
I0301 06:38:35.034259       1 leaderelection.go:250] attempting to acquire leader lease longhorn-system/longhorn-manager-upgrade-lock...
I0301 06:38:35.049979       1 leaderelection.go:260] successfully acquired lease longhorn-system/longhorn-manager-upgrade-lock
time="2024-03-01T06:38:35Z" level=info msg="Start upgrading" func=upgrade.upgrade.func1 file="upgrade.go:130"
time="2024-03-01T06:38:35Z" level=info msg="No API version upgrade is needed" func=upgrade.doAPIVersionUpgrade file="upgrade.go:176"
time="2024-03-01T06:38:35Z" level=info msg="Walking through the resource upgrade path v1.4.x to v1.5.0" func=upgrade.doResourceUpgrade file="upgrade.go:237"
time="2024-03-01T06:38:35Z" level=info msg="Walking through the resource upgrade path v1.5.1 to v1.5.2" func=upgrade.doResourceUpgrade file="upgrade.go:243"
time="2024-03-01T06:38:35Z" level=info msg="Walking through the resource upgrade path v1.5.3 to v1.5.4" func=upgrade.doResourceUpgrade file="upgrade.go:247"
time="2024-03-01T06:38:36Z" level=error msg="Upgrade failed: upgrade resources failed: create not allowed while custom resource definition is terminating" func=upgrade.upgrade.func1.1 file="upgrade.go:125"
time="2024-03-01T06:38:36Z" level=info msg="Upgrade leader lost: silkdo-dev-06" func=upgrade.upgrade.func2 file="upgrade.go:139"
time="2024-03-01T06:38:36Z" level=fatal msg="Error starting manager: upgrade resources failed: create not allowed while custom resource definition is terminating" func=main.main.DaemonCmd.func3 file="daemon.go:84"

[derekbit]

time="2024-03-01T06:38:36Z" level=info msg="Upgrade leader lost: silkdo-dev-06" func=upgrade.upgrade.func2 file="upgrade.go:139"

Is the network not stable?

[jangrui]

It seems that it is different from his problem. After the backup is removed, there is still this problem.

[jangrui]

Is there a debug method?

[derekbit]

It seems that it is different from his problem. After the backup is removed, there is still this problem.

I mean the context related to create not allowed while custom resource definition is terminating. :)
Anyway, can you provide the result of kubectl -n longhorn-system get crds -o yaml? Thank you.

[jangrui]

Uploading crd-1.5.4.yaml.zip…

[jangrui]

@derekbit