Longhorn v1.5.4-rc4

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Highlight

• [BACKPORT][v1.5.4][FEATURE] Add a new settings that allows Longhorn to evict replicas automatically when a node is drained 7421 - @yangchiu @ejweber

Improvement

• [BACKPORT][v1.5.4][IMPROVEMENT] Use HEAD instead of a GET to fetch the Content-Length of an resource via URL 7981 - @votdev @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Change support-bundle-manager image pull policy to PullIfNotPresent 7999 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Use HEAD instead of a GET to fetch the Content-Length of an resource via URL 7974 - @votdev
• [BACKPORT][v1.5.4][FEATURE] Update base image of Longhorn components to BCI 15.5 7134 - @nitendra-suse
• [BACKPORT][v1.5.4]Allow to set mount options for storageclass via values.yaml in helm chart 7593 - @ChanYiLin @mantissahz
• [BACKPORT][v1.5.4][IMPROVEMENT] Remove startup probe of CSI driver after liveness probe conn fix ready 7933 - @ejweber @chriscchien
• [IMPROVEMENT] Make environment_check look for a global default K8s priority class in those releases that it affects. 7831 - @mantissahz @james-munson
• [BACKPORT][v1.5.4][IMPROVEMENT] Allow deployment of Prometheus ServiceMonitor with the Longhorn helm chart 7864 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Remove unused process manager connection in longhorn-manager 7785 - @derekbit @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Clean up backup target in IM-R pod if the backup target setting is unset 7145 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] BackingImage should be compressed when downloading and use the name as filename instead of UUID 7397 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Automatically remount read-only RWO volume to read-write 7500 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] deploy: driver deployer shouldn't cleanup previous deployment if Kubernetes version changes 7345 - @PhanLe1010 @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Only restarts pods with volumes in the unexpected Read-Only state 7729 - @yangchiu @ChanYiLin
• [BACKPORT][v1.5.4][IMPROVEMENT] Improve handling of 16TiB+ volumes with ext4 as the underlying file system 7429 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Volumes: metrics for snapshots include (size and type: system vs user) 7725 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Improve the profiler of longhorn-engine for runtime profiling 7545 - @Vicente-Cheng @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Don't crash the migration engine when kubelet restarts 7328 - @yangchiu @ejweber
• [BACKPORT][v1.5.4][IMPROVEMENT] Upgrade CSI components to the latest patch release 7492 - @c3y1huang @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Reject the last replica deletion if its volume.spec.deletionTimestamp is not set 7432 - @yangchiu @derekbit
• [BACKPORT][v1.5.4][IMPROVEMENT] Upgrade support bundle kit version to v0.0.33 7279 - @c3y1huang
• [BACKPORT][v1.5.4][IMPROVEMENT] Review and simplify longhorn component image build 7162 - @ChanYiLin
• [BACKPORT][v1.5.4][IMPROVEMENT] Replace deprecated grpc.WithInsecure 7364 - @c3y1huang
• [BACKPORT][v1.5.4][IMPROVEMENT] Have a setting to disable snapshot purge for maintenance purpose 7265 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Bypass upgrade when installing a fresh setup 7283 - @mantissahz @roger-ryao

Bug

• [BUG][v1.5.x] DR volume unable to be activated if the latest backup's been deleted 7997 - @yangchiu @shuo-wu
• [BACKPORT][v1.5.4][BUG][v1.5.x] Recurring job fails to create backup when volume detached 8015 - @yangchiu @mantissahz @PhanLe1010 @c3y1huang
• [BACKPORT][v1.5.4][BUG] The activated DR volume do not contain the latest data. 7947 - @shuo-wu @roger-ryao
• [BACKPORT][v1.5.4][BUG] Deadlock for RWX volume if an error occurs in its share-manager pod 7186 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][BUG] Deadlock is possible in v1.6.0 instance manager 7941 - @ejweber @roger-ryao
• [BACKPORT][v1.5.4][BUG] Longhorn may keep corrupted salvaged replicas and discard good ones 7801 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][BUG] Deadlock between volume migration and upgrade after Longhorn upgrade 7869 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][BUG] Executing fstrim while rebuilding causes IO errors 7867 - @ejweber @chriscchien
• [BUG] Backup related test cases failed 7989 - @yangchiu @shuo-wu
• [BACKPORT][v1.5.4][BUG] BackingImage does not download URL correctly in some situation 7986 - @yangchiu
• [BACKPORT][v1.5.4][BUG] The feature of auto remount read only volume not work on a single node cluster. 7844 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][BUG] Volumes stuck upgrading after 1.5.3 -> 1.6.0 upgrade. 7901 - @yangchiu @ejweber
• [BUG][v1.5.4-rc1] V2 volume have engine upgrade option on UI after upgrade from v1.5.3 to v1.5.4-rc1 7863 - @chriscchien @scures
• [BACKPORT][v1.5.4][BUG] longhorn manager pod fails to start in container-based K3s 7848 - @ChanYiLin
• [BACKPORT][v1.5.4][BUG] Relax S3 client retry intervals, for throttled requests 7098 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][BUG][v1.6.0-rc1] Negative test case failed: Stop Volume Node Kubelet For More Than Pod Eviction Timeout While Workload Heavy Writing 7761 - @yangchiu @c3y1huang
• [BACKPORT][v1.5.4][BUG] Volumes don't mount with mTLS enabled 7789 - @derekbit @roger-ryao
• [BACKPORT][v1.5.4][BUG] supportbundle/kubelet.log empty in k3s environment 7123 - @c3y1huang @chriscchien @roger-ryao
• [BUG] v1.5.x/v1.4.x BackingImage download fails if URL has query parameters 7822 - @ChanYiLin @mantissahz
• [BACKPORT][v1.5.4][BUG] Metric totalVolumeSize and totalVolumeActualSize incorrect due to v2 volume counts 7392 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][BUG] Continuously auto-balancing replicas when zone does not have enough space 7306 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][BUG] Unable to list backups when backuptarget resource is picked up by a cordoned node 7621 - @mantissahz @c3y1huang
• [BUG] The wrong template in default-setting.yaml of the Longhorn chart in v1.5 and v1.4 7459 - @mantissahz @roger-ryao
• [BACKPORT][v1.5.4][BUG] Failed to check_volume_data after volume engine upgrade/migration 7402 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.5.4][BUG] Volume conditions are not represented in the UI for v1.4.x and newer 7242 - @m-ildefons @roger-ryao
• [BACKPORT][v1.5.4][BUG] Confusing logging when trying to attach a new volume with no scheduled replicas 7245 - @ejweber @roger-ryao
• [BACKPORT][v1.5.4][BUG] Environment check script claims success when kubectl fails. 7216 - @james-munson @roger-ryao
• [BACKPORT][v1.5.4][BUG] Backup volume attachment tickets might not be cleaned up after completion. 7604 - @james-munson @chriscchien
• [BACKPORT][v1.5.4][BUG][v1.6.0-rc1] Some Longhorn resources remaining after longhorn-uninstall job completed 7663 - @yangchiu @PhanLe1010
• [BACKPORT][v1.5.4][BUG] Backi...

Longhorn v1.5.4-rc3

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Highlight

• [BACKPORT][v1.5.4][FEATURE] Add a new settings that allows Longhorn to evict replicas automatically when a node is drained 7421 - @yangchiu @ejweber

Improvement

• [BACKPORT][v1.5.4]Allow to set mount options for storageclass via values.yaml in helm chart 7593 - @ChanYiLin @mantissahz
• [BACKPORT][v1.5.4][FEATURE] Update base image of Longhorn components to BCI 15.5 7134 - @nitendra-suse
• [BACKPORT][v1.5.4][IMPROVEMENT] Use HEAD instead of a GET to fetch the Content-Length of an resource via URL 7981 - @votdev @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Use HEAD instead of a GET to fetch the Content-Length of an resource via URL 7974 - @votdev
• [BACKPORT][v1.5.4][IMPROVEMENT] Improve environment_check script for NFS protocol bug and the host system self diagnosis 7969 - @james-munson
• [BACKPORT][v1.5.4][IMPROVEMENT] Remove startup probe of CSI driver after liveness probe conn fix ready 7933 - @ejweber @chriscchien
• [IMPROVEMENT] Make environment_check look for a global default K8s priority class in those releases that it affects. 7831 - @mantissahz @james-munson
• [BACKPORT][v1.5.4][IMPROVEMENT] Allow deployment of Prometheus ServiceMonitor with the Longhorn helm chart 7864 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Remove unused process manager connection in longhorn-manager 7785 - @derekbit @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Clean up backup target in IM-R pod if the backup target setting is unset 7145 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] BackingImage should be compressed when downloading and use the name as filename instead of UUID 7397 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Automatically remount read-only RWO volume to read-write 7500 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] deploy: driver deployer shouldn't cleanup previous deployment if Kubernetes version changes 7345 - @PhanLe1010 @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Only restarts pods with volumes in the unexpected Read-Only state 7729 - @yangchiu @ChanYiLin
• [BACKPORT][v1.5.4][IMPROVEMENT] Improve handling of 16TiB+ volumes with ext4 as the underlying file system 7429 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Volumes: metrics for snapshots include (size and type: system vs user) 7725 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Improve the profiler of longhorn-engine for runtime profiling 7545 - @Vicente-Cheng @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Don't crash the migration engine when kubelet restarts 7328 - @yangchiu @ejweber
• [BACKPORT][v1.5.4][IMPROVEMENT] Upgrade CSI components to the latest patch release 7492 - @c3y1huang @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Reject the last replica deletion if its volume.spec.deletionTimestamp is not set 7432 - @yangchiu @derekbit
• [BACKPORT][v1.5.4][IMPROVEMENT] Upgrade support bundle kit version to v0.0.33 7279 - @c3y1huang
• [BACKPORT][v1.5.4][IMPROVEMENT] Review and simplify longhorn component image build 7162 - @ChanYiLin
• [BACKPORT][v1.5.4][IMPROVEMENT] Replace deprecated grpc.WithInsecure 7364 - @c3y1huang
• [BACKPORT][v1.5.4][IMPROVEMENT] Have a setting to disable snapshot purge for maintenance purpose 7265 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Bypass upgrade when installing a fresh setup 7283 - @mantissahz @roger-ryao

Bug

• [BUG][v1.5.x] Recurring job fails to create backup when volume detached 7937 - @mantissahz @c3y1huang
• [BUG] Backup related test cases failed 7989 - @yangchiu @shuo-wu
• [BACKPORT][v1.5.4][BUG] The activated DR volume do not contain the latest data. 7947 - @shuo-wu @roger-ryao
• [BACKPORT][v1.5.4][BUG] BackingImage does not download URL correctly in some situation 7986 - @yangchiu
• [BACKPORT][v1.5.4][BUG] The feature of auto remount read only volume not work on a single node cluster. 7844 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][BUG] Deadlock is possible in v1.6.0 instance manager 7941 - @ejweber @roger-ryao
• [BACKPORT][v1.5.4][BUG] Volumes stuck upgrading after 1.5.3 -> 1.6.0 upgrade. 7901 - @yangchiu @ejweber
• [BUG][v1.5.4-rc1] V2 volume have engine upgrade option on UI after upgrade from v1.5.3 to v1.5.4-rc1 7863 - @chriscchien @scures
• [BACKPORT][v1.5.4][BUG] longhorn manager pod fails to start in container-based K3s 7848 - @ChanYiLin
• [BACKPORT][v1.5.4][BUG] Deadlock between volume migration and upgrade after Longhorn upgrade 7869 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][BUG] Executing fstrim while rebuilding causes IO errors 7867 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][BUG] Relax S3 client retry intervals, for throttled requests 7098 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][BUG][v1.6.0-rc1] Negative test case failed: Stop Volume Node Kubelet For More Than Pod Eviction Timeout While Workload Heavy Writing 7761 - @yangchiu @c3y1huang
• [BACKPORT][v1.5.4][BUG] Volumes don't mount with mTLS enabled 7789 - @derekbit @roger-ryao
• [BACKPORT][v1.5.4][BUG] supportbundle/kubelet.log empty in k3s environment 7123 - @c3y1huang @chriscchien @roger-ryao
• [BACKPORT][v1.5.4][BUG] Longhorn may keep corrupted salvaged replicas and discard good ones 7801 - @ejweber @chriscchien
• [BUG] v1.5.x/v1.4.x BackingImage download fails if URL has query parameters 7822 - @ChanYiLin @mantissahz
• [BACKPORT][v1.5.4][BUG] Metric totalVolumeSize and totalVolumeActualSize incorrect due to v2 volume counts 7392 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][BUG] Continuously auto-balancing replicas when zone does not have enough space 7306 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][BUG] Unable to list backups when backuptarget resource is picked up by a cordoned node 7621 - @mantissahz @c3y1huang
• [BUG] The wrong template in default-setting.yaml of the Longhorn chart in v1.5 and v1.4 7459 - @mantissahz @roger-ryao
• [BACKPORT][v1.5.4][BUG] Failed to check_volume_data after volume engine upgrade/migration 7402 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.5.4][BUG] Volume conditions are not represented in the UI for v1.4.x and newer 7242 - @m-ildefons @roger-ryao
• [BACKPORT][v1.5.4][BUG] Confusing logging when trying to attach a new volume with no scheduled replicas 7245 - @ejweber @roger-ryao
• [BACKPORT][v1.5.4][BUG] Environment check script claims success when kubectl fails. 7216 - @james-munson @roger-ryao
• [BACKPORT][v1.5.4][BUG] Backup volume attachment tickets might not be cleaned up after completion. 7604 - @james-munson @chriscchien
• [BACKPORT][v1.5.4][BUG][v1.6.0-rc1] Some Longhorn resources remaining after longhorn-uninstall job completed 7663 - @yangchiu @PhanLe1010
• [BACKPORT][v1.5.4][BUG] Backing Image Data Inconsistency if it's Exported from a Backing Image Backed Volume 7701 - @yangchiu @ChanYiLin
• [BACKPORT][v1.5.4][BUG] CSI components CrashLoopBackOff, failed to connect to unix://csi/csi.sock after cluster restart 7426 - @ejweber @roger-ryao
• [BA...

Longhorn v1.5.4-rc2

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Highlight

• [BACKPORT][v1.5.4][FEATURE] Add a new settings that allows Longhorn to evict replicas automatically when a node is drained 7421 - @yangchiu @ejweber

Feature

• [BACKPORT][v1.5.4]Allow to set mount options for storageclass via values.yaml in helm chart 7593 - @ChanYiLin @mantissahz
• [BACKPORT][v1.5.4][FEATURE] Update base image of Longhorn components to BCI 15.5 7134 - @nitendra-suse

Improvement

• [BACKPORT][v1.5.4][IMPROVEMENT] Allow deployment of Prometheus ServiceMonitor with the Longhorn helm chart 7864 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Remove unused process manager connection in longhorn-manager 7785 - @derekbit @roger-ryao
• [IMPROVEMENT] Make environment_check look for a global default K8s priority class in those releases that it affects. 7831 - @mantissahz @james-munson
• [BACKPORT][v1.5.4][IMPROVEMENT] Clean up backup target in IM-R pod if the backup target setting is unset 7145 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] BackingImage should be compressed when downloading and use the name as filename instead of UUID 7397 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Automatically remount read-only RWO volume to read-write 7500 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] deploy: driver deployer shouldn't cleanup previous deployment if Kubernetes version changes 7345 - @PhanLe1010 @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Only restarts pods with volumes in the unexpected Read-Only state 7729 - @yangchiu @ChanYiLin
• [BACKPORT][v1.5.4][IMPROVEMENT] Improve handling of 16TiB+ volumes with ext4 as the underlying file system 7429 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Volumes: metrics for snapshots include (size and type: system vs user) 7725 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Improve the profiler of longhorn-engine for runtime profiling 7545 - @Vicente-Cheng @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Don't crash the migration engine when kubelet restarts 7328 - @yangchiu @ejweber
• [BACKPORT][v1.5.4][IMPROVEMENT] Upgrade CSI components to the latest patch release 7492 - @c3y1huang @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Reject the last replica deletion if its volume.spec.deletionTimestamp is not set 7432 - @yangchiu @derekbit
• [BACKPORT][v1.5.4][IMPROVEMENT] Upgrade support bundle kit version to v0.0.33 7279 - @c3y1huang
• [BACKPORT][v1.5.4][IMPROVEMENT] Review and simplify longhorn component image build 7162 - @ChanYiLin
• [BACKPORT][v1.5.4][IMPROVEMENT] Replace deprecated grpc.WithInsecure 7364 - @c3y1huang
• [BACKPORT][v1.5.4][IMPROVEMENT] Have a setting to disable snapshot purge for maintenance purpose 7265 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Bypass upgrade when installing a fresh setup 7283 - @mantissahz @roger-ryao

Bug

• [BACKPORT][v1.5.4][BUG] Volumes don't mount with mTLS enabled 7789 - @derekbit @roger-ryao
• [BACKPORT][v1.5.4][BUG][v1.5.4-rc1] Recurring job failed to create/delete backups after node reboot 7856 - @james-munson
• [BUG][v1.5.4-rc1] V2 volume have engine upgrade option on UI after upgrade from v1.5.3 to v1.5.4-rc1 7863 - @scures
• [BACKPORT][v1.5.4][BUG] longhorn manager pod fails to start in container-based K3s 7848 - @ChanYiLin
• [BACKPORT][v1.5.4][BUG] The feature of auto remount read only volume not work on a single node cluster. 7844 - @ChanYiLin
• [BACKPORT][v1.5.4][BUG] Create backup failed: failed lock lock-*.lck type 1 acquisition 7876 - @yangchiu @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][BUG] supportbundle/kubelet.log empty in k3s environment 7123 - @c3y1huang @chriscchien @roger-ryao
• [BACKPORT][v1.5.4][BUG] Longhorn may keep corrupted salvaged replicas and discard good ones 7801 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][BUG] Deadlock between volume migration and upgrade after Longhorn upgrade 7869 - @ejweber
• [BACKPORT][v1.5.4][BUG] Executing fstrim while rebuilding causes IO errors 7867 - @ejweber
• [BACKPORT][v1.5.4][BUG] Relax S3 client retry intervals, for throttled requests 7098 - @mantissahz @chriscchien
• [BUG] v1.5.x/v1.4.x BackingImage download fails if URL has query parameters 7822 - @ChanYiLin @mantissahz
• [BACKPORT][v1.5.4][BUG] Metric totalVolumeSize and totalVolumeActualSize incorrect due to v2 volume counts 7392 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][BUG] Continuously auto-balancing replicas when zone does not have enough space 7306 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][BUG] Unable to list backups when backuptarget resource is picked up by a cordoned node 7621 - @mantissahz @c3y1huang
• [BUG] The wrong template in default-setting.yaml of the Longhorn chart in v1.5 and v1.4 7459 - @mantissahz @roger-ryao
• [BACKPORT][v1.5.4][BUG] Failed to check_volume_data after volume engine upgrade/migration 7402 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.5.4][BUG][v1.6.0-rc1] Negative test case failed: Stop Volume Node Kubelet For More Than Pod Eviction Timeout While Workload Heavy Writing 7761 - @yangchiu @c3y1huang
• [BACKPORT][v1.5.4][BUG] Volume conditions are not represented in the UI for v1.4.x and newer 7242 - @m-ildefons @roger-ryao
• [BACKPORT][v1.5.4][BUG] Confusing logging when trying to attach a new volume with no scheduled replicas 7245 - @ejweber @roger-ryao
• [BACKPORT][v1.5.4][BUG] Environment check script claims success when kubectl fails. 7216 - @james-munson @roger-ryao
• [BACKPORT][v1.5.4][BUG] Backup volume attachment tickets might not be cleaned up after completion. 7604 - @james-munson @chriscchien
• [BACKPORT][v1.5.4][BUG][v1.6.0-rc1] Some Longhorn resources remaining after longhorn-uninstall job completed 7663 - @yangchiu @PhanLe1010
• [BACKPORT][v1.5.4][BUG] Backing Image Data Inconsistency if it's Exported from a Backing Image Backed Volume 7701 - @yangchiu @ChanYiLin
• [BACKPORT][v1.5.4][BUG] CSI components CrashLoopBackOff, failed to connect to unix://csi/csi.sock after cluster restart 7426 - @ejweber @roger-ryao
• [BACKPORT][v1.5.4][BUG] Volume could not be remounted after engine process killed 7772 - @ChanYiLin @shuo-wu @roger-ryao
• [BACKPORT][v1.5.4][BUG] Enabling replica-auto-balance tries to replicate to disabled nodes causing lots of errors in the logs and in the UI 7275 - @yangchiu @c3y1huang
• [BACKPORT][v1.5.4][BUG] allow-collecting-longhorn-usage-metrics setting is missing from chart settings 7250 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.5.4][BUG] When disabling revision counter, salvaging a faulty volume not work as expected 7732 - @james-munson @roger-ryao
• [BACKPORT][v1.5.4][BUG] During volume live engine upgrade, delete replica with old engine image will make volume degraded forever 7334 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.5.4][BUG] Uninstallation job stuck forever if the MutatingWebhookConfigurations or ValidatingWebhookConfigurations already deleted 7658 - @PhanLe1010 @roger-ryao
• [BACKPORT][v1.5.4][BUG] Volume encryption doesn't work on Amazon Linux 2 [7165](https://github.com/longhorn/longhorn/issue...

Longhorn v1.6.0

Longhorn v1.6.0 Release Notes

This latest version of Longhorn introduces several features, enhancements, and bug fixes that are intended to improve system quality and the overall user experience. Highlights include new V2 Data Engine features, platform-agnostic deployment, node maintenance, and improvements to stability, performance, and resilience.

The Longhorn team appreciates your contributions and anticipates receiving feedback regarding this release.

Note:
For more information about release-related terminology, see Releases.

Primary Highlights

New V2 Data Engine Features

Although the V2 Data Engine is still considered a preview feature in this release, the core functions have been significantly enhanced. For example, you can now seamlessly perform volume backup and restore operations between the V1 and V2 Data Engines, paving the way for volume migration between the two data engines in the future.

• Volume Snapshot and Revert
• Volume Backup and Restore
• Separate Data Plane for v1 and v2 Data Engines
• ARM64 Support

The Longhorn team will continue to develop features for the V1 Data Engine and to prepare the V2 Data Engine for use in all types of environments.

Platform-Agnostic Deployment

Longhorn is designed to seamlessly operate on general-purpose Linux distributions, and on certain container-optimized systems such as SLE Micro. In response to numerous requests, v1.6.0 was enhanced to allow installation of Longhorn components on Talos, which is a secure, immutable, and minimal Kubernetes OS. v1.6.0 also includes OKD support, which was contributed by community member @ArthurVardevanyan.

• Talos Support
• OKD (OpenShift Origin) Support

The Longhorn team is committed to making Longhorn an adaptive storage solution and anticipates receiving feedback regarding your preferred platforms.

Space Efficiency

Starting with v1.6.0, Longhorn allows you to configure the maximum snapshot count and the maximum aggregate snapshot size for all volumes and for specific volumes. Both settings, whether applied globally or individually, aid in space estimation and management. Earlier Longhorn versions do not provide mechanisms for controlling or predicting the quantity and size of volume snapshots.

• Snapshot Space Management

GitOps Friendly

Longhorn has been validated with popular GitOps solutions, including Flux, Argo CD, and Fleet. Future releases will include enhancements that further solidify Longhorn's status as a GitOps-aware storage solution.

Data Protection

Longhorn now supports block volume encryption, which is particularly beneficial in virtualization use cases such as Harvester and KubeVirt.

Node Maintenance

v1.6.0 includes two new node drain policy options: Block For Eviction and Block For Eviction If Contains Last Replica. Both options allow automatic eviction and relocation of healthy replicas from draining nodes (before the nodes are cordoned).

The Longhorn team recommends enabling these options only during planned maintenance to minimize impact on data movement. For more information about the advantages and disavantages of all options, see Node Drain Policy Recommendations in the Longhorn documentation.

Backing Image Management

Longhorn now allows you to create and restore backups of backing images, which can streamline the management of backing images across clusters. This feature is particularly beneficial in virtualization use cases such as Harvester and KubeVirt.

Installation

Ensure that your cluster is running Kubernetes v1.21 or later before installing Longhorn v1.6.0.

You can install Longhorn using a variety of tools, including Rancher, Kubectl, and Helm. For more information about installation methods and requirements, see Quick Installation in the Longhorn documentation.

Upgrade

Ensure that your cluster is running Kubernetes v1.21 or later before upgrading from Longhorn v1.5.x to v1.6.0.

Longhorn only allows upgrades from supported versions. For more information about upgrade paths and procedures, see Upgrade in the Longhorn documentation.

Deprecation & Incompatibilities

For information about important changes, including feature incompatibility, deprecation, and removal, see Important Notes in the Longhorn documentation.

Post-Release Known Issues

For information about issues identified after this release, see Release-Known-Issues.

Resolved Issues

Highlights

• [FEATURE] Longhorn snapshot space management 6563 - @FrankYang0529 @yangchiu
• [FEATURE] v2 data engine volume snapshot and revert 6137 - @shuo-wu @roger-ryao
• [FEATURE] Support eventual danger zone setting update 7173 - @mantissahz @chriscchien
• [FEATURE] Engine upgrade enforcement 5842 - @yangchiu @mantissahz @c3y1huang
• [FEATURE] Selective V2 Data Engine Activation 7015 - @derekbit @chriscchien @roger-ryao
• [FEATURE] Have default priorityClass to prevent unexpected longhorn pods eviction 6528 - @mantissahz @chriscchien
• [FEATURE] v2 volume supports volume backup/restore 6138 - @yangchiu @derekbit
• [IMPROVEMENT] Remove or Change Helm pre-upgrade hook to support ArgoCD 6415 - @mantissahz
• [FEATURE] Restore BackingImage for BackupVolume in a new cluster 4165 - @ChanYiLin @roger-ryao
• [FEATURE] Talos support 3161 - @yangchiu @c3y1huang
• [FEATURE] Support v2 volume on ARM64 platform 6021 - @derekbit @chriscchien @roger-ryao
• [IMPROVEMENT] Add a new setting that allows Longhorn to evict replicas automatically when a node is drained 2238 - @ejweber @chriscchien
• [FEATURE] Add linear dm device on the top of v2 volume 7357 - @derekbit @chriscchien
• [FEATURE] Support Encryption for VolumeMode Block 4883 - @derekbit @roger-ryao
• [TASK] Add install/upgrade longhorn by gitops (flux) pipeline 6343 - @yangchiu
• [FEATURE] OKD/Openshift support 1831 - @mantissahz @ArthurVardevanyan @roger-ryao

Features

• [UI][FEATURE] Longhorn snapshot space management 7522 - @yangchiu @scures
• [FEATURE] RWX volume supports different NFS version (4.2) and mount options 7638 - @james-munson
• [FEATURE] Introduce upgradeVersionCheck to decide version upgrade enforcement 7539 - @mantissahz @chriscchien
• [FEATURE] v2 volume replica management 5420 - @DamiaSan
• [FEATURE] Update nfs-genesha to 5.x for share manager 6000 - @james-munson @chriscchien
• [FEATURE] Allow to set mount options for storageclass via values.yaml in helm chart 7351 - @ChanYiLin @chriscchien
• [FEATURE] Flush on-the-fly IOs in the queue before snapshotting 5648 - @DamiaSan
• [FEATURE] Update base image of Longhorn components to BCI 15.5 6206 - @nitendra-suse
• [FEATURE] Customize MaxRecurringJobRetain 5713 - @mantissahz @chriscchien
• [FEATURE] Replica rebuild over SPDK 5216 - @shuo-wu @DamiaSan
• [FEATURE] Allow kubectl drain to stop manually attached volumes 6978 - @ChanYiLin @chriscchien
• [FEATURE] Single Node Disk affinity 3823 - @ejweber @roger-ryao
• [FEATURE] Storage network support for Multus v4.0 thick-plugin 5048 - @c3y1huang @chriscchien
• [FEATURE] Add disk status prometheus metrics 6858 - @c3y1huang @chriscchien
• [FEATURE] Add a brand new/empty bdev with WriteOnly mode to the RAID1 bdev 5865 - @DamiaSan
• [FEATURE] Add ...

Longhorn v1.5.4-rc1

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Highlight

• [BACKPORT][v1.5.4][FEATURE] Add a new settings that allows Longhorn to evict replicas automatically when a node is drained 7421 - @yangchiu @ejweber

Feature

• [BACKPORT][v1.5.4]Allow to set mount options for storageclass via values.yaml in helm chart 7593 - @ChanYiLin
• [BACKPORT][v1.5.4][FEATURE] Update base image of Longhorn components to BCI 15.5 7134 - @nitendra-suse

Improvement

• [BACKPORT][v1.5.4][IMPROVEMENT] Remove unused process manager connection in longhorn-manager 7785 - @derekbit @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Only restarts pods with volumes in the unexpected Read-Only state 7729 - @yangchiu @ChanYiLin
• [BACKPORT][v1.5.4][IMPROVEMENT] Upgrade CSI components to the latest patch release 7492 - @c3y1huang @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Reject the last replica deletion if its volume.spec.deletionTimestamp is not set 7432 - @yangchiu @derekbit
• [BACKPORT][v1.5.4][IMPROVEMENT] Automatically remount read-only RWO volume to read-write 7500 - @ChanYiLin @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Improve the profiler of longhorn-engine for runtime profiling 7545 - @Vicente-Cheng @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Improve handling of 16TiB+ volumes with ext4 as the underlying file system 7429 - @mantissahz @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Upgrade support bundle kit version to v0.0.33 7279 - @c3y1huang
• [BACKPORT][v1.5.4][IMPROVEMENT] Review and simplify longhorn component image build 7162 - @ChanYiLin
• [BACKPORT][v1.5.4][IMPROVEMENT] Volumes: metrics for snapshots include (size and type: system vs user) 7725 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] BackingImage should be compressed when downloading and use the name as filename instead of UUID 7397 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] deploy: driver deployer shouldn't cleanup previous deployment if Kubernetes version changes 7345 - @PhanLe1010 @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Don't crash the migration engine when kubelet restarts 7328 - @yangchiu @ejweber
• [BACKPORT][v1.5.4][IMPROVEMENT] Replace deprecated grpc.WithInsecure 7364 - @c3y1huang
• [BACKPORT][v1.5.4][IMPROVEMENT] Have a setting to disable snapshot purge for maintenance purpose 7265 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][IMPROVEMENT] Bypass upgrade when installing a fresh setup 7283 - @mantissahz @roger-ryao
• [BACKPORT][v1.5.4][IMPROVEMENT] Clean up backup target in IM-R pod if the backup target setting is unset 7145 - @ChanYiLin @chriscchien

Bug

• [BACKPORT][v1.5.4][BUG][v1.6.0-rc1] Negative test case failed: Stop Volume Node Kubelet For More Than Pod Eviction Timeout While Workload Heavy Writing 7761 - @c3y1huang
• [BACKPORT][v1.5.4][BUG] Relax S3 client retry intervals, for throttled requests 7098 - @mantissahz
• [BUG] longhorn manager pod fails to start in container-based K3s 5693 - @ChanYiLin @andrewd-zededa
• [BACKPORT][v1.5.4][BUG] Backing Image Data Inconsistency if it's Exported from a Backing Image Backed Volume 7701 - @yangchiu @ChanYiLin
• [BACKPORT][v1.5.4][BUG] Volumes don't mount with mTLS enabled 7789 - @derekbit
• [BACKPORT][v1.5.4][BUG] When disabling revision counter, salvaging a faulty volume not work as expected 7732 - @james-munson @roger-ryao
• [BACKPORT][v1.5.4][BUG] During volume live engine upgrade, delete replica with old engine image will make volume degraded forever 7334 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.5.4][BUG] Uninstallation job stuck forever if the MutatingWebhookConfigurations or ValidatingWebhookConfigurations already deleted 7658 - @PhanLe1010 @roger-ryao
• [BACKPORT][v1.5.4][BUG] Longhorn may keep corrupted salvaged replicas and discard good ones 7801 - @ejweber
• [BACKPORT][v1.5.4][BUG] Volume encryption doesn't work on Amazon Linux 2 7165 - @derekbit @chriscchien
• [BACKPORT][v1.5.4][BUG] Rancher cannot import longhorn 1.5 charts due to "error converting YAML to JSON: yaml: line 699: did not find expected key" 7776 - @mantissahz @PhanLe1010
• [BACKPORT][v1.5.4][BUG] allow-collecting-longhorn-usage-metrics setting is missing from chart settings 7250 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.5.4][BUG] Deadlock for RWX volume if an error occurs in its share-manager pod 7186 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][BUG] Delete kubernetes node did not remove node.longhorn.io 7538 - @ejweber @chriscchien
• [BACKPORT][v1.5.4][BUG] Volume could not be remounted after engine process killed 7772 - @ChanYiLin @shuo-wu @roger-ryao
• [BACKPORT][v1.5.4][BUG] Enabling replica-auto-balance tries to replicate to disabled nodes causing lots of errors in the logs and in the UI 7275 - @yangchiu @c3y1huang
• [BUG] The wrong template in default-setting.yaml of the Longhorn chart in v1.5 and v1.4 7459 - @mantissahz @roger-ryao
• [BACKPORT][v1.5.4][BUG] Unable to list backups when backuptarget resource is picked up by a cordoned node 7621 - @c3y1huang
• [BACKPORT][v1.5.4][BUG][v1.6.0-rc1] Some Longhorn resources remaining after longhorn-uninstall job completed 7663 - @yangchiu @PhanLe1010
• [BACKPORT][v1.5.4][BUG] Backup volume attachment tickets might not be cleaned up after completion. 7604 - @james-munson @chriscchien
• [BACKPORT][v1.5.4][BUG] backingimage download server error 7381 - @scures @roger-ryao
• [BACKPORT][v1.5.4][BUG] Longhorn-manager does not deploy CSI driver when integrated with linkerd service mesh 7391 - @yangchiu @mantissahz
• [BACKPORT][v1.5.4][BUG] Failed to check_volume_data after volume engine upgrade/migration 7402 - @PhanLe1010 @chriscchien
• [BACKPORT][v1.5.4][BUG] Metric totalVolumeSize and totalVolumeActualSize incorrect due to v2 volume counts 7392 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][BUG] CSI components CrashLoopBackOff, failed to connect to unix://csi/csi.sock after cluster restart 7426 - @ejweber @roger-ryao
• [BACKPORT][v1.5.4][BUG] Helm2 install error: 'lookup' function not defined in validate-psp-install.yaml 7435 - @roger-ryao
• [BACKPORT][v1.5.4][BUG] Warning events are being spammed by Longhorn - CRD 7309 - @m-ildefons @roger-ryao
• [BACKPORT][v1.5.4][BUG] Persistent volume is not ready for workloads 7314 - @james-munson @roger-ryao
• [BACKPORT][v1.5.4][BUG] Download backing image failed with HTTP 502 error if Storage Network configured 7239 - @ChanYiLin @roger-ryao
• [BACKPORT][v1.5.4][BUG] supportbundle/kubelet.log empty in k3s environment 7123 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][BUG] Errors found by static checker in volume controller 7269 - @m-ildefons
• [BACKPORT][v1.5.4][BUG] Continuously auto-balancing replicas when zone does not have enough space 7306 - @c3y1huang @chriscchien
• [BACKPORT][v1.5.4][BUG] Volume conditions are not represented in the UI for v1.4.x and newer 7242 - @m-ildefons @roger-ryao
• [BACKPORT][v1.5.4][BUG] Confusing logging when trying to attach a new volume with no scheduled replicas 7245 - @ejweber @roger-ryao
• [BACKPORT][v1.5.4][BUG] Environment check script claims...

Longhorn v1.6.0-rc3

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved Issues in this release

Highlight

• [FEATURE] Longhorn snapshot space management 6563 - @FrankYang0529 @yangchiu
• [FEATURE] v2 data engine volume snapshot and revert 6137 - @shuo-wu @roger-ryao
• [FEATURE] Support eventual danger zone setting update 7173 - @mantissahz @chriscchien
• [FEATURE] Engine upgrade enforcement 5842 - @yangchiu @mantissahz @c3y1huang
• [FEATURE] Selective V2 Data Engine Activation 7015 - @derekbit @chriscchien @roger-ryao
• [FEATURE] Have default priorityClass to prevent unexpected longhorn pods eviction 6528 - @mantissahz @chriscchien
• [FEATURE] v2 volume supports volume backup/restore 6138 - @yangchiu @derekbit
• [IMPROVEMENT] Remove or Change Helm pre-upgrade hook to support ArgoCD 6415 - @mantissahz
• [FEATURE] Restore BackingImage for BackupVolume in a new cluster 4165 - @ChanYiLin @roger-ryao
• [FEATURE] Talos support 3161 - @yangchiu @c3y1huang
• [FEATURE] Support v2 volume on ARM64 platform 6021 - @derekbit @chriscchien @roger-ryao
• [IMPROVEMENT] Add a new settings that allows Longhorn to evict replicas automatically when a node is drained 2238 - @ejweber @chriscchien
• [FEATURE] Add linear dm device on the top of v2 volume 7357 - @derekbit @chriscchien
• [FEATURE] Support Encryption for VolumeMode Block 4883 - @derekbit @roger-ryao
• [TASK] Bump the versions of dependent libs or components 7001 - @c3y1huang @chriscchien
• [TASK] Add install/upgrade longhorn by gitops (flux) pipeline 6343 - @yangchiu
• [FEATURE] OKD/Openshift support 1831 - @mantissahz @ArthurVardevanyan @roger-ryao

Feature

• [UI][FEATURE] Longhorn snapshot space management 7522 - @yangchiu @scures
• [FEATURE] RWX volume supports different NFS version (4.2) and mount options 7638 - @james-munson
• [FEATURE] Introduce upgradeVersionCheck to decide version upgrade enforcement 7539 - @mantissahz @chriscchien
• [FEATURE] v2 volume replica management 5420 - @DamiaSan
• [FEATURE] Update nfs-genesha to 5.x for share manager 6000 - @james-munson @chriscchien
• [FEATURE] Allow to set mount options for storageclass via values.yaml in helm chart 7351 - @ChanYiLin @chriscchien
• [FEATURE] Flush on-the-fly IOs in the queue before snapshotting 5648 - @DamiaSan
• [FEATURE] Update base image of Longhorn components to BCI 15.5 6206 - @nitendra-suse
• [FEATURE] Customize MaxRecurringJobRetain 5713 - @mantissahz @chriscchien
• [FEATURE] Replica rebuild over SPDK 5216 - @shuo-wu @DamiaSan
• [FEATURE] Allow kubectl drain to stop manually attached volumes 6978 - @ChanYiLin @chriscchien
• [FEATURE] Single Node Disk affinity 3823 - @ejweber @roger-ryao
• [FEATURE] Storage network support for Multus v4.0 thick-plugin 5048 - @c3y1huang @chriscchien
• [FEATURE] Add disk status prometheus metrics 6858 - @c3y1huang @chriscchien
• [FEATURE] Add a brand new/empty bdev with WriteOnly mode to the RAID1 bdev 5865 - @DamiaSan
• [FEATURE] Add a script to identify the valid volumes to recover given s3 backup url and secret 1523 - @weizhe0422
• [FEATURE] Pause IO when raid1 bdev snapshotting 5421 - @DamiaSan
• [FEATURE] Change the replica selector behavior so that an absent selector is able to select nodes without a TAG 4826 - @ChanYiLin @roger-ryao
• [FEATURE] Helm Chart make loglevel configurable 3655 - @mantissahz

Improvement

• [IMPROVEMENT] Use ensureFolderPath rather than ensureMount for checking folder path in CSI plugin 7784 - @derekbit @roger-ryao
• [IMPROVEMENT] Remove unused process manager connection in longhorn-manager 7783 - @derekbit @chriscchien
• [IMPROVEMENT] Revert RWX volume back to NFS v4.1 7741 - @yangchiu @james-munson
• [IMPROVEMENT] Remove static sessionAffinity: ClientIP set in most services if not required 7399 - @yangchiu @ejweber
• [IMPROVEMENT] Automatically remount read-only RWO volume to read-write 6386 - @ChanYiLin @chriscchien
• [IMPROVEMENT] Only restarts pods with volumes in the unexpected Read-Only state 7728 - @ChanYiLin @chriscchien
• [IMPROVEMENT] Volumes: metrics for snapshots include (size and type: system vs user) 5869 - @c3y1huang @chriscchien
• [IMPROVEMENT] Have a clear message when reverting the parent of a volume-head snapshot for a v2 volume 7630 - @derekbit @chriscchien
• [IMPROVEMENT] Flooding error messages failed to sync setting for..... 7654 - @mantissahz @chriscchien
• [IMPROVEMENT] Enhance the code quality in the instance-manager instance and disk gRPC server methods. 7628 - @derekbit
• [IMPROVEMENT] Increase the hugepage size for spdk_tgt to 2GiB 7606 - @derekbit @chriscchien
• [IMPROVEMENT] Reject DR volume creation for v2 volume 7627 - @derekbit @roger-ryao
• [IMPROVEMENT] Do not use --force for dmsetup remove command 7615 -
• [IMPROVEMENT] Update nvme-cli to v2.7.1 in instance-manager pod 7609 - @derekbit
• [IMPROVEMENT] Prevent from complains in spdk_tgt when deleting a v2 volume 7568 - @yangchiu @derekbit @roger-ryao
• [IMPROVEMENT] Expose actual size of a logical volume 5947 - @derekbit @shuo-wu @chriscchien @DamiaSan
• [IMPROVEMENT] UI backup restoration supports v1 and v2 Data Engine 6597 - @derekbit @scures @roger-ryao
• [IMPROVEMENT][UI] Display v2 volume actual size 7524 - @derekbit @chriscchien
• [IMPROVEMENT] Recreate instance manager pod for v2 volume when spdk_tgt is dead 7551 - @derekbit @chriscchien
• [IMPROVEMENT] Add reserve storage percentage of nodes setting in helm chart 5958 - @mantissahz @roger-ryao
• [IMPROVEMENT] Reconcile engine/replica instance state of v2 volume like v1 volume 7326 - @derekbit @chriscchien
• [IMPROVEMENT] Improve handling of 16TiB+ volumes with ext4 as the underlying file system 7423 - @mantissahz @chriscchien
• [IMPROVEMENT] Rename backendStoreDriver to dataEngin in instance-manager and associated components 7480 - @yangchiu @derekbit
• [IMPROVEMENT][UI] Validate volume creation according to the enabled data engines 7505 - @derekbit @chriscchien
• [IMPROVEMENT] Add guaranteed instanceManager CPU setting for v2 volume 7361 - @derekbit @roger-ryao
• [IMPROVEMENT] Support backup list if there is only v2-data-engine enabled 7486 - @derekbit @chriscchien
• [IMPROVEMENT] Upgrade CSI components to the latest patch release 7384 - @c3y1huang @roger-ryao
• [IMPROVEMENT] Add global setting for enable v1 or v2 volume support 7095 - @yangchiu @derekbit
• [IMPROVEMENT] Blindly stop raid bdev exposure before exposing it for V2 volume 7324 - @yangchiu @derekbit @roger-ryao
• [IMPROVEMENT] instance-managers for v1 and v2 volumes resp...

Longhorn v1.6.0-rc2

DON'T UPGRADE from/to any RC/Preview/Sprint releases because the operation is not supported.

Resolved issues in this release

Highlight

• [FEATURE] Support eventual danger zone setting update 7173 - @mantissahz @chriscchien
• [FEATURE] Longhorn snapshot space management 6563 - @FrankYang0529 @yangchiu
• [FEATURE] Engine upgrade enforcement 5842 - @yangchiu @mantissahz @c3y1huang
• [FEATURE] Selective V2 Data Engine Activation 7015 - @derekbit @chriscchien @roger-ryao
• [FEATURE] Have default priorityClass to prevent unexpected longhorn pods eviction 6528 - @mantissahz @chriscchien
• [FEATURE] v2 volume supports volume backup/restore 6138 - @yangchiu @derekbit
• [IMPROVEMENT] Remove or Change Helm pre-upgrade hook to support ArgoCD 6415 - @mantissahz
• [FEATURE] Restore BackingImage for BackupVolume in a new cluster 4165 - @ChanYiLin @roger-ryao
• [FEATURE] Talos support 3161 - @yangchiu @c3y1huang
• [FEATURE] Support v2 volume on ARM64 platform 6021 - @derekbit @chriscchien @roger-ryao
• [IMPROVEMENT] Add a new settings that allows Longhorn to evict replicas automatically when a node is drained 2238 - @ejweber @chriscchien
• [FEATURE] Add linear dm device on the top of v2 volume 7357 - @derekbit @chriscchien
• [FEATURE] Support Encryption for VolumeMode Block 4883 - @derekbit @roger-ryao
• [TASK] Bump the versions of dependent libs or components 7001 - @c3y1huang @chriscchien
• [TASK] Add install/upgrade longhorn by gitops (flux) pipeline 6343 - @yangchiu
• [FEATURE] OKD/Openshift support 1831 - @mantissahz @ArthurVardevanyan @roger-ryao

Feature

• [FEATURE] RWX volume supports different NFS version (4.2) and mount options 7638 - @james-munson
• [UI][FEATURE] Longhorn snapshot space management 7522 - @scures
• [FEATURE] Update nfs-genesha to 5.x for share manager 6000 - @james-munson @chriscchien
• [FEATURE] v2 data engine volume snapshot and revert 6137 - @shuo-wu @roger-ryao
• [FEATURE] Allow to set mount options for storageclass via values.yaml in helm chart 7351 - @ChanYiLin @chriscchien
• [FEATURE] Flush on-the-fly IOs in the queue before snapshotting 5648 - @DamiaSan
• [FEATURE] Update base image of Longhorn components to BCI 15.5 6206 - @nitendra-suse
• [FEATURE] Customize MaxRecurringJobRetain 5713 - @mantissahz @chriscchien
• [FEATURE] Replica rebuild over SPDK 5216 - @shuo-wu @DamiaSan
• [FEATURE] Allow kubectl drain to stop manually attached volumes 6978 - @ChanYiLin @chriscchien
• [FEATURE] Single Node Disk affinity 3823 - @ejweber @roger-ryao
• [FEATURE] Storage network support for Multus v4.0 thick-plugin 5048 - @c3y1huang @chriscchien
• [FEATURE] Add disk status prometheus metrics 6858 - @c3y1huang @chriscchien
• [FEATURE] Add a brand new/empty bdev with WriteOnly mode to the RAID1 bdev 5865 - @DamiaSan
• [FEATURE] Manage replicas of RAID1 bdev 5420 - @DamiaSan
• [FEATURE] Add a script to identify the valid volumes to recover given s3 backup url and secret 1523 - @weizhe0422
• [FEATURE] Pause IO when raid1 bdev snapshotting 5421 - @DamiaSan
• [FEATURE] Change the replica selector behavior so that an absent selector is able to select nodes without a TAG 4826 - @ChanYiLin @roger-ryao
• [FEATURE] Helm Chart make loglevel configurable 3655 - @mantissahz

Improvement

• [IMPROVEMENT] Only restarts pods with volumes in the unexpected Read-Only state 7728 - @ChanYiLin
• [TASK] Introduce upgradeVersionCheck to decide version upgrade enforcement 7539 - @mantissahz @chriscchien
• [IMPROVEMENT] Volumes: metrics for snapshots include (size and type: system vs user) 5869 - @c3y1huang @chriscchien
• [IMPROVEMENT] v2 volume snapshot supports UserCreated flag 7578 - @DamiaSan
• [IMPROVEMENT] Have a clear message when reverting the parent of a volume-head snapshot for a v2 volume 7630 - @derekbit @chriscchien
• [IMPROVEMENT] Flooding error messages failed to sync setting for..... 7654 - @mantissahz @chriscchien
• [IMPROVEMENT] Enhance the code quality in the instance-manager instance and disk gRPC server methods. 7628 - @derekbit
• [IMPROVEMENT] Increase the hugepage size for spdk_tgt to 2GiB 7606 - @derekbit @chriscchien
• [IMPROVEMENT] Reject DR volume creation for v2 volume 7627 - @derekbit @roger-ryao
• [IMPROVEMENT] Do not use --force for dmsetup remove command 7615 -
• [IMPROVEMENT] Update nvme-cli to v2.7.1 in instance-manager pod 7609 - @derekbit
• [IMPROVEMENT] Prevent from complains in spdk_tgt when deleting a v2 volume 7568 - @yangchiu @derekbit @roger-ryao
• [IMPROVEMENT] Expose actual size of a logical volume 5947 - @derekbit @shuo-wu @chriscchien @DamiaSan
• [IMPROVEMENT] UI backup restoration supports v1 and v2 Data Engine 6597 - @derekbit @scures @roger-ryao
• [IMPROVEMENT][UI] Display v2 volume actual size 7524 - @derekbit @chriscchien
• [IMPROVEMENT] Recreate instance manager pod for v2 volume when spdk_tgt is dead 7551 - @derekbit @chriscchien
• [IMPROVEMENT] Add reserve storage percentage of nodes setting in helm chart 5958 - @mantissahz @roger-ryao
• [IMPROVEMENT] Reconcile engine/replica instance state of v2 volume like v1 volume 7326 - @derekbit @chriscchien
• [IMPROVEMENT] Improve handling of 16TiB+ volumes with ext4 as the underlying file system 7423 - @mantissahz @chriscchien
• [IMPROVEMENT] Rename backendStoreDriver to dataEngin in instance-manager and associated components 7480 - @yangchiu @derekbit
• [IMPROVEMENT][UI] Validate volume creation according to the enabled data engines 7505 - @derekbit @chriscchien
• [IMPROVEMENT] Add guaranteed instanceManager CPU setting for v2 volume 7361 - @derekbit @roger-ryao
• [IMPROVEMENT] Automatically remount read-only RWO volume to read-write 6386 - @ChanYiLin @chriscchien
• [IMPROVEMENT] Support backup list if there is only v2-data-engine enabled 7486 - @derekbit @chriscchien
• [IMPROVEMENT] Upgrade CSI components to the latest patch release 7384 - @c3y1huang @roger-ryao
• [IMPROVEMENT] Add global setting for enable v1 or v2 volume support 7095 - @yangchiu @derekbit
• [IMPROVEMENT] Blindly stop raid bdev exposure before exposing it for V2 volume 7324 - @yangchiu @derekbit @roger-ryao
• [IMPROVEMENT] instance-managers for v1 and v2 volumes respectively 6984 - @yangchiu @derekbit
• [IMPROVEMENT] BackingImage should be compressed when downloading and use the name as filename instead of UUID 7295 - @ChanYiLin @chriscchien
• [IMPROVEMENT] Reject the creation of encrypted v2 volume in validating webhook 7404 - @derekbit @chriscchien
• [IMPROVEMENT] Longhorn-engine processes should refuse to serve requests not intended for...

Longhorn v1.6.0-rc1

DON'T UPGRADE to/from any RC/Preview/Sprint releases because the operation is not supported!

Changes in this release

Highlight

• [FEATURE] Support eventual danger zone setting update 7173 - @mantissahz
• [FEATURE] Engine upgrade enforcement 5842 - @yangchiu @c3y1huang
• [FEATURE] Support instance-manager pod for v2 volumes on selected nodes 7015 - @derekbit @chriscchien @roger-ryao
• [FEATURE] Longhorn snapshot space management 6563 - @FrankYang0529 @yangchiu
• [FEATURE] v2 volume supports volume backup/restore 6138 - @yangchiu @derekbit
• [IMPROVEMENT] Remove or Change Helm pre-upgrade hook to support ArgoCD 6415 - @mantissahz
• [FEATURE] Restore BackingImage for BackupVolume in a new cluster 4165 - @ChanYiLin @roger-ryao
• [FEATURE] Talos support 3161 - @yangchiu @c3y1huang
• [FEATURE] Support v2 volume on ARM64 platform 6021 - @derekbit @chriscchien @roger-ryao
• [FEATURE] Have default priorityClass to prevent unexpected longhorn pods eviction 6528 - @mantissahz
• [IMPROVEMENT] Add a new settings that allows Longhorn to evict replicas automatically when a node is drained 2238 - @ejweber @chriscchien
• [FEATURE] Add linear dm device on the top of v2 volume 7357 - @derekbit @chriscchien
• [TASK] Bump up the minimum supported Kubernetes version 7224 - @c3y1huang @roger-ryao
• [FEATURE] Support Encryption for VolumeMode Block 4883 - @derekbit @roger-ryao
• [TASK] Bump the versions of dependent libs or components 7001 - @c3y1huang @chriscchien
• [TASK] Add install/upgrade longhorn by gitops (flux) pipeline 6343 - @yangchiu
• [FEATURE] OKD/Openshift support 1831 - @mantissahz @ArthurVardevanyan @roger-ryao

Feature

• [FEATURE] v2 data engine volume snapshot and revert 6137 - @shuo-wu @roger-ryao
• [FEATURE] Allow to set mount options for storageclass via values.yaml in helm chart 7351 - @ChanYiLin @chriscchien
• [UI][FEATURE] Longhorn snapshot space management 7522 - @scures
• [FEATURE] Flush on-the-fly IOs in the queue before snapshotting 5648 - @DamiaSan
• [FEATURE] Update base image of Longhorn components to BCI 15.5 6206 - @nitendra-suse
• [FEATURE] Customize MaxRecurringJobRetain 5713 - @mantissahz @chriscchien
• [FEATURE] Replica rebuild over SPDK 5216 - @shuo-wu @DamiaSan
• [FEATURE] Allow kubectl drain to stop manually attached volumes 6978 - @ChanYiLin @chriscchien
• [FEATURE] Single Node Disk affinity 3823 - @ejweber @roger-ryao
• [FEATURE] Storage network support for Multus v4.0 thick-plugin 5048 - @c3y1huang @chriscchien
• [FEATURE] Add disk status prometheus metrics 6858 - @c3y1huang @chriscchien
• [FEATURE] Add a brand new/empty bdev with WriteOnly mode to the RAID1 bdev 5865 - @DamiaSan
• [FEATURE] Manage replicas of RAID1 bdev 5420 - @DamiaSan
• [BUG] when volume degraded and rebuild, the filesystem in every mounted pod were read-only 1500 - @PhanLe1010
• [FEATURE] Add a script to identify the valid volumes to recover given s3 backup url and secret 1523 - @weizhe0422
• [FEATURE] Manage snapshot number or quota for storage capacity planning 6077 -
• [FEATURE] Pause IO when raid1 bdev snapshotting 5421 - @DamiaSan
• [FEATURE] Change the replica selector behavior so that an absent selector is able to select nodes without a TAG 4826 - @ChanYiLin @roger-ryao
• [FEATURE] Introduce write only mode in RAID1 5783 - @DamiaSan
• [FEATURE] Helm Chart make loglevel configurable 3655 - @mantissahz

Improvement

• [IMPROVEMENT] Have a clear message when reverting the parent of a volume-head snapshot for a v2 volume 7630 - @derekbit @shuo-wu
• [IMPROVEMENT] Enhance the code quality in the instance-manager instance and disk gRPC server methods. 7628 - @derekbit
• [TASK] Introduce upgradeVersionCheck to decide version upgrade enforcement 7539 - @mantissahz @chriscchien
• [IMPROVEMENT] Increase the hugepage size for spdk_tgt to 2GiB 7606 - @derekbit @chriscchien
• [IMPROVEMENT] Reject DR volume creation for v2 volume 7627 - @derekbit @roger-ryao
• [IMPROVEMENT] Do not use --force for dmsetup remove command 7615 -
• [IMPROVEMENT] Update nvme-cli to v2.7.1 in instance-manager pod 7609 - @derekbit
• [IMPROVEMENT] v2 volume snapshot supports UserCreated flag 7578 - @DamiaSan
• [IMPROVEMENT] Prevent from complains in spdk_tgt when deleting a v2 volume 7568 - @yangchiu @derekbit @roger-ryao
• [IMPROVEMENT] Expose actual size of a logical volume 5947 - @derekbit @shuo-wu @chriscchien @DamiaSan
• [IMPROVEMENT] UI backup restoration supports v1 and v2 Data Engine 6597 - @derekbit @scures @roger-ryao
• [IMPROVEMENT][UI] Display v2 volume actual size 7524 - @derekbit @chriscchien
• [IMPROVEMENT] Recreate instance manager pod for v2 volume when spdk_tgt is dead 7551 - @derekbit @chriscchien
• [IMPROVEMENT] Add reserve storage percentage of nodes setting in helm chart 5958 - @mantissahz @roger-ryao
• [IMPROVEMENT] Reconcile engine/replica instance state of v2 volume like v1 volume 7326 - @derekbit @chriscchien
• [IMPROVEMENT] Improve handling of 16TiB+ volumes with ext4 as the underlying file system 7423 - @mantissahz @chriscchien
• [IMPROVEMENT] Rename backendStoreDriver to dataEngin in instance-manager and associated components 7480 - @yangchiu @derekbit
• [IMPROVEMENT][UI] Validate volume creation according to the enabled data engines 7505 - @derekbit @chriscchien
• [IMPROVEMENT] Add guaranteed instanceManager CPU setting for v2 volume 7361 - @derekbit @roger-ryao
• [IMPROVEMENT] Automatically remount read-only RWO volume to read-write 6386 - @ChanYiLin @chriscchien
• [IMPROVEMENT] Support backup list if there is only v2-data-engine enabled 7486 - @derekbit @chriscchien
• [IMPROVEMENT] Upgrade CSI components to the latest patch release 7384 - @c3y1huang @roger-ryao
• [IMPROVEMENT] Add global setting for enable v1 or v2 volume support 7095 - @yangchiu @derekbit
• [IMPROVEMENT] Blindly stop raid bdev exposure before exposing it for V2 volume 7324 - @yangchiu @derekbit @roger-ryao
• [IMPROVEMENT] instance-managers for v1 and v2 volumes respectively 6984 - @yangchiu @derekbit
• [IMPROVEMENT] Reject volume.spec.image update for v2 volumes before introducing live upgrade 7446 - @derekbit @chriscchien
• [IMPROVEMENT] BackingImage should be compressed when downloading and use the name as filename instead of UUID 7295 - @ChanYiLin @chriscchien
• [IMPROVEMENT] Reject the creation of encrypted v2 volume in validating webhook 7404 - @derekbit @chriscchien
• [IMPROVEMENT] Longhorn-engine processes should refuse to serve requests not intended for them [5845](#5845...

Longhorn v1.6.0-dev-20240105

This is a sprint release for internal testing and development purposes only. DO NOT INSTALL or UPGRADE to any sprint releases, as this operation is not supported.

Issues in the sprint

Misc

• [BUG] Randomly failed to create volume with backing image 7543 - @yangchiu @ChanYiLin
• [UI][FEATURE] Longhorn snapshot space management 7522 - @scures
• [FEATURE] Restore BackingImage for BackupVolume in a new cluster 4165 - @ChanYiLin @roger-ryao
• [IMPROVEMENT] Recreate instance manager pod for v2 volume when spdk_tgt is dead 7551 - @derekbit @chriscchien
• [FEATURE] Longhorn snapshot space management 6563 - @FrankYang0529 @yangchiu
• [BUG] After upgrade to master-head, existing volume won't rebuild replica if one deleted, and the volume keeps healthy instead of degraded 7555 - @FrankYang0529 @derekbit
• [BUG] volume engine failed to live upgrade 7548 - @derekbit @chriscchien
• [BUG] Failed to check_volume_data after volume engine upgrade/migration 7396 - @PhanLe1010 @james-munson @roger-ryao
• [BUG] test case test_engine_image_not_fully_deployed_perform_auto_upgrade_engine failed 7540 - @c3y1huang
• [IMPROVEMENT] Expose actual size of a logical volume 5947 - @derekbit @shuo-wu @DamiaSan
• [BUG] Warning events are being spammed by Longhorn - CRD 7290 - @m-ildefons @roger-ryao
• [TASK] Introduce upgradeVersionCheck to decide version upgrade enforcement 7539 - @mantissahz
• [BUG] Delete kubernetes node did not remove node.longhorn.io 7475 - @ejweber @chriscchien
• [IMPROVEMENT] Add reserve storage percentage of nodes setting in helm chart 5958 - @mantissahz @roger-ryao
• [BUG] Failed RWX mount due to connection timeout still happening 7301 - @james-munson
• [BUG] test case test_metric_longhorn_snapshot_actual_size_bytes failed 7532 - @yangchiu @c3y1huang
• [BUG] Test case test_volume_reattach_after_engine_sigkill failed 6751 - @c3y1huang
• [IMPROVEMENT] Improve the profiler of longhorn-engine for runtime profiling 6282 - @Vicente-Cheng
• [IMPROVEMENT] Reconcile engine/replica instance state of v2 volume like v1 volume 7326 - @derekbit @chriscchien
• [TASK] Security vulnerabilities in docker images 7523 - @c3y1huang
• [FEATURE] Support instance-manager pod for v2 volumes on selected nodes 7015 - @derekbit @chriscchien @roger-ryao
• [BUG] V2 volume is attached to a node first, the V1 volume will fails to attach. 7511 - @c3y1huang @roger-ryao
• [FEATURE] v2 volume supports volume backup/restore 6138 - @yangchiu @derekbit
• [TASK][UI] v2 volume does not support engine image upgrade 7445 - @chriscchien @scures @roger-ryao
• [BUG] v2 volume always displays engine upgrade available on UI 7489 - @scures
• [BUG] Create volume(v1) faulted 7536 - @FrankYang0529 @chriscchien
• [BUG] Backing Image Data Inconsistency if it's Exported from a Backing Image Backed Volume 6899 - @ChanYiLin
• [BUG] Creating volume randomly failed: failed to find a node that is ready and has the default engine image 7413 - @yangchiu @PhanLe1010
• [DOC] Add missing descriptions for Helm 7485 - @mantissahz
• [IMPROVEMENT][UI] Display v2 volume actual size 7524 - @derekbit
• [IMPROVEMENT] Improve handling of 16TiB+ volumes with ext4 as the underlying file system 7423 - @mantissahz @chriscchien
• [FEATURE] Talos support 3161 - @yangchiu @c3y1huang
• [IMPROVEMENT] Rename backendStoreDriver to dataEngin in instance-manager and associated components 7480 - @yangchiu @derekbit
• [FEATURE] Flush on-the-fly IOs in the queue before snapshotting 5648 - @DamiaSan
• [IMPROVEMENT][UI] Validate volume creation according to the enabled data engines 7505 - @derekbit @chriscchien
• [FEATURE] Support v2 volume on ARM64 platform 6021 - @derekbit @chriscchien @roger-ryao
• [FEATURE] Have default priorityClass to prevent unexpected longhorn pods eviction 6528 - @mantissahz
• [CI] Add .drone.yaml in go-spdk-helper 7509 - @derekbit
• [IMPROVEMENT] Add guaranteed instanceManager CPU setting for v2 volume 7361 - @derekbit @roger-ryao
• [TASK] Update protoc to v24.3 6666 - @FrankYang0529
• [IMPROVEMENT] UI backup restoration supports v1 and v2 Data Engine 6597 - @derekbit @scures
• [BUG] Running new e2e test pipeline always failed with error not ready 7484 - @PhanLe1010 @chriscchien
• [BUG] Persistent volume is not ready for workloads 6776 - @james-munson @roger-ryao
• [IMPROVEMENT] Automatically remount read-only RWO volume to read-write 6386 - @ChanYiLin @chriscchien
• [IMPROVEMENT] Support backup list if there is only v2-data-engine enabled 7486 - @derekbit @chriscchien
• [IMPROVEMENT] Upgrade CSI components to the latest patch release 7384 - @c3y1huang @roger-ryao
• [IMPROVEMENT] Add global setting for enable v1 or v2 volume support 7095 - @yangchiu @derekbit
• [BUG] Unable to create snapshot: cannot get engine client because it isn't deployed 7438 - @yangchiu @PhanLe1010
• [IMPROVEMENT] Blindly stop raid bdev exposure before exposing it for V2 volume 7324 - @yangchiu @derekbit @roger-ryao
• [IMPROVEMENT] instance-managers for v1 and v2 volumes respectively 6984 - @yangchiu @derekbit
• [FEATURE] Enable resource profiling for IM 6377 - @derekbit @roger-ryao
• [BUG] Deadlock for RWX volume if an error occurs in its share-manager pod 7183 - @derekbit @chriscchien
• [TASK] Synchronize version of CSI components in longhorn/longhorn and longhorn/longhorn-manager 7377 - @c3y1huang @roger-ryao
• [CI] longhorn-engine CI can't publish-image after longhorn-engine#954 7476 - @FrankYang0529
• [BUG] Volume conditions are not represented in the UI for v1.4.x and newer 7241 - @m-ildefons @chriscchien
• [BUG] backingimage download server error 7288 - @scures @roger-ryao
• [FEATURE] Engine upgrade enforcement 5842 - @yangchiu @c3y1huang
• [IMPROVEMENT] Add a new settings that allows Longhorn to evict replicas automatically when a node is drained 2238 - @ejweber @chriscchien
• [TASK] Upgrade csi-snapshotter to mitigate rapid retry bug 6506 - @ejweber
• [BUG] CSI components CrashLoopBackOff, failed to connect to unix://csi/csi.sock after cluster restart 7116 - @yangchiu @ejweber
• [IMPROVEMENT] Reject volume.spec.image update for v2 volumes before introducing live upgrade 7446 - @derekbit @chriscchien
• [BUG] Kubelet cannot finish terminating a pod that uses a PVC with volumeMode: Block when restarting the node 6919 - @PhanLe1010 @chriscchien
• [BUG] Test case test_node_default_disk_labeled failed 7385 - @derekbit @roger-ryao
• [BUG] Helm2 install error: 'lookup' function not defined in validate-psp-install.yaml 6318 - @innobead @roger-ryao
• [FEATURE] Support non-disruptive volume-related setting updates [7173](https://gith...

Longhorn v1.6.0-dev-20231217

This is a sprint release for internal testing and development purposes only. DO NOT INSTALL or UPGRADE to any sprint releases, as this operation is not supported.

Handled issues in the sprint

• [BUG] Warning events are being spammed by Longhorn - CRD 7290 - @m-ildefons
• [TASK] Add linear dm device on the top of v2 volume 7357 - @derekbit
• [BUG] Persistent volume is not ready for workloads 6776 - @james-munson @roger-ryao
• [DOC] Fix erronous value for default StorageMinimalAvailablePercentage setting. 7342 - @james-munson
• [BUG] A race after a node reboot leads to I/O errors with migratable volumes 6961 - @yangchiu @ejweber
• [BUG] CSI components CrashLoopBackOff, failed to connect to unix://csi/csi.sock after cluster restart 7116 - @ejweber
• [IMPROVEMENT] Don't crash the migration engine when kubelet restarts 7302 - @ejweber
• [TASK] Ensure the Go version in Dockerfile.dapper matches go.mod for all Longhorn components 7341 - @ejweber
• [BUG] Kubelet cannot finish terminating a pod that uses a PVC with volumeMode: Block when restarting the node 6919 - @PhanLe1010
• [IMPROVEMENT] deploy: driver deployer shouldn't cleanup previous deployment if Kubernetes version changes 5474 - @PhanLe1010
• [IMPROVEMENT] BackingImage should be compressed when downloading and use the name as filename instead of UUID 7295 - @ChanYiLin
• [FEATURE] Allow kubectl drain to stop manually attached volumes 6978 - @ChanYiLin @chriscchien
• [IMPROVEMENT] Reconcile replica instance state after its associated block-type disk is ready 7338 - @derekbit
• [BUG] Helm2 install error: 'lookup' function not defined in validate-psp-install.yaml 6318 -
• [BUG] multiple "for-cloning-volume" snapshots created after cloning volume 5835 - @PhanLe1010 @chriscchien
• [IMPROVEMENT] Collect SPDK related info for the usage metrics 6033 - @c3y1huang
• [BUG] Test case test_node_eviction failed 7210 - @ejweber @roger-ryao
• [TASK] Revisit volume automatic recover mechanism 6962 - @PhanLe1010
• [BUG] During volume live engine upgrade, delete replica with old engine image will make volume degraded forever 7012 - @PhanLe1010 @chriscchien
• [FEATURE] Add a new settings that allows Longhorn to evict replicas automatically when a node is drained 2238 - @ejweber
• [BUG] backingimage download server error 7288 - @ChanYiLin @scures
• [IMPROVEMENT] Allow deployment of Prometheus ServiceMonitor with the Longhorn helm chart 7041 - @mantissahz @chriscchien
• [TEST] Introduce Negative Testing 5240 - @yangchiu @khushboo-rancher
• [DOC] FS Trim for RWX is supported, but docs are out of date. 6733 - @james-munson
• [TASK] Collect coverage profiles for Go components during e2e test cases 5681 - @FrankYang0529
• [IMPROVEMENT] Reconcile engine/replica instance state of v2 volume like v1 volume 7326 - @derekbit
• [BUG] Backing image disk state unknown after unmount disk 6443 - @ChanYiLin
• [IMPROVEMENT] add build script to generate gRPC related code more convenient 6973 - @Vicente-Cheng
• [BUG] Backup NFS - Operation not permitted during mount 6114 - @mantissahz
• [BUG] Failed RWX mount due to connection timeout still happening 7301 - @james-munson
• [IMPROVEMENT] Disable CGO in longhorn components if not used 7135 - @derekbit
• [IMPROVEMENT] Environment setup and check tooling container 6657 - @derekbit
• [BUG] Cannot add block-type disk to node resource due to timeout error 7253 - @shuo-wu
• [IMPROVEMENT] Add test for longhorn-spdk-engine 6060 - @shuo-wu
• [IMPROVEMENT] Thread-safe SPDK JSON client 6106 - @shuo-wu
• [FEATURE] Support non-disruptive volume-related setting updates 7173 - @mantissahz
• [BUG] Test case test_volume_reattach_after_engine_sigkill failed 6751 - @c3y1huang
• [FEATURE] Container-Optimized OS support 6165 - @c3y1huang
• [IMPROVEMENT] Blindly stop raid bdev exposure before exposing it for V2 volume 7324 - @derekbit
• [IMPROVEMENT] BackingImage UI improvement 7293 - @scures
• [BUG] Volume has 2 active engines at the same time that blocks the volume controller reconciliation loop 4827 - @PhanLe1010 @chriscchien @roger-ryao
• [BUG] Volume UI displays only the last backup when using the recurring job 2997 - @mantissahz @chriscchien @roger-ryao
• [BUG] Volume gets stuck in an unknown state forever if created in an engine not fully deployed environment 6131 - @yangchiu @PhanLe1010
• [BUG] Continuously auto-balancing replicas when zone does not have enough space 6671 - @yangchiu @c3y1huang @roger-ryao
• [REFACTOR] Abstract the disk/lvol file operations in backupstore 6576 - @derekbit @chriscchien
• [BUG] backing-image-manager- hostPath selection exception 7062 - @ChanYiLin @chriscchien
• [IMPROVEMENT] Upgrade support bundle kit version to v0.0.33 7277 - @c3y1huang
• [BUG] GET error for volume attachment on node reboot 4188 - @PhanLe1010
• [BUG] MultiUnmapper floods logs with warnings about size mismatch. 6406 - @james-munson
• [TASK] Remove namespaceOverride from the chart values 6527 - @mantissahz @roger-ryao
• [IMPROVEMENT] Volumes: metrics for backups and snapshots include (count and size and type: system vs user), to better understand how the actual size is summed up. 5869 - @c3y1huang
• [TASK] Bump up the minimum supported Kubernetes version 7224 - @c3y1huang @roger-ryao
• [TASK] Implement xattr get and set operations on SPDK logical volumes (lvol) 6604 - @derekbit
• [DOC] Stress using object store as best practice for backups. 6773 - @james-munson
• [BUG] Errors found by static checker in volume controller 7009 - @m-ildefons
• [FEATURE] Have default priorityClass to prevent unexpected longhorn pods eviction 6528 - @mantissahz
• [BUG] Deadlock for RWX volume if an error occurs in its share-manager pod 7183 - @derekbit @chriscchien
• [TASK] Remove namespaceOverride from values.yaml in charts repo 7311 - @innobead
• [FEATURE] Talos support 3161 - @c3y1huang
• [IMPROVEMENT] Bypass upgrade when installing a fresh setup 6988 - @mantissahz @roger-ryao
• [IMPROVEMENT] Replace deprecated grpc.WithInsecure 7291 - @c3y1huang
• [BUG] Download backing image failed with HTTP 502 error if Storage Network configured 7236 - @ChanYiLin
• [TASK] Investigate and update the Rancher chart annotation requirements 7024 - @PhanLe1010 @chriscchien
• [BUG] Enabling replica-auto-balance tries to replicate to disabled nodes causing lots of errors in the logs and in the UI 6508 - @c3y1huang @chriscchien
• [IMPROVEMENT] Gracefully shut down spdk_tgt 7263 - @derekbit
• [FEATURE] Single Node Disk affinity 3823 - @ejw...