Longhorn v1.4.3 release

Release Note

v1.4.3 released! 🎆

Longhorn v1.4.3 is the latest stable version of Longhorn 1.4.
It introduces improvements and bug fixes in the areas of stability, resilience, and so on. Please try it out and provide feedback. Thanks for all the contributions!

For the definition of stable or latest release, please check here.

Installation

Please ensure your Kubernetes cluster is at least v1.21 before installing v1.4.3.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please read the important notes first and ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.4.3 from v1.3.x/v1.4.x, which are only supported source versions.

Follow the upgrade instructions here.

Deprecation & Incompatibilities

N/A

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Improvement

• [IMPROVEMENT] Assign the pods to the same node where the strict-local volume is present (5448) - @c3y1huang @chriscchien

Resilience

• [BUG] filesystem corrupted after delete instance-manager-r for a locality best-effort volume (5801) - @yangchiu @ChanYiLin @mantissahz

Bugs

• [BUG] 'Upgrade Engine' still shows up in a specific situation when engine already upgraded (3063) - @weizhe0422 @PhanLe1010 @smallteeths
• [BUG] DR volume even after activation remains in standby mode if there are one or more failed replicas. (3069) - @yangchiu @mantissahz
• [BUG] Prevent Longhorn uninstallation from getting stuck due to backups in error (5868) - @ChanYiLin @mantissahz
• [BUG] Unable to create support bundle if the previous one stayed in ReadyForDownload phase (5882) - @c3y1huang @roger-ryao
• [BUG] share-manager for a given pvc keep restarting (other pvc are working fine) (5954) - @yangchiu @derekbit
• [BUG] Replica auto-rebalance doesn't respect node selector (5971) - @c3y1huang @roger-ryao
• [BUG] Extra snapshot generated when clone from a detached volume (5986) - @weizhe0422 @ejweber
• [BUG] User created snapshot deleted after node drain and uncordon (5992) - @yangchiu @mantissahz
• [BUG] In some specific situation, system backup auto deleted when creating another one (6045) - @c3y1huang @chriscchien
• [BUG] Backing Image deletion stuck if it's deleted during uploading process and bids is ready-for-transfer state (6086) - @WebberHuang1118 @chriscchien
• [BUG] Backing image manager fails when SELinux is enabled (6108) - @ejweber @chriscchien
• [BUG] test_dr_volume_with_restore_command_error failed (6130) - @mantissahz @roger-ryao
• [BUG] Longhorn doesn't remove the system backups crd on uninstallation (6185) - @c3y1huang @khushboo-rancher
• [BUG] Test case test_ha_backup_deletion_recovery failed in rhel or rockylinux arm64 environment (6213) - @yangchiu @ChanYiLin @mantissahz
• [BUG] Engine continues to attempt to rebuild replica while detaching (6217) - @yangchiu @ejweber
• [BUG] Unable to receive support bundle from UI when it's large (400MB+) (6256) - @c3y1huang @chriscchien
• [BUG] Migration test case failed: unable to detach volume migration is not ready yet (6238) - @yangchiu @PhanLe1010 @khushboo-rancher
• [BUG] Restored Volumes stuck in attaching state (6239) - @derekbit @roger-ryao

Contributors

• @ChanYiLin
• @PhanLe1010
• @WebberHuang1118
• @c3y1huang
• @chriscchien
• @derekbit
• @ejweber
• @innobead
• @khushboo-rancher
• @mantissahz
• @roger-ryao
• @smallteeths
• @weizhe0422
• @yangchiu

Longhorn v1.4.3-rc2 release

Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.4.3-rc2 released! 🎆

This release introduces improvements and bug fixes as described below about stability, resilience, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Please ensure your Kubernetes cluster is at least v1.21 before installing v1.4.3.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please read the important notes first and ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.4.3 from v1.3.x/v1.4.x, which are only supported source versions.

The live upgrade from v1.3.x/v1.4.x is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

N/A

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Improvement

• [IMPROVEMENT] Assign the pods to the same node where the strict-local volume is present (5448) - @c3y1huang @chriscchien

Resilience

• [BUG] filesystem corrupted after delete instance-manager-r for a locality best-effort volume (5801) - @yangchiu @ChanYiLin @mantissahz

Bugs

• [BUG] 'Upgrade Engine' still shows up in a specific situation when engine already upgraded (3063) - @weizhe0422 @PhanLe1010 @smallteeths
• [BUG] DR volume even after activation remains in standby mode if there are one or more failed replicas. (3069) - @yangchiu @mantissahz
• [BUG] Prevent Longhorn uninstallation from getting stuck due to backups in error (5868) - @ChanYiLin @mantissahz
• [BUG] Unable to create support bundle if the previous one stayed in ReadyForDownload phase (5882) - @c3y1huang @roger-ryao
• [BUG] share-manager for a given pvc keep restarting (other pvc are working fine) (5954) - @yangchiu @derekbit
• [BUG] Replica auto-rebalance doesn't respect node selector (5971) - @c3y1huang @roger-ryao
• [BUG] Extra snapshot generated when clone from a detached volume (5986) - @weizhe0422 @ejweber
• [BUG] User created snapshot deleted after node drain and uncordon (5992) - @yangchiu @mantissahz
• [BUG] In some specific situation, system backup auto deleted when creating another one (6045) - @c3y1huang @chriscchien
• [BUG] Backing Image deletion stuck if it's deleted during uploading process and bids is ready-for-transfer state (6086) - @WebberHuang1118 @chriscchien
• [BUG] Backing image manager fails when SELinux is enabled (6108) - @ejweber @chriscchien
• [BUG] test_dr_volume_with_restore_command_error failed (6130) - @mantissahz @roger-ryao
• [BUG] Longhorn doesn't remove the system backups crd on uninstallation (6185) - @c3y1huang @khushboo-rancher
• [BUG] Test case test_ha_backup_deletion_recovery failed in rhel or rockylinux arm64 environment (6213) - @yangchiu @ChanYiLin @mantissahz
• [BUG] Engine continues to attempt to rebuild replica while detaching (6217) - @yangchiu @ejweber
• [BUG] Unable to receive support bundle from UI when it's large (400MB+) (6256) - @c3y1huang @chriscchien
• [BUG] Migration test case failed: unable to detach volume migration is not ready yet (6238) - @yangchiu @PhanLe1010 @khushboo-rancher
• [BUG] Restored Volumes stuck in attaching state (6239) - @derekbit @roger-ryao

Contributors

• @ChanYiLin
• @PhanLe1010
• @WebberHuang1118
• @c3y1huang
• @chriscchien
• @derekbit
• @ejweber
• @innobead
• @khushboo-rancher
• @mantissahz
• @roger-ryao
• @smallteeths
• @weizhe0422
• @yangchiu

Longhorn v1.5.0 release

Release Note

v1.5.0 released! 🎆

Longhorn v1.5.0 is the latest version of Longhorn 1.5.
It introduces many enhancements, improvements, and bug fixes as described below including performance, stability, maintenance, resilience, and so on. Please try it and feedback. Thanks for all the contributions!

For the definition of stable or latest release, please check here.

• v2 Data Engine based on SPDK - Preview

  Please note that this is a preview feature, so should not be used in any production environment. A preview feature is disabled by default and would be changed in the following versions until it becomes general availability.

  In addition to the existing iSCSI stack (v1) data engine, we are introducing the v2 data engine based on SPDK (Storage Performance Development Kit). This release includes the introduction of volume lifecycle management, degraded volume handling, offline replica rebuilding, block device management, and orphaned replica management. For the performance benchmark and comparison with v1, check the report here.

• Longhorn Volume Attachment
  Introducing the new Longhorn VolumeAttachment CR, which ensures exclusive attachment and supports automatic volume attachment and detachment for various headless operations such as volume cloning, backing image export, and recurring jobs.

• Cluster Autoscaler - GA
  Cluster Autoscaler was initially introduced as an experimental feature in v1.3. After undergoing automatic validation on different public cloud Kubernetes distributions and receiving user feedback, it has now reached general availability.

• Instance Manager Engine & Replica Consolidation
  Previously, there were two separate instance manager pods responsible for volume engine and replica process management. However, this setup required high resource usage, especially during live upgrades. In this release, we have merged these pods into a single instance manager, reducing the initial resource requirements.

• Volume Backup Compression Methods
  Longhorn supports different compression methods for volume backups, including lz4, gzip, or no compression. This allows users to choose the most suitable method based on their data type and usage requirements.

• Automatic Volume Trim Recurring Job
  While volume filesystem trim was introduced in v1.4, users had to perform the operation manually. From this release, users can create a recurring job that automatically runs the trim process, improving space efficiency without requiring human intervention.

• RWX Volume Trim
  Longhorn supports filesystem trim for RWX (Read-Write-Many) volumes, expanding the trim functionality beyond RWO (Read-Write-Once) volumes only.

• Upgrade Path Enforcement & Downgrade Prevention
  To ensure compatibility after an upgrade, we have implemented upgrade path enforcement. This prevents unintended downgrades and ensures the system and data remain intact.

• Backing Image Management via CSI VolumeSnapshot
  Users can now utilize the unified CSI VolumeSnapshot interface to manage Backing Images similar to volume snapshots and backups.

• Snapshot Cleanup & Delete Recurring Job
  Introducing two new recurring job types specifically designed for snapshot cleanup and deletion. These jobs allow users to remove unnecessary snapshots for better space efficiency.

• CIFS Backup Store & Azure Backup Store
  To enhance users' backup strategies and align with data governance policies, Longhorn now supports additional backup storage protocols, including CIFS and Azure.

• Kubernetes Upgrade Node Drain Policy
  The new Node Drain Policy provides flexible strategies to protect volume data during Kubernetes upgrades or node maintenance operations. This ensures the integrity and availability of your volumes.

Installation

Please ensure your Kubernetes cluster is at least v1.21 before installing Longhorn v1.5.0.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.5.0 from v1.4.x. Only support upgrading from 1.4.x.

Follow the upgrade instructions here.

Deprecation & Incompatibilities

Please check the important notes to know more about deprecated, removed, incompatible features and important changes. If you upgrade indirectly from an older version like v1.3.x, please also check the corresponding important note for each upgrade version path.

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

• [DOC] Provide the user guide for Kubernetes upgrade (494) - @PhanLe1010
• [FEATURE] Backups to Azure Blob Storage (1309) - @mantissahz @chriscchien
• [IMPROVEMENT] Use PDB to protect Longhorn components from unexpected drains (3304) - @yangchiu @PhanLe1010
• [FEATURE] CIFS Backup Store Support (3599) - @derekbit @chriscchien
• [IMPROVEMENT] Consolidate volume attach/detach implementation (3715) - @yangchiu @PhanLe1010
• [IMPROVEMENT] Periodically clean up volume snapshots (3836) - @c3y1huang @chriscchien
• [IMPROVEMENT] Introduce timeout mechanism for the sparse file syncing service (4305) - @yangchiu @ChanYiLin
• [IMPROVEMENT] Recurring jobs create new snapshots while being not able to clean up old ones (4898) - @mantissahz @chriscchien
• [FEATURE] BackingImage Management via VolumeSnapshot (5005) - @ChanYiLin @chriscchien
• [FEATURE] Upgrade path enforcement & downgrade prevention (5131) - @yangchiu @mantissahz
• [FEATURE] Support RWX volume trim (5143) - @derekbit @chriscchien
• [FEATURE] Auto Trim via recurring job (5186) - @c3y1huang @chriscchien
• [FEATURE] Introduce faster compression and multiple threads for volume backup & restore (5189) - @derekbit @roger-ryao
• [FEATURE] Consolidate Instance Manager Engine & Replica for resource consumption reduction (5208) - @yangchiu @c3y1huang
• [FEATURE] Cluster Autoscaler Support GA (5238) - @yangchiu @c3y1huang
• [FEATURE] Update K8s version support and component/pkg/build dependencies for Longhorn 1.5 (5595) - @yangchiu @ejweber
• [FEATURE] Support SPDK Data Engine - Preview (5751) - @derekbit @shuo-wu @DamiaSan

Enhancements

• [FEATURE] Allow users to directly activate a restoring/DR volume as long as there is one ready replica. (1512) - @mantissahz @weizhe0422
• [REFACTOR] volume controller refactoring/split up, to simplify the control flow (2527) - @PhanLe1010 @chriscchien
• [FEATURE] Import and export SPDK longhorn volumes to longhorn sparse file directory (4100) - @DamiaSan
• [FEATURE] Add a global storage reserved setting for newly created longhorn nodes' disks (4773) - @mantissahz @chriscchien
• [FEATURE] Support backup volumes during system backup (5011) - @c3y1huang @chriscchien
• [FEATURE] Support SPDK lvol shallow copy for newly replica creation (5217) - @DamiaSan
• [FEATURE] Introduce longhorn-spdk-engine for SPDK volume management (5282) - @shuo-wu
• [FEATURE] Support replica-zone-soft-anti-affinity setting per volume (5358) - @ChanYiLin @smallteeths @chriscchien
• [FEATURE] Install Opt-In NetworkPolicies (5403) - @yangchiu @ChanYiLin
• [FEATURE] Create Longhorn SPDK Engine component with basic fundamental functions (5406) - @shuo-wu
• [FEATURE] Add status APIs for shallow copy and...

Longhorn v1.4.3-rc1 release

Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.4.3-rc1 released! 🎆

This release introduces improvements and bug fixes as described below about stability, security, resilience, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Please ensure your Kubernetes cluster is at least v1.21 before installing v1.4.3.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please read the important notes first and ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.4.3 from v1.3.x/v1.4.x, which are only supported source versions.

The live upgrade from v1.3.x/v1.4.x is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

N/A

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Improvement

• [IMPROVEMENT] Assign the pods to the same node where the strict-local volume is present (5448) - @c3y1huang @chriscchien

Bugs

• [BUG] 'Upgrade Engine' still shows up in a specific situation when engine already upgraded (3063) - @weizhe0422 @PhanLe1010 @smallteeths
• [BUG] DR volume even after activation remains in standby mode if there are one or more failed replicas. (3069) - @yangchiu @mantissahz
• [BUG] Prevent Longhorn uninstallation from getting stuck due to backups in error (5868) - @ChanYiLin @mantissahz
• [BUG] Unable to create support bundle if the previous one stayed in ReadyForDownload phase (5882) - @c3y1huang @roger-ryao
• [BUG] share-manager for a given pvc keep restarting (other pvc are working fine) (5954) - @yangchiu @derekbit
• [BUG] Replica auto-rebalance doesn't respect node selector (5971) - @c3y1huang @roger-ryao
• [BUG] Extra snapshot generated when clone from a detached volume (5986) - @weizhe0422 @ejweber
• [BUG] User created snapshot deleted after node drain and uncordon (5992) - @yangchiu @mantissahz
• [BUG] In some specific situation, system backup auto deleted when creating another one (6045) - @c3y1huang @chriscchien
• [BUG] Backing Image deletion stuck if it's deleted during uploading process and bids is ready-for-transfer state (6086) - @WebberHuang1118 @chriscchien
• [BUG] Backing image manager fails when SELinux is enabled (6108) - @ejweber @chriscchien
• [BUG] test_dr_volume_with_restore_command_error failed (6130) - @mantissahz @roger-ryao
• [BUG] Longhorn doesn't remove the system backups crd on uninstallation (6185) - @c3y1huang @khushboo-rancher
• [BUG] Test case test_ha_backup_deletion_recovery failed in rhel or rockylinux arm64 environment (6213) - @yangchiu @ChanYiLin @mantissahz
• [BUG] Engine continues to attempt to rebuild replica while detaching (6217) - @ejweber

Contributors

• @ChanYiLin
• @PhanLe1010
• @WebberHuang1118
• @c3y1huang
• @chriscchien
• @derekbit
• @ejweber
• @innobead
• @khushboo-rancher
• @mantissahz
• @roger-ryao
• @smallteeths
• @weizhe0422
• @yangchiu

Longhorn v1.5.0-rc3 release

Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.5.0-rc3 released! 🎆

Longhorn v1.5.0 is the latest version of Longhorn 1.5.
It introduces many enhancements, improvements, and bug fixes as described below including performance, stability, maintenance, resilience, and so on.

Please try it and feedback. Thanks for all the contributions!

For the definition of stable or latest release, please check here.

• SPDK Data Engine - Preview

  Please note that this is a preview feature, so should not be used in any production environment.

  In addition to the existing iSCSI stack (v1) data engine, we are introducing the v2 data engine based on SPDK (Storage Performance Development Kit). This release includes the introduction of volume lifecycle management, degraded volume handling, offline replica rebuilding, block device management and orphaned replica management.

• Longhorn Volume Attachment
  Introducing the new Longhorn VolumeAttachment CR, which ensures exclusive attachment and supports automatic volume attachment and detachment for various headless operations such as volume cloning, backing image export, and recurring jobs.

• Cluster Autoscaler - GA
  Cluster Autoscaler was initially introduced as an experimental feature in v1.3. After undergoing automatic validation on different public cloud Kubernetes distributions and receiving user feedback, it has now reached general availability.

• Instance Manager Engine & Replica Consolidation
  Previously, there were two separate instance manager pods responsible for volume engine and replica process management. However, this setup required high resource usage, especially during live upgrades. In this release, we have merged these pods into a single instance manager, reducing the initial resource requirements.

• Volume Backup Compression Methods
  Longhorn supports different compression methods for volume backups, including lz4, gzip, or no compression. This allows users to choose the most suitable method based on their data type and usage requirements.

• Automatic Volume Trim Recurring Job
  While volume filesystem trim was introduced in v1.4, users had to perform the operation manually. From this release, users can create a recurring job that automatically runs the trim process, improving space efficiency without requiring human intervention.

• RWX Volume Trim
  Longhorn supports filesystem trim for RWX (Read-Write-Many) volumes, expanding the trim functionality beyond RWO (Read-Write-Once) volumes only.

• Upgrade Path Enforcement & Downgrade Prevention
  To ensure compatibility after an upgrade, we have implemented upgrade path enforcement. This prevents unintended downgrades and ensures the system and data remain intact.

• Backing Image Management via CSI VolumeSnapshot
  Users can now utilize the unified CSI VolumeSnapshot interface to manage Backing Images similar to volume snapshots and backups.

• Snapshot Cleanup & Delete Recurring Job
  Introducing two new recurring job types specifically designed for snapshot cleanup and deletion. These jobs allow users to remove unnecessary snapshots for better space efficiency.

• CIFS Backup Store & Azure Backup Store
  To enhance users' backup strategies and align with data governance policies, Longhorn now supports additional backup storage protocols, including CIFS and Azure.

• Kubernetes Upgrade Node Drain Policy
  The new Node Drain Policy provides flexible strategies to protect volume data during Kubernetes upgrades or node maintenance operations. This ensures the integrity and availability of your volumes.

Installation

Please ensure your Kubernetes cluster is at least v1.21 before installing Longhorn v1.5.0.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.5.0 from v1.4.x. Only support upgrading from 1.4.x.

The live upgrade from v1.4.x is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

Please check the important notes to know more about deprecated, removed, incompatible features.

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

• [DOC] Provide the user guide for Kubernetes upgrade (494) - @PhanLe1010
• [FEATURE] Backups to Azure Blob Storage (1309) - @mantissahz @chriscchien
• [IMPROVEMENT] Use PDB to protect Longhorn components from unexpected drains (3304) - @yangchiu @PhanLe1010
• [FEATURE] CIFS Backup Store Support (3599) - @derekbit @chriscchien
• [IMPROVEMENT] Consolidate volume attach/detach implementation (3715) - @yangchiu @PhanLe1010
• [IMPROVEMENT] Periodically clean up volume snapshots (3836) - @c3y1huang @chriscchien
• [IMPROVEMENT] Introduce timeout mechanism for the sparse file syncing service (4305) - @yangchiu @ChanYiLin
• [IMPROVEMENT] Recurring jobs create new snapshots while being not able to clean up old ones (4898) - @mantissahz @chriscchien
• [FEATURE] BackingImage Management via VolumeSnapshot (5005) - @ChanYiLin @chriscchien
• [FEATURE] Upgrade path enforcement & downgrade prevention (5131) - @yangchiu @mantissahz
• [FEATURE] Support RWX volume trim (5143) - @derekbit @chriscchien
• [FEATURE] Auto Trim via recurring job (5186) - @c3y1huang @chriscchien
• [FEATURE] Introduce faster compression and multiple threads for volume backup & restore (5189) - @derekbit @roger-ryao
• [FEATURE] Consolidate Instance Manager Engine & Replica for resource consumption reduction (5208) - @yangchiu @c3y1huang
• [FEATURE] Cluster Autoscaler Support GA (5238) - @yangchiu @c3y1huang
• [FEATURE] Update K8s version support and component/pkg/build dependencies for Longhorn 1.5 (5595) - @yangchiu @ejweber
• [FEATURE] Support SPDK Data Engine - Preview (5751) - @derekbit @shuo-wu @DamiaSan

Enhancements

• [FEATURE] Allow users to directly activate a restoring/DR volume as long as there is one ready replica. (1512) - @mantissahz @weizhe0422
• [REFACTOR] volume controller refactoring/split up, to simplify the control flow (2527) - @PhanLe1010 @chriscchien
• [FEATURE] Import and export SPDK longhorn volumes to longhorn sparse file directory (4100) - @DamiaSan
• [FEATURE] Add a global storage reserved setting for newly created longhorn nodes' disks (4773) - @mantissahz @chriscchien
• [FEATURE] Support backup volumes during system backup (5011) - @c3y1huang @chriscchien
• [FEATURE] Support SPDK lvol shallow copy for newly replica creation (5217) - @DamiaSan
• [FEATURE] Introduce longhorn-spdk-engine for SPDK volume management (5282) - @shuo-wu
• [FEATURE] Support replica-zone-soft-anti-affinity setting per volume (5358) - @ChanYiLin @smallteeths @chriscchien
• [FEATURE] Install Opt-In NetworkPolicies (5403) - @yangchiu @ChanYiLin
• [FEATURE] Create Longhorn SPDK Engine component with basic fundamental functions (5406) - @shuo-wu
• [FEATURE] Add status APIs for shallow copy and IO pause/resume (5647) - @DamiaSan
• [FEATURE] Introduce a new disk type, disk management and replica scheduler for SPDK volumes ([5683](https://github.com/longhorn/longhorn...

Longhorn v1.5.0-rc2 release

Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.5.0-rc2 released! 🎆

Longhorn v1.5.0 is the latest version of Longhorn 1.5.
It introduces many enhancements, improvements, and bug fixes as described below about SPDK data engine (preview), stability, maintenance, performance, resilience, and so on. Please try it and feedback. Thanks for all the contributions!

For the definition of stable or latest release, please check here.

• SPDK Data Engine - Preview

  Please note that this is a preview feature, so should not be used in any production environment.

  In addition to the existing iSCSI stack data engine, we are introducing a second engine based on SPDK. This release includes the introduction of volume lifecycle management, degraded volume handling, offline replica rebuilding, block device management and orphaned replica management.

• Longhorn Volume Attachment
  Introducing the new Longhorn VolumeAttachment CR, which ensures exclusive attachment and supports automatic volume attachment and detachment for various headless operations such as volume cloning, backing image export, and recurring jobs.

• Cluster Autoscaler - GA
  Cluster Autoscaler was initially introduced as an experimental feature in v1.3. After undergoing automatic validation on different public cloud Kubernetes distributions and receiving user feedback, it has now reached general availability.

• Instance Manager Engine & Replica Consolidation
  Previously, there were two separate instance manager pods responsible for volume engine and replica process management. However, this setup required high resource usage, especially during live upgrades. In this release, we have merged these pods into a single instance manager, reducing the initial resource requirements.

• Volume Backup Compression Methods
  Longhorn supports different compression methods for volume backups, including lz4, gzip, or no compression. This allows users to choose the most suitable method based on their data type and usage requirements.

• Automatic Volume Trim Recurring Job
  While volume filesystem trim was introduced in v1.4, users had to perform the operation manually. From this release, users can create a recurring job that automatically runs the trim process, improving space efficiency without requiring human intervention.

• RWX Volume Trim
  Longhorn supports filesystem trim for RWX (Read-Write-Many) volumes, expanding the trim functionality beyond RWO (Read-Write-Once) volumes only.

• Upgrade Path Enforcement & Downgrade Prevention
  To ensure compatibility after an upgrade, we have implemented upgrade path enforcement. This prevents unintended downgrades and ensures the system and data remain intact.

• Backing Image Management via CSI VolumeSnapshot
  Users can now utilize the unified CSI VolumeSnapshot interface to manage Backing Images similar to volume snapshots and backups.

• Snapshot Cleanup & Delete Recurring Job
  Introducing two new recurring job types specifically designed for snapshot cleanup and deletion. These jobs allow users to remove unnecessary snapshots for better space efficiency.

• CIFS Backup Store & Azure Backup Store
  To enhance users' backup strategies and align with data governance policies, Longhorn now supports additional backup storage protocols, including CIFS and Azure.

• Kubernetes Upgrade Node Drain Policy
  The new Node Drain Policy provides flexible strategies to protect volume data during Kubernetes upgrades or node maintenance operations. This ensures the integrity and availability of your volumes.

Installation

Please ensure your Kubernetes cluster is at least v1.21 before installing Longhorn v1.5.0.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.5.0 from v1.4.x. Only support upgrading from 1.4.x.

The live upgrade from v1.4.x is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

Please check the important notes to know more about deprecated, removed, incompatible features.

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

• [DOC] Provide the user guide for Kubernetes upgrade (494) - @PhanLe1010
• [FEATURE] Backups to Azure Blob Storage (1309) - @mantissahz @chriscchien
• [IMPROVEMENT] Use PDB to protect Longhorn components from unexpected drains (3304) - @yangchiu @PhanLe1010
• [FEATURE] CIFS Backup Store Support (3599) - @derekbit @chriscchien
• [IMPROVEMENT] Consolidate volume attach/detach implementation (3715) - @yangchiu @PhanLe1010
• [IMPROVEMENT] Periodically clean up volume snapshots (3836) - @c3y1huang @chriscchien
• [IMPROVEMENT] Introduce timeout mechanism for the sparse file syncing service (4305) - @yangchiu @ChanYiLin
• [IMPROVEMENT] Recurring jobs create new snapshots while being not able to clean up old ones (4898) - @mantissahz @chriscchien
• [FEATURE] BackingImage Management via VolumeSnapshot (5005) - @ChanYiLin @chriscchien
• [FEATURE] Upgrade path enforcement & downgrade prevention (5131) - @yangchiu @mantissahz
• [FEATURE] Support RWX volume trim (5143) - @derekbit @chriscchien
• [FEATURE] Auto Trim via recurring job (5186) - @c3y1huang @chriscchien
• [FEATURE] Introduce faster compression and multiple threads for volume backup & restore (5189) - @derekbit @roger-ryao
• [FEATURE] Consolidate Instance Manager Engine & Replica for resource consumption reduction (5208) - @yangchiu @c3y1huang
• [FEATURE] Cluster Autoscaler Support GA (5238) - @yangchiu @c3y1huang
• [FEATURE] Update K8s version support and component/pkg/build dependencies for Longhorn 1.5 (5595) - @yangchiu @ejweber
• [FEATURE] Support SPDK Data Engine - Preview (5751) - @derekbit @shuo-wu @DamiaSan

Enhancements

• [FEATURE] Allow users to directly activate a restoring/DR volume as long as there is one ready replica. (1512) - @mantissahz @weizhe0422
• [REFACTOR] volume controller refactoring/split up, to simplify the control flow (2527) - @PhanLe1010 @chriscchien
• [FEATURE] Import and export SPDK longhorn volumes to longhorn sparse file directory (4100) - @DamiaSan
• [FEATURE] Add a global storage reserved setting for newly created longhorn nodes' disks (4773) - @mantissahz @chriscchien
• [FEATURE] Support backup volumes during system backup (5011) - @c3y1huang @chriscchien
• [FEATURE] Support SPDK lvol shallow copy for newly replica creation (5217) - @DamiaSan
• [FEATURE] Introduce longhorn-spdk-engine for SPDK volume management (5282) - @shuo-wu
• [FEATURE] Support replica-zone-soft-anti-affinity setting per volume (5358) - @ChanYiLin @smallteeths @chriscchien
• [FEATURE] Install Opt-In NetworkPolicies (5403) - @yangchiu @ChanYiLin
• [FEATURE] Create Longhorn SPDK Engine component with basic fundamental functions (5406) - @shuo-wu
• [FEATURE] Add status APIs for shallow copy and IO pause/resume (5647) - @DamiaSan
• [FEATURE] Introduce a new disk type, disk management and replica scheduler for SPDK volumes (5683) - @derekbit @...

Longhorn v1.5.0-rc1 release

Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.5.0-rc1 released! 🎆

Longhorn v1.5.0 is the latest version of Longhorn 1.5.
It introduces many enhancements, improvements, and bug fixes as described below about the SPDK (Storage Performance Development Kit) data engine (preview), stability, maintenance, performance, resilience, and so on. Please try it and feedback. Thanks for all the contributions!

For the definition of the stable or latest release, please check here.

• SPDK Data Engine - Preview

  Please note that this is a preview feature, so should not be used in any production environment.

  In addition to the existing iSCSI stack data engine, we are introducing a second engine based on SPDK. This release includes the introduction of volume lifecycle management, degraded volume handling, offline replica rebuilding, block device management and orphaned replica management.

• Longhorn Volume Attachment
  Introducing the new Longhorn VolumeAttachment CR, which ensures exclusive attachment and supports automatic volume attachment and detachment for various headless operations such as volume cloning, backing image export, and recurring jobs.

• Cluster Autoscaler - GA
  Cluster Autoscaler was initially introduced as an experimental feature in v1.3. After undergoing automatic validation on different public cloud Kubernetes distributions and receiving user feedback, it has now reached general availability.

• Instance Manager Engine & Replica Consolidation
  Previously, there were two separate instance manager pods responsible for volume engine and replica process management. However, this setup required high resource usage, especially during live upgrades. In this release, we have merged these pods into a single instance manager, reducing the initial resource requirements.

• Volume Backup Compression Methods
  Longhorn supports different compression methods for volume backups, including lz4, gzip, or no compression. This allows users to choose the most suitable method based on their data type and usage requirements.

• Automatic Volume Trim Recurring Job
  While volume filesystem trim was introduced in v1.4, users had to perform the operation manually. From this release, users can create a recurring job that automatically runs the trim process, improving space efficiency without requiring human intervention.

• RWX Volume Trim
  Longhorn supports filesystem trim for RWX (Read-Write-Many) volumes, expanding the trim functionality beyond RWO (Read-Write-Once) volumes only.

• Upgrade Path Enforcement & Downgrade Prevention
  To ensure compatibility after an upgrade, we have implemented upgrade path enforcement. This prevents unintended downgrades and ensures the system and data remain intact.

• Backing Image Management via CSI VolumeSnapshot
  Users can now utilize the unified CSI VolumeSnapshot interface to manage Backing Images similar to volume snapshots and backups.

• Snapshot Cleanup & Delete Recurring Job
  Introducing two new recurring job types specifically designed for snapshot cleanup and deletion. These jobs allow users to remove unnecessary snapshots for better space efficiency.

• CIFS Backup Store

• Azure Backup Store
  To enhance users' backup strategies and align with data governance policies, Longhorn now supports additional backup storage protocols, including CIFS and Azure.

• Kubernetes Upgrade Node Drain Policy
  The new Node Drain Policy provides flexible strategies to protect volume data during Kubernetes upgrades or node maintenance operations. This ensures the integrity and availability of your volumes.

Installation

Please ensure your Kubernetes cluster is at least v1.21 before installing Longhorn v1.5.0.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.5.0 from v1.4.x. Only support upgrading from 1.4.x.

The live upgrade from v1.4.x is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

Please check the important notes to know more about deprecated, removed, incompatible features.

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

• [DOC] Provide the user guide for Kubernetes upgrade (494) - @PhanLe1010
• [FEATURE] Backups to Azure Blob Storage (1309) - @mantissahz @chriscchien
• [IMPROVEMENT] Use PDB to protect Longhorn components from unexpected drains (3304) - @yangchiu @PhanLe1010
• [FEATURE] CIFS Backup Store Support (3599) - @derekbit @chriscchien
• [IMPROVEMENT] Consolidate volume attach/detach implementation (3715) - @yangchiu @PhanLe1010
• [IMPROVEMENT] Periodically clean up volume snapshots (3836) - @c3y1huang @chriscchien
• [IMPROVEMENT] Introduce timeout mechanism for the sparse file syncing service (4305) - @yangchiu @ChanYiLin
• [IMPROVEMENT] Recurring jobs create new snapshots while being not able to clean up old ones (4898) - @mantissahz @chriscchien
• [FEATURE] BackingImage Management via VolumeSnapshot (5005) - @ChanYiLin @chriscchien
• [FEATURE] Upgrade path enforcement & downgrade prevention (5131) - @yangchiu @mantissahz
• [FEATURE] Support RWX volume trim (5143) - @derekbit @chriscchien
• [FEATURE] Auto Trim via recurring job (5186) - @c3y1huang @chriscchien
• [FEATURE] Introduce faster compression and multiple threads for volume backup & restore (5189) - @derekbit @roger-ryao
• [FEATURE] Consolidate Instance Manager Engine & Replica for resource consumption reduction (5208) - @yangchiu @c3y1huang
• [FEATURE] Cluster Autoscaler Support GA (5238) - @yangchiu @c3y1huang
• [FEATURE] Update K8s version support and component/pkg/build dependencies for Longhorn 1.5 (5595) - @yangchiu @ejweber
• [FEATURE] Support SPDK Data Engine - Preview (5751) - @derekbit @shuo-wu @DamiaSan

Enhancements

• [FEATURE] Allow users to directly activate a restoring/DR volume as long as there is one ready replica. (1512) - @mantissahz @weizhe0422
• [REFACTOR] volume controller refactoring/split up, to simplify the control flow (2527) - @PhanLe1010 @chriscchien
• [FEATURE] Import and export SPDK longhorn volumes to longhorn sparse file directory (4100) - @DamiaSan
• [FEATURE] Add a global storage reserved setting for newly created longhorn nodes' disks (4773) - @mantissahz @chriscchien
• [FEATURE] Support backup volumes during system backup (5011) - @c3y1huang @chriscchien
• [FEATURE] Support SPDK lvol shallow copy for newly replica creation (5217) - @DamiaSan
• [FEATURE] Introduce longhorn-spdk-engine for SPDK volume management (5282) - @shuo-wu
• [FEATURE] Support replica-zone-soft-anti-affinity setting per volume (5358) - @ChanYiLin @smallteeths @chriscchien
• [FEATURE] Install Opt-In NetworkPolicies (5403) - @yangchiu @ChanYiLin
• [FEATURE] Create Longhorn SPDK Engine component with basic fundamental functions (5406) - @shuo-wu
• [FEATURE] Add status APIs for shallow copy and IO pause/resume (5647) - @DamiaSan
• [FEATURE] Introduce a new disk type, disk management and replica scheduler for SPDK v...

Longhorn v1.4.2 release

Release Note

v1.4.2 released! 🎆

Longhorn v1.4.2 is the latest stable version of Longhorn 1.4.
It introduces improvements and bug fixes in the areas of stability, performance, space efficiency, resilience, and so on. Please try it out and provide feedback. Thanks for all the contributions!

For the definition of stable or latest release, please check here.

Installation

Please ensure your Kubernetes cluster is at least v1.21 before installing v1.4.2.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please read the important notes first and ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.4.2 from v1.3.x/v1.4.x, which are only supported source versions.

Follow the upgrade instructions here.

Deprecation & Incompatibilities

N/A

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

• [IMPROVEMENT] Use PDB to protect Longhorn components from unexpected drains (3304) - @yangchiu @PhanLe1010
• [IMPROVEMENT] Introduce timeout mechanism for the sparse file syncing service (4305) - @yangchiu @ChanYiLin
• [IMPROVEMENT] Recurring jobs create new snapshots while being not able to clean up old ones (4898) - @mantissahz @chriscchien

Improvement

• [IMPROVEMENT] Support bundle collects dmesg, syslog and related information of longhorn nodes (5073) - @weizhe0422 @roger-ryao
• [IMPROVEMENT] Fix BackingImage uploading/downloading flow to prevent client timeout (5443) - @ChanYiLin @chriscchien
• [IMPROVEMENT] Create a new setting so that Longhorn removes PDB for instance-manager-r that doesn't have any running instance inside it (5549) - @PhanLe1010 @khushboo-rancher
• [IMPROVEMENT] Deprecate the setting allow-node-drain-with-last-healthy-replica and replace it by node-drain-policy setting (5585) - @yangchiu @PhanLe1010
• [IMPROVEMENT][UI] Recurring jobs create new snapshots while being not able to clean up old one (5610) - @mantissahz @smallteeths @roger-ryao
• [IMPROVEMENT] Only activate replica if it doesn't have deletion timestamp during volume engine upgrade (5632) - @PhanLe1010 @roger-ryao
• [IMPROVEMENT] Clean up backup target if the backup target setting is unset (5655) - @yangchiu @ChanYiLin

Resilience

• [BUG] Directly mark replica as failed if the node is deleted (5542) - @weizhe0422 @roger-ryao
• [BUG] RWX volume is stuck at detaching when the attached node is down (5558) - @derekbit @roger-ryao
• [BUG] Backup monitor gets stuck in an infinite loop if backup isn't found (5662) - @derekbit @chriscchien
• [BUG] Resources such as replicas are somehow not mutated when network is unstable (5762) - @derekbit @roger-ryao
• [BUG] Instance manager may not update instance status for a minute after starting (5809) - @ejweber @chriscchien

Bugs

• [BUG] Delete a uploading backing image, the corresponding LH temp file is not deleted (3682) - @ChanYiLin @chriscchien
• [BUG] Can not create backup in engine image not fully deployed cluster (5248) - @ChanYiLin @roger-ryao
• [BUG] Upgrade engine --> spec.restoreVolumeRecurringJob and spec.snapshotDataIntegrity Unsupported value (5485) - @yangchiu @derekbit
• [BUG] Bulk backup deletion cause restoring volume to finish with attached state. (5506) - @ChanYiLin @roger-ryao
• [BUG] volume expansion starts for no reason, gets stuck on current size > expected size (5513) - @mantissahz @roger-ryao
• [BUG] RWX volume attachment failed if tried more enough times (5537) - @yangchiu @derekbit
• [BUG] instance-manager-e emits Wait for process pvc-xxxx to shutdown constantly (5575) - @derekbit @roger-ryao
• [BUG] Support bundle kit should respect node selector & taint toleration (5614) - @yangchiu @c3y1huang
• [BUG] Value overlapped in page Instance Manager Image (5622) - @smallteeths @chriscchien
• [BUG] Instance manager PDB created with wrong selector thus blocking the draining of the wrongly selected node forever (5680) - @PhanLe1010 @chriscchien
• [BUG] During volume live engine upgrade, if the replica pod is killed, the volume is stuck in upgrading forever (5684) - @yangchiu @PhanLe1010
• [BUG] Instance manager PDBs cannot be removed if the longhorn-manager pod on its spec node is not available (5688) - @PhanLe1010 @roger-ryao
• [BUG] Rebuild rebuilding is possibly issued to a wrong replica (5709) - @ejweber @roger-ryao
• [BUG] longhorn upgrade is not upgrading engineimage (5740) - @shuo-wu @chriscchien
• [BUG] test_replica_auto_balance_when_replica_on_unschedulable_node Error in creating volume with nodeSelector and dataLocality parameters (5745) - @c3y1huang @roger-ryao
• [BUG] Unable to backup volume after NFS server IP change (5856) - @derekbit @roger-ryao

Misc

• [TASK] Check and update the networking doc & example YAMLs (5651) - @yangchiu @shuo-wu

Contributors

• @ChanYiLin
• @PhanLe1010
• @c3y1huang
• @chriscchien
• @derekbit
• @ejweber
• @innobead
• @khushboo-rancher
• @mantissahz
• @roger-ryao
• @shuo-wu
• @smallteeths
• @weizhe0422
• @yangchiu

Longhorn v1.4.2-rc1 release

Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.4.2-rc1 released! 🎆

This release introduces improvements and bug fixes as described below about stability, performance, space efficiency, resilience, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Please ensure your Kubernetes cluster is at least v1.21 before installing v1.4.2.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please read the important notes first and ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.4.2 from v1.3.x/v1.4.x, which are only supported source versions.

The live upgrade from v1.3.x/v1.4.x is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

N/A

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

• [IMPROVEMENT] Use PDB to protect Longhorn components from unexpected drains (3304) - @yangchiu @PhanLe1010
• [IMPROVEMENT] Introduce timeout mechanism for the sparse file syncing service (4305) - @yangchiu @ChanYiLin
• [IMPROVEMENT] Recurring jobs create new snapshots while being not able to clean up old ones (4898) - @mantissahz @chriscchien

Improvement

• [IMPROVEMENT] Support bundle collects dmesg, syslog and related information of longhorn nodes (5073) - @weizhe0422 @roger-ryao
• [IMPROVEMENT] Fix BackingImage uploading/downloading flow to prevent client timeout (5443) - @ChanYiLin @chriscchien
• [IMPROVEMENT] Create a new setting so that Longhorn removes PDB for instance-manager-r that doesn't have any running instance inside it (5549) - @PhanLe1010 @khushboo-rancher
• [IMPROVEMENT] Deprecate the setting allow-node-drain-with-last-healthy-replica and replace it by node-drain-policy setting (5585) - @yangchiu @PhanLe1010
• [IMPROVEMENT][UI] Recurring jobs create new snapshots while being not able to clean up old one (5610) - @mantissahz @smallteeths @roger-ryao
• [IMPROVEMENT] Only activate replica if it doesn't have deletion timestamp during volume engine upgrade (5632) - @PhanLe1010 @roger-ryao
• [IMPROVEMENT] Clean up backup target if the backup target setting is unset (5655) - @ChanYiLin

Resilience

• [BUG] Directly mark replica as failed if the node is deleted (5542) - @weizhe0422
• [BUG] RWX volume is stuck at detaching when the attached node is down (5558) - @derekbit @roger-ryao
• [BUG] Resources such as replicas are somehow not mutated when network is unstable (5762) - @derekbit @roger-ryao
• [BUG] Instance manager may not update instance status for a minute after starting (5809) - @ejweber @roger-ryao

Bugs

• [BUG] Delete a uploading backing image, the corresponding LH temp file is not deleted (3682) - @ChanYiLin @chriscchien
• [BUG] Can not create backup in engine image not fully deployed cluster (5248) - @ChanYiLin @roger-ryao
• [BUG] Upgrade engine --> spec.restoreVolumeRecurringJob and spec.snapshotDataIntegrity Unsupported value (5485) - @yangchiu @derekbit
• [BUG] volume expansion starts for no reason, gets stuck on current size > expected size (5513) - @mantissahz
• [BUG] RWX volume attachment failed if tried more enough times (5537) - @yangchiu @derekbit
• [BUG] instance-manager-e emits Wait for process pvc-xxxx to shutdown constantly (5575) - @derekbit @roger-ryao
• [BUG] Support bundle kit should respect node selector & taint toleration (5614) - @yangchiu @c3y1huang
• [BUG] Value overlapped in page Instance Manager Image (5622) - @smallteeths @chriscchien
• [BUG] Backup monitor gets stuck in an infinite loop if backup isn't found (5662) - @derekbit @chriscchien
• [BUG] Instance manager PDB created with wrong selector thus blocking the draining of the wrongly selected node forever (5680) - @PhanLe1010 @chriscchien
• [BUG] During volume live engine upgrade, if the replica pod is killed, the volume is stuck in upgrading forever (5684) - @yangchiu @PhanLe1010
• [BUG] Instance manager PDBs cannot be removed if the longhorn-manager pod on its spec node is not available (5688) - @PhanLe1010 @roger-ryao
• [BUG] longhorn upgrade is not upgrading engineimage (5740) - @shuo-wu @chriscchien
• [BUG] test_replica_auto_balance_when_replica_on_unschedulable_node Error in creating volume with nodeSelector and dataLocality parameters (5745) - @c3y1huang @roger-ryao
• [BUG] Unable to backup volume after NFS server IP change (5856) - @derekbit @roger-ryao

Misc

• [TASK] Check and update the networking doc & example YAMLs (5651) - @yangchiu @shuo-wu

Contributors

• @ChanYiLin
• @PhanLe1010
• @c3y1huang
• @chriscchien
• @derekbit
• @ejweber
• @innobead
• @khushboo-rancher
• @mantissahz
• @roger-ryao
• @shuo-wu
• @smallteeths
• @weizhe0422
• @yangchiu

Longhorn v1.3.3 release

Release Note

v1.3.3 released! 🎆

This release introduces improvements and bug fixes as described below about stability, performance, space efficiency, resilience, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Please ensure your Kubernetes cluster is >= v1.18 and <= v1.24 before installing Longhorn v1.3.3.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please read the important notes first and ensure your Kubernetes cluster is >= v1.18 and <= v1.24 before upgrading to Longhorn v1.3.3 from v1.2.x or v1.3.x. Only support upgrading from v1.2.x and v1.3.x.

Follow the upgrade instructions here.

Deprecation & Incompatibilities

N/A

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

• [IMPROVEMENT] Use PDB to protect Longhorn components from unexpected drains (3304) - @yangchiu @PhanLe1010
• [IMPROVEMENT] Periodically clean up volume snapshots (3836) - @c3y1huang @chriscchien
• [IMPROVEMENT] Recurring jobs create new snapshots while being not able to clean up old ones (4898) - @mantissahz @chriscchien

Improvement

• [IMPROVEMENT] Change the script into a docker run command mentioned in 'recovery from longhorn backup without system installed' doc (1521) - @weizhe0422 @chriscchien
• [IMPROVEMENT] liveness and readiness probes with longhorn csi plugin daemonset (3907) - @c3y1huang @roger-ryao
• [IMPROVEMENT] Too many debug-level log messages in engine instance-manager (4427) - @derekbit @chriscchien
• [IMPROVEMENT] share-manager pod bypasses the kubernetes scheduler (4789) - @joshimoo @chriscchien
• [IMPROVEMENT] Unify the format of returned error messages in longhorn-engine (4828) - @derekbit
• [IMPROVEMENT] Affinity in the longhorn-ui deployment within the helm chart (4987) - @mantissahz @chriscchien
• [IMPROVEMENT] Upgrade tcmalloc in longhorn-engine (5050) - @derekbit
• [IMPROVEMENT] Fix Guaranteed Engine Manager CPU recommendation forumula in UI (5338) - @c3y1huang @smallteeths @roger-ryao
• [IMPROVEMENT] Set write-cache of longhorn block device to off explicitly (5382) - @derekbit @chriscchien
• [DOC] Update Kubernetes version info to have consistent description from the longhorn documentation in chart (5399) - @ChanYiLin @roger-ryao
• [IMPROVEMENT] Fix BackingImage uploading/downloading flow to prevent client timeout (5443) - @ChanYiLin @chriscchien
• [IMPROVEMENT] Create a new setting so that Longhorn removes PDB for instance-manager-r that doesn't have any running instance inside it (5549) - @PhanLe1010 @khushboo-rancher
• [IMPROVEMENT] Deprecate the setting allow-node-drain-with-last-healthy-replica and replace it by node-drain-policy setting (5585) - @PhanLe1010
• [IMPROVEMENT][UI] Recurring jobs create new snapshots while being not able to clean up old one (5610) - @mantissahz @smallteeths @roger-ryao
• [IMPROVEMENT] Only activate replica if it doesn't have deletion timestamp during volume engine upgrade (5632) - @PhanLe1010 @roger-ryao

Performance

• [TASK] Disable tcmalloc in data path because newer tcmalloc version leads to performance drop (5096) - @derekbit @chriscchien

Stability

• [BUG] Longhorn won't fail all replicas if there is no valid backend during the engine starting stage (1330) - @derekbit @roger-ryao
• [BUG] Engine binary cannot be recovered after being removed accidentally (4380) - @yangchiu @c3y1huang
• [BUG] volume is stuck in attaching/detaching loop with error Failed to init frontend: device... (4959) - @derekbit @PhanLe1010 @chriscchien
• [BUG] Memory leak in CSI plugin caused by stuck umount processes if the RWX volume is already gone (5296) - @derekbit @roger-ryao
• [BUG] share-manager pod failed to restart after kubelet restart (5507) - @yangchiu @derekbit
• [BUG] RWX volume is stuck at detaching when the attached node is down (5558) - @derekbit @roger-ryao

Bugs

• [BUG] Restoring volume stuck forever if the backup is already deleted. (1867) - @mantissahz @chriscchien
• [BUG] Duplicated default instance manager leads to engine/replica cannot be started (3000) - @PhanLe1010 @roger-ryao
• [BUG] Delete a uploading backing image, the corresponding LH temp file is not deleted (3682) - @ChanYiLin @chriscchien
• [BUG] Replica rebuilding failure with error "Replica must be closed, Can not add in state: open" (3828) - @mantissahz @roger-ryao
• [BUG] Max length of volume name not consist between frontend and backend (3917) - @weizhe0422 @roger-ryao
• [BUG] Can't delete volumesnapshot if backup removed first (4107) - @weizhe0422 @chriscchien
• [BUG] LH continuously reports invalid customized default setting taint-toleration (4554) - @weizhe0422 @roger-ryao
• [BUG] longhorn-engine integration test test_restore_to_file_with_backing_file failed after upgrade to sles 15.4 (4632) - @mantissahz
• [BUG] The old instance-manager-r Pods are not deleted after upgrade (4726) - @mantissahz @chriscchien
• [BUG] Replica Auto Balance repeatedly delete the local replica and trigger rebuilding (4761) - @c3y1huang @roger-ryao
• [BUG] Unable to reuse existing failed replica causes test case test_allow_volume_creation_with_degraded_availability_restore failed (4791) - @yangchiu @mantissahz
• [BUG] Volume metafile getting deleted or empty results in a detach-attach loop (4846) - @mantissahz @chriscchien
• [BUG] Backing image is stuck at in-progress status if the provided checksum is incorrect (4852) - @FrankYang0529 @chriscchien
• [BUG] Duplicate channel close error in the backing image manage related components (4865) - @weizhe0422 @roger-ryao
• [BUG] The node ID of backing image data source somehow get changed then lead to file handling failed (4887) - @shuo-wu @chriscchien
• [BUG] Cannot upload a backing image larger than 10G (4902) - @smallteeths @shuo-wu @chriscchien
• [BUG] System backup showing wrong age (5047) - @smallteeths @khushboo-rancher
• [BUG] Longhorn 1.3.2 fails to backup & restore volumes behind Internet proxy (5054) - @mantissahz @chriscchien
• [BUG] Sync up with backup target during DR volume activation (5292) - @yangchiu @weizhe0422
• [BUG] environment_check.sh does not handle differnt kernel versions in cluster correctly (5304) - @achims311 @roger-ryao
• [BUG] Replica rebuilding caused by rke2/kubelet restart (5340) - @derekbit @chriscchien
• [BUG] Error message not consistent between create/update recurring job when retain number greater than 50 (5434) - @c3y1huang @chriscchien
• [BUG] Do not copy Host header to API requests forwarded to Longhorn Manager (5438) - @yangchiu @smallteeths
• [BUG] test case test_backup_lock_deletion_during_restoration failed (5458) - @yangchiu @derekbit
• [BUG] Volume restoration will never complete if attached node is down (5464) - @derekbit @weizhe0422 @chriscchien
• [BUG] Physical node do...