Longhorn v1.3.3 release

Release Note

v1.3.3 released! 🎆

This release introduces improvements and bug fixes as described below about stability, performance, space efficiency, resilience, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Please ensure your Kubernetes cluster is >= v1.18 and <= v1.24 before installing Longhorn v1.3.3.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please read the important notes first and ensure your Kubernetes cluster is >= v1.18 and <= v1.24 before upgrading to Longhorn v1.3.3 from v1.2.x or v1.3.x. Only support upgrading from v1.2.x and v1.3.x.

Follow the upgrade instructions here.

Deprecation & Incompatibilities

N/A

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

• [IMPROVEMENT] Use PDB to protect Longhorn components from unexpected drains (3304) - @yangchiu @PhanLe1010
• [IMPROVEMENT] Periodically clean up volume snapshots (3836) - @c3y1huang @chriscchien
• [IMPROVEMENT] Recurring jobs create new snapshots while being not able to clean up old ones (4898) - @mantissahz @chriscchien

Improvement

• [IMPROVEMENT] Change the script into a docker run command mentioned in 'recovery from longhorn backup without system installed' doc (1521) - @weizhe0422 @chriscchien
• [IMPROVEMENT] liveness and readiness probes with longhorn csi plugin daemonset (3907) - @c3y1huang @roger-ryao
• [IMPROVEMENT] Too many debug-level log messages in engine instance-manager (4427) - @derekbit @chriscchien
• [IMPROVEMENT] share-manager pod bypasses the kubernetes scheduler (4789) - @joshimoo @chriscchien
• [IMPROVEMENT] Unify the format of returned error messages in longhorn-engine (4828) - @derekbit
• [IMPROVEMENT] Affinity in the longhorn-ui deployment within the helm chart (4987) - @mantissahz @chriscchien
• [IMPROVEMENT] Upgrade tcmalloc in longhorn-engine (5050) - @derekbit
• [IMPROVEMENT] Fix Guaranteed Engine Manager CPU recommendation forumula in UI (5338) - @c3y1huang @smallteeths @roger-ryao
• [IMPROVEMENT] Set write-cache of longhorn block device to off explicitly (5382) - @derekbit @chriscchien
• [DOC] Update Kubernetes version info to have consistent description from the longhorn documentation in chart (5399) - @ChanYiLin @roger-ryao
• [IMPROVEMENT] Fix BackingImage uploading/downloading flow to prevent client timeout (5443) - @ChanYiLin @chriscchien
• [IMPROVEMENT] Create a new setting so that Longhorn removes PDB for instance-manager-r that doesn't have any running instance inside it (5549) - @PhanLe1010 @khushboo-rancher
• [IMPROVEMENT] Deprecate the setting allow-node-drain-with-last-healthy-replica and replace it by node-drain-policy setting (5585) - @PhanLe1010
• [IMPROVEMENT][UI] Recurring jobs create new snapshots while being not able to clean up old one (5610) - @mantissahz @smallteeths @roger-ryao
• [IMPROVEMENT] Only activate replica if it doesn't have deletion timestamp during volume engine upgrade (5632) - @PhanLe1010 @roger-ryao

Performance

• [TASK] Disable tcmalloc in data path because newer tcmalloc version leads to performance drop (5096) - @derekbit @chriscchien

Stability

• [BUG] Longhorn won't fail all replicas if there is no valid backend during the engine starting stage (1330) - @derekbit @roger-ryao
• [BUG] Engine binary cannot be recovered after being removed accidentally (4380) - @yangchiu @c3y1huang
• [BUG] volume is stuck in attaching/detaching loop with error Failed to init frontend: device... (4959) - @derekbit @PhanLe1010 @chriscchien
• [BUG] Memory leak in CSI plugin caused by stuck umount processes if the RWX volume is already gone (5296) - @derekbit @roger-ryao
• [BUG] share-manager pod failed to restart after kubelet restart (5507) - @yangchiu @derekbit
• [BUG] RWX volume is stuck at detaching when the attached node is down (5558) - @derekbit @roger-ryao

Bugs

• [BUG] Restoring volume stuck forever if the backup is already deleted. (1867) - @mantissahz @chriscchien
• [BUG] Duplicated default instance manager leads to engine/replica cannot be started (3000) - @PhanLe1010 @roger-ryao
• [BUG] Delete a uploading backing image, the corresponding LH temp file is not deleted (3682) - @ChanYiLin @chriscchien
• [BUG] Replica rebuilding failure with error "Replica must be closed, Can not add in state: open" (3828) - @mantissahz @roger-ryao
• [BUG] Max length of volume name not consist between frontend and backend (3917) - @weizhe0422 @roger-ryao
• [BUG] Can't delete volumesnapshot if backup removed first (4107) - @weizhe0422 @chriscchien
• [BUG] LH continuously reports invalid customized default setting taint-toleration (4554) - @weizhe0422 @roger-ryao
• [BUG] longhorn-engine integration test test_restore_to_file_with_backing_file failed after upgrade to sles 15.4 (4632) - @mantissahz
• [BUG] The old instance-manager-r Pods are not deleted after upgrade (4726) - @mantissahz @chriscchien
• [BUG] Replica Auto Balance repeatedly delete the local replica and trigger rebuilding (4761) - @c3y1huang @roger-ryao
• [BUG] Unable to reuse existing failed replica causes test case test_allow_volume_creation_with_degraded_availability_restore failed (4791) - @yangchiu @mantissahz
• [BUG] Volume metafile getting deleted or empty results in a detach-attach loop (4846) - @mantissahz @chriscchien
• [BUG] Backing image is stuck at in-progress status if the provided checksum is incorrect (4852) - @FrankYang0529 @chriscchien
• [BUG] Duplicate channel close error in the backing image manage related components (4865) - @weizhe0422 @roger-ryao
• [BUG] The node ID of backing image data source somehow get changed then lead to file handling failed (4887) - @shuo-wu @chriscchien
• [BUG] Cannot upload a backing image larger than 10G (4902) - @smallteeths @shuo-wu @chriscchien
• [BUG] System backup showing wrong age (5047) - @smallteeths @khushboo-rancher
• [BUG] Longhorn 1.3.2 fails to backup & restore volumes behind Internet proxy (5054) - @mantissahz @chriscchien
• [BUG] Sync up with backup target during DR volume activation (5292) - @yangchiu @weizhe0422
• [BUG] environment_check.sh does not handle differnt kernel versions in cluster correctly (5304) - @achims311 @roger-ryao
• [BUG] Replica rebuilding caused by rke2/kubelet restart (5340) - @derekbit @chriscchien
• [BUG] Error message not consistent between create/update recurring job when retain number greater than 50 (5434) - @c3y1huang @chriscchien
• [BUG] Do not copy Host header to API requests forwarded to Longhorn Manager (5438) - @yangchiu @smallteeths
• [BUG] test case test_backup_lock_deletion_during_restoration failed (5458) - @yangchiu @derekbit
• [BUG] Volume restoration will never complete if attached node is down (5464) - @derekbit @weizhe0422 @chriscchien
• [BUG] Physical node do...

Longhorn v1.3.3-rc3 release

Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.3.3-rc3 released! 🎆

This release introduces improvements and bug fixes as described below about stability, performance, space efficiency, resilience, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Please ensure your Kubernetes cluster is >= v1.18 and <= v1.24 before installing Longhorn v1.3.3.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please ensure your Kubernetes cluster is >= v1.18 and <= v1.24 before upgrading to Longhorn v1.3.3 from v1.2.x or v1.3.x. Only support upgrading from v1.2.x and v1.3.x.

The live upgrade from v1.2.x/v1.3.x is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

N/A

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

• [IMPROVEMENT] Use PDB to protect Longhorn components from unexpected drains (3304) - @yangchiu @PhanLe1010
• [IMPROVEMENT] Periodically clean up volume snapshots (3836) - @c3y1huang @chriscchien
• [IMPROVEMENT] Recurring jobs create new snapshots while being not able to clean up old ones (4898) - @mantissahz @chriscchien

Improvement

• [IMPROVEMENT] Change the script into a docker run command mentioned in 'recovery from longhorn backup without system installed' doc (1521) - @weizhe0422 @chriscchien
• [IMPROVEMENT] liveness and readiness probes with longhorn csi plugin daemonset (3907) - @c3y1huang @roger-ryao
• [IMPROVEMENT] Too many debug-level log messages in engine instance-manager (4427) - @derekbit @chriscchien
• [IMPROVEMENT] share-manager pod bypasses the kubernetes scheduler (4789) - @joshimoo @chriscchien
• [IMPROVEMENT] Unify the format of returned error messages in longhorn-engine (4828) - @derekbit
• [IMPROVEMENT] Affinity in the longhorn-ui deployment within the helm chart (4987) - @mantissahz @chriscchien
• [IMPROVEMENT] Upgrade tcmalloc in longhorn-engine (5050) - @derekbit
• [IMPROVEMENT] Fix Guaranteed Engine Manager CPU recommendation forumula in UI (5338) - @c3y1huang @smallteeths @roger-ryao
• [IMPROVEMENT] Set write-cache of longhorn block device to off explicitly (5382) - @derekbit @chriscchien
• [DOC] Update Kubernetes version info to have consistent description from the longhorn documentation in chart (5399) - @ChanYiLin @roger-ryao
• [IMPROVEMENT] Fix BackingImage uploading/downloading flow to prevent client timeout (5443) - @ChanYiLin @chriscchien
• [IMPROVEMENT] Create a new setting so that Longhorn removes PDB for instance-manager-r that doesn't have any running instance inside it (5549) - @PhanLe1010 @khushboo-rancher
• [IMPROVEMENT][UI] Recurring jobs create new snapshots while being not able to clean up old one (5610) - @mantissahz @smallteeths @roger-ryao
• [IMPROVEMENT] Only activate replica if it doesn't have deletion timestamp during volume engine upgrade (5632) - @PhanLe1010 @roger-ryao

Performance

• [TASK] Disable tcmalloc in data path because newer tcmalloc version leads to performance drop (5096) - @derekbit @chriscchien

Stability

• [BUG] Longhorn won't fail all replicas if there is no valid backend during the engine starting stage (1330) - @derekbit @roger-ryao
• [BUG] Engine binary cannot be recovered after being removed accidentally (4380) - @yangchiu @c3y1huang
• [BUG] volume is stuck in attaching/detaching loop with error Failed to init frontend: device... (4959) - @derekbit @PhanLe1010 @chriscchien
• [BUG] Memory leak in CSI plugin caused by stuck umount processes if the RWX volume is already gone (5296) - @derekbit @roger-ryao
• [BUG] share-manager pod failed to restart after kubelet restart (5507) - @yangchiu @derekbit
• [BUG] RWX volume is stuck at detaching when the attached node is down (5558) - @derekbit @roger-ryao

Bugs

• [BUG] Restoring volume stuck forever if the backup is already deleted. (1867) - @mantissahz @chriscchien
• [BUG] Duplicated default instance manager leads to engine/replica cannot be started (3000) - @PhanLe1010 @roger-ryao
• [BUG] Delete a uploading backing image, the corresponding LH temp file is not deleted (3682) - @ChanYiLin @chriscchien
• [BUG] Replica rebuilding failure with error "Replica must be closed, Can not add in state: open" (3828) - @mantissahz @roger-ryao
• [BUG] Max length of volume name not consist between frontend and backend (3917) - @weizhe0422 @roger-ryao
• [BUG] Can't delete volumesnapshot if backup removed first (4107) - @weizhe0422 @chriscchien
• [BUG] LH continuously reports invalid customized default setting taint-toleration (4554) - @weizhe0422 @roger-ryao
• [BUG] longhorn-engine integration test test_restore_to_file_with_backing_file failed after upgrade to sles 15.4 (4632) - @mantissahz
• [BUG] The old instance-manager-r Pods are not deleted after upgrade (4726) - @mantissahz @chriscchien
• [BUG] Replica Auto Balance repeatedly delete the local replica and trigger rebuilding (4761) - @c3y1huang @roger-ryao
• [BUG] Unable to reuse existing failed replica causes test case test_allow_volume_creation_with_degraded_availability_restore failed (4791) - @yangchiu @mantissahz
• [BUG] Volume metafile getting deleted or empty results in a detach-attach loop (4846) - @mantissahz @chriscchien
• [BUG] Backing image is stuck at in-progress status if the provided checksum is incorrect (4852) - @FrankYang0529 @chriscchien
• [BUG] Duplicate channel close error in the backing image manage related components (4865) - @weizhe0422 @roger-ryao
• [BUG] The node ID of backing image data source somehow get changed then lead to file handling failed (4887) - @shuo-wu @chriscchien
• [BUG] Cannot upload a backing image larger than 10G (4902) - @smallteeths @shuo-wu @chriscchien
• [BUG] System backup showing wrong age (5047) - @smallteeths @khushboo-rancher
• [BUG] Longhorn 1.3.2 fails to backup & restore volumes behind Internet proxy (5054) - @mantissahz @chriscchien
• [BUG] Sync up with backup target during DR volume activation (5292) - @yangchiu @weizhe0422
• [BUG] environment_check.sh does not handle differnt kernel versions in cluster correctly (5304) - @achims311 @roger-ryao
• [BUG] Replica rebuilding caused by rke2/kubelet restart (5340) - @derekbit @chriscchien
• [BUG] Error message not consistent between create/update recurring job when retain number greater than 50 (5434) - @c3y1huang @chriscchien
• [BUG] Do not copy Host header to API requests forwarded to Longhorn Manager (5438) - @yangchiu @smallteeths
• [BUG] test case test_backup_lock_deletion_during_restoration failed (5458) - @yangchiu @derekbit
• [BUG] Volume restoration will never complete if attached node is down (5464) - @derekbit @weizhe0422 @chriscchien
• [BUG] Physical node down test failed (5477) - @derekbit @chriscchien
  ...

Longhorn v1.3.3-rc2 release

Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.3.3-rc2 released! 🎆

This release introduces improvements and bug fixes as described below about stability, performance, space efficiency, resilience, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Please ensure your Kubernetes cluster is >= v1.18 and <= v1.24 before installing Longhorn v1.3.3.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please ensure your Kubernetes cluster is >= v1.18 and <= v1.24 before upgrading to Longhorn v1.3.3 from v1.2.x or v1.3.x. Only support upgrading from v1.2.x and v1.3.x.

The live upgrade from v1.2.x/v1.3.x is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

N/A

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

• [IMPROVEMENT] Use PDB to protect Longhorn components from unexpected drains (3304) - @PhanLe1010
• [IMPROVEMENT] Periodically clean up volume snapshots (3836) - @c3y1huang @chriscchien
• [IMPROVEMENT] Recurring jobs create new snapshots while being not able to clean up old ones (4898) - @mantissahz @chriscchien

Improvement

• [IMPROVEMENT] Change the script into a docker run command mentioned in 'recovery from longhorn backup without system installed' doc (1521) - @weizhe0422 @chriscchien
• [IMPROVEMENT] liveness and readiness probes with longhorn csi plugin daemonset (3907) - @c3y1huang @roger-ryao
• [IMPROVEMENT] Too many debug-level log messages in engine instance-manager (4427) - @derekbit @chriscchien
• [IMPROVEMENT] share-manager pod bypasses the kubernetes scheduler (4789) - @joshimoo @chriscchien
• [IMPROVEMENT] Unify the format of returned error messages in longhorn-engine (4828) - @derekbit
• [IMPROVEMENT] Affinity in the longhorn-ui deployment within the helm chart (4987) - @mantissahz @chriscchien
• [IMPROVEMENT] Upgrade tcmalloc in longhorn-engine (5050) - @derekbit
• [IMPROVEMENT] Fix Guaranteed Engine Manager CPU recommendation forumula in UI (5338) - @c3y1huang @smallteeths @roger-ryao
• [IMPROVEMENT] Set write-cache of longhorn block device to off explicitly (5382) - @derekbit @chriscchien
• [DOC] Update Kubernetes version info to have consistent description from the longhorn documentation in chart (5399) - @ChanYiLin @roger-ryao
• [IMPROVEMENT] Fix BackingImage uploading/downloading flow to prevent client timeout (5443) - @ChanYiLin @chriscchien
• [IMPROVEMENT] Create a new setting so that Longhorn removes PDB for instance-manager-r that doesn't have any running instance inside it (5549) - @PhanLe1010 @khushboo-rancher
• [IMPROVEMENT][UI] Recurring jobs create new snapshots while being not able to clean up old one (5610) - @mantissahz @smallteeths @roger-ryao
• [IMPROVEMENT] Only active replica if it doesn't have deletion timestamp (5632) - @PhanLe1010

Performance

• [TASK] Disable tcmalloc in data path because newer tcmalloc version leads to performance drop (5096) - @derekbit @chriscchien

Stability

• [BUG] Longhorn won't fail all replicas if there is no valid backend during the engine starting stage (1330) - @derekbit @roger-ryao
• [BUG] Engine binary cannot be recovered after being removed accidentally (4380) - @yangchiu @c3y1huang
• [BUG] volume is stuck in attaching/detaching loop with error Failed to init frontend: device... (4959) - @derekbit @PhanLe1010 @chriscchien
• [BUG] Memory leak in CSI plugin caused by stuck umount processes if the RWX volume is already gone (5296) - @derekbit @roger-ryao
• [BUG] share-manager pod failed to restart after kubelet restart (5507) - @yangchiu @derekbit
• [BUG] RWX volume is stuck at detaching when the attached node is down (5558) - @derekbit @roger-ryao

Bugs

• [BUG] Restoring volume stuck forever if the backup is already deleted. (1867) - @mantissahz @chriscchien
• [BUG] Duplicated default instance manager leads to engine/replica cannot be started (3000) - @PhanLe1010 @roger-ryao
• [BUG] Delete a uploading backing image, the corresponding LH temp file is not deleted (3682) - @ChanYiLin @chriscchien
• [BUG] Replica rebuilding failure with error "Replica must be closed, Can not add in state: open" (3828) - @mantissahz @roger-ryao
• [BUG] Max length of volume name not consist between frontend and backend (3917) - @weizhe0422 @roger-ryao
• [BUG] Can't delete volumesnapshot if backup removed first (4107) - @weizhe0422 @chriscchien
• [BUG] LH continuously reports invalid customized default setting taint-toleration (4554) - @weizhe0422 @roger-ryao
• [BUG] longhorn-engine integration test test_restore_to_file_with_backing_file failed after upgrade to sles 15.4 (4632) - @mantissahz
• [BUG] The old instance-manager-r Pods are not deleted after upgrade (4726) - @mantissahz @chriscchien
• [BUG] Replica Auto Balance repeatedly delete the local replica and trigger rebuilding (4761) - @c3y1huang @roger-ryao
• [BUG] Unable to reuse existing failed replica causes test case test_allow_volume_creation_with_degraded_availability_restore failed (4791) - @yangchiu @mantissahz
• [BUG] Volume metafile getting deleted or empty results in a detach-attach loop (4846) - @mantissahz @chriscchien
• [BUG] Backing image is stuck at in-progress status if the provided checksum is incorrect (4852) - @FrankYang0529 @chriscchien
• [BUG] Duplicate channel close error in the backing image manage related components (4865) - @weizhe0422 @roger-ryao
• [BUG] The node ID of backing image data source somehow get changed then lead to file handling failed (4887) - @shuo-wu @chriscchien
• [BUG] Cannot upload a backing image larger than 10G (4902) - @smallteeths @shuo-wu @chriscchien
• [BUG] System backup showing wrong age (5047) - @smallteeths @khushboo-rancher
• [BUG] Longhorn 1.3.2 fails to backup & restore volumes behind Internet proxy (5054) - @mantissahz @chriscchien
• [BUG] Sync up with backup target during DR volume activation (5292) - @yangchiu @weizhe0422
• [BUG] environment_check.sh does not handle differnt kernel versions in cluster correctly (5304) - @achims311 @roger-ryao
• [BUG] Replica rebuilding caused by rke2/kubelet restart (5340) - @derekbit @chriscchien
• [BUG] Error message not consistent between create/update recurring job when retain number greater than 50 (5434) - @c3y1huang @chriscchien
• [BUG] Do not copy Host header to API requests forwarded to Longhorn Manager (5438) - @yangchiu @smallteeths
• [BUG] test case test_backup_lock_deletion_during_restoration failed (5458) - @yangchiu @derekbit
• [BUG] Volume restoration will never complete if attached node is down (5464) - @derekbit @weizhe0422 @chriscchien
• [BUG] Physical node down test failed (5477) - @derekbit @chriscchien
• [BUG] Backing image with sync failure ([5481](http...

Longhorn v1.3.3-rc1 release

Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.3.3-rc1 released! 🎆

This release introduces improvements and bug fixes as described below about stability, performance, space efficiency, resilience, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Please ensure your Kubernetes cluster is >= v1.18 and <= v1.24 before installing Longhorn v1.3.3.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please ensure your Kubernetes cluster is >= v1.18 and <= v1.24 before upgrading to Longhorn v1.3.3 from v1.2.x or v1.3.x. Only support upgrading from v1.2.x and v1.3.x.

The live upgrade from v1.2.x/v1.3.x is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

N/A

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

• [IMPROVEMENT] Use PDB to protect Longhorn components from unexpected drains (3304) - @PhanLe1010
• [IMPROVEMENT] Periodically clean up volume snapshots (3836) - @c3y1huang @chriscchien
• [IMPROVEMENT] Recurring jobs create new snapshots while being not able to clean up old ones (4898) - @mantissahz @chriscchien

Improvement

• [IMPROVEMENT] Change the script into a docker run command mentioned in 'recovery from longhorn backup without system installed' doc (1521) - @weizhe0422 @chriscchien
• [IMPROVEMENT] liveness and readiness probes with longhorn csi plugin daemonset (3907) - @c3y1huang @roger-ryao
• [IMPROVEMENT] Too many debug-level log messages in engine instance-manager (4427) - @derekbit @chriscchien
• [IMPROVEMENT] share-manager pod bypasses the kubernetes scheduler (4789) - @joshimoo @chriscchien
• [IMPROVEMENT] Unify the format of returned error messages in longhorn-engine (4828) - @derekbit
• [IMPROVEMENT] Affinity in the longhorn-ui deployment within the helm chart (4987) - @mantissahz @chriscchien
• [IMPROVEMENT] Upgrade tcmalloc in longhorn-engine (5050) - @derekbit
• [IMPROVEMENT] Fix Guaranteed Engine Manager CPU recommendation forumula in UI (5338) - @c3y1huang @smallteeths @roger-ryao
• [IMPROVEMENT] Set write-cache of longhorn block device to off explicitly (5382) - @derekbit @chriscchien
• [DOC] Update Kubernetes version info to have consistent description from the longhorn documentation in chart (5399) - @ChanYiLin @roger-ryao
• [IMPROVEMENT] Fix BackingImage uploading/downloading flow to prevent client timeout (5443) - @ChanYiLin @chriscchien
• [IMPROVEMENT] Create a new setting so that Longhorn removes PDB for instance-manager-r that doesn't have any running instance inside it (5549) - @PhanLe1010 @khushboo-rancher

Performance

• [TASK] Disable tcmalloc in data path because newer tcmalloc version leads to performance drop (5096) - @derekbit @chriscchien

Stability

• [BUG] Longhorn won't fail all replicas if there is no valid backend during the engine starting stage (1330) - @derekbit @roger-ryao
• [BUG] Engine binary cannot be recovered after being removed accidentally (4380) - @yangchiu @c3y1huang
• [BUG] volume is stuck in attaching/detaching loop with error Failed to init frontend: device... (4959) - @derekbit @PhanLe1010 @chriscchien
• [BUG] Memory leak in CSI plugin caused by stuck umount processes if the RWX volume is already gone (5296) - @derekbit @roger-ryao
• [BUG] share-manager pod failed to restart after kubelet restart (5507) - @yangchiu @derekbit
• [BUG] RWX volume is stuck at detaching when the attached node is down (5558) - @derekbit @roger-ryao

Bugs

• [BUG] Restoring volume stuck forever if the backup is already deleted. (1867) - @mantissahz @chriscchien
• [BUG] Duplicated default instance manager leads to engine/replica cannot be started (3000) - @PhanLe1010 @roger-ryao
• [BUG] Delete a uploading backing image, the corresponding LH temp file is not deleted (3682) - @ChanYiLin @chriscchien
• [BUG] Replica rebuilding failure with error "Replica must be closed, Can not add in state: open" (3828) - @mantissahz @roger-ryao
• [BUG] Max length of volume name not consist between frontend and backend (3917) - @weizhe0422 @roger-ryao
• [BUG] Can't delete volumesnapshot if backup removed first (4107) - @weizhe0422 @chriscchien
• [BUG] LH continuously reports invalid customized default setting taint-toleration (4554) - @weizhe0422 @roger-ryao
• [BUG] longhorn-engine integration test test_restore_to_file_with_backing_file failed after upgrade to sles 15.4 (4632) - @mantissahz
• [BUG] The old instance-manager-r Pods are not deleted after upgrade (4726) - @mantissahz @chriscchien
• [BUG] Replica Auto Balance repeatedly delete the local replica and trigger rebuilding (4761) - @c3y1huang @roger-ryao
• [BUG] Unable to reuse existing failed replica causes test case test_allow_volume_creation_with_degraded_availability_restore failed (4791) - @yangchiu @mantissahz
• [BUG] Volume metafile getting deleted or empty results in a detach-attach loop (4846) - @mantissahz @chriscchien
• [BUG] Backing image is stuck at in-progress status if the provided checksum is incorrect (4852) - @FrankYang0529 @chriscchien
• [BUG] Duplicate channel close error in the backing image manage related components (4865) - @weizhe0422 @roger-ryao
• [BUG] The node ID of backing image data source somehow get changed then lead to file handling failed (4887) - @shuo-wu @chriscchien
• [BUG] Cannot upload a backing image larger than 10G (4902) - @smallteeths @shuo-wu @chriscchien
• [BUG] System backup showing wrong age (5047) - @smallteeths @khushboo-rancher
• [BUG] Longhorn 1.3.2 fails to backup & restore volumes behind Internet proxy (5054) - @mantissahz @chriscchien
• [BUG] Sync up with backup target during DR volume activation (5292) - @yangchiu @weizhe0422
• [BUG] environment_check.sh does not handle differnt kernel versions in cluster correctly (5304) - @achims311 @roger-ryao
• [BUG] Replica rebuilding caused by rke2/kubelet restart (5340) - @derekbit @chriscchien
• [BUG] Error message not consistent between create/update recurring job when retain number greater than 50 (5434) - @c3y1huang @chriscchien
• [BUG] Do not copy Host header to API requests forwarded to Longhorn Manager (5438) - @yangchiu @smallteeths
• [BUG] test case test_backup_lock_deletion_during_restoration failed (5458) - @yangchiu @derekbit
• [BUG] [master] Volume restoration will never complete if attached node is down (5464) - @derekbit @weizhe0422 @chriscchien
• [BUG] Physical node down test failed (5477) - @derekbit @chriscchien
• [BUG] Backing image with sync failure (5481) - @ChanYiLin @roger-ryao
• [BUG] Example of data migration doesn't work for hidden/./dot-files) (5484) - @hedefalk @shuo-wu @chriscchien
• [BUG] test case test_dr_volume_with_backup_block_deletion failed ([5489](https://github.com/lon...

Longhorn v1.4.1 release

Release Note

v1.4.1 released! 🎆

This release introduces improvements and bug fixes as described below about stability, performance, space efficiency, resilience, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Please ensure your Kubernetes cluster is at least v1.21 before installing Longhorn v1.4.1.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.4.1 from v1.3.x/v1.4.0, which are only supported source versions.

Follow the upgrade instructions here.

Deprecation & Incompatibilities

N/A

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

• [IMPROVEMENT] Periodically clean up volume snapshots (3836) - @c3y1huang @chriscchien

Improvement

• [IMPROVEMENT] Do not count the failure replica reuse failure caused by the disconnection (1923) - @yangchiu @mantissahz
• [IMPROVEMENT] Update uninstallation info to include the 'Deleting Confirmation Flag' in chart (5250) - @PhanLe1010 @roger-ryao
• [IMPROVEMENT] Fix Guaranteed Engine Manager CPU recommendation forumula in UI (5338) - @c3y1huang @smallteeths @roger-ryao
• [IMPROVEMENT] Update PSP validation in the Longhorn upstream chart (5339) - @yangchiu @PhanLe1010
• [IMPROVEMENT] Update ganesha nfs to 4.2.3 (5356) - @derekbit @roger-ryao
• [IMPROVEMENT] Set write-cache of longhorn block device to off explicitly (5382) - @derekbit @chriscchien

Stability

• [BUG] Memory leak in CSI plugin caused by stuck umount processes if the RWX volume is already gone (5296) - @derekbit @roger-ryao
• [BUG] share-manager pod failed to restart after kubelet restart (5507) - @yangchiu @derekbit

Bugs

• [BUG] Longhorn 1.3.2 fails to backup & restore volumes behind Internet proxy (5054) - @mantissahz @chriscchien
• [BUG] RWX doesn't work with release 1.4.0 due to end grace update error from recovery backend (5183) - @derekbit @chriscchien
• [BUG] Incorrect indentation of charts/questions.yaml (5196) - @mantissahz @roger-ryao
• [BUG] Updating option "Allow snapshots removal during trim" for old volumes failed (5218) - @shuo-wu @roger-ryao
• [BUG] Incorrect router retry mechanism (5259) - @mantissahz @chriscchien
• [BUG] System Backup is stuck at Uploading if there are PVs not provisioned by CSI driver (5286) - @c3y1huang @chriscchien
• [BUG] Sync up with backup target during DR volume activation (5292) - @yangchiu @weizhe0422
• [BUG] environment_check.sh does not handle differnt kernel versions in cluster correctly (5304) - @achims311 @roger-ryao
• [BUG] instance-manager-r high memory consumption (5312) - @derekbit @roger-ryao
• [BUG] Replica rebuilding caused by rke2/kubelet restart (5340) - @derekbit @chriscchien
• [BUG] Error message not consistent between create/update recurring job when retain number greater than 50 (5434) - @c3y1huang @chriscchien
• [BUG] Do not copy Host header to API requests forwarded to Longhorn Manager (5438) - @yangchiu @smallteeths
• [BUG] RWX Volume attachment is getting Failed (5456) - @derekbit
• [BUG] test case test_backup_lock_deletion_during_restoration failed (5458) - @yangchiu @derekbit
• [BUG] [master] [v1.4.1-rc1] Volume restoration will never complete if attached node is down (5464) - @derekbit @weizhe0422 @chriscchien
• [BUG] Unable to create support bundle agent pod in air-gap environment (5467) - @yangchiu @c3y1huang
• [BUG] Node disconnection test failed (5476) - @yangchiu @derekbit
• [BUG] Physical node down test failed (5477) - @derekbit @chriscchien
• [BUG] Backing image with sync failure (5481) - @ChanYiLin @roger-ryao
• [BUG] Example of data migration doesn't work for hidden/./dot-files) (5484) - @hedefalk @shuo-wu @chriscchien
• [BUG] test case test_dr_volume_with_backup_block_deletion failed (5489) - @yangchiu @derekbit

Misc

• [TASK][UI] add new recurring job tasks (5272) - @smallteeths @chriscchien

Contributors

• @ChanYiLin
• @PhanLe1010
• @achims311
• @c3y1huang
• @chriscchien
• @derekbit
• @hedefalk
• @innobead
• @mantissahz
• @roger-ryao
• @shuo-wu
• @smallteeths
• @weizhe0422
• @yangchiu

Longhorn v1.4.1-rc2 release

Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.4.1-rc2 released! 🎆

This release introduces improvements and bug fixes as described below about stability, performance, space efficiency, resilience, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Please ensure your Kubernetes cluster is at least v1.21 before installing Longhorn v1.4.1.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.4.1 from v1.3.x/v1.4.0, which are only supported source versions.

The live upgrade from v1.3.x/v1.4.0 is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

N/A

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

• [IMPROVEMENT] Periodically clean up volume snapshots (3836) - @c3y1huang @chriscchien

Improvement

• [IMPROVEMENT] Do not count the failure replica reuse failure caused by the disconnection (1923) - @yangchiu @mantissahz
• [IMPROVEMENT] Update uninstallation info to include the 'Deleting Confirmation Flag' in chart (5250) - @PhanLe1010 @roger-ryao
• [IMPROVEMENT] Fix Guaranteed Engine Manager CPU recommendation forumula in UI (5338) - @c3y1huang @smallteeths @roger-ryao
• [IMPROVEMENT] Update PSP validation in the Longhorn upstream chart (5339) - @yangchiu @PhanLe1010
• [IMPROVEMENT] Update ganesha nfs to 4.2.3 (5356) - @derekbit @roger-ryao
• [IMPROVEMENT] Set write-cache of longhorn block device to off explicitly (5382) - @derekbit @chriscchien

Stability

• [BUG] Memory leak in CSI plugin caused by stuck umount processes if the RWX volume is already gone (5296) - @derekbit @roger-ryao

Bugs

• [BUG] Longhorn 1.3.2 fails to backup & restore volumes behind Internet proxy (5054) - @mantissahz @chriscchien
• [BUG] RWX doesn't work with release 1.4.0 due to end grace update error from recovery backend (5183) - @derekbit @chriscchien
• [BUG] Incorrect indentation of charts/questions.yaml (5196) - @mantissahz @roger-ryao
• [BUG] Updating option "Allow snapshots removal during trim" for old volumes failed (5218) - @shuo-wu @roger-ryao
• [BUG] Incorrect router retry mechanism (5259) - @mantissahz @roger-ryao
• [BUG] System Backup is stuck at Uploading if there are PVs not provisioned by CSI driver (5286) - @c3y1huang @chriscchien
• [BUG] Sync up with backup target during DR volume activation (5292) - @yangchiu @weizhe0422
• [BUG] environment_check.sh does not handle differnt kernel versions in cluster correctly (5304) - @achims311 @roger-ryao
• [BUG] instance-manager-r high memory consumption (5312) - @derekbit @roger-ryao
• [BUG] Replica rebuilding caused by rke2/kubelet restart (5340) - @derekbit @chriscchien
• [BUG] Error message not consistent between create/update recurring job when retain number greater than 50 (5434) - @c3y1huang @chriscchien
• [BUG] Do not copy Host header to API requests forwarded to Longhorn Manager (5438) - @yangchiu @smallteeths
• [BUG][v1.4.x] test case test_backup_lock_deletion_during_restoration failed (5458) - @derekbit
• [BUG] [master] [v1.4.1-rc1] Volume restoration will never complete if attached node is down (5464) - @derekbit @weizhe0422 @chriscchien
• [BUG] Unable to create support bundle agent pod in air-gap environment (5467) - @yangchiu @c3y1huang
• [BUG][v1.4.1-rc1] Node disconnection test failed (5476) - @yangchiu @derekbit
• [BUG] [v1.4.1-rc1] Physical node down test failed (5477) - @derekbit @chriscchien
• [BUG][v1.4.1-rc1] Backing image with sync failure (5481) - @ChanYiLin @roger-ryao
• [BUG] test case test_dr_volume_with_backup_block_deletion failed (5489) - @yangchiu @derekbit

Misc

• [TASK][UI] add new recurring job tasks (5272) - @smallteeths @chriscchien

Contributors

• @ChanYiLin
• @PhanLe1010
• @achims311
• @c3y1huang
• @chriscchien
• @derekbit
• @innobead
• @mantissahz
• @roger-ryao
• @shuo-wu
• @smallteeths
• @weizhe0422
• @yangchiu

Longhorn v1.4.1-rc1 release

Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.4.1-rc1 released! 🎆

This release introduces improvements and bug fixes as described below about stability, performance, space efficiency, resilience, and so on. Please try it and feedback. Thanks for all the contributions!

Installation

Please ensure your Kubernetes cluster is at least v1.21 before installing Longhorn v1.4.1.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.4.1 from v1.3.x/v1.4.0, which are only supported source versions.

The live upgrade from v1.3.x/v1.4.0 is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

N/A

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

• [IMPROVEMENT] Periodically clean up volume snapshots (3836) - @c3y1huang @chriscchien

Improvement

• [IMPROVEMENT] Do not count the failure replica reuse failure caused by the disconnection (1923) - @yangchiu @mantissahz
• [IMPROVEMENT] Update uninstallation info to include the 'Deleting Confirmation Flag' in chart (5250) - @PhanLe1010 @roger-ryao
• [IMPROVEMENT] Fix Guaranteed Engine Manager CPU recommendation forumula in UI (5338) - @c3y1huang @smallteeths @roger-ryao
• [IMPROVEMENT] Update PSP validation in the Longhorn upstream chart (5339) - @yangchiu @PhanLe1010
• [IMPROVEMENT] Update ganesha nfs to 4.2.3 (5356) - @derekbit @roger-ryao
• [IMPROVEMENT] Set write-cache of longhorn block device to off explicitly (5382) - @derekbit @chriscchien

Stability

• [BUG] Memory leak in CSI plugin caused by stuck umount processes if the RWX volume is already gone (5296) - @derekbit @roger-ryao

Bugs

• [BUG] Longhorn 1.3.2 fails to backup & restore volumes behind Internet proxy (5054) - @mantissahz @chriscchien
• [BUG] RWX doesn't work with release 1.4.0 due to end grace update error from recovery backend (5183) - @derekbit @chriscchien
• [BUG] Incorrect indentation of charts/questions.yaml (5196) - @mantissahz @roger-ryao
• [BUG] Updating option "Allow snapshots removal during trim" for old volumes failed (5218) - @shuo-wu @roger-ryao
• [BUG] Incorrect router retry mechanism (5259) - @mantissahz @roger-ryao
• [BUG] System Backup is stuck at Uploading if there are PVs not provisioned by CSI driver (5286) - @c3y1huang @chriscchien
• [BUG] Sync up with backup target during DR volume activation (5292) - @yangchiu @weizhe0422
• [BUG] environment_check.sh does not handle differnt kernel versions in cluster correctly (5304) - @achims311 @roger-ryao
• [BUG] instance-manager-r high memory consumption (5312) - @derekbit @roger-ryao
• [BUG] Replica rebuilding caused by rke2/kubelet restart (5340) - @derekbit @chriscchien
• [BUG] Error message not consistent between create/update recurring job when retain number greater than 50 (5434) - @c3y1huang @chriscchien
• [BUG] Do not copy Host header to API requests forwarded to Longhorn Manager (5438) - @yangchiu @smallteeths

Misc

• [TASK][UI] add new recurring job tasks (5272) - @smallteeths @chriscchien

Contributors

• @PhanLe1010
• @achims311
• @c3y1huang
• @chriscchien
• @derekbit
• @innobead
• @mantissahz
• @roger-ryao
• @shuo-wu
• @smallteeths
• @weizhe0422
• @yangchiu

Longhorn v1.4.0 release

Release Note

v1.4.0 released! 🎆

This release introduces many enhancements, improvements, and bug fixes as described below about stability, performance, data integrity, troubleshooting, and so on. Please try it and feedback. Thanks for all the contributions!

• Kubernetes 1.25 Support [doc]
  In the previous versions, Longhorn relies on Pod Security Policy (PSP) to authorize Longhorn components for privileged operations. From Kubernetes 1.25, PSP has been removed and replaced with Pod Security Admission (PSA). Longhorn v1.4.0 supports opt-in PSP enablement, so it can support Kubernetes versions with or without PSP.

• ARM64 GA
  ARM64 has been experimental from Longhorn v1.1.0. After receiving more user feedback and increasing testing coverage, ARM64 distribution has been stabilized with quality as per our regular regression testing, so it is qualified for general availability.

• RWX GA [lep][doc]
  RWX has been experimental from Longhorn v1.1.0, but it lacks availability support when the Longhorn Share Manager component behind becomes unavailable. Longhorn v1.4.0 supports NFS recovery backend based on Kubernetes built-in resource, ConfigMap, for recovering NFS client connection during the fail-over period. Also, the NFS client hard mode introduction will further avoid previous potential data loss. For the detail, please check the issue and enhancement proposal.

• Volume Snapshot Checksum [lep][doc]
  Data integrity is a continuous effort for Longhorn. In this version, Snapshot Checksum has been introduced w/ some settings to allow users to enable or disable checksum calculation with different modes.

• Volume Bit-rot Protection [lep][doc]
  When enabling the Volume Snapshot Checksum feature, Longhorn will periodically calculate and check the checksums of volume snapshots, find corrupted snapshots, then fix them.

• Volume Replica Rebuilding Speedup
  When enabling the Volume Snapshot Checksum feature, Longhorn will use the calculated snapshot checksum to avoid needless snapshot replication between nodes for improving replica rebuilding speed and resource consumption.

• Volume Trim [lep][doc]
  Longhorn engine supports UNMAP SCSI command to reclaim space from the block volume.

• Online Volume Expansion [doc]
  Longhorn engine supports optional parameters to pass size expansion requests when updating the volume frontend to support online volume expansion and resize the filesystem via CSI node driver.

• Local Volume via Data Locality Strict Mode [lep][doc]
  Local volume is based on a new Data Locality setting, Strict Local. It will allow users to create one replica volume staying in a consistent location, and the data transfer between the volume frontend and engine will be through a local socket instead of the TCP stack to improve performance and reduce resource consumption.

• Volume Recurring Job Backup Restore [lep][doc]
  Recurring jobs binding to a volume can be backed up to the remote backup target together with the volume backup metadata. They can be restored back as well for a better operation experience.

• Volume IO Metrics [doc]
  Longhorn enriches Volume metrics by providing real-time IO stats including IOPS, latency, and throughput of R/W IO. Users can set up a monotoning solution like Prometheus to monitor volume performance.

• Longhorn System Backup & Restore [lep][doc]
  Users can back up the longhorn system to the remote backup target. Afterward, it's able to restore back to an existing cluster in place or a new cluster for specific operational purposes.

• Support Bundle Enhancement [lep]
  Longhorn introduces a new support bundle integration based on a general support bundle kit solution. This can help us collect more complete troubleshooting info and simulate the cluster environment.

• Tunable Timeout between Engine and Replica [doc]
  In the current Longhorn versions, the default timeout between the Longhorn engine and replica is fixed without any exposed user settings. This will potentially bring some challenges for users having a low-spec infra environment. By exporting the setting configurable, it will allow users adaptively tune the stability of volume operations.

Installation

Please ensure your Kubernetes cluster is at least v1.21 before installing Longhorn v1.4.0.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.4.0 from v1.3.x. Only support upgrading from 1.3.x.

Follow the upgrade instructions here.

Deprecation & Incompatibilities

• Pod Security Policy is an opt-in setting. If installing Longhorn with PSP support, need to enable it first.
• The built-in CSI Snapshotter sidecar is upgraded to v5.0.1. The v1beta1 version of Volume Snapshot custom resource is deprecated but still supported. However, it will be removed after upgrading CSI Snapshotter to 6.1 or later versions in the future, so please start using v1 version instead before the deprecated version is removed.

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

• [FEATURE] Reclaim/Shrink space of volume (836) - @yangchiu @derekbit @smallteeths @shuo-wu
• [FEATURE] Backup/Restore Longhorn System (1455) - @c3y1huang @khushboo-rancher
• [FEATURE] Online volume expansion (1674) - @shuo-wu @chriscchien
• [FEATURE] Record recurring schedule in the backups and allow user choose to use it for the restored volume (2227) - @yangchiu @mantissahz
• [FEATURE] NFS support (RWX) GA (2293) - @derekbit @chriscchien
• [FEATURE] Support metrics for Volume IOPS, throughput and latency real time (2406) - @derekbit @roger-ryao
• [FEATURE] Support bundle enhancement (2759) - @c3y1huang @chriscchien
• [FEATURE] Automatic identifying of corrupted replica (bit rot detection) (3198) - @yangchiu @derekbit
• [FEATURE] Local volume for distributed data workloads (3957) - @derekbit @chriscchien
• [IMPROVEMENT] Support K8s 1.25 by updating removed deprecated resource versions like PodSecurityPolicy (4003) - @PhanLe1010 @chriscchien
• [IMPROVEMENT] Faster resync time for fresh replica rebuilding (4092) - @yangchiu @derekbit
• [FEATURE] Introduce checksum for snapshots (4210) - @derekbit @roger-ryao
• [FEATURE] Update K8s version support and component/pkg/build dependencies (4239) - @yangc...

Longhorn v1.4.0-rc3 release

Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.4.0-rc3 released! 🎆

This release introduces many enhancements, improvements, and bug fixes as described below about stability, performance, data integrity, troubleshooting, and so on. Please try it and feedback. Thanks for all the contributions!

• Kubernetes 1.25 Support [doc]
  In the previous versions, Longhorn relies on Pod Security Policy (PSP) to authorize Longhorn components for privileged operations. From Kubernetes 1.25, PSP has been removed and replaced with Pod Security Admission (PSA). Longhorn v1.4.0 supports opt-in PSP enablement, so it can support Kubernetes versions with or without PSP.

• ARM64 GA
  ARM64 has been experimental from Longhorn v1.1.0. After receiving more user feedback and increasing testing coverage, ARM64 distribution has been stabilized with quality as per our regular regression testing, so it is qualified for general availability.

• RWX GA [lep][doc]
  RWX has been experimental from Longhorn v1.1.0, but it lacks availability support when the Longhorn Share Manager component behind becomes unavailable. Longhorn v1.4.0 supports NFS recovery backend based on Kubernetes built-in resource, ConfigMap, for recovering NFS client connection during the fail-over period. Also, the NFS client hard mode introduction will further avoid previous potential data loss. For the detail, please check the issue and enhancement proposal.

• Volume Snapshot Checksum [lep][doc]
  Data integrity is a continuous effort for Longhorn. In this version, Snapshot Checksum has been introduced w/ some settings to allow users to enable or disable checksum calculation with different modes.

• Volume Bit-rot Protection [lep][doc]
  When enabling the Volume Snapshot Checksum feature, Longhorn will periodically calculate and check the checksums of volume snapshots, find corrupted snapshots, then fix them.

• Volume Replica Rebuilding Speedup
  When enabling the Volume Snapshot Checksum feature, Longhorn will use the calculated snapshot checksum to avoid needless snapshot replication between nodes for improving replica rebuilding speed and resource consumption.

• Volume Trim [lep][doc]
  Longhorn engine supports UNMAP SCSI command to reclaim space from the block volume.

• Online Volume Expansion
  Longhorn engine supports optional parameters to pass size expansion requests when updating the volume frontend to support online volume expansion and resize the filesystem via CSI node driver.

• Local Volume via Data Locality Strict Mode [lep][doc]
  Local volume is based on a new Data Locality setting, Strict Local. It will allow users to create one replica volume staying in a consistent location, and the data transfer between the volume frontend and engine will be through a local socket instead of the TCP stack to improve performance and reduce resource consumption.

• Volume Recurring Job Backup Restore [lep][doc]
  Recurring jobs binding to a volume can be backed up to the remote backup target together with the volume backup metadata. They can be restored back as well for a better operation experience.

• Volume IO Metrics [doc]
  Longhorn enriches Volume metrics by providing real-time IO stats including IOPS, latency, and throughput of R/W IO. Users can set up a monotoning solution like Prometheus to monitor volume performance.

• Longhorn System Backup & Restore [lep][doc]
  Users can back up the longhorn system to the remote backup target. Afterward, it's able to restore back to an existing cluster in place or a new cluster for specific operational purposes.

• Support Bundle Enhancement [lep]
  Longhorn introduces a new support bundle integration based on a general support bundle kit solution. This can help us collect more complete troubleshooting info and simulate the cluster environment.

• Tunable Timeout between Engine and Replica [doc]
  In the current Longhorn versions, the default timeout between the Longhorn engine and replica is fixed without any exposed user settings. This will potentially bring some challenges for users having a low-spec infra environment. By exporting the setting configurable, it will allow users adaptively tune the stability of volume operations.

Installation

Please ensure your Kubernetes cluster is at least v1.21 before installing Longhorn v1.4.0.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.4.0 from v1.3.x. Only support upgrading from 1.3.x.

The live upgrade from v1.3.x is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

• Pod Security Policy is an opt-in setting. If installing Longhorn with PSP support, need to enable it first.

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

• [FEATURE] Reclaim/Shrink space of volume (836) - @yangchiu @derekbit @smallteeths @shuo-wu
• [FEATURE] Backup/Restore Longhorn System (1455) - @c3y1huang @khushboo-rancher
• [FEATURE] Online volume expansion (1674) - @shuo-wu
• [FEATURE] Record recurring schedule in the backups and allow user choose to use it for the restored volume (2227) - @yangchiu @mantissahz
• [FEATURE] NFS support (RWX) GA (2293) - @derekbit @chriscchien
• [FEATURE] Support metrics for Volume IOPS, throughput and latency real time (2406) - @derekbit @roger-ryao
• [FEATURE] Support bundle enhancement (2759) - @c3y1huang @chriscchien
• [FEATURE] Automatic identifying of corrupted replica (bit rot detection) (3198) - @yangchiu @derekbit
• [FEATURE] Local volume for distributed data workloads (3957) - @derekbit @chriscchien
• [IMPROVEMENT] Support K8s 1.25 by updating removed deprecated resource versions like PodSecurityPolicy (4003) - @PhanLe1010 @chriscchien
• [IMPROVEMENT] Faster resync time for fresh replica rebuilding (4092) - @yangchiu @derekbit
• [FEAUTRE] Introduce checksum for snapshots (4210) - @derekbit @roger-ryao
• [FEATURE] Align with upstream K8s version support from Longhorn 1.4 and update component/pkg/build dependencies (4239) - @yangchiu @PhanLe1010
• [BUG] data corruption due to COW and block size not being aligned during rebuilding replicas (...

Longhorn v1.4.0-rc2 release

Release Note

DON'T UPGRADE from/to any RC/Preview releases, because the operation is not supported.

v1.4.0-rc2 released! 🎆

This release introduces many enhancements, improvements, and bug fixes as described below about stability, performance, data integrity, troubleshooting, and so on. Please try it and feedback. Thanks for all the contributions!

• Kubernetes 1.25 Support [doc]
  In the previous versions, Longhorn relies on Pod Security Policy (PSP) to authorize Longhorn components for privileged operations. From Kubernetes 1.25, PSP has been removed and replaced with Pod Security Admission (PSA). Longhorn v1.4.0 supports opt-in PSP enablement, so it can support Kubernetes versions with or without PSP.

• ARM64 GA
  ARM64 has been experimental from Longhorn v1.1.0. After receiving more user feedback and increasing testing coverage, ARM64 distribution has been stabilized with quality as per our regular regression testing, so it is qualified for general availability.

• RWX GA [lep][doc]
  RWX has been experimental from Longhorn v1.1.0, but it lacks availability support when the Longhorn Share Manager component behind becomes unavailable. Longhorn v1.4.0 supports NFS recovery backend based on Kubernetes built-in resource, ConfigMap, for recovering NFS client connection during the fail-over period. Also, the NFS client hard mode introduction will further avoid previous potential data loss. For the detail, please check the issue and enhancement proposal.

• Volume Snapshot Checksum [lep][doc]
  Data integrity is a continuous effort for Longhorn. In this version, Snapshot Checksum has been introduced w/ some settings to allow users to enable or disable checksum calculation with different modes.

• Volume Bit-rot Protection [lep][doc]
  When enabling the Volume Snapshot Checksum feature, Longhorn will periodically calculate and check the checksums of volume snapshots, find corrupted snapshots, then fix them.

• Volume Replica Rebuilding Speedup
  When enabling the Volume Snapshot Checksum feature, Longhorn will use the calculated snapshot checksum to avoid needless snapshot replication between nodes for improving replica rebuilding speed and resource consumption.

• Volume Trim [lep][doc]
  Longhorn engine supports UNMAP SCSI command to reclaim space from the block volume.

• Online Volume Expansion
  Longhorn engine supports optional parameters to pass size expansion requests when updating the volume frontend to support online volume expansion and resize the filesystem via CSI node driver.

• Local Volume via Data Locality Strict Mode [lep][doc]
  Local volume is based on a new Data Locality setting, Strict Local. It will allow users to create one replica volume staying in a consistent location, and the data transfer between the volume frontend and engine will be through a local socket instead of the TCP stack to improve performance and reduce resource consumption.

• Volume Recurring Job Backup Restore [lep][doc]
  Recurring jobs binding to a volume can be backed up to the remote backup target together with the volume backup metadata. They can be restored back as well for a better operation experience.

• Volume IO Metrics
  Longhorn enriches Volume metrics by providing real-time IO stats including IOPS, latency, and throughput of R/W IO. Users can set up a monotoning solution like Prometheus to monitor volume performance.

• Longhorn System Backup & Restore [lep]
  Users can back up the longhorn system to the remote backup target. Afterward, it's able to restore back to an existing cluster in place or a new cluster for specific operational purposes.

• Support Bundle Enhancement[lep]
  Longhorn introduces a new support bundle integration based on a general support bundle kit solution. This can help us collect more complete troubleshooting info and simulate the cluster environment.

• Tunable Timeout between Engine and Replica [doc]
  In the current Longhorn versions, the default timeout between the Longhorn engine and replica is fixed without any exposed user settings. This will potentially bring some challenges for users having a low-spec infra environment. By exporting the setting configurable, it will allow users adaptively tune the stability of volume operations.

Installation

Please ensure your Kubernetes cluster is at least v1.21 before installing Longhorn v1.4.0.

Longhorn supports 3 installation ways including Rancher App Marketplace, Kubectl, and Helm. Follow the installation instructions here.

Upgrade

Please ensure your Kubernetes cluster is at least v1.21 before upgrading to Longhorn v1.4.0 from v1.3.x. Only support upgrading from 1.3.x.

The live upgrade from v1.3.x is not supported until the GA release. When available, follow the upgrade instructions here. here.

Deprecation & Incompatibilities

• Pod Security Policy is an opt-in setting. If installing Longhorn with PSP support, need to enable it first.

Known Issues after Release

Please follow up on here about any outstanding issues found after this release.

Highlights

• [FEATURE] Reclaim/Shrink space of volume (836) - @smallteeths @shuo-wu @roger-ryao
• [FEATURE] Backup/Restore Longhorn System (1455) - @c3y1huang @khushboo-rancher
• [FEATURE] Online volume expansion (1674) - @shuo-wu
• [FEATURE] Record recurring schedule in the backups and allow user choose to use it for the restored volume (2227) - @yangchiu @mantissahz
• [FEATURE] NFS support (RWX) GA (2293) - @derekbit @chriscchien
• [FEATURE] Support metrics for IOPS and bandwidth real time (2406) - @derekbit
• [FEATURE] Support bundle enhancement (2759) - @c3y1huang @chriscchien
• [FEATURE] Automatic identifying of corrupted replica (bit rot detection) (3198) - @yangchiu @derekbit
• [FEATURE] Local volume for distributed data workloads (3957) - @derekbit @chriscchien
• [IMPROVEMENT] Support K8s 1.25 by updating removed deprecated resource versions like PodSecurityPolicy (4003) - @PhanLe1010 @chriscchien
• [IMPROVEMENT] Faster resync time for fresh replica rebuilding (4092) - @derekbit
• [FEAUTRE] Introduce checksum for snapshots (4210) - @derekbit @roger-ryao
• [FEATURE] Align with upstream K8s version support from Longhorn 1.4 and update component/pkg/build dependencies (4239) - @yangchiu @PhanLe1010
• [BUG] data corruption due to COW and block size not being aligned during rebuilding replicas (4354) - @PhanLe1010 @chriscchien
• [IMPROVEMENT] Adjust the iSCSI timeout and the engine-to-replica timeout settings ([4491](https://github....