feat(tiering): Defragmentation #3021

dranikpg · 2024-05-07T12:02:42Z

No description provided.

src/server/tiered_storage.cc

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>

dranikpg · 2024-05-28T18:27:13Z

Finally with all the issues resolved, we can have proper defrag 🤓

src/server/tiered_storage.cc

romange

LGTM, minor comments

src/server/tiering/op_manager.cc

romange · 2024-05-29T13:16:49Z

src/server/tiering/op_manager.cc

  ReadOp* info = &pending_reads_.at(offset);

+  // Reorder base read (offset 0) to be last, so reads for defragmentation are handled last


can you please explain this?

If we already have a page read for defragmentation pending and some other read for the sub-segment is enqueued, we first must handle the sub-segment read, only then the full page read

I suggest copying this explanation into the comment

romange · 2024-05-29T13:17:45Z

src/server/tiering/op_manager.cc

  bool deleting_full = false;
  std::string key_value;
-  for (auto& ko : info->key_ops) {
+  for (size_t i = 0; i < info->key_ops.size(); i++) {  // more items can be read while iterating


what does it mean more items can be read while iterating ?

When we call ReportDelete, it might issue a read for the whole page to trigger defragmentation, which will be added to this list

please add a comment:
// Note that functions in the loop may append items to info->key_ops during the traversal.

src/server/tiering/op_manager.h

romange · 2024-05-29T13:21:29Z

src/server/tiering/small_bins.cc

+      return {full_segment, false /* fragmented */, true /* empty */};
+    }
+
+    if (bin.bytes < kPageSize / 2) {


is it the condition to trigger fragmentation?

Yes 🤓 Not very sophisticated

src/server/tiering/small_bins.h

romange

LGTM, but please add comments that I asked.

dranikpg · 2024-05-29T16:11:31Z

Added comments

…nfly ( v1.18.1 → v1.19.0 ) (#3784) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [docker.dragonflydb.io/dragonflydb/dragonfly](https://togithub.com/dragonflydb/dragonfly) | minor | `v1.18.1` -> `v1.19.0` | --- ### Release Notes <details> <summary>dragonflydb/dragonfly (docker.dragonflydb.io/dragonflydb/dragonfly)</summary> ### [`v1.19.0`](https://togithub.com/dragonflydb/dragonfly/releases/tag/v1.19.0) [Compare Source](https://togithub.com/dragonflydb/dragonfly/compare/v1.18.1...v1.19.0) ##### Dragonfly v1.19.0 Some prominent changes include: - SPOP and SRANDMEMBER return truly randomized results ([#3022](https://togithub.com/dragonflydb/dragonfly/issues/3022)) - Fix memory blowup in some cases during replication ([#3084](https://togithub.com/dragonflydb/dragonfly/issues/3084) [#3103](https://togithub.com/dragonflydb/dragonfly/issues/3103)) - Snapshotting works with data tiering ([#3073](https://togithub.com/dragonflydb/dragonfly/issues/3073)) - Fix incompatibility issue of RDB snapshot with Redis 6.x ([#3121](https://togithub.com/dragonflydb/dragonfly/issues/3121)) - Memory utilization improvements + better introspection of memory usage via /metrics ##### What's Changed - chore: update versions and remove caching code by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3030 - chore: fix macos tiering by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3032 - feat: support subrange expressions in jsonpathv2 by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3036 - fix: reduce delay when stop replica [#3020](https://togithub.com/dragonflydb/dragonfly/issues/3020) by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3028 - fix(set): fix random in SRANDMEMBER and SPOP commands by [@BagritsevichStepan](https://togithub.com/BagritsevichStepan) in [dragonflydb/dragonfly#3022 - chore: prevent updating gcc on macos by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3037 - chore: add half-range indices by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3041 - chore: update helio together with new mimalloc version by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3040 - fix: header for oom_errors_total by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3043 - chore: reset enable_direct_fd to avoid socket leakage in kernel by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3051 - chore: update deprecated actions by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3052 - chore: remove unneeded check-fails by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3054 - chore: Export replication memory stats by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3062 - <chore>!: Update grafana panel by [@MaoMaoCake](https://togithub.com/MaoMaoCake) in [dragonflydb/dragonfly#3064 - chore: add replication memory stats to the dashboard by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3065 - chore: bpop prints by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3076 - fix(tiering): wait for IO before test teardown by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3079 - chore: small replayer fixes by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3081 - fix: increase lua stack limit to 8KB by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3080 - chore: minor fixes by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3082 - fix: fix cluster incorrect keys status by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3083 - fix(acl): return -NOPERM instead of response error by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#3049 - feat(server): Expose serialization bytes via `INFO` and `/metrics` by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#3087 - chore(streams): Some refactoring by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3088 - feat(server): remove multi shard sync from replication by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#3085 - fix(replication): fullsync phase write to sync on noop by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#3084 - fix(tiering): Async delete for small bins by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3068 - fix: prevent crashing if error happened during snapshot moving by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3092 - fix: add timeout for DFLYMIGRATE ACK to prevent deadlock by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3093 - chore: improve Migration() by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#3033 - feat(streams): Stream optimizations by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3091 - feat(tiering): Defragmentation by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3021 - fix: bit shift of kTagMask by [@haodongnj](https://togithub.com/haodongnj) in [dragonflydb/dragonfly#3099 - feat(cluster_mgr): Allow attaching replicas by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#3105 - bug(server): fix replication stuck in full sync by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#3103 - fix(server): Sync FLUSH with tiering by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3098 - chore: export pipeline related metrics by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3104 - chore: zset error log (from 1.18 branch) by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3109 - New test for cluster migration: connection issue by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3102 - chore: minor fixes + remove redundant DCHECK by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3115 - fix: fix cluster_fuzzy_migration test by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3116 - feat(server): Pipeline and dispatch on `/metrics` by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#3117 - chore: pull helio add test for tls deadlock by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#3111 - feat(cluster_mgr): Improvements to `cluster_mgr.py` by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#3118 - refactor client tracking, fix atomicity, squashing and multi/exec by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2970 - fix(server): fix compatibility with rdb snapshot by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#3121 - Revert "refactor client tracking, fix atomicity, squashing and multi/… by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#3122 - feat(cluster_mgr): Take over command by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#3120 - fix: fix RestoreStreamer to prevent buckets skipping [#2830](https://togithub.com/dragonflydb/dragonfly/issues/2830) by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3119 - feat(cluster_mgr): Fix migration action by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#3124 - chore: prevent Dispatch fiber to be launched during migration by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3123 - feat(tiering): Simple snapshotting by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3073 - fix: upload action in version release by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3127 ##### New Contributors - [@MaoMaoCake](https://togithub.com/MaoMaoCake) made their first contribution in [dragonflydb/dragonfly#3064 - [@haodongnj](https://togithub.com/haodongnj) made their first contribution in [dragonflydb/dragonfly#3099 ##### Huge thanks to all the contributors! ❤️ **Full Changelog**: dragonflydb/dragonfly@v1.18.0...v1.19.0 </details>  Co-authored-by: repo-jeeves[bot] <106431701+repo-jeeves[bot]@users.noreply.github.com>

…nfly ( v1.18.1 → v1.19.0 ) (#629) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [docker.dragonflydb.io/dragonflydb/dragonfly](https://togithub.com/dragonflydb/dragonfly) | minor | `v1.18.1` -> `v1.19.0` | --- ### Release Notes <details> <summary>dragonflydb/dragonfly (docker.dragonflydb.io/dragonflydb/dragonfly)</summary> ### [`v1.19.0`](https://togithub.com/dragonflydb/dragonfly/releases/tag/v1.19.0) [Compare Source](https://togithub.com/dragonflydb/dragonfly/compare/v1.18.1...v1.19.0) ##### Dragonfly v1.19.0 Some prominent changes include: - SPOP and SRANDMEMBER return truly randomized results ([#3022](https://togithub.com/dragonflydb/dragonfly/issues/3022)) - Fix memory blowup in some cases during replication ([#3084](https://togithub.com/dragonflydb/dragonfly/issues/3084) [#3103](https://togithub.com/dragonflydb/dragonfly/issues/3103)) - Snapshotting works with data tiering ([#3073](https://togithub.com/dragonflydb/dragonfly/issues/3073)) - Fix incompatibility issue of RDB snapshot with Redis 6.x ([#3121](https://togithub.com/dragonflydb/dragonfly/issues/3121)) - Memory utilization improvements + better introspection of memory usage via /metrics ##### What's Changed - chore: update versions and remove caching code by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3030 - chore: fix macos tiering by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3032 - feat: support subrange expressions in jsonpathv2 by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3036 - fix: reduce delay when stop replica [#3020](https://togithub.com/dragonflydb/dragonfly/issues/3020) by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3028 - fix(set): fix random in SRANDMEMBER and SPOP commands by [@BagritsevichStepan](https://togithub.com/BagritsevichStepan) in [dragonflydb/dragonfly#3022 - chore: prevent updating gcc on macos by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3037 - chore: add half-range indices by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3041 - chore: update helio together with new mimalloc version by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3040 - fix: header for oom_errors_total by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3043 - chore: reset enable_direct_fd to avoid socket leakage in kernel by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3051 - chore: update deprecated actions by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3052 - chore: remove unneeded check-fails by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3054 - chore: Export replication memory stats by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3062 - <chore>!: Update grafana panel by [@MaoMaoCake](https://togithub.com/MaoMaoCake) in [dragonflydb/dragonfly#3064 - chore: add replication memory stats to the dashboard by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3065 - chore: bpop prints by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3076 - fix(tiering): wait for IO before test teardown by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3079 - chore: small replayer fixes by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3081 - fix: increase lua stack limit to 8KB by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3080 - chore: minor fixes by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3082 - fix: fix cluster incorrect keys status by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3083 - fix(acl): return -NOPERM instead of response error by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#3049 - feat(server): Expose serialization bytes via `INFO` and `/metrics` by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#3087 - chore(streams): Some refactoring by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3088 - feat(server): remove multi shard sync from replication by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#3085 - fix(replication): fullsync phase write to sync on noop by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#3084 - fix(tiering): Async delete for small bins by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3068 - fix: prevent crashing if error happened during snapshot moving by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3092 - fix: add timeout for DFLYMIGRATE ACK to prevent deadlock by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3093 - chore: improve Migration() by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#3033 - feat(streams): Stream optimizations by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3091 - feat(tiering): Defragmentation by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3021 - fix: bit shift of kTagMask by [@haodongnj](https://togithub.com/haodongnj) in [dragonflydb/dragonfly#3099 - feat(cluster_mgr): Allow attaching replicas by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#3105 - bug(server): fix replication stuck in full sync by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#3103 - fix(server): Sync FLUSH with tiering by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3098 - chore: export pipeline related metrics by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3104 - chore: zset error log (from 1.18 branch) by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3109 - New test for cluster migration: connection issue by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3102 - chore: minor fixes + remove redundant DCHECK by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3115 - fix: fix cluster_fuzzy_migration test by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3116 - feat(server): Pipeline and dispatch on `/metrics` by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#3117 - chore: pull helio add test for tls deadlock by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#3111 - feat(cluster_mgr): Improvements to `cluster_mgr.py` by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#3118 - refactor client tracking, fix atomicity, squashing and multi/exec by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2970 - fix(server): fix compatibility with rdb snapshot by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#3121 - Revert "refactor client tracking, fix atomicity, squashing and multi/… by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#3122 - feat(cluster_mgr): Take over command by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#3120 - fix: fix RestoreStreamer to prevent buckets skipping [#2830](https://togithub.com/dragonflydb/dragonfly/issues/2830) by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3119 - feat(cluster_mgr): Fix migration action by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#3124 - chore: prevent Dispatch fiber to be launched during migration by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3123 - feat(tiering): Simple snapshotting by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3073 - fix: upload action in version release by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3127 ##### New Contributors - [@MaoMaoCake](https://togithub.com/MaoMaoCake) made their first contribution in [dragonflydb/dragonfly#3064 - [@haodongnj](https://togithub.com/haodongnj) made their first contribution in [dragonflydb/dragonfly#3099 ##### Huge thanks to all the contributors! ❤️ **Full Changelog**: dragonflydb/dragonfly@v1.18.0...v1.19.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).  Co-authored-by: kireque-bot[bot] <143391978+kireque-bot[bot]@users.noreply.github.com>

…19.0 ) (#4821) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/dragonflydb/dragonfly](https://togithub.com/dragonflydb/dragonfly) | minor | `v1.18.1` -> `v1.19.0` | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>dragonflydb/dragonfly (ghcr.io/dragonflydb/dragonfly)</summary> ### [`v1.19.0`](https://togithub.com/dragonflydb/dragonfly/releases/tag/v1.19.0) [Compare Source](https://togithub.com/dragonflydb/dragonfly/compare/v1.18.1...v1.19.0) ##### Dragonfly v1.19.0 Some prominent changes include: - SPOP and SRANDMEMBER return truly randomized results ([#3022](https://togithub.com/dragonflydb/dragonfly/issues/3022)) - Fix memory blowup in some cases during replication ([#3084](https://togithub.com/dragonflydb/dragonfly/issues/3084) [#3103](https://togithub.com/dragonflydb/dragonfly/issues/3103)) - Snapshotting works with data tiering ([#3073](https://togithub.com/dragonflydb/dragonfly/issues/3073)) - Fix incompatibility issue of RDB snapshot with Redis 6.x ([#3121](https://togithub.com/dragonflydb/dragonfly/issues/3121)) - Memory utilization improvements + better introspection of memory usage via /metrics ##### What's Changed - chore: update versions and remove caching code by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3030 - chore: fix macos tiering by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3032 - feat: support subrange expressions in jsonpathv2 by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3036 - fix: reduce delay when stop replica [#3020](https://togithub.com/dragonflydb/dragonfly/issues/3020) by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3028 - fix(set): fix random in SRANDMEMBER and SPOP commands by [@BagritsevichStepan](https://togithub.com/BagritsevichStepan) in [dragonflydb/dragonfly#3022 - chore: prevent updating gcc on macos by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3037 - chore: add half-range indices by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3041 - chore: update helio together with new mimalloc version by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3040 - fix: header for oom_errors_total by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3043 - chore: reset enable_direct_fd to avoid socket leakage in kernel by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3051 - chore: update deprecated actions by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3052 - chore: remove unneeded check-fails by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3054 - chore: Export replication memory stats by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3062 - <chore>!: Update grafana panel by [@MaoMaoCake](https://togithub.com/MaoMaoCake) in [dragonflydb/dragonfly#3064 - chore: add replication memory stats to the dashboard by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3065 - chore: bpop prints by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3076 - fix(tiering): wait for IO before test teardown by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3079 - chore: small replayer fixes by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3081 - fix: increase lua stack limit to 8KB by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3080 - chore: minor fixes by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3082 - fix: fix cluster incorrect keys status by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3083 - fix(acl): return -NOPERM instead of response error by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#3049 - feat(server): Expose serialization bytes via `INFO` and `/metrics` by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#3087 - chore(streams): Some refactoring by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3088 - feat(server): remove multi shard sync from replication by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#3085 - fix(replication): fullsync phase write to sync on noop by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#3084 - fix(tiering): Async delete for small bins by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3068 - fix: prevent crashing if error happened during snapshot moving by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3092 - fix: add timeout for DFLYMIGRATE ACK to prevent deadlock by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3093 - chore: improve Migration() by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#3033 - feat(streams): Stream optimizations by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3091 - feat(tiering): Defragmentation by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3021 - fix: bit shift of kTagMask by [@haodongnj](https://togithub.com/haodongnj) in [dragonflydb/dragonfly#3099 - feat(cluster_mgr): Allow attaching replicas by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#3105 - bug(server): fix replication stuck in full sync by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#3103 - fix(server): Sync FLUSH with tiering by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3098 - chore: export pipeline related metrics by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3104 - chore: zset error log (from 1.18 branch) by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3109 - New test for cluster migration: connection issue by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3102 - chore: minor fixes + remove redundant DCHECK by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3115 - fix: fix cluster_fuzzy_migration test by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3116 - feat(server): Pipeline and dispatch on `/metrics` by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#3117 - chore: pull helio add test for tls deadlock by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#3111 - feat(cluster_mgr): Improvements to `cluster_mgr.py` by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#3118 - refactor client tracking, fix atomicity, squashing and multi/exec by [@kostasrim](https://togithub.com/kostasrim) in [dragonflydb/dragonfly#2970 - fix(server): fix compatibility with rdb snapshot by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#3121 - Revert "refactor client tracking, fix atomicity, squashing and multi/… by [@adiholden](https://togithub.com/adiholden) in [dragonflydb/dragonfly#3122 - feat(cluster_mgr): Take over command by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#3120 - fix: fix RestoreStreamer to prevent buckets skipping [#2830](https://togithub.com/dragonflydb/dragonfly/issues/2830) by [@BorysTheDev](https://togithub.com/BorysTheDev) in [dragonflydb/dragonfly#3119 - feat(cluster_mgr): Fix migration action by [@chakaz](https://togithub.com/chakaz) in [dragonflydb/dragonfly#3124 - chore: prevent Dispatch fiber to be launched during migration by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3123 - feat(tiering): Simple snapshotting by [@dranikpg](https://togithub.com/dranikpg) in [dragonflydb/dragonfly#3073 - fix: upload action in version release by [@romange](https://togithub.com/romange) in [dragonflydb/dragonfly#3127 ##### New Contributors - [@MaoMaoCake](https://togithub.com/MaoMaoCake) made their first contribution in [dragonflydb/dragonfly#3064 - [@haodongnj](https://togithub.com/haodongnj) made their first contribution in [dragonflydb/dragonfly#3099 ##### Huge thanks to all the contributors! ❤️ **Full Changelog**: dragonflydb/dragonfly@v1.18.0...v1.19.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://togithub.com/renovatebot/renovate).  Co-authored-by: lumiere-bot[bot] <98047013+lumiere-bot[bot]@users.noreply.github.com>

adiholden reviewed May 7, 2024

View reviewed changes

src/server/tiered_storage.cc Outdated Show resolved Hide resolved

dranikpg mentioned this pull request May 8, 2024

chore(dash): Replace comparator with predicate #3025

Merged

dranikpg force-pushed the tiering-defrag branch from 81e847d to 163f024 Compare May 9, 2024 06:33

dranikpg force-pushed the tiering-defrag branch from 163f024 to 6180934 Compare May 22, 2024 11:40

dranikpg mentioned this pull request May 27, 2024

fix(tiering): Async delete for small bins #3068

Merged

dranikpg added 2 commits May 28, 2024 19:04

feat(tiering): Defragmentation

59fb693

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>

feat: Use FindFirst for defrag

a1768b7

Signed-off-by: Vladislav Oleshko <vlad@dragonflydb.io>

dranikpg force-pushed the tiering-defrag branch from 6180934 to a1768b7 Compare May 28, 2024 16:39

fix: fixes

461f8cb

dranikpg requested review from adiholden and romange May 28, 2024 18:27

dranikpg added 2 commits May 28, 2024 21:27

chore: stats

6de1bd7

fix: small comments

8a49b0e

dranikpg changed the title ~~WIP feat(tiering): Defragmentation~~ feat(tiering): Defragmentation May 29, 2024

romange reviewed May 29, 2024

View reviewed changes

src/server/tiered_storage.cc Show resolved Hide resolved

romange reviewed May 29, 2024

View reviewed changes

fix: small fixes

21a0c49

romange previously approved these changes May 29, 2024

View reviewed changes

dranikpg dismissed romange’s stale review via 01b8c39 May 29, 2024 16:11

dranikpg requested a review from romange May 29, 2024 16:11

chore: commnets

01b8c39

romange approved these changes May 29, 2024

View reviewed changes

dranikpg merged commit 99949ca into dragonflydb:main May 29, 2024
7 checks passed

dranikpg deleted the tiering-defrag branch June 4, 2024 07:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(tiering): Defragmentation #3021

feat(tiering): Defragmentation #3021

dranikpg commented May 7, 2024

dranikpg commented May 28, 2024

romange left a comment

romange May 29, 2024

dranikpg May 29, 2024

romange May 29, 2024

romange May 29, 2024

dranikpg May 29, 2024

romange May 29, 2024 •

edited

romange May 29, 2024

dranikpg May 29, 2024

romange left a comment

dranikpg commented May 29, 2024

		ReadOp* info = &pending_reads_.at(offset);

		// Reorder base read (offset 0) to be last, so reads for defragmentation are handled last

feat(tiering): Defragmentation #3021

feat(tiering): Defragmentation #3021

Conversation

dranikpg commented May 7, 2024

dranikpg commented May 28, 2024

romange left a comment

Choose a reason for hiding this comment

romange May 29, 2024

Choose a reason for hiding this comment

dranikpg May 29, 2024

Choose a reason for hiding this comment

romange May 29, 2024

Choose a reason for hiding this comment

romange May 29, 2024

Choose a reason for hiding this comment

dranikpg May 29, 2024

Choose a reason for hiding this comment

romange May 29, 2024 • edited

Choose a reason for hiding this comment

romange May 29, 2024

Choose a reason for hiding this comment

dranikpg May 29, 2024

Choose a reason for hiding this comment

romange left a comment

Choose a reason for hiding this comment

dranikpg commented May 29, 2024

romange May 29, 2024 •

edited