v1.16.0

[romange]

Dragonfly v1.16.0

Our spring release. We are getting closer to 2.0 with some very exciting features ahead. Stay tuned!

Some prominent changes include:

• Improved memory accounting of client connections (feat(server): Use mimalloc in SSL calls #2710 feat(server): Add TLS usage to /metrics and INFO MEMORY #2755 and chore: add malloc-based stats and decommit #2692 )
• FT.AGGREGATE call (Basic FT.AGGREGATE #2413)
• Properly handle and replicate Memcache flags (chore: Fix memcached flags not updated #2787 fix: propagate memcached flags to replica #2807)
• Intoduce BF.AGGREGATE BD.(M)ADD and BF.(M)EXISTS methods (feat: add bf.(m)add and bf.(m)exists commands #2801). Note, that it does not work with snapshots and replication yet.
• Dragonfly builds natively on MacOS. We would love some help with extending the release pipeline to create a proper macos binary.
• Following the requests from the Edge developers community, we added a basic HTTP API support! Try running Dragonfly with:
  --expose_http_api flag and then call curl -X POST -d '["ping"]' localhost:6379/api. We will follow up with more extensive docs later this month.
• Lots of stability fixes, especially around Sidekiq and BullMQ workloads.

What's Changed

• chore: make usan asan optional and enable them on CI by @kostasrim in chore: make usan asan optional and enable them on CI #2631
• fix: missing manual trigger for daily sanitizers by @kostasrim in fix: missing manual trigger for daily sanitizers #2682
• bug(tiering): fix overflow in page offset calculation and wrong hash offset calculation by @theyueli in bug(tiering): fix overflow in page offset calculation and wrong hash offset calculation #2683
• Chore: Fixed Docker Health Check by @manojks1999 in Chore: Fixed Docker Health Check #2659
• chore: Increase disk space in the coverage runs by @romange in chore: Increase disk space in the coverage runs #2684
• fix(flushall): Decommit memory after releasing tables. by @chakaz in fix(flushall): Decommit memory after releasing tables. #2691
• feat(server): Account for serializer's temporary buffer size by @chakaz in feat(server): Account for serializer's temporary buffer size #2689
• refactor: remove FULL-SYNC-CUT cmd remove Full-sync-cut cmd #2687 by @BorysTheDev in refactor: remove FULL-SYNC-CUT cmd #2687 #2688
• chore: add malloc-based stats and decommit by @romange in chore: add malloc-based stats and decommit #2692
• feat(cluster): automatic slot migration finalization do Finalyze for migration automatically #2697 by @BorysTheDev in feat(cluster): automatic slot migration finalization #2697 #2698
• Basic FT.AGGREGATE by @dranikpg in Basic FT.AGGREGATE #2413
• chore: little transaction cleanup by @dranikpg in chore: little transaction cleanup #2608
• fix(channel store): add acquire/release pair in fast update path by @dranikpg in fix(channel store): add acquire/release pair in fast update path #2704
• chore: add ubuntu22 devcontainer by @romange in chore: add ubuntu22 devcontainer #2700
• feat(cluster): Add --cluster_id flag by @chakaz in feat(cluster): Add --cluster_id flag #2695
• feat(server): Use mimalloc in SSL calls by @chakaz in feat(server): Use mimalloc in SSL calls #2710
• chore: Pull helio with new BlockingCounter by @dranikpg in chore: Pull helio with new BlockingCounter #2711
• chore(transaction): Simplify PollExecution by @dranikpg in chore(transaction): Simplify PollExecution #2712
• chore(transaction): Don't call GetLocalMask from blocking controller by @dranikpg in chore(transaction): Don't call GetLocalMask from blocking controller #2715
• chore: improve compatibility of EXPIRE functions with Redis by @romange in chore: improve compatibility of EXPIRE functions with Redis #2696
• chore: disable flaky fuzzy migration test by @kostasrim in chore: disable flaky fuzzy migration test #2716
• chore: update sanitizers workflow by @kostasrim in chore: update sanitizers workflow #2686
• chore: Use c-ares for resolving hosts in ProtocolClient by @chakaz in chore: Use c-ares for resolving hosts in ProtocolClient #2719
• Remove check-fail in ExpireIfNeeded and introduce DFLY LOAD by @romange in Remove check-fail in ExpireIfNeeded and introduce DFLY LOAD #2699
• chore: Record cmd stat from invoke by @dranikpg in chore: Record cmd stat from invoke #2720
• fix(transaction): nullptr access on non transactional commands by @kostasrim in fix(transaction): nullptr access on non transactional commands #2724
• fix(BgSave): async from sync by @kostasrim in fix(BgSave): async from sync #2702
• chore: remove core/fibers by @romange in chore: remove core/fibers #2723
• fix(transaction): Replace with armed sync point by @dranikpg in fix(transaction): Replace with armed sync point #2708
• feat(json): Deserialize ReJSON format by @chakaz in feat(json): Deserialize ReJSON format #2725
• feat: add flag masteruser by @kostasrim in feat: add flag masteruser #2693
• refactor: block list refactoring Check code with static analyzers #2580 by @BorysTheDev in refactor: block list refactoring #2580 #2732
• chore: fix DeduceExecMode by @dranikpg in chore: fix DeduceExecMode #2733
• fix(cluster): Reply with correct \n / \r\n from CLUSTER sub cmd by @chakaz in fix(cluster): Reply with correct \n / \r\n from CLUSTER sub cmd #2731
• chore: Introduce fiber stack allocator by @romange in chore: Introduce fiber stack allocator #2730
• fix(cluster): Save replica ID per replica by @chakaz in fix(cluster): Save replica ID per replica #2735
• fix(ssl): Proper cleanup by @chakaz in fix(ssl): Proper cleanup #2742
• chore: add skeleton files for flat_dfs code by @romange in chore: add skeleton files for flat_dfs code #2738
• chore: better error reporting when connecting to tls with plain socket by @romange in chore: better error reporting when connecting to tls with plain socket #2740
• chore: Support json paths without root selector by @dranikpg in chore: Support json paths without root selector #2747
• chore: journal cleanup by @romange in chore: journal cleanup #2749
• refactor: remove start-slot-migration cmd remove start-slot-migration command #2727 by @BorysTheDev in refactor: remove start-slot-migration cmd #2727 #2728
• feat(server): Add TLS usage to /metrics and INFO MEMORY by @chakaz in feat(server): Add TLS usage to /metrics and INFO MEMORY #2755
• chore: preparations for adding flat json support by @romange in chore: preparations for adding flat json support #2752
• chore(transaction): Introduce RunCallback by @dranikpg in chore(transaction): Introduce RunCallback #2760
• feat(replication): Do not auto replicate different master by @chakaz in feat(replication): Do not auto replicate different master #2753
• Improve Helm chart to be rendered locally and on machines where is not the application target by @fafg in Improve Helm chart to be rendered locally and on machines where is not the application target #2706
• chore: preparation for basic http api by @romange in chore: preparation for basic http api #2764
• feat(server): Add metric for RDB save duration. by @chakaz in feat(server): Add metric for RDB save duration. #2768
• chore: fix flat_dfs read tests. by @romange in chore: fix flat_dfs read tests. #2772
• fix(ci): do not overwrite last_log_file among tests by @kostasrim in fix(ci): do not overwrite last_log_file among tests #2759
• feat(server): support cluster replication by @adiholden in feat(server): support cluster replication #2748
• fix: fiber preempts on read path and OnCbFinish() clears fetched_items_ by @kostasrim in fix: fiber preempts on read path and OnCbFinish() clears fetched_items_ #2763
• chore(ci): open last_log_file in append mode by @kostasrim in chore(ci): open last_log_file in append mode #2776
• doc(README): fix outdated expiry ranges description by @enjoy-binbin in doc(README): fix outdated expiry ranges description #2779
• Benchmark runner by @adiholden in Benchmark runner #2780
• chore(replication-tests): add cache_mode on test replication all by @kostasrim in chore(replication-tests): add cache_mode on test replication all #2685
• feat(tiering): DiskStorage by @dranikpg in feat(tiering): DiskStorage #2770
• chore: add a boilerplate for bloom filter family by @romange in chore: add a boilerplate for bloom filter family #2782
• chore: introduce conversion routines between JsonType and FlatJson by @romange in chore: introduce conversion routines between JsonType and FlatJson #2785
• chore: Fix memcached flags not updated by @dranikpg in chore: Fix memcached flags not updated #2787
• chore: remove duplicate code from dash and simplify by @kostasrim in chore: remove duplicate code from dash and simplify #2765
• chore: disable test_cluster_slot_migration by @kostasrim in chore: disable test_cluster_slot_migration #2788
• fix: new[] delete[] missmatch in disk_storage by @kostasrim in fix: new[] delete[] missmatch in disk_storage #2792
• fix: sanitizers clang build and clean up some warnings by @kostasrim in fix: sanitizers clang build and clean up some warnings #2793
• chore: add bloom filter class by @romange in chore: add bloom filter class #2791
• chore: add SBF data structure by @romange in chore: add SBF data structure #2795
• chore(tiering): Disable compilation for MacOs by @dranikpg in chore(tiering): Disable compilation for MacOs #2799
• chore: fix daily build by @romange in chore: fix daily build #2798
• chore: expose SBF via compact_object by @romange in chore: expose SBF via compact_object #2797
• fix(ci): malloc trim on sanitizers workflow by @kostasrim in fix(ci): malloc trim on sanitizers workflow #2794
• fix(cluster): Don't miss updates in FLUSHSLOTS by @chakaz in fix(cluster): Don't miss updates in FLUSHSLOTS #2783
• feat: add bf.(m)add and bf.(m)exists commands by @romange in feat: add bf.(m)add and bf.(m)exists commands #2801
• fix: SBF memory leaks by @kostasrim in fix: SBF memory leaks #2803
• chore: refactor StringFamily::Set to use CmdArgParser by @romange in chore: refactor StringFamily::Set to use CmdArgParser #2800
• fix: propagate memcached flags to replica by @romange in fix: propagate memcached flags to replica #2807
• DFLYMIGRATE ACK refactoring by @BorysTheDev in DFLYMIGRATE ACK refactoring #2790
• feat: add master lsn and journal_executed dcheck in replica via ping by @kostasrim in feat: add master lsn and journal_executed dcheck in replica via ping #2778
• fix: correct json response for errors by @romange in fix: correct json response for errors #2813
• chore: bloom test - cover corner cases by @romange in chore: bloom test - cover corner cases #2806
• bug(server): do not write lsn opcode to journal by @adiholden in bug(server): do not write lsn opcode to journal #2814
• chore: Fix build by disabling the tests. by @romange in chore: Fix build by disabling the tests. #2821
• fix(replication): replication with multi shard sync enabled lagging by @adiholden in fix(replication): replication with multi shard sync enabled lagging #2823
• fix: io_uring/fibers bug in DnsResolve by @romange in fix: io_uring/fibers bug in DnsResolve #2825

New Contributors

• @manojks1999 made their first contribution in Chore: Fixed Docker Health Check #2659
• @fafg made their first contribution in Improve Helm chart to be rendered locally and on machines where is not the application target #2706
• @enjoy-binbin made their first contribution in doc(README): fix outdated expiry ranges description #2779

Huge thanks to all the contributors! ❤️

🇮🇱 🇺🇦

Full Changelog: v1.15.0...v1.16.0

---

This discussion was created from the release v1.16.0.