Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Shard placement table persistence #18283

Merged
merged 17 commits into from
May 20, 2024
Merged

Shard placement table persistence #18283

merged 17 commits into from
May 20, 2024

Commits on May 17, 2024

  1. cluster: track group id in shard_placement_table 

    We track group ids in partition assignments and current state because we
want to use them as kvstore keys - group id is just a number so it is
better in this regard than the ntp and it uniquely identifies the
current incarnation of the ntp (i.e. each group id is uniquely
mapped to ntp, but ntp can be mapped to several group ids, though only
one will be current).
    @ztlpn
    ztlpn committed May 17, 2024
    Configuration menu
    Copy the full SHA
    19690d1 View commit details
    Browse the repository at this point in the history

  2. c/shard_placement_table: persist changes to kvstore

    @ztlpn
    ztlpn committed May 17, 2024
    Configuration menu
    Copy the full SHA
    d7ec7c7 View commit details
    Browse the repository at this point in the history

  3. offline_log_viewer: add support for shard_placement kvstore items

    @ztlpn
    ztlpn committed May 17, 2024
    Configuration menu
    Copy the full SHA
    3928208 View commit details
    Browse the repository at this point in the history

  4. kvstore: add for_each method

    @ztlpn
    ztlpn committed May 17, 2024
    Configuration menu
    Copy the full SHA
    94d3543 View commit details
    Browse the repository at this point in the history

  5. c/shard_placement_table: implement migration to kvstore persistence

    @ztlpn
    ztlpn committed May 17, 2024
    Configuration menu
    Copy the full SHA
    bbac4e9 View commit details
    Browse the repository at this point in the history

  6. bytes: add bytes_to_iobuf(bytes_view)

    @ztlpn
    ztlpn committed May 17, 2024
    Configuration menu
    Copy the full SHA
    2978557 View commit details
    Browse the repository at this point in the history

  7. c/shard_placement_table: initialize from kvstore

    @ztlpn
    ztlpn committed May 17, 2024
    Configuration menu
    Copy the full SHA
    a3d0bfe View commit details
    Browse the repository at this point in the history

  8. c/shard_placement_table: update class comment

    @ztlpn
    ztlpn committed May 17, 2024
    Configuration menu
    Copy the full SHA
    3088707 View commit details
    Browse the repository at this point in the history

  9. c/shard_placement_table: add get_target(ntp) method

    @ztlpn
    ztlpn committed May 17, 2024
    Configuration menu
    Copy the full SHA
    7710756 View commit details
    Browse the repository at this point in the history

  10. c/shard_placement_table/ut: improve logging 

    Make different components log with different prefixes in test to more
easily disambiguate their messages.
    @ztlpn
    ztlpn committed May 17, 2024
    Configuration menu
    Copy the full SHA
    ed1e25b View commit details
    Browse the repository at this point in the history

  11. c/shard_placement_table/ut: restart in between ops

    @ztlpn
    ztlpn committed May 17, 2024
    Configuration menu
    Copy the full SHA
    c65c602 View commit details
    Browse the repository at this point in the history

  12. c/shard_placement_table/ut: add helper to assert key set equality

    @ztlpn
    ztlpn committed May 17, 2024
    Configuration menu
    Copy the full SHA
    1e353b9 View commit details
    Browse the repository at this point in the history

  13. c/shard_placement_table/ut: add after-restart checks

    @ztlpn
    ztlpn committed May 17, 2024
    Configuration menu
    Copy the full SHA
    8d67f79 View commit details
    Browse the repository at this point in the history

  14. c/shard_balancer: use local shard_placement_table instance everywhere

    @ztlpn
    ztlpn committed May 17, 2024
    Configuration menu
    Copy the full SHA
    26366e2 View commit details
    Browse the repository at this point in the history

  15. c/shard_balancer: initialize from kvstore if persistence is enabled

    @ztlpn
    ztlpn committed May 17, 2024
    Configuration menu
    Copy the full SHA
    91fb69b View commit details
    Browse the repository at this point in the history

  16. features: add shard_placement_persistence feature flag 

    This flag will signal nodes that they can switch to persisting their
shard_placement_table locally in the kvstore.
    @ztlpn
    ztlpn committed May 17, 2024
    Configuration menu
    Copy the full SHA
    caa89ca View commit details
    Browse the repository at this point in the history

  17. c/shard_balancer: enable persistence when the feature barrier allows

    @ztlpn
    ztlpn committed May 17, 2024
    Configuration menu
    Copy the full SHA
    faa76fd View commit details
    Browse the repository at this point in the history