Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Zookeeper ha #3971

Draft
wants to merge 71 commits into
base: main
Choose a base branch
from
Draft

Zookeeper ha #3971

wants to merge 71 commits into from

Commits on Aug 2, 2023

  1. REMOVE: add docker compôse for tests

    @ManyTheFish
    ManyTheFish committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    2ce8b42 View commit details
    Browse the repository at this point in the history

  2. WIP

    @ManyTheFish
    ManyTheFish committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    dc38da9 View commit details
    Browse the repository at this point in the history

  3. makes zk available inside the auth-controller with config coming from… 

    … the cli, it compiles
    @irevoire
    irevoire committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    97e3dfd View commit details
    Browse the repository at this point in the history

  4. send the creation of api-key to zookeeper

    @irevoire
    irevoire committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    84d56f3 View commit details
    Browse the repository at this point in the history

  5. fix analitics compilation

    @ManyTheFish
    ManyTheFish committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    49f976c View commit details
    Browse the repository at this point in the history

  6. Create api keys

    @ManyTheFish
    ManyTheFish committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    3eb6f4b View commit details
    Browse the repository at this point in the history

  7. Event Listener: delete local key if deleted on ZK

    @ManyTheFish
    ManyTheFish committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    b0ff595 View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2023

  1. Forward the keys update to zookeeper

    @irevoire
    irevoire committed Aug 3, 2023
    Configuration menu
    Copy the full SHA
    0cd8157 View commit details
    Browse the repository at this point in the history

  2. Forward the key deletions to zookeeper

    @irevoire
    irevoire committed Aug 3, 2023
    Configuration menu
    Copy the full SHA
    a325ddf View commit details
    Browse the repository at this point in the history

  3. implement the watcher for all kind of operations

    @irevoire
    irevoire committed Aug 3, 2023
    Configuration menu
    Copy the full SHA
    57dc4b1 View commit details
    Browse the repository at this point in the history

  4. Import the already existing api keys on startup

    @irevoire
    irevoire committed Aug 3, 2023
    Configuration menu
    Copy the full SHA
    fe7a312 View commit details
    Browse the repository at this point in the history

  5. fix the tests

    @irevoire
    irevoire committed Aug 3, 2023
    Configuration menu
    Copy the full SHA
    d5523cc View commit details
    Browse the repository at this point in the history

  6. add logs

    @irevoire
    irevoire committed Aug 3, 2023
    Configuration menu
    Copy the full SHA
    5ce01bc View commit details
    Browse the repository at this point in the history

  7. fix auto-synchronization with zk

    @ManyTheFish
    ManyTheFish committed Aug 3, 2023
    Configuration menu
    Copy the full SHA
    ad7f8ed View commit details
    Browse the repository at this point in the history

  8. update docker compose as an example

    @ManyTheFish
    ManyTheFish committed Aug 3, 2023
    Configuration menu
    Copy the full SHA
    3d46e84 View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2023

  1. Update zookeeper client

    @ManyTheFish
    ManyTheFish committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    b311089 View commit details
    Browse the repository at this point in the history

  2. Comment Meilisearch container by default

    @ManyTheFish
    ManyTheFish committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    b2f36b9 View commit details
    Browse the repository at this point in the history

  3. Create a task on zookeeper side when a task is created locally

    @ManyTheFish
    ManyTheFish committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    b66bf04 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2023

  1. fix a few warnings

    @irevoire
    irevoire committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    0d20d08 View commit details
    Browse the repository at this point in the history

  2. fix the register task watcher

    @irevoire
    irevoire committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    1191ec5 View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2023

  1. Start leader election and task processing (WIP)

    @ManyTheFish
    ManyTheFish committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    8e437ed View commit details
    Browse the repository at this point in the history

  2. fix the leader election

    @irevoire
    irevoire committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    8c20d6e View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2023

  1. implement the deletion of tasks after processing a batch 

    add a lot of comments and logs
    @irevoire
    irevoire committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    f0c4d36 View commit details
    Browse the repository at this point in the history

  2. wake up after registering a task

    @irevoire
    irevoire committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    61ccfaf View commit details
    Browse the repository at this point in the history

  3. starts creating snapshot, the import is still missing

    @irevoire
    irevoire committed Aug 10, 2023
    Configuration menu
    Copy the full SHA
    777eebb View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2023

  1. wip: starts working on importing the snapshots

    @irevoire
    irevoire committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    854745c View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2023

  1. WIP moving to the sync zookeeper API

    @Kerollmops
    Kerollmops committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    0c7d7c6 View commit details
    Browse the repository at this point in the history

  2. Fixup a lot of small issues on the ZK config

    @Kerollmops
    Kerollmops committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    c488a4a View commit details
    Browse the repository at this point in the history

  3. Keep the ZK flow when enqueuing tasks

    @Kerollmops
    Kerollmops committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    2d1434d View commit details
    Browse the repository at this point in the history

  4. React to changes towards the cluster members

    @Kerollmops
    Kerollmops committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    8c3ad57 View commit details
    Browse the repository at this point in the history

  5. Fix the watcher ordering of the auth/ node

    @Kerollmops
    Kerollmops committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    9dd4423 View commit details
    Browse the repository at this point in the history

  6. WIP fix the tests

    @Kerollmops
    Kerollmops committed Aug 30, 2023
    Configuration menu
    Copy the full SHA
    e257710 View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2023

  1. Wrap the IndexScheduler fields into an inner struct

    @Kerollmops
    Kerollmops committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    95a011a View commit details
    Browse the repository at this point in the history

  2. Make things to compile again

    @Kerollmops
    Kerollmops committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    d7233ec View commit details
    Browse the repository at this point in the history

  3. WIP making the final snapshot swap

    @Kerollmops
    Kerollmops committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    0c68b9e View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2023

  1. make the tests compile again

    @irevoire
    irevoire committed Sep 4, 2023
    Configuration menu
    Copy the full SHA
    966cbda View commit details
    Browse the repository at this point in the history

  2. Add the options into the IndexScheduler

    @Kerollmops
    Kerollmops committed Sep 4, 2023
    Configuration menu
    Copy the full SHA
    76657af View commit details
    Browse the repository at this point in the history

  3. Make the snapshot download work

    @Kerollmops
    Kerollmops committed Sep 4, 2023
    Configuration menu
    Copy the full SHA
    7d85753 View commit details
    Browse the repository at this point in the history

  4. Introduce the zk-tasks folder

    @Kerollmops
    Kerollmops committed Sep 4, 2023
    Configuration menu
    Copy the full SHA
    41697c4 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2023

  1. starts using s3

    @irevoire
    irevoire committed Sep 5, 2023
    Configuration menu
    Copy the full SHA
    5b89276 View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2023

  1. Mastering minio

    @Kerollmops
    Kerollmops committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    01c13c9 View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2023

  1. Fix and crash when the tasks path is unknown

    @Kerollmops
    Kerollmops committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    719fdd7 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2023

  1. Store content files into the S3

    @Kerollmops
    Kerollmops committed Sep 11, 2023
    Configuration menu
    Copy the full SHA
    a53a0fd View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2023

  1. start a script to run everything

    @irevoire
    irevoire committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    b7109c0 View commit details
    Browse the repository at this point in the history

  2. Fix internal error

    @Kerollmops
    Kerollmops committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    c158d03 View commit details
    Browse the repository at this point in the history

  3. Remove tasks and content file on the s3

    @Kerollmops
    Kerollmops committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    f544cfa View commit details
    Browse the repository at this point in the history

  4. Use slashes instead of dots for the s3 paths separators

    @Kerollmops
    Kerollmops committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    f37fdce View commit details
    Browse the repository at this point in the history

  5. Move the load snapshot step into a function

    @Kerollmops
    Kerollmops committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    9b01506 View commit details
    Browse the repository at this point in the history

  6. Fix again the dots

    @Kerollmops
    Kerollmops committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    309c33a View commit details
    Browse the repository at this point in the history

  7. Load the latest snapshot when we start the engine

    @Kerollmops
    Kerollmops committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    8a2e8a8 View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2023

  1. exposes all the s3 arguments

    @irevoire
    irevoire committed Sep 13, 2023
    Configuration menu
    Copy the full SHA
    ecd36b1 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2023

  1. fix build in release mode

    @irevoire
    irevoire committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    c71ba72 View commit details
    Browse the repository at this point in the history

  2. bump charabia

    @irevoire
    irevoire committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    6325cda View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2023

  1. move to our new S3 lib

    @irevoire
    irevoire committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    98b67f2 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2023

  1. bump strois version

    @irevoire
    irevoire committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    dfb84f8 View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2023

  1. bump strois to a temporary version while we wait for rusty_s3 to do a… 

    … new release
    @irevoire
    irevoire committed Oct 13, 2023
    Configuration menu
    Copy the full SHA
    b22f126 View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2023

  1. No more create the S3 Bucket

    @Kerollmops
    Kerollmops committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    b9983a4 View commit details
    Browse the repository at this point in the history

  2. bump strois to a temporary version while we wait for rusty_s3 to do a… 

    … new release again
    @Kerollmops
    Kerollmops committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    8f04353 View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2023

  1. WIP: start updating the zookeeper client => leader election is missing

    @irevoire
    irevoire committed Oct 31, 2023
    Configuration menu
    Copy the full SHA
    c573261 View commit details
    Browse the repository at this point in the history

  2. ICE

    @irevoire
    irevoire committed Oct 31, 2023
    Configuration menu
    Copy the full SHA
    c3a8d4b View commit details
    Browse the repository at this point in the history

  3. ICE: use a git version of my crate so rust team can pull the repo

    @irevoire
    irevoire committed Oct 31, 2023
    Configuration menu
    Copy the full SHA
    03b5109 View commit details
    Browse the repository at this point in the history

Commits on Nov 2, 2023

  1. WIT: it compiles but the processing of tasks and loading of snapshots… 

    … is still not implemented
    @irevoire
    irevoire committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    3661441 View commit details
    Browse the repository at this point in the history

  2. it compiles, now tokio is complaining that we block the main thread (… 

    …and is right), I need to patch the zookeeper client to not rely on tokio anymore
    @irevoire
    irevoire committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    0c18962 View commit details
    Browse the repository at this point in the history

  3. no deadlock on start

    @irevoire
    irevoire committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    d0a3582 View commit details
    Browse the repository at this point in the history

  4. WIP fix a lot of bugs: The follower do not wake up when a new snapsho… 

    …t is available
    @irevoire
    irevoire committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    c5ec817 View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2023

  1. fix all the bugs on the snapshots export/import pipeline

    @irevoire
    irevoire committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    d1bc7ec View commit details
    Browse the repository at this point in the history

  2. fix the leader election

    @irevoire
    irevoire committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    41178e5 View commit details
    Browse the repository at this point in the history

  3. fix a few warnings

    @irevoire
    irevoire committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    c48f72e View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2023

  1. re-implement the snapshot import at startup

    @irevoire
    irevoire committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    1585b3e View commit details
    Browse the repository at this point in the history

  2. fix the path of the deps

    @irevoire
    irevoire committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    e4adf7f View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2023

  1. use the zookeeper-client-sync from meilisearch instead of tamo

    @irevoire
    irevoire committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    a6863a7 View commit details
    Browse the repository at this point in the history