Zookeeper ha

[irevoire]

What works

We synchronize the API-keys through Zookeeper.

Usage

One new cli flag has been introduced. It’s called zk-url and must specify the URL of one of the Zookeeper machines in the cluster.

Example usage:

You must create your bucket beforehand.
If needed you can also specify the region of your bucket with the --s3-region parameter.

./meilisearch -- --s3-secret-key iLXhXdFkXGaZUeSSV05YA8ysvPdjw6LIjEC1gqi0 --s3-access-key y7fvx1ne7DexQtiPtRES --s3-url http://169.254.233.107:9000 --s3-bucket mieli --zk-url localhost:2181

Todos

• Expose the thing as an experimental feature
• Should we do something about the encryption of the exchanges with zookeeper, or is it ok like that?
• What about analytics
• Stops registering the tasks and API-keys locally + sending it to zookeeper, which makes us register it a second time. We should register it only once. (send it to zookeeper and then let the watcher insert it in the DB for us, probably?)
• Index-scheduler
  • Patch the IndexScheduler::new to really be async (i.e.: use all the tokio thingy instead of using, for example, std::fs and spawn blocking when needed).
  • When setting up the run function, we should be able to return errors instead of panicking
  • We don’t handle the task cancellation in the register

What zookeepers should look like

/
  auth/
    {uuid of api-keys} -> json associated with the key
  tasks/
    task-{XX} -> XX is an Incremental number sync through the whole cluster
  election/
    node-{XX} -> XX is an incremental ephemeral number created at the start of each instance. The instance with the lowest number is the leader
  snapshots/
    snapshot-{XX} -> XX is a persistent, incremental number. The latest number is the most up-to-date snapshot  

• Every instance should install a watcher on auth to update its api-key accordingly to the zookeeper's fs in case of creation, update, or deletion.
• Every instance should install a watcher on tasks to insert the last enqueued tasks to its task queue
• election is used to run a leader election. Read more about the leader election process here: https://zookeeper.apache.org/doc/current/recipes.html#sc_leaderElection
• snapshots should be watched by everyone to pull the latest snapshot every time the leader finishes processing a task

What the S3 should look like

/
  updates_files/
    {uuidv4} -> the files associated with the document addition updates
  snapshots/
    snapshot-{XX}/ -> XX correspond to the number stored on zookeeper
      tasks.mdb -> the task queue
      indexes/
        {index-uid}.mdb -> the database of each indexes

• The update files can be written by everyone asynchronously without asking the leader.
• Every time a batch of tasks is processed, the leader must create a new snapshot-{XX} directory with all the databases.

---

related to #3494