$$\      $$\           $$\ $$\ $$\                                                   $$\                                                                             $$\
$$$\    $$$ |          \__|$$ |\__|                                                  $$ |                                                                            $$ |
$$$$\  $$$$ | $$$$$$\  $$\ $$ |$$\  $$$$$$$\  $$$$$$\   $$$$$$\   $$$$$$\   $$$$$$$\ $$$$$$$\           $$$$$$$\  $$$$$$\  $$$$$$$\  $$$$$$$\   $$$$$$\   $$$$$$$\ $$$$$$\    $$$$$$\   $$$$$$\
$$\$$\$$ $$ |$$  __$$\ $$ |$$ |$$ |$$  _____|$$  __$$\  \____$$\ $$  __$$\ $$  _____|$$  __$$\ $$$$$$\ $$  _____|$$  __$$\ $$  __$$\ $$  __$$\ $$  __$$\ $$  _____|\_$$  _|  $$  __$$\ $$  __$$\
$$ \$$$  $$ |$$$$$$$$ |$$ |$$ |$$ |\$$$$$$\  $$$$$$$$ | $$$$$$$ |$$ |  \__|$$ /      $$ |  $$ |\______|$$ /      $$ /  $$ |$$ |  $$ |$$ |  $$ |$$$$$$$$ |$$ /        $$ |    $$$$$$$$ |$$ |  \__|
$$ |\$  /$$ |$$   ____|$$ |$$ |$$ | \____$$\ $$   ____|$$  __$$ |$$ |      $$ |      $$ |  $$ |        $$ |      $$ |  $$ |$$ |  $$ |$$ |  $$ |$$   ____|$$ |        $$ |$$\ $$   ____|$$ |
$$ | \_/ $$ |\$$$$$$$\ $$ |$$ |$$ |$$$$$$$  |\$$$$$$$\ \$$$$$$$ |$$ |      \$$$$$$$\ $$ |  $$ |        \$$$$$$$\ \$$$$$$  |$$ |  $$ |$$ |  $$ |\$$$$$$$\ \$$$$$$$\   \$$$$  |\$$$$$$$\ $$ |
\__|     \__| \_______|\__|\__|\__|\_______/  \_______| \_______|\__|       \_______|\__|  \__|         \_______| \______/ \__|  \__|\__|  \__| \_______| \_______|   \____/  \_______|\__|

                                                                                        Version: 0.1
                                                                                        Author: dattd

Features:

1. Synchronize data from your data source to Meilisearch.
2. Perform change data capture (CDC) to remain data consistency.
3. Refresh data in case of inconsistency between your data source and Meilisearch instance using Meilisearch's swap index feature (performing cost no downtime).
4. Show consistent status between your datasource and Meilisearch instance (record wise).
5. Config search policy of each index. For more documentation please refer to official Meilisearch documentation here .

Synchronize your datasource (currently support PostgresSQL) with Meilisearch instance.

PostgresSQL:

Requirement:

• Support Postgres version: above version 12
• Plugin wal2json must be installed associate with Postgres version on your local machine:

  In Red Hat/CentOS:

  sudo yum install wal2json14

  In Debian/Ubuntu:

  sudo apt-get install postgresql-14-wal2json

  In MacOS:

  brew install wal2json

• Make sure to config Postgres wal_level and have enough replication slot:

wal_level = logical
#
# these parameters only need to set in versions 9.4, 9.5 and 9.6
# default values are ok in version 10 or later
#

# each table config to sync with Meilisearch required 1 replication slot.
max_replication_slots = 10
max_wal_senders = 10

Meilisearch:

Make sure to cover core concepts of Meilisearch such as index, document, search policy and more at: https://www.meilisearch.com/docs

How to build

cargo build --release

How to run

• Show help

cargo run -- -h

• Sync data and perform CDC

  cargo run -- --config <path> sync

• Check for consistency

  cargo run -- --config <path> status

• Refresh data by swap index

  cargo run -- --config <path> refresh

Configuration file example

meilisearch:
  apiUrl: http://localhost:7700
  adminApiKey: "4d210c85a27082683758d083d6c1298ee7e1e97385edeaad3250b9cd8aebd276"
dataSource:
  sourceType: Postgres
  host: localhost
  port: 5432
  username: dattran
  password: dattran
  database: postgres
synchronizeTables:
  - indexName: users
    primaryKey: user_id
    limit: 1000
    syncFields:
      - user_id
      - username
      - created_at
    meiliSetting:
      displayedAttributes:
        - user_id
        - username
      searchableAttributes:
        - username
  - indexName: employees
    primaryKey: employee_id
    limit: 10
    syncFields:
      - employee_id
      - first_name

* Note: This is a POC and still in development process. Base source can change drastically in the future, please consider to use in production environment with real data.