Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Speed up PostgreSQL to Citus migration #8134

Merged
merged 16 commits into from
Apr 29, 2024
Merged

Speed up PostgreSQL to Citus migration #8134

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
caecc11
checkpoint
jnels124 Apr 18, 2024
7699f34
checkpoint
jnels124 Apr 19, 2024
ab6a89a
checkpoint
jnels124 Apr 20, 2024
ceb0aea
working bookmark
jnels124 Apr 21, 2024
f4b7af6
fix race condition with transaction start and batch fetch
jnels124 Apr 22, 2024
cd7e587
fix race condition with transaction start and batch fetch
jnels124 Apr 23, 2024
8cf354a
switch from cursor to multi-binary copy jobs
jnels124 Apr 23, 2024
2053af9
cleanup
jnels124 Apr 24, 2024
a5b0bc3
Merge remote-tracking branch 'origin/main' into 7855-research-faster-…
jnels124 Apr 24, 2024
b10f022
bug fixes
jnels124 Apr 24, 2024
a47381b
clean up from testing
jnels124 Apr 25, 2024
d511679
address pr feedback
jnels124 Apr 26, 2024
18784c3
Merge remote-tracking branch 'origin/main' into 7855-research-faster-…
jnels124 Apr 26, 2024
3a56bd1
uncomment code
jnels124 Apr 26, 2024
b99d60c
fix typos
jnels124 Apr 26, 2024
1743697
update run example
jnels124 Apr 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 0 additions & 1 deletion charts/hedera-mirror/ci/v2-values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ graphql:
importer:
env:
SPRING_FLYWAY_PLACEHOLDERS_HASHSHARDCOUNT: "2"
SPRING_FLYWAY_PLACEHOLDERS_IDPARTITIONSIZE: "1000000000000000"
SPRING_FLYWAY_PLACEHOLDERS_PARTITIONTIMEINTERVAL: "'10 years'"
SPRING_FLYWAY_PLACEHOLDERS_SHARDCOUNT: "2"
SPRING_PROFILES_ACTIVE: v2
Expand Down
14 changes: 8 additions & 6 deletions docs/database/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -268,14 +268,16 @@ Following are the prerequisites and steps for migrating V1 data to V2.

1. Create a Citus cluster with enough resources (Disk, CPU and memory). For GKE, an e2-custom-6-32768 can be used.
2. Ensure the source and target schemas are compatible by deploying the same version to both.
3. Create a VM with a disk big enough to contain the source data and checkout the source code to it.
4. Populate correct values for the source and target configuration in the
3. Populate correct values for the source and target configuration in the
[migration.config](/hedera-mirror-importer/src/main/resources/db/scripts/v2/migration.config).
5. Get the correct version of [flyway](https://flywaydb.org/documentation/usage/commandline/) based on your OS and
4. Get the correct version of [flyway](https://flywaydb.org/documentation/usage/commandline/) based on your OS and
update it in the `FLYWAY_URL` field in the `migration.config` file. The default is set to the linux version.
6. Stop the [Importer](/docs/importer/README.md) process.
7. Run the [migration.sh](/hedera-mirror-importer/src/main/resources/db/scripts/v2/migration.sh) script.
8. Update the mirror node configuration to point to the new Citus DB and start it.
5. Stop the [Importer](/docs/importer/README.md) process.
6. Run the [migration.sh](/hedera-mirror-importer/src/main/resources/db/scripts/v2/migration.sh) script.
7. Update the mirror node configuration to point to the new Citus DB and start it.

### Migration notes
1. It is recommended to run the migration script in a screen/tmux session and pipe stderr and stdout to separate locations.
steven-sheehy marked this conversation as resolved.
Show resolved Hide resolved

## Citus Backup and Restore

Expand Down
6 changes: 5 additions & 1 deletion hedera-mirror-importer/src/main/resources/db/scripts/v2/migration.config
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
ASYNC_TABLES=("'crypto_transfer'","'transaction'")
ASYNC_TABLE_SPLITS=
CONCURRENCY=
EXCLUDED_TABLES=("'entity_state_start'","'flyway_schema_history'","'transaction_hash'")
CONCURRENT_COPIES_PER_TABLE=
CREATE_IDEXES_BEFORE_MIGRATION=
steven-sheehy marked this conversation as resolved.
Show resolved Hide resolved
steven-sheehy marked this conversation as resolved.
Show resolved Hide resolved
EXCLUDED_TABLES=("'entity_state_start'","'entity_transaction'","'flyway_schema_history'","'transaction_hash'")
FLYWAY_URL=https://repo1.maven.org/maven2/org/flywaydb/flyway-commandline/9.22.3/flyway-commandline-9.22.3-linux-x64.tar.gz
MAX_TIMESTAMP=
SOURCE_DB_HOST=
Expand Down