Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Speed up PostgreSQL to Citus migration #8134

Merged
merged 16 commits into from Apr 29, 2024
Merged

Speed up PostgreSQL to Citus migration #8134

merged 16 commits into from Apr 29, 2024

Conversation

jnels124
Copy link
Contributor

@jnels124 jnels124 commented Apr 19, 2024 •
edited

Description:

  • add ability to execute migrations in batches
  • add ability to restart migration
  • use direct pipe instead of download/upload of csv
  • add config option to parallelize batch copies

Related issue(s):

Fixes #7855

Notes for reviewer:

Checklist

  • Documented (Code comments, README, etc.)
  • Tested (unit, integration, etc.)

@jnels124
checkpoint
caecc11 
Signed-off-by: Jesse Nelson <jesse@swirldslabs.com>
@jnels124
checkpoint
7699f34 
Signed-off-by: Jesse Nelson <jesse@swirldslabs.com>
@jnels124 jnels124 added enhancement Type: New feature parser Area: File parsing database Area: Database labels Apr 19, 2024
@jnels124 jnels124 self-assigned this Apr 19, 2024
@jnels124 jnels124 linked an issue Apr 19, 2024 that may be closed by this pull request
Research faster PostgreSQL to Citus migration approach #7855
Closed
@steven-sheehy steven-sheehy changed the title 7855 research faster postgresql to citus migration approach Speed up PostgreSQL to Citus migration Apr 19, 2024
@steven-sheehy steven-sheehy added this to the 0.104.0 milestone Apr 19, 2024
@jnels124
checkpoint
ab6a89a 
Signed-off-by: Jesse Nelson <jesse@swirldslabs.com>
@codecov Codecov
Copy link

codecov bot commented Apr 20, 2024 •
edited

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.39%. Comparing base (a65acaf) to head (1743697).

Additional details and impacted files 
@@             Coverage Diff              @@
##               main    #8134      +/-   ##
============================================
+ Coverage     92.26%   92.39%   +0.13%     
+ Complexity     7263     7154     -109     
============================================
  Files           897      879      -18     
  Lines         29244    28970     -274     
  Branches       3559     3536      -23     
============================================
- Hits          26981    26768     -213     
+ Misses         1448     1392      -56     
+ Partials        815      810       -5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jnels124
working bookmark
ceb0aea 
Signed-off-by: Jesse Nelson <jesse@swirldslabs.com>
@jnels124
fix race condition with transaction start and batch fetch
f4b7af6 
Signed-off-by: Jesse Nelson <jesse@swirldslabs.com>
@jnels124
fix race condition with transaction start and batch fetch
cd7e587 
Signed-off-by: Jesse Nelson <jesse@swirldslabs.com>
@jnels124 jnels124 requested a review from a team April 23, 2024 18:54
@jnels124 jnels124 force-pushed the 7855-research-faster-postgresql-to-citus-migration-approach branch 2 times, most recently from dac8871 to 1f8fc71 Compare April 23, 2024 19:05
@jnels124
switch from cursor to multi-binary copy jobs
8cf354a 
Signed-off-by: Jesse Nelson <jesse@swirldslabs.com>
@jnels124 jnels124 force-pushed the 7855-research-faster-postgresql-to-citus-migration-approach branch from 1f8fc71 to 8cf354a Compare April 23, 2024 19:12
@jnels124 jnels124 marked this pull request as ready for review April 23, 2024 19:13
@jnels124 jnels124 marked this pull request as draft April 23, 2024 22:02
@jnels124
cleanup
2053af9 
Signed-off-by: Jesse Nelson <jesse@swirldslabs.com>
@jnels124 jnels124 marked this pull request as ready for review April 24, 2024 03:27
@jnels124 jnels124 marked this pull request as draft April 24, 2024 15:14
@jnels124
Merge remote-tracking branch 'origin/main' into 7855-research-faster-…
a5b0bc3 
…postgresql-to-citus-migration-approach
@jnels124
bug fixes
b10f022 
Signed-off-by: Jesse Nelson <jesse@swirldslabs.com>
@jnels124 jnels124 marked this pull request as ready for review April 24, 2024 18:59
@jnels124
clean up from testing
a47381b 
Signed-off-by: Jesse Nelson <jesse@swirldslabs.com>
@jnels124 jnels124 force-pushed the 7855-research-faster-postgresql-to-citus-migration-approach branch from 3af0b5e to a47381b Compare April 25, 2024 16:40
steven-sheehy
steven-sheehy requested changes Apr 25, 2024
View reviewed changes
docs/database/README.md Outdated Show resolved Hide resolved
hedera-mirror-importer/src/main/resources/db/scripts/v2/migration.config Outdated Show resolved Hide resolved
hedera-mirror-importer/src/main/resources/db/scripts/v2/migration.sh Outdated Show resolved Hide resolved
hedera-mirror-importer/src/main/resources/db/scripts/v2/migration.config Outdated Show resolved Hide resolved
hedera-mirror-importer/src/main/resources/db/scripts/v2/migration.sh Outdated Show resolved Hide resolved
hedera-mirror-importer/src/main/resources/db/scripts/v2/migration.sh Outdated Show resolved Hide resolved
hedera-mirror-importer/src/main/resources/db/scripts/v2/migration.sh Outdated Show resolved Hide resolved
hedera-mirror-importer/src/main/resources/db/scripts/v2/migration.sh Outdated Show resolved Hide resolved
@jnels124
address pr feedback
d511679 
Signed-off-by: Jesse Nelson <jesse@swirldslabs.com>
@jnels124
Merge remote-tracking branch 'origin/main' into 7855-research-faster-…
18784c3 
…postgresql-to-citus-migration-approach
@jnels124
uncomment code
3a56bd1 
Signed-off-by: Jesse Nelson <jesse@swirldslabs.com>
@jnels124
fix typos
b99d60c 
Signed-off-by: Jesse Nelson <jesse@swirldslabs.com>
steven-sheehy
steven-sheehy reviewed Apr 26, 2024
View reviewed changes
docs/database/README.md Outdated Show resolved Hide resolved
docs/database/README.md Outdated Show resolved Hide resolved
@jnels124
update run example
1743697 
Signed-off-by: Jesse Nelson <jesse@swirldslabs.com>
@sonarcloud SonarCloud
Copy link

sonarcloud bot commented Apr 29, 2024

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

steven-sheehy
steven-sheehy approved these changes Apr 29, 2024
View reviewed changes
Copy link
Member

@steven-sheehy steven-sheehy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

xin-hedera
xin-hedera approved these changes Apr 29, 2024
View reviewed changes
Copy link
Collaborator

@xin-hedera xin-hedera left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jnels124 jnels124 merged commit f70bc44 into main Apr 29, 2024
31 checks passed
@jnels124 jnels124 deleted the 7855-research-faster-postgresql-to-citus-migration-approach branch April 29, 2024 16:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@steven-sheehy steven-sheehy steven-sheehy approved these changes

@xin-hedera xin-hedera xin-hedera approved these changes

Assignees

@jnels124 jnels124

Labels
database Area: Database enhancement Type: New feature parser Area: File parsing
Projects
None yet
Milestone
0.104.0
Development

Successfully merging this pull request may close these issues.

Research faster PostgreSQL to Citus migration approach
3 participants
@jnels124 @steven-sheehy @xin-hedera