Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Research faster PostgreSQL to Citus migration approach #7855

Closed
steven-sheehy opened this issue Mar 4, 2024 · 0 comments · Fixed by #8134
Closed

Research faster PostgreSQL to Citus migration approach #7855

steven-sheehy opened this issue Mar 4, 2024 · 0 comments · Fixed by #8134
Assignees
@jnels124
Labels
database Area: Database enhancement Type: New feature performance
Milestone
0.104.0

Comments

@steven-sheehy
Copy link
Member

Problem

The current approach to pg_dump from Cloud SQL to files then pg_restore those files to Citus takes a very long time.

Solution

  • Investigate feasibility of using dimitri/pgcopydb
  • Explore Kafka as a sink
  • Consider optimizations to current approach

Alternatives

No response
@steven-sheehy steven-sheehy added enhancement Type: New feature performance database Area: Database labels Mar 4, 2024
@steven-sheehy steven-sheehy added this to the 0.101.0 milestone Mar 4, 2024
@jnels124 jnels124 self-assigned this Mar 8, 2024
@steven-sheehy steven-sheehy modified the milestones: 0.101.0, 0.102.0 Mar 19, 2024
@steven-sheehy steven-sheehy modified the milestones: 0.102.0, 0.103.0 Apr 1, 2024
@steven-sheehy steven-sheehy modified the milestones: 0.103.0, 0.104.0 Apr 15, 2024
@jnels124 jnels124 mentioned this issue Apr 19, 2024
Merged
2 tasks
@jnels124 jnels124 linked a pull request Apr 19, 2024 that will close this issue
Speed up PostgreSQL to Citus migration #8134
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jnels124 jnels124

Labels
database Area: Database enhancement Type: New feature performance
Projects
Mirror Node
Status: ✅ Done
Milestone
0.104.0
Development

Successfully merging a pull request may close this issue.

Speed up PostgreSQL to Citus migration hashgraph/hedera-mirror-node
2 participants
@jnels124 @steven-sheehy