Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WITH max parallel create index 1 is still parallelly run many CREATE INDEX and the importing of next table #1576

Open
5 tasks done
n0099 opened this issue Apr 12, 2024 · 0 comments

Comments

@n0099
Copy link

n0099 commented Apr 12, 2024

  • pgloader --version
    pgloader version "3.6.af8c3c1"
    compiled with SBCL 2.1.11.debian
    
  • did you test a fresh compile from the source tree?
  • did you search for other similar issues?
  • how can I reproduce the bug?
    LOAD DATABASE
         FROM mysql://user:@localhost/tbm
         INTO pgsql://user:@localhost/tbm
    
    WITH quote identifiers, max parallel create index 1
    ;
    
  • pgloader output you obtain
    2024-04-12T22:17:18.008001+08:00 LOG pgloader version "3.6.af8c3c1"                                                     
    2024-04-12T22:17:18.092001+08:00 LOG Migrating from #<MYSQL-CONNECTION mysql://user@localhost:3306/tbm {1007F8F823}>
    2024-04-12T22:17:18.092001+08:00 LOG Migrating into #<PGSQL-CONNECTION pgsql://user@localhost:5432/tbm {1007F90443}>
    2024-04-12T22:17:21.728043+08:00 LOG report summary reset
    

By periodically running

SELECT * FROM pg_stat_activity WHERE application_name = 'pgloader';

I can see when the COPY for first table is complete there will be multiple(as many as the count of indexes on that table) of CREATE TABLE running and the COPY of next table, this behavior will slow down database and all these queries.
Then I tried decreasing the parallelism by max parallel create index 1 https://pgloader.readthedocs.io/en/latest/batches.html

By default, as many CREATE INDEX threads as the maximum number of indexes per table are found in your source schema. It is possible to set the max parallel create index WITH option to another number in case there’s just too many of them to create.

but it's still the same.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant