WIP: supports distributed acceleration of COPY [skip ci]

[Weijun-H]

What?

This pr is for #246

This feature will assist PolarDB for PostgreSQL in implementing distributed acceleration of COPY so that PolarDB PG can significantly improve performance when executing COPY statements in the case of large data volumes and fully utilise the HTAP framework.

The implementation includes

• For the COPY ... TO ... statement, multiple machines should be read in parallel for the read operation,
• For the COPY ... FROM ... statement should be written by multiple processes on a single read/write node.

How?

• global variable for the user to turn on this feature set polar_px_enable_copy=on
• In Copy query to File, make query use the PX optimizer
• Copy ... To ...
• Copy ... From ...

Benchmark

• Copy query to File
  

• Copy ... To ...
  

Screenshots (optional)

[polardb-bot]

Hi @Weijun-H ~ Congratulations to your first PR to PolarDB. 🎉

Please make sure that your PR conforms the standard, and has passed all the checks.

We will review your PR as soon as possible.

[polardb-bot]

Hey @Weijun-H :

Your checks have been cancelled ⛔️. Please re-run the checks if you want to merge this PR.

[polardb-bot]

Hey @Weijun-H :

Something wrong occuried during the checks of your commit 😟, please check the detail:

⚠️ regression (OLAP) container:polardb/polardb_pg_devel:centos7 View more details

✅ 01:10 clone
❌ 57:52 main

test polar-px-dev/plpgsql         ... ok (1.51 s)
test polar-px-dev/olap_setup      ... ok (0.24 s)
test polar-px-dev/qp_functions_in_contexts_setup ... ok (0.01 s)
test polar-px-dev/bfv_olap        ... ok (11.54 s)
test polar-px-dev/bfv_aggregate   ... ok (8.20 s)
test polar-px-dev/aggregate_with_groupingsets ... ok (0.29 s)
test polar-px-dev/gporca          ... ok (121.40 s)
test polar-px-dev/explain_analyze ... ok (1.45 s)
test polar-px-dev/polar_px_btbuild ... ok (21.51 s)
test polar-px-dev/with_clause     ... ok (25.63 s)
test polar-px-dev/qp_with_clause  ... ok (51.82 s)
test polar-px-dev/qp_with_functional_inlining ... ok (23.80 s)
test polar-px-dev/qp_with_functional_noinlining ... ok (30.25 s)
test polar-px-dev/qp_functions_in_with ... ok (0.04 s)
test polar-px-dev/shared_scan     ... ok (16.04 s)
test polar-px-dev/polar_px_btbuild_bulkwrite ... ok (32.89 s)
test polar-px-dev/bfv_partition_plans ... ok (32.32 s)
test polar-px-dev/partition_indexing ... ok (15.91 s)
test polar-px-dev/partition_subquery ... ok (1.11 s)
test polar-px-dev/bfv_index       ... ok (17.21 s)
test polar-px-dev/polar_px_btbuild_concurrently ... ok (24.68 s)
test polar-px-dev/left_semi_join  ... ok (6.79 s)
test polar-px-dev/polar_global_function ... ok (2.35 s)
test polar-px-dev/tableless_scan  ... ok (7.22 s)
test polar-px-dev/dynamic_partition_pruning_range ... ok (35.74 s)
test polar-px-dev/dynamic_partition_pruning_hash ... ok (16.96 s)
test polar-px-dev/result_hash_filter ... ok (1.42 s)
test polar-px-dev/const_expr_in_nested_loop ... ok (3.37 s)
test polar-px-dev/crash_in_nested_loop ... ok (0.64 s)
test polar-px-dev/assertop        ... ok (1.67 s)
test polar-px-dev/px_parallel_dml ... ok (125.67 s)
test polar-px-dev/polar_partition ... ok (26.31 s)
test polar-px-dev/left_index_nestloop_join ... ok (2.31 s)
test polar-px-dev/polar_partition_hash ... ok (15.58 s)
test polar-px-dev/polar_partition_alter_px_workers ... ok (12.37 s)
test polar-px-dev/squelch_deadlock ... ok (0.04 s)
test polar-px-dev/polar_partitionwise_join ... ok (22.50 s)
test polar-px-dev/px_parallel_update ... ok (105.52 s)
test polar-px-dev/polar_px_adaptive_scan ... ok (10.42 s)
test polar-px-dev/px_bugfix       ... ok (10.44 s)
test polar-px-dev/polar_multilevel_partition ... ok (150.80 s)
test polar-px-dev/polar_multilevel_hash_partition ... ok (29.41 s)
test polar-px-dev/polar_partition1_fromorca ... ok (43.51 s)
test polar-px-dev/px_spi_select   ... ok (1.51 s)


===========================================================
 All 64 tests passed. 

 POLARDB:
 All 65 tests, 0 tests in ignore, 1 tests in polar ignore. 
===========================================================

rm -f temp_config
make[1]: Leaving directory '/tmp/cirrus-ci-build/src/test/regress'
������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������

[polardb-bot]

Hey @Weijun-H :

Congratulations~ 🎉 Your commit has passed all the checks. Please wait for further manual review.

[polardb-bot]

Hey @Weijun-H :

Something wrong occuried during the checks of your commit 😟, please check the detail:

⚠️ regression (OLAP) container:polardb/polardb_pg_devel:centos7 View more details

✅ 01:01 clone
❌ 54:06 main

test polar-px-dev/plpgsql         ... ok (1.41 s)
test polar-px-dev/olap_setup      ... ok (0.28 s)
test polar-px-dev/qp_functions_in_contexts_setup ... ok (0.02 s)
test polar-px-dev/bfv_olap        ... ok (11.18 s)
test polar-px-dev/bfv_aggregate   ... ok (20.90 s)
test polar-px-dev/aggregate_with_groupingsets ... ok (0.37 s)
test polar-px-dev/gporca          ... ok (166.57 s)
test polar-px-dev/explain_analyze ... ok (2.27 s)
test polar-px-dev/polar_px_btbuild ... ok (20.96 s)
test polar-px-dev/with_clause     ... ok (8.64 s)
test polar-px-dev/qp_with_clause  ... ok (37.45 s)
test polar-px-dev/qp_with_functional_inlining ... ok (16.11 s)
test polar-px-dev/qp_with_functional_noinlining ... ok (14.95 s)
test polar-px-dev/qp_functions_in_with ... ok (0.04 s)
test polar-px-dev/shared_scan     ... ok (15.72 s)
test polar-px-dev/polar_px_btbuild_bulkwrite ... ok (17.40 s)
test polar-px-dev/bfv_partition_plans ... ok (25.40 s)
test polar-px-dev/partition_indexing ... ok (14.41 s)
test polar-px-dev/partition_subquery ... ok (0.05 s)
test polar-px-dev/bfv_index       ... ok (11.12 s)
test polar-px-dev/polar_px_btbuild_concurrently ... ok (16.12 s)
test polar-px-dev/left_semi_join  ... ok (1.93 s)
test polar-px-dev/polar_global_function ... ok (0.59 s)
test polar-px-dev/tableless_scan  ... ok (0.46 s)
test polar-px-dev/dynamic_partition_pruning_range ... ok (17.33 s)
test polar-px-dev/dynamic_partition_pruning_hash ... ok (18.10 s)
test polar-px-dev/result_hash_filter ... ok (1.87 s)
test polar-px-dev/const_expr_in_nested_loop ... ok (0.86 s)
test polar-px-dev/crash_in_nested_loop ... ok (0.54 s)
test polar-px-dev/assertop        ... ok (1.08 s)
test polar-px-dev/px_parallel_dml ... ok (197.68 s)
test polar-px-dev/polar_partition ... ok (47.89 s)
test polar-px-dev/left_index_nestloop_join ... ok (5.35 s)
test polar-px-dev/polar_partition_hash ... ok (21.62 s)
test polar-px-dev/polar_partition_alter_px_workers ... ok (20.72 s)
test polar-px-dev/squelch_deadlock ... ok (0.04 s)
test polar-px-dev/polar_partitionwise_join ... ok (31.84 s)
test polar-px-dev/px_parallel_update ... ok (106.07 s)
test polar-px-dev/polar_px_adaptive_scan ... ok (8.23 s)
test polar-px-dev/px_bugfix       ... ok (9.34 s)
test polar-px-dev/polar_multilevel_partition ... ok (83.96 s)
test polar-px-dev/polar_multilevel_hash_partition ... ok (23.32 s)
test polar-px-dev/polar_partition1_fromorca ... ok (19.81 s)
test polar-px-dev/px_spi_select   ... ok (2.00 s)


===========================================================
 All 64 tests passed. 

 POLARDB:
 All 65 tests, 0 tests in ignore, 1 tests in polar ignore. 
===========================================================

rm -f temp_config
make[1]: Leaving directory '/tmp/cirrus-ci-build/src/test/regress'
����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������

[HBKO]

增加 Copy 的回归 Case

[mrdrivingduck]

@Weijun-H Please add some regression test case, and undraft this MR, so that our CI can run all the cases.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ mrdrivingduck
❌ Weijun-H
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[mrdrivingduck]

@Weijun-H Any update about this PR?

[Weijun-H]

@Weijun-H Any update about this PR?

@mrdrivingduck I apologize for the delay in my response. I understand that there is currently a bug with the COPY FROM feature. I am working on fixing it, but my progress may be slower due to heavy coursework on campus.

[mrdrivingduck]

@Weijun-H Any update about this PR?

@mrdrivingduck I apologize for the delay in my response. I understand that there is currently a bug with the COPY FROM feature. I am working on fixing it, but my progress may be slower due to heavy coursework on campus.

That's ok. Please let us know if you need some help.

[mrdrivingduck]

@Weijun-H Any updates in this thread?

[Weijun-H]

@Weijun-H Any updates in this thread?

Sorry for this late response. I just picked this up recently.

[mrdrivingduck]

@Weijun-H Any updates in this thread?

Sorry for this late response. I just picked this up recently.

Glad to here that. Your dev branch is behind POLARDB_11_DEV for a lot. You can rebase your changes to the latest POLARDB_11_DEV, instead of merge with it.

[Weijun-H]

Would it be possible to compile the project in parallel? I've noticed that it's compiling very slowly on my machine.

[HBKO]

Use ./rds_build.sh --initpx command to complier in parallel.
Or you can use make -j after configure

[HBKO]

Or you can use
make -j
sudo make install
pg_ctl -D /home/postgres/tmp_master_dir_polardb_pg_1100_bld/ restart
pg_ctl -D /home/postgres/tmp_replica_dir_polardb_pg_1100_bld1/ restart
pg_ctl -D /home/postgres/tmp_replica_dir_polardb_pg_1100_bld2/ restart

after you rds_build.sh --initpx is done

[Weijun-H]

Use ./rds_build.sh --initpx command to complier in parallel. Or you can use make -j after configure

I did not find rds_build .sh

[HBKO]

polardb_build.sh

[HBKO]

You can follow the document.
https://apsaradb.github.io/PolarDB-for-PostgreSQL/zh/deploying/db-localfs.html#%E6%9C%AC%E5%9C%B0%E5%A4%9A%E8%8A%82%E7%82%B9%E5%B8%A6%E5%A4%87%E5%BA%93%E5%AE%9E%E4%BE%8B

[mrdrivingduck]

Use ./rds_build.sh --initpx command to complier in parallel. Or you can use make -j after configure

I did not find rds_build .sh

@Weijun-H polardb_build.sh is a script for building PolarDB's binary and also initdb.

Once your database is initialized, if you want to modify some code without initializing database again, you can directly make -j on the root path of source code to build new binary, and make install to copy the binary to install path. After that, you need to restart the database service from the new binary by pg_ctl -D $datapath restart. Hope it helps.

[Weijun-H]

Apologies for the delay in progress. I've been working on resolving the bug in my spare time, but unfortunately, I'm stuck and unsure of how to proceed. Could anyone lend me a hand and provide me with some hints or guidance?

When I tested the COPY TO, I met the following error because of a comment,

server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

But no idea how to modify PxDispatchCopyEnd to fit PolarDB architecture.
Thank you in advance. 🙇‍♂️

[mrdrivingduck]

Is there any *.core files in ~/tmp_master_xxx directory? If so, use gdb --core to see the call stack when the error happens.