Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Add config for memory pool init capacity to reduce arbitration times #5815

Merged
merged 2 commits into from
May 22, 2024

Conversation

Yohahaha
Copy link
Contributor

@Yohahaha Yohahaha commented May 20, 2024

What changes were proposed in this pull request?

Add new config spark.gluten.sql.columnar.backend.velox.memInitCapacity to reduce memory arbitration times, default value is 8M.

Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

Copy link

Run Gluten Clickhouse CI

@Yohahaha Yohahaha force-pushed the arbitrator-memory-init-capacity branch from 6b92da8 to 12d4024 Compare May 20, 2024 13:39
Copy link

Run Gluten Clickhouse CI

@Yohahaha Yohahaha changed the title [VL] Add Arbitrator memory init capacity [VL] Add config for memory pool init capacity to reduce arbitration times May 20, 2024
@FelixYBW
Copy link
Contributor

We previously hardcoded it as 8M, this PR make it configurable, right?

@Yohahaha
Copy link
Contributor Author

We previously hardcoded it as 8M, this PR make it configurable, right?

preciously is 0.

@zhztheplayer
Copy link
Member

Add new config spark.gluten.sql.columnar.backend.velox.memInitCapacity to reduce memory arbitration times

Is there any difference on performance in your test observed within the change? @Yohahaha

@Yohahaha
Copy link
Contributor Author

Add new config spark.gluten.sql.columnar.backend.velox.memInitCapacity to reduce memory arbitration times

Is there any difference on performance in your test observed within the change? @Yohahaha

currently, no, but it can reduce the frequency of enter arbitration in theory, and may have benefits when set this config larger, eg 128M.

zhztheplayer
zhztheplayer previously approved these changes May 21, 2024
Copy link

Run Gluten Clickhouse CI

@Yohahaha
Copy link
Contributor Author

/Benchmark Velox

1 similar comment
@Yohahaha
Copy link
Contributor Author

/Benchmark Velox

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_5815_time.csv log/native_master_05_13_2024_33f993554_time.csv difference percentage
q1 33.80 36.42 2.619 107.75%
q2 23.80 23.70 -0.103 99.57%
q3 36.63 36.61 -0.019 99.95%
q4 38.34 37.83 -0.513 98.66%
q5 70.22 70.62 0.391 100.56%
q6 7.69 7.47 -0.222 97.12%
q7 83.23 85.08 1.850 102.22%
q8 84.52 84.30 -0.220 99.74%
q9 123.03 121.00 -2.030 98.35%
q10 48.44 45.37 -3.074 93.65%
q11 20.49 19.49 -1.004 95.10%
q12 26.99 26.52 -0.465 98.28%
q13 53.52 55.08 1.556 102.91%
q14 17.87 21.53 3.663 120.50%
q15 29.86 29.02 -0.844 97.17%
q16 13.01 14.00 0.990 107.61%
q17 103.31 103.62 0.310 100.30%
q18 144.04 146.24 2.194 101.52%
q19 13.51 13.35 -0.165 98.78%
q20 28.75 29.06 0.311 101.08%
q21 283.60 282.54 -1.057 99.63%
q22 15.21 14.61 -0.599 96.06%
total 1299.87 1303.44 3.570 100.27%

@Yohahaha Yohahaha merged commit 1c6c7fc into apache:main May 22, 2024
40 checks passed
@Yohahaha Yohahaha deleted the arbitrator-memory-init-capacity branch May 22, 2024 01:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants