create_table() overhead time #6822
Replies: 1 comment 9 replies
-
@vmsaipreeth Thanks for opening a discussion! Can you confirm the cost of executing the queries without calling I'd like to be able to isolate the cost of the SQL query from the cost of Can you show the time to run |
Beta Was this translation helpful? Give feedback.
9 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi folks!
We have the latest version of ibis-framework which is
6.1.0
We have trino as our backend and we were exploring
create_table()
in order to write tables to trino.Below is the sample code.
We have also tried using
create_table()
with only providingschema
parameter and then usingconnection.insert()
approachWe tried multiple tests with increasing no. of rows and noticed some overhead time(roughly 1 min) while creating tables in trino using above mentioned approached
Here are some appropriate write times with no. of rows:
Test 1:
No. of rows: 14
Time took for writing tables: 1min 10s
Test 2:
No. of rows: ~36,000
Time took for writing tables: 1min 15s
Test 3:
No. of rows: 1.8Million
Time took for writing tables: 2min 30s
Test 4:
No. of rows: 2.3Million
Time took for writing tables: 2min 55s
Is there any way that we can reduce the overhead time here?
Please free to open an issue regarding the same.
Thanks!
Beta Was this translation helpful? Give feedback.
All reactions