-
Notifications
You must be signed in to change notification settings - Fork 146
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
I would love to snowflake to support supports_multivalues_insert
#55
Comments
I think Snowflake supports it but the flag is not enabled for some reason. Trying add the flag. |
@wxianxin can you please educate how this flag impacts the functionality? I'm mostly sure I can add
to the base class, but I'm not clear what test case I should add to verify the impact. In my understanding, this flag indicates INSERT statement can have multiple values, e.g.,
My question is how SQLAlchemy transforms the objects into the above multi-values INSERT statement. |
@smtakeda Thanks for your attention. I am afraid that the answer to your question is: I don't know. I am not familiar with the design of SQLAlchemy. Why do I raise this issue? I am just a pandas and snowflake user. I often need to upload large dataframe to snowflake. Ideally I hope to use I have been tracking the development of pandas and I think they will add a feature to speed up the upload. Which is related to I can point you to related issues in pandas repo. For the flag, I think at one time I see it used in pandas to check whether the dialect supports this feature and behave accordingly. But now I can't find the specific code anymore. Maybe it doesn't really matter. As I am not an expert in this, feel free to close this issue in any way you see fit. Thanks! |
Actually Can you try adding the following two lines before the connection?
For example:
instead of:
|
Thanks. I have noticed 25% improvement in speed. |
In terms of performance, there is no much we can do at the moment. We are experimenting a scalable bulk bind but the performance is not good enough to enable for general purpose. About the hang, can you please file a ticket to the support including the log?. This code snippet can generate
|
Thanks for the update. I did 3 test run. I will separate the test info multiple messages:
My company's IT security stuff is pretty complicated. That could also contribute to the failure. So this is just for your reference. |
|
@wxianxin thanks for the perf test. will look into it when I have a chance. I would recommend to communicate with the support especially if we need to share the logs. We do our best not to include sensitive information in the log but cannot guarantee it never happen at any point of time. It would be safer to upload logs directly to the support ticket. Thanks. |
I agree. Thanks for the help! |
Please answer these questions before submitting your issue. Thanks!
What version of Python are you using (
python --version
)?python 3.7
What operating system and processor architecture are you using (
python -c 'import platform; print(platform.platform())'
)?Mac OS. X86
What are the component versions in the environment (
pip list
)?All latest as of today: 20180814
What did you do?
I got
No
. There is no support forsupports_multivalues_insert
. I feel heartbroken.A complete runnable program is good.
What did you expect to see?
Add support for pandas sql to fast upload dataframe to snowflake table.
What did you see instead?
Slow ORM unit of work.
Can you set logging to DEBUG and collect the logs?
Not necessary
The text was updated successfully, but these errors were encountered: