Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: improve streaming performance #240

Merged
merged 2 commits into from Feb 23, 2021
Merged

Conversation

larkee
Copy link
Contributor

@larkee larkee commented Feb 22, 2021

This PR fixes performance issues that were introduced during the mirgration to v2 and optimizes several streaming methods.

From v2 onwards, the library uses proto-plus to produce user friendly Python object to represent proto messages. Unfortunately, it significantly impacted the performance of streaming for cases with many columns as the library must unnecessarily parse the types many times. By reverting the streaming changes to use the raw protobuf messages, the streaming speed is greatly improved.

Fixes #207, fixes #234

@product-auto-label product-auto-label bot added the api: spanner Issues related to the googleapis/python-spanner API. label Feb 22, 2021
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Feb 22, 2021
@larkee larkee marked this pull request as ready for review February 23, 2021 05:35
@larkee larkee requested a review from a team as a code owner February 23, 2021 05:35

def _merge_values(self, values):
"""Merge values into rows.

:type values: list of :class:`~google.protobuf.struct_pb2.Value`
:param values: non-chunked values from partial result set.
"""
width = len(self.fields)
print(self.fields)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did this make it into the final code?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: spanner Issues related to the googleapis/python-spanner API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
3 participants