feat: convert StreamedResultSet to Pandas Dataframe #226

Hardikr23 · 2021-02-08T07:22:10Z

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

Fixes #155 🦕

google-cla · 2021-02-08T07:22:13Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

Hardikr23 · 2021-02-08T07:32:55Z

@tswast Please review the code and let us know your feedback

tswast

You're missing the following:

Unit tests -- show a DataFrame can be created successfully with 0 rows, 1 rows, many rows
System tests -- show that a DataFrame can be created from a real query. (Also with the emulator)
Update setup.py to document the "extras" (optional dependency) on pandas, including which versions of pandas are supported.

google/cloud/spanner_v1/_pandas_helpers.py

google/cloud/spanner_v1/streamed.py

google/cloud/spanner_v1/_pandas_helpers.py

google-cla · 2021-02-26T13:42:32Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-02-26T13:52:05Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-02-26T17:49:09Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

Hardikr23 · 2021-03-02T16:25:43Z

@googlebot I fixed it.

google-cla · 2021-03-02T16:25:50Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

tswast

Nits re: license headers

tests/system/test_system_pandasDf.py

tests/unit/test__pandas_helpers.py

Co-authored-by: Tim Swast <swast@google.com>

google-cla · 2021-03-08T11:37:39Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-03-08T11:37:39Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

Co-authored-by: Tim Swast <swast@google.com>

google-cla · 2021-03-08T11:38:00Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-03-08T11:38:00Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-03-10T16:37:35Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-03-18T18:11:32Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

tswast · 2021-03-22T14:17:40Z

tests/system/test_system_pandasdf.py

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from google.cloud import spanner_v1_mod


Tests are failing because this import is incorrect.

Suggested change

from google.cloud import spanner_v1_mod

from google.cloud import spanner_v1

Got it. Changed

tswast · 2021-03-22T14:20:21Z

google/cloud/spanner_v1/_pandas_helpers.py

+        if v.name == "TIMESTAMP" or v.name == "DATE":
+            try:
+                df[k] = df[k].to_datetime()
+            except Exception as e:


Can we please use a more specific exception? Exception is way too broad.

Also, please add some comments about why we have these two different methods of converting to datetime. I don't understand it.

Also also, please add back in the "localize" logic to the TIMESTAMP type. TIMESTAMP is intended to represent UTC datetimes.

Also also also, we need tests for the datetime64[ns, UTC] data type. I want to make sure the data type is the one we expect in the test (not just that we got a row and it didn't error)

For example, in pandas-gbq we test that the dataframe we got matches one that we manually created and localized in the test.

https://github.com/pydata/pandas-gbq/blob/46c579ac21879b431c8568b49e68624f4a5ea25e/tests/unit/test_timestamp.py

Also also, please add back in the "localize" logic to the TIMESTAMP type. TIMESTAMP is intended to represent UTC datetimes.

added

tswast · 2021-03-22T14:24:11Z

tests/system/test_system_pandasdf.py

+TABLE_NAME_1 = 'Functional_Alltypes'
+COLUMNS_1 = ['id','bool_col','date','float_col','string_col','timestamp_col']
+VALUES_1 = [
+        [1,True,'2016-02-09',2.2,'David','2002-02-10T15:30:00.45Z'],


Indentation looks off. Can you run nox -s blacken ?

tswast · 2021-03-22T14:26:37Z

tests/system/test_system_pandasdf.py

+VALUES = [[1, 'Alice'], [2, 'Bob']]
+
+
+TABLE_NAME_1 = 'Functional_Alltypes'


This looks like the name from the BigQuery Ibis tests. Please use the existing all_types table from

python-spanner/tests/_fixtures.py

Line 31 in e990ff7

CREATE TABLE all_types (

if possible.

[Optional] In fact, we may be able to import test data from here:

python-spanner/tests/system/test_system.py

Line 1096 in e990ff7

ALL_TYPES_ROWDATA = (

For example, the DB-API tests import some useful test fixtures.

python-spanner/tests/system/test_system_dbapi.py

Lines 32 to 39 in e990ff7

from .test_system import (

CREATE_INSTANCE,

EXISTING_INSTANCES,

INSTANCE_ID,

USE_EMULATOR,

_list_instances,

Config,

)

tswast · 2021-03-22T14:27:36Z

tests/system/test_system_pandasdf.py

+import pytest
+
+# for referrence
+TABLE_NAME = 'testTable'


Reusing the contacts table would be more appropriate than creating our own.

python-spanner/tests/_fixtures.py

Line 19 in e990ff7

CREATE TABLE contacts (

google/cloud/spanner_v1/_pandas_helpers.py

larkee · 2021-03-18T07:22:29Z

google/cloud/spanner_v1/streamed.py

+        """Creates a pandas DataFrame of all rows in the result set
+
+        :rtype: pandas.DataFrame
+


Suggested change

google-cla · 2021-03-25T17:59:42Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-03-25T18:05:04Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-03-25T18:52:51Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-03-25T19:22:58Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-03-25T19:22:58Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-04-07T15:58:31Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-04-07T16:57:56Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

google-cla · 2021-04-08T18:11:09Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

yaozhang09 · 2023-09-25T23:53:46Z

any update on this feature? ^

Hardik Rathod added 2 commits February 8, 2021 12:13

Added feature to return result in pandas-df

483e9fb

Added feature to return result in pandas-df

362c143

Hardikr23 requested a review from a team as a code owner February 8, 2021 07:22

google-cla bot added the cla: no This human has *not* signed the Contributor License Agreement. label Feb 8, 2021

product-auto-label bot added the api: spanner Issues related to the googleapis/python-spanner API. label Feb 8, 2021

Hardikr23 changed the title ~~Feature to Convert StreamedResultSet to Pandas Dataframe~~ feat :Convert StreamedResultSet to Pandas Dataframe Feb 8, 2021

Hardikr23 marked this pull request as draft February 8, 2021 07:29

Hardikr23 marked this pull request as ready for review February 8, 2021 07:33

larkee requested a review from tswast February 15, 2021 11:20

larkee changed the title ~~feat :Convert StreamedResultSet to Pandas Dataframe~~ feat: convert StreamedResultSet to Pandas Dataframe Feb 17, 2021

tswast requested changes Feb 17, 2021

View reviewed changes

push check

fc5ab60

added unit and system testcases and PR feedback

cb5733a

made changes in test.skip() logic

0aee1d8

tswast reviewed Mar 5, 2021

View reviewed changes

tests/system/test_system_pandasDf.py Outdated Show resolved Hide resolved

tests/unit/test__pandas_helpers.py Outdated Show resolved Hide resolved

Update tests/system/test_system_pandasDf.py

e7a4d4c

Co-authored-by: Tim Swast <swast@google.com>

Update tests/unit/test__pandas_helpers.py

537d498

Co-authored-by: Tim Swast <swast@google.com>

Merge branch 'master' into master

b4d715e

Added test cases for null values

a83a7ce

tswast reviewed Mar 22, 2021

View reviewed changes

larkee reviewed Mar 24, 2021

View reviewed changes

hardikPR added 2 commits March 25, 2021 23:03

removed _mod from the imports

be5c241

cosmetic changes done

4057e76

added to_localize()

ad1d039

tswast added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Mar 25, 2021

Merge branch 'master' into master

f460174

tswast added kokoro:force-run Add this label to force Kokoro to re-run the tests. and removed kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Mar 25, 2021

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Mar 25, 2021

revert grpcio intersphinx change

7906b14

tswast added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Mar 31, 2021

tswast mentioned this pull request Apr 1, 2021

cleanup spanner connector to delete duplicate code from google-cloud-spanner library GoogleCloudPlatform/professional-services-data-validator#225

Closed

2 tasks

Merge remote-tracking branch 'upstream/master' into Hardikr23/master

f919276

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Apr 7, 2021

add pandas extras to noxfile

0028b7b

tswast added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Apr 7, 2021

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Apr 7, 2021

move import to more standard location

0104348

	from google.cloud import spanner_v1_mod
	from google.cloud import spanner_v1

		VALUES = [[1, 'Alice'], [2, 'Bob']]


		TABLE_NAME_1 = 'Functional_Alltypes'

	from .test_system import (
	CREATE_INSTANCE,
	EXISTING_INSTANCES,
	INSTANCE_ID,
	USE_EMULATOR,
	_list_instances,
	Config,
	)

		"""Creates a pandas DataFrame of all rows in the result set

		:rtype: pandas.DataFrame

feat: convert StreamedResultSet to Pandas Dataframe #226

Are you sure you want to change the base?

feat: convert StreamedResultSet to Pandas Dataframe #226

Conversation

Hardikr23 commented Feb 8, 2021 • edited by tswast

google-cla bot commented Feb 8, 2021

Hardikr23 commented Feb 8, 2021

tswast left a comment

Choose a reason for hiding this comment

google-cla bot commented Feb 26, 2021

google-cla bot commented Feb 26, 2021

google-cla bot commented Feb 26, 2021

Hardikr23 commented Mar 2, 2021

google-cla bot commented Mar 2, 2021

tswast left a comment

Choose a reason for hiding this comment

google-cla bot commented Mar 8, 2021

google-cla bot commented Mar 8, 2021

google-cla bot commented Mar 8, 2021

google-cla bot commented Mar 8, 2021

google-cla bot commented Mar 10, 2021

google-cla bot commented Mar 18, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

google-cla bot commented Mar 25, 2021

google-cla bot commented Mar 25, 2021

google-cla bot commented Mar 25, 2021

google-cla bot commented Mar 25, 2021

google-cla bot commented Mar 25, 2021

google-cla bot commented Apr 7, 2021

google-cla bot commented Apr 7, 2021

google-cla bot commented Apr 8, 2021

yaozhang09 commented Sep 25, 2023

Hardikr23 commented Feb 8, 2021 •

edited by tswast