feat(pydeck): support v9 in bindings #8577

Jesus89 · 2024-03-04T08:22:47Z

Fixes feat(pydeck): support v9 in bindings #8227

Implements the new @deck.gl/carto interface in pydeck-carto.

pydeck 0.9.0b0 → deck.gl v9.0-beta
pydeck-carto 0.2.0b0

Notes on `globalThis.CartoLibrary` fix

There is a limitation to publishing pydeck 0.8.1, which relies on the following change introduced in deck.gl v8.9: https://github.com/visgl/deck.gl/pull/7546/files#diff-06477719d4b3355d7e49ba5a2a05b02da47047163a0f0cdce08df83a8dc42d38L12

// Export carto layer library for pydeck integration
// More info: https://github.com/ajduberstein/pydeck_custom_layer
globalThis.CartoLibrary = CartoUtils;

This PR includes the fix, but it's likely to land in separate PRs first:

coveralls · 2024-03-04T08:27:46Z

coverage: 90.024%. remained the same
when pulling 5a98376 on jarroyo/pydeck-carto-v9
into 3773e9f on master.

bindings/pydeck-carto/pydeck_carto/sources.py

donmccurdy · 2024-03-28T20:37:12Z

I'm working on testing this PR locally, but currently hitting the same errors (AttributeError: 'NoneType' object has no attribute 'get'...) described in:

[Bug]: build failure pydeck #8469

I'll share updates when I resolve that, but just wanted to post the update here in case it's something you'd seen already.

EDIT: Resolved by deleting pyproject.toml.

donmccurdy · 2024-04-01T20:14:12Z

Updates on issues with local builds:

[Bug]: build failure pydeck #8469 (comment)

)

….gl v9 (#8806)

donmccurdy · 2024-04-22T18:39:44Z

bindings/pydeck-carto/examples/scripts/carto_layer_geo_query.py

@@ -26,4 +31,4 @@
 view_state = pdk.ViewState(latitude=0, longitude=0, zoom=1)

 r = pdk.Deck(layer, map_style=pdk.map_styles.ROAD, initial_view_state=view_state)
-r.to_html("carto_layer_geo_query.html", open_browser=True)
+r.to_html(join(dirname(__file__), "carto_layer_geo_query.html"))


Changed this line — in all example scripts — to ensure example output is written to the same folder as the example itself. This makes it easier to run all the example scripts together ...

make test-scripts

... and verify the output ...

npx serve examples/scripts

donmccurdy · 2024-04-22T19:05:22Z

bindings/pydeck-carto/requirements/requirements-dev.txt

 black==22.3.0
+carto-auth>=0.2.0
+flake8==4.0.1
 ipython>=7.8.0
-tokenize-rt>=3.2.0
-twine==4.0.0
+pytest-cov==3.0.0
+pytest-mock==3.8.2
 pytest==7.1.2
 requests-mock==1.9.3
-pytest-mock==3.8.2
-pytest-cov==3.0.0
+tokenize-rt>=3.2.0
+twine==4.0.0
+typing-extensions>=4.0.0
+wheel==0.37.1


Changes:

Add carto-auth

Add typing-extensions

Sort dependencies

donmccurdy · 2024-04-22T19:07:33Z

bindings/pydeck-carto/setup.py

@@ -20,7 +20,8 @@
    python_requires=">=3.7",
    install_requires=[
        "pydeck>=0.8.0",
-        "carto-auth>=0.1.0",
+        "carto-auth>=0.2.0",
+        "typing-extensions>=4.0.0",


typing-extensions is here to support type checking in Python versions < 3.10. Currently Pydeck supports Python 3.7+. As an alternative to the dependency, we could raise the minimum version, or do without the types.

This is a good time to bump the version requirements. 3.8 is a no-brainer as 3.7 is end-of-life: https://devguide.python.org/versions/

Dumb question: Do we need to support type checking in 3.8 and 3.9? Or could we just not offer that feature in those versions and document the limitation?

I looked more into which features of typing-extensions landed in which versions of Python, it turned out that while some typing features landed in 3.10, features we're using here landed in 3.11 and 3.12. Here are the versions for each, and what they do in pydeck:

typing.NotRequired: Python 3.11+, allows us to mark specific options required vs. optional easily. Without this we would give up the ability for the type hints to catch missing required dependencies.

typing.Unpack: Python 3.12+, Allows **kwargs to be typed, see PEP 692. Without this we would either give up the type hints, or need to explicitly duplicate all options for each of the foo_bar_source() functions.

typing.assert_type: Python 3.11+, convenient, but not that important to the PR.

It's too early to require Python 3.12+. I'm not sure that it's possible to support type checking in newer versions of Python in a backward-compatible way, without either publishing separate packages (non-starter) or including typing-extensions to polyfill support, as I've done here. I'm new to type-checking in Python though, there may be options I've overlooked.

But I agree it makes sense to bump the minimum version from 3.7 → 3.8 at least, regardless of any of the above — I'll open a separate PR for that.

EDIT: Opened chore(pydeck): Require Python 3.8+ #8811.

donmccurdy · 2024-04-22T19:09:07Z

modules/carto/bundle.ts

+
+// Export CARTO library for pydeck integration.
+// More info: https://github.com/ajduberstein/pydeck_custom_layer
+globalThis.CartoLibrary = CartoUtils;


This line is also included in #8799. I've filed the separate PR because the deck.gl v9.0.x patch release could potentially happen before the pydeck changes land.

donmccurdy · 2024-04-22T19:26:13Z

@Jesus89 @felixpalmer ready for review! If this PR will require more time, it would be helpful if we could go ahead and land #8799 to get the missing export into the next @deck.gl/carto 9.0.x patch release.

donmccurdy · 2024-04-24T19:54:22Z

Published beta releases to pydeck v0.9.0b0 and pydeck-carto v0.2.0b0. I've done limited testing so far, but both appear to be working as expected in Google Colab. Colab is using Python 3.10 (this can be changed, but I'm not sure how...) and isn't really benefiting from the type hints at this point, but works fine without them.

Jesus89 · 2024-04-25T07:21:33Z

It seems that the latest versions of jupyter + ipywidgets are not compatible with pydeck (even older versions). Also, deck.gl does not use webpack anymore to build the jupyter-widget package, so setting up jupyter-widget support will require and extra effort.

Failed to create view for 'ErrorWidgetView' from module '@jupyter-widgets/base' with model 'JupyterTransportModel' from module '@deck.gl/jupyter-widget'
TypeError: Cannot read properties of undefined (reading 'length')

In the past, we fallback the function show() into to_html() for Google Colab because ipywidgets can not work in that environment.

I would consider dropping support for ipywidgets in pydeck 0.9.0, fallback always show() into to_html() and simplify the code/docs regarding nbextension and @deck.gl/jupyter-widget.

Jesus89

@donmccurdy great job. LGTM ✔️

donmccurdy · 2024-04-26T17:55:44Z

Further tests for Deck v9 in Google Colab and Streamlit environments have been working well, so merging the PR to master now. As @Jesus89 mentions above, all Jupyter-specific features remain broken in any pydeck version (using Deck v8 or v9) newer than v0.8.0. Given...

no clear path to fix Jupyter-specific features for the older Jupyter versions Pydeck previously supported
those older Jupyter versions reach EOL next month
newer Jupyter versions will require very large efforts to rebuild support for the Jupyter-specific features

... my preference is also to disable the current Jupyter-specific features, publish Pydeck v0.9 without them, and regroup with pydeck community input on what could be done — both technically, and in terms of maintainer availability — to support two-way bindings moving forward.

* feat(pydeck): Update to Deck v9 * feat(pydeck-carto): Add new CARTO layers, sources, and auth flow (#8791, #8806) --------- Co-authored-by: Don McCurdy <donmccurdy@cartodb.com>

draft: new pydeck-carto layers/sources

ae63bc5

felixpalmer reviewed Mar 4, 2024

View reviewed changes

bindings/pydeck-carto/pydeck_carto/sources.py Outdated Show resolved Hide resolved

donmccurdy added the pydeck label Mar 29, 2024

donmccurdy added this to the v9.0 milestone Mar 29, 2024

donmccurdy added 3 commits April 11, 2024 22:32

Merge branch 'master' into jarroyo/pydeck-carto-v9

f01e49f

Bump pydeck semver constants to 9.0.x

3b427c8

Fix typo in CARTO sources test

6ddaeb8

This was referenced Apr 15, 2024

feat(pydeck-carto): Implement Deck v9 sources and type validation #8791

Merged

fix(pydeck-carto): Restore global required by pydeck-carto #8799

Merged

feat(pydeck-carto): Implement Deck v9 sources and type validation (#8791

2520d02

)

donmccurdy mentioned this pull request Apr 17, 2024

feat(pydeck-carto): Update H3, Quadbin, and styling examples for deck.gl v9 #8806

Merged

donmccurdy added 3 commits April 17, 2024 15:54

feat(pydeck-carto): Update H3, Quadbin, and styling examples for deck…

e9c1a94

….gl v9 (#8806)

Merge branch 'master' into jarroyo/pydeck-carto-v9

7547dba

Merge branch 'master' into jarroyo/pydeck-carto-v9

b5e1010

donmccurdy reviewed Apr 22, 2024

View reviewed changes

donmccurdy added 2 commits April 22, 2024 14:54

Clean up inline comments

22eb302

Clean up inline comments

4c18a97

donmccurdy reviewed Apr 22, 2024

View reviewed changes

donmccurdy marked this pull request as ready for review April 22, 2024 19:13

donmccurdy mentioned this pull request Apr 22, 2024

feat(pydeck): support v9 in bindings #8227

Closed

Merge branch 'master' into jarroyo/pydeck-carto-v9

e473251

padawannn approved these changes Apr 23, 2024

View reviewed changes

donmccurdy added 2 commits April 24, 2024 15:12

Merge branch 'master' into jarroyo/pydeck-carto-v9

7a7b03a

pydeck v0.9.0b0 + pydeck-carto v0.2.0b0

2475ff4

Pessimistress force-pushed the master branch 4 times, most recently from 0a083b6 to eb85da3 Compare April 25, 2024 03:09

Jesus89 commented Apr 25, 2024

View reviewed changes

Merge branch 'master' into jarroyo/pydeck-carto-v9

5a98376

donmccurdy approved these changes Apr 26, 2024

View reviewed changes

donmccurdy merged commit 9be0e4a into master Apr 26, 2024
4 checks passed

donmccurdy deleted the jarroyo/pydeck-carto-v9 branch April 26, 2024 17:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(pydeck): support v9 in bindings #8577

feat(pydeck): support v9 in bindings #8577

Jesus89 commented Mar 4, 2024 •

edited by donmccurdy

coveralls commented Mar 4, 2024 •

edited

donmccurdy commented Mar 28, 2024 •

edited

donmccurdy commented Apr 1, 2024

donmccurdy Apr 22, 2024 •

edited

donmccurdy Apr 22, 2024

donmccurdy Apr 22, 2024

ibgreen Apr 22, 2024

donmccurdy Apr 22, 2024 •

edited

donmccurdy Apr 22, 2024

donmccurdy commented Apr 22, 2024

donmccurdy commented Apr 24, 2024

Jesus89 commented Apr 25, 2024

Jesus89 left a comment

donmccurdy commented Apr 26, 2024 •

edited

feat(pydeck): support v9 in bindings #8577

feat(pydeck): support v9 in bindings #8577

Conversation

Jesus89 commented Mar 4, 2024 • edited by donmccurdy

coveralls commented Mar 4, 2024 • edited

donmccurdy commented Mar 28, 2024 • edited

donmccurdy commented Apr 1, 2024

donmccurdy Apr 22, 2024 • edited

Choose a reason for hiding this comment

donmccurdy Apr 22, 2024

Choose a reason for hiding this comment

donmccurdy Apr 22, 2024

Choose a reason for hiding this comment

ibgreen Apr 22, 2024

Choose a reason for hiding this comment

donmccurdy Apr 22, 2024 • edited

Choose a reason for hiding this comment

donmccurdy Apr 22, 2024

Choose a reason for hiding this comment

donmccurdy commented Apr 22, 2024

donmccurdy commented Apr 24, 2024

Jesus89 commented Apr 25, 2024

Jesus89 left a comment

Choose a reason for hiding this comment

donmccurdy commented Apr 26, 2024 • edited

Jesus89 commented Mar 4, 2024 •

edited by donmccurdy

coveralls commented Mar 4, 2024 •

edited

donmccurdy commented Mar 28, 2024 •

edited

donmccurdy Apr 22, 2024 •

edited

donmccurdy Apr 22, 2024 •

edited

donmccurdy commented Apr 26, 2024 •

edited