[pydeck] Create a prebuilt extension for JupyterLab #7030

dakoop · 2022-06-11T00:47:33Z

The general idea is to create a prebuilt extension that can be distributed as part of the pydeck python package. There is no need for a separate @deck.gl/jupyter-widget install as it is bundled in the pydeck python package, and the extension does not require jupyterlab to be rebuilt as it uses webpack 5's module federation. Just run pip install pydeck. This is WIP as it doesn't currently check for compatibility with classic notebook support, etc. To build,

yarn bootstrap
Install build and jupyter_packaging for python
Run python -m build in bindings/pydeck
a. If this doesn't work, try jlpm build in modules/jupyter-widget
b. This should create a wheel in dist
pip install dist/*.whl

Change List

Integrates @jupyterlab/builder and webpack 5 into the jupyter-widget build process. This is a one-step build now as the jupyter labextension build uses webpack to create a federated module (no separate dist/index.js)
Removes acorn resolution in the root package.json. Some part of the new build process requires a newer version of acorn, as the build just hangs otherwise. I am not sure what the effect is on other parts of deck.gl.
Writes the generated labextension to pydeck/labextension in bindings/pydeck so it can be distributed as part of the python wheel.
Supports the python -m build method for building pydeck and the labextension
Updates some metadata suggested in jupyterlab extensions
Updates pydeck version to 0.8.0
Builds on [pydeck] Update jupyter-widget to work with JupyterLab 3 #7026

Todo

Fix issue with jupyter labextension develop . (something with pyproject.toml?)
Check whether other pieces of pydeck and jupyter-widget function as before
Determine impact of acorn resolution
Fix other pydeck setup/build code (e.g. Makefile)
Update documentation

See https://ipywidgets.readthedocs.io/en/stable/changelog.html#updates-for-widget-maintainers

Note that this ties it to 8.8.0-alpha.6; see visgl#7025 for comments

bump_version.py generates these files so could be updated

The main thing is modifying package.json and webpack.config.js to work with the jupyterlab builder. Since the previous build was just using webpack, we let the builder integrate the webpack config from that build with the labextension webpack. Required some changes to remove the original dist directory so everything just goes to a pydeck/labextension directory now.

kylebarron · 2022-06-13T18:36:08Z

install as it is bundled in the pydeck python package

iirc this is only valid for specific jupyter versions. Only for notebook as of version 5.3 and Jupyter lab as of version 3.

kylebarron · 2022-06-13T18:42:12Z

bindings/pydeck/pydeck/_version.py

@@ -1 +1 @@
-__version__ = "0.7.1"
+__version__ = "0.8.0"


Nit: I'd bump the version in a follow-up PR

kylebarron · 2022-06-13T18:43:22Z

bindings/pydeck/pyproject.toml

-  "setuptools",
-  "wheel",
-]
+requires = ["jupyter_packaging~=0.10,<2", "jupyterlab~=3.1"]


Why do you include jupyterlab as a build dependency? The jupyter-packaging docs don't seem to require it: https://github.com/jupyter/jupyter-packaging/#as-a-build-backend

kylebarron · 2022-06-13T18:43:47Z

bindings/pydeck/pyproject.toml


-[tool.black]


Why delete the black config section?

kylebarron · 2022-06-13T18:47:10Z

bindings/pydeck/RELEASE.md

+
+```bash
+npm login
+npm publish --access public


Nit: we should be able to put --access public in the package.json I think so we don't need to specify it manually. Ref
https://docs.npmjs.com/cli/v8/configuring-npm/package-json#publishconfig and https://docs.npmjs.com/cli/v8/using-npm/config#access

kylebarron · 2022-06-13T18:48:15Z

bindings/pydeck/RELEASE.md

+python -m build
+```
+
+> `python setup.py sdist bdist_wheel` is deprecated and will not work for this package.


Can you update the Makefile with the new commands as well?

kylebarron · 2022-06-13T18:53:00Z

modules/jupyter-widget/pyproject.toml

+
+[tool.jupyter-packaging.build-args]
+build_cmd = "build:prod"
+source_dir = "../../modules/jupyter-widget"


This path doesn't make sense here or at least is unnecessary

kylebarron · 2022-06-13T18:54:35Z

bindings/pydeck/pyproject.toml

+build_dir = "pydeck/labextension"
+npm = ["jlpm"]
+
+[tool.check-manifest]


The check-manifest config is defined here but it seems there's instructions on how to use it.

kylebarron · 2022-06-13T18:55:00Z

bindings/pydeck/pyproject.toml

+factory = "jupyter_packaging.npm_builder"
+
+[tool.jupyter-packaging.build-args]
+build_cmd = "build:labextension"


This only builds the labextension?

kylebarron · 2022-06-13T18:58:37Z

bindings/pydeck/setup.py


+    logging.basicConfig(format="%(levelname)s: %(message)s")
+    logging.warning(
+        "Build tool `jupyter-packaging` is missing. Install it with pip or conda."


It's not clear to me why this should be a warning not an error. It seems like it could lead to hard-to-debug issues where the extension isn't updating as expected. Additionally, I would expect that jupyter-packaging is always installed given that it's defined in build-requirements in the pyproject.toml.

kylebarron · 2022-06-13T19:01:54Z

bindings/pydeck/setup.py

+    author_email=pkg_json["author"]["email"],
+    description=pkg_json["description"],
+    license=pkg_json["license"],
+    license_file="LICENSE.txt",


Given that in this PR you're now defining more of the build process in pyproject.toml, move the non-dynamic parts of the setuptools config there too? https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html

kylebarron · 2022-06-13T19:04:24Z

package.json

@@ -20,7 +20,6 @@
    "prettier: 2.3.2 breaks on `import type`, remove when ocular bumps"
  ],
  "resolutions": {
-    "acorn": "7.4.1",


cc @Pessimistress on this. I don't know why it was previously pinned

kylebarron · 2022-06-13T19:05:20Z

modules/jupyter-widget/style/index.js

@@ -0,0 +1 @@
+import './base.css';


Is this folder used? Seems like it just imports an empty CSS file.

dakoop · 2022-06-13T19:10:53Z

Correct, although even notebook will now be built on jupyterlab infrastructure (ref). I think you could keep the other method (current install process) working for the other modes, but these would have to be tested. JupyterLab developers are aware of some difficulties for users finding and installing the new prebuilt extensions: jupyterlab/jupyterlab#10554

Perhaps a better approach in structuring the code would be to structure the jupyter-widget components as currently done, and create a separate build process for the JupyterLab 3 plugin so that different dependency versions (specific to JupyterLab's build/plugin process) could be integrated without impacting the deck.gl's current dependencies. I didn't end up going that route because I couldn't figure out what was going on with the original approach that just included dist/index.js, but in retrospect, this may have been due to the dependency issues (acorn).

davidbrochart · 2022-07-25T18:25:36Z

@dakoop do you plan on continuing that work? It would be awesome.

donmccurdy · 2024-03-29T15:12:00Z

My impression (not a regular Jupyter user!) is that nbextensions for Jupyter are dying a slow death at this point, and with Jupyter Notebooks v7+ (released July 2023), you'll get errors like ...

Jupyter command `jupyter-nbextension` not found.

... for which old solutions will no longer work. Likely a prebuilt extension for JupyterLab is becoming necessary. Are there other alternatives we should consider, or anything I can do to help land this PR?

Otherwise, I think pydeck may be stuck in Jupyter Notebook 6.4 or lower, or nbclassic.

kylebarron · 2024-03-29T15:21:55Z

I think the main difference is that Jupyter Notebook v7 is built upon JupyterLab architecture, but in a way that looks like the classic notebook. So I think you need to make JupyterLab extensions to support the latest notebook

dakoop · 2024-03-29T15:45:24Z

As @kylebarron noted, Notebook 7 introduces extension changes as it moved to lab architecture. This PR was a push to update the pydeck extension a while back, but I haven't had time to reexamine this. There is other work on deck.gl Jupyter integration: lonboard and ipydeck.

dakoop added 7 commits June 10, 2022 16:30

Make changes to support JupyterLab 3

90d724c

See https://ipywidgets.readthedocs.io/en/stable/changelog.html#updates-for-widget-maintainers

Bump version so pydeck is tied to current jupyter-widget version

e1b18d4

Note that this ties it to 8.8.0-alpha.6; see visgl#7025 for comments

Run black on auto-generated version files

a246e84

bump_version.py generates these files so could be updated

Don't resolve acorn to allow jupyterlab builder to work

954598d

Integrate jupyter-widget with pydeck as prebuilt extension

cddac6e

Update package version in toml

b49b646

Pessimistress requested review from ajduberstein and kylebarron June 11, 2022 04:35

kylebarron reviewed Jun 13, 2022

View reviewed changes

This was referenced Mar 29, 2024

Fix Python CI #6777

Merged

[Bug]: build failure pydeck #8469

Open

donmccurdy added feature pydeck labels Mar 29, 2024

donmccurdy added this to the v9.0 milestone Mar 29, 2024

Pessimistress force-pushed the master branch 4 times, most recently from 0a083b6 to eb85da3 Compare April 25, 2024 03:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[pydeck] Create a prebuilt extension for JupyterLab #7030

[pydeck] Create a prebuilt extension for JupyterLab #7030

dakoop commented Jun 11, 2022

kylebarron commented Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

kylebarron Jun 13, 2022

dakoop commented Jun 13, 2022

davidbrochart commented Jul 25, 2022

donmccurdy commented Mar 29, 2024

kylebarron commented Mar 29, 2024

dakoop commented Mar 29, 2024

		@@ -1 +1 @@
		__version__ = "0.7.1"
		__version__ = "0.8.0"


		[tool.black]

[pydeck] Create a prebuilt extension for JupyterLab #7030

Are you sure you want to change the base?

[pydeck] Create a prebuilt extension for JupyterLab #7030

Conversation

dakoop commented Jun 11, 2022

Change List

Todo

kylebarron commented Jun 13, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dakoop commented Jun 13, 2022

davidbrochart commented Jul 25, 2022

donmccurdy commented Mar 29, 2024

kylebarron commented Mar 29, 2024

dakoop commented Mar 29, 2024