Tracking issue for using WebGPU to render on the window of the desktop environment

[chirsz-ever]

So far we have WebGPU and BYOW, but it need more work to use the front-end graphics ecology.

Repository for testing and integrating: https://github.com/chirsz-ever/deno-webgpu-window-demos

Seamlessly use WebGPU-backended frameworks

• three.js would be useful since r164 (WebGPUBackend: Allows the device to be created externally mrdoob/three.js#28192)
• pixijs: TODO
• Babylon.js: TODO

Bugs and requirements

• Calling createImageBitmap with a Blob, "TypeError: expected typed ArrayBufferView" occurred. #22649
  • fix(ext/web): pass createImageBitmap when the argument type is Blob #23518
  • support formats other than RGBA8, at least RGB
  • support more MIME types, like "image/jpeg"
• WebGPU: not able to use the alphaMode member of GPUCanvasConfiguration #23509
• WebGPU: Deno's GPUCanvasConfiguration is not standard #23508
• Deno.UnsafeWindowSurface panics with x11 #23433
• WebGPU: "Error: Invalid Surface Status" triggered randomly #23407
• WebGPU: Implement copyExternalImageToTexture #23576

Creating and managing native windows

We already have deno_sdl2 and dwm for creating windows. But they rely on third-party libraries SDL2 and GLFW.

Maybe we need some methods to create windows without any third-party libraries? I have two ideas:

• import winit and design a set of interfaces to use it;¹
• only write JavaScript code, use FFI to create window on respective platform, as a third-party library.

The first idea is easier to implement, but add a built-in component increases deno's compatibility costs and technical debt.

Canvas, CanvasRenderingContext2D and WebGL

Related issues: #5701 #1629

If deno support Canvas, CanvasRenderingContext2D and WebGL, it would be more easy to use front-end graphics libraries.

Deno.UnsafeWindowSurface is analog to Canvas, and it can use getContext("webgpu") to create a GPUCanvasContext. It is worthwhile to support other contexts, getContext("2d"), getContext("webgl"), getContext("webgl2") and getContext("bitmaprenderer").

How to implement CanvasRenderingContext2D (Canvas.getContext("2d"))? There are many methods in #5701, which mostly introduces skia or similar graphics libraries like Vello.

For WebGL, we can just use the native OpenGL implementation (gluton)[https://github.com/deno-windowing/gluten], or import ANGLE to maximize compliance of the WebGL standard.¹

Maybe we can do something named "Canvas2D over WebGPU" and "WebGL over WebGPU", which only use JavaScript to implement CanvasRenderingContext2D and WebGL with WebGPU.

If we resolve all above, implementing OffscreenCanvas and ImageBitmapRenderingContext (Canvas.getContext("bitmaprenderer")) seems not hard.

Footnotes

1. Removed by https://github.com/denoland/deno/issues/23563#issuecomment-2077777430 ↩ ↩²

[crowlKats]

We will not add a dependency on winit or any other windowing system, the way we have it currently is the best compromise.

Regarding Canvas and related:
for OffscreenCanvas, I have a local branch that needs some further work. The idea is that UnsafeWindowSurface will extend OffscreenCavas.
We will not implement a webgl context or any webgl related APIs, however we likely will implemented 2dcontext via Vello.

[chirsz-ever]

Why don't allow "none" colorSpaceConversion when createImageBitmap?

https://github.com/denoland/deno/blame/c5193556242117737fc3ba95ee3692b7831ca04a/ext/canvas/01_image.js#L307-L309

[chirsz-ever]

Is this a bug? I think the alpha value do not need to multiply 255. And why there need a is_not_premultiplied flag calculated from every pixels?

https://github.com/denoland/deno/blame/c5193556242117737fc3ba95ee3692b7831ca04a/ext/canvas/lib.rs#L86

[chirsz-ever]

For ImageBitmap :

• ImageBitmap is designed for immutable bitmap data stored in GPU (but not necessary, for example, Firefox do not store the data in GPU)¹², so it has no interface to get the underlying pixel data.
• The following functions use ImageBitmap:
  • createImageBitmap()
  • CanvasRenderingContext2D.drawImage()
  • CanvasRenderingContext2D.createPattern()
  • WebGLRenderingContext.tex(Sub)Image2D()
  • ImageBitmapRenderingContext.transferFromImageBitmap()
  • OffscreenCanvas.transferToImageBitmap()
  • GPUQueue.copyExternalImageToTexture()
• So we could make the following improvements:
  • Use pixel format other than RGBA, such as that in image::DynamicImage. This would save memory usage.
  • Use a GPUTexture as the underlying data resource when possible. This improves performance because you don't need to transfer data from and to the GPU.

Footnotes

1. https://stackoverflow.com/a/60033582 ↩

2. https://github.com/gpuweb/gpuweb/issues/1154 ↩

[crowlKats]

one you missed that uses ImageBitmap is https://gpuweb.github.io/gpuweb/#gpuimagecopyexternalimage

[crowlKats]

• Use pixel format other than RGBA, such as that in image::DynamicImage. This would save memory usage.

sounds reasonable

• Use a GPUTexture as the underlying data resource when possible. This improves performance because you don't need to transfer data from and to the GPU.

I don't think this is viable, since to create a GPUTexture you need a GPUDevice, which is not feasible since a user could use a different GPUDevice, or same GPUDevice but with different features.

[chirsz-ever]

We could hold the GPUDevice object in a ImageBitmap object when the underlying resource is a GPUTexture. When we want to perform some operations, we first check whether the devices on both sides are the same, and then directly perform the operation, or copy the data to the CPU side and then transfer it to the target.

This optimization would not be implemented in the near future, so it is recorded here just for reference.