BREAKING: WebGPU support (MToonNodeMaterial) #1384

0b5vr · 2024-03-25T10:19:33Z

This PR supports WebGPURenderer by adding MToonNodeMaterial, which is a NodeMaterial port of MToonMaterial.

After the change is published, you can use three-vrm on WebGPU by using MToonNodeMaterialLoaderPlugin.
To use the loader, specify the loader in the option of VRMLoaderPlugin.
See the webgpu-dnd.html example for details.

BREAKING

gltf.userData.vrmMToonMaterials might have materials other than MToonMaterial (namely, MToonNodeMaterial in this case).
- This should not affect many users since the interface of MToonNodeMaterial is almost identical to MToonMaterial.

TODOs

Since three-vrm-material-mtoon imports three/addons/nodes/Nodes.js, the initial module loading time became longer when you are using certain CDNs like unpkg. We are currently seeking for a workaround.
- See also: https://discourse.threejs.org/t/nodematerial-on-unpkg-slow-to-load-all-the-dependencies/63213
- Replaced the CDN from unpkg to jsdelivr and the problem is resolved.
  - See: chore: replace the cdn, unpkg -> jsdelivr mrdoob/three.js#28006
  - See: chore: change cdn from unpkg to jsDelivr #1385
Several type definitions on @types/three are patched by patch-package
- We might want to fix by three-ts-types side before we ship this
The Type of property 'cache' circularly references itself ... error (TS2615) occurs
- We might want to bump our TypeScript version to 5.1 or later
- The issue is resolved, at least in the patch
  - See: feat (Nodes): more tslFn related typings three-types/three-ts-types#744 (comment)

- The type parameter of `THREE.Object3D` is no longer required (used to be `THREE.Object3D<THREE.Event>` - `THREE.UniformsUtils.merge` now has more appropriate typing which breaks the existing code, workarounded with `as any`

Starting from r155, the light unit is set to physically based one by default. We need to multiply the directional light intensity by PI because of this change. See: mrdoob/three.js#26392

also removed an impossible compat code path

Built the MToon shader using NodeMaterial feature I had to add `three/examples/jsm/nodes/Nodes.js` to rollup's `external` config. Is the solution legit? We have a problem on the current `@types/three`, I had to patch-package. Some of them are already addressed as PR on `three-ts-types` repo I applied temporary changes to several example html files, this should be reverted before merge

…ead of local space

`parameters` have `depthWrite = false` when it's transparent, we have to overwrite this

It seems model normal was multiplied by 100 in UniGLTF 1.27, 1.28. Low confidence

also emit a warning if the model contains vertex colors

The outline was too bright when the outlineLightingMix is zero

This prevents opaque / cutout materials to go white

…utlineWidthTexture

…LightingMix factor is 1 it should be multiplied by (1.0 / PI)

outline didn't respect alpha when outlineLightingMixFactor is not 1

outlineWidthMode is not required when it's not outline pass also add comma

…cheKey it's included in cacheKey in either way. See: https://github.com/mrdoob/three.js/blob/r160/examples/jsm/renderers/common/RenderObject.js#L115

We should have the parametric rim result to a temporary variable

WebGPURenderer creates shader variants for each skeleton uuid so we want to unify skeletons as much as possible If there is a superset skeleton, reuse it

This reverts commit 33fc057. boneInverse might be different between skeletons and the new method does not work well with such cases.

The `Type of property 'cache' circularly references itself in mapped type` still persists The error disappears once I bump TypeScript to v5.1.3 or above

revert several examples back to WebGL `three/examples/jsm/*` now referred by `three/addons/*`; see `tsconfig.json` and `rollup.config.js` `RotateUV` doesn't work properly with UV Animation Mask Texture on r162. Spin using a handmade rotation matrix instead

tested using VRMC_materials_mtoon_UV_Animation_Test.vrm

This is an option to decrease initial rendering time when the model has many materials

If you want to use three-vrm on WebGPU, use `MToonNodeMaterialLoaderPlugin` by specifying the loader in the option of `VRMLoaderPlugin`. See the `webgpu-dnd.html` example for details BREAKING: MToonMaterialLoaderPlugin might put materials other than MToonMaterial (namely, MToonNodeMaterial in this case) to `gltf.userData.vrmMToonMaterials`. This shouldn't affect many users since the interface of MToonNodeMaterial is almost identical to MToonMaterial

sequel of 4de75c1

…hree` See: three-types/three-ts-types#744 (comment)

…arately We need to fix the issue from the `@types/three` side someday This change and the previous r164 bump removes all patches to `@types/three`, the commit also removes `patch-package` 🎉

0b5vr · 2024-05-14T09:59:15Z

Ready to merge!

We are going to bump the major since the breaking change occurs.

The spec says that the animation should not accelerate over time when we apply uv scroll and rotation over time See: https://github.com/vrm-c/vrm-specification/tree/master/specification/VRMC_materials_mtoon-1.0#transform-order The GLSL code is also wrong, we have to fix this later

0b5vr · 2024-05-15T08:03:15Z

日本語で恐縮ですが、NodeMaterialを継承したマテリアルのコードを読む上で、NodeMaterialの動作の流れ（r164現在のもの）を以下のページにまとめてあります。レビューの際にご活用ください。

https://scrapbox.io/0b5vr/Three.js:_NodeMaterial

packages/three-vrm-materials-mtoon/src/nodes/MToonNodeMaterialLoaderPlugin.ts

packages/three-vrm-materials-mtoon/src/MToonMaterialLoaderPlugin.ts

ke456-png · 2024-05-17T10:47:48Z

The rendering result differs when there is no texture and only a normal map.

examples/feature-test

webgpu-feature-test

ke456-png · 2024-05-17T10:54:03Z

Perhaps MToonMaterial was wrong?

three-vrm/packages/three-vrm-materials-mtoon/src/MToonMaterial.ts

Lines 607 to 614 in 8a89449

    
           const useUvInFrag = 
        
             this.map !== null || 
        
             this.emissiveMap !== null || 
        
             this.shadeMultiplyTexture !== null || 
        
             this.shadingShiftTexture !== null || 
        
             this.rimMultiplyTexture !== null || 
        
             this.uvAnimationMaskTexture !== null;

 const useUvInFrag = 
...
      this.uvAnimationMaskTexture !== null ||
      this.normalMap !== null;

0b5vr · 2024-05-20T08:03:01Z

The rendering result differs when there is no texture and only a normal map.

It seems to be, yes! The blue ball in the example should have bumps on the surface since the normal map is assigned.
I will fix this in another PR.

See: #1384 (comment)

ke456-png · 2024-05-22T07:40:17Z

If possible, please incorporate this change.
25cc2ba

ke456-png · 2024-05-22T07:45:59Z

packages/three-vrm-materials-mtoon/src/nodes/MToonNodeMaterial.ts

+    // Set alpha to 1 if it is not transparent
+    if (this.transparent === false) {
+      Nodes.diffuseColor.a.assign(1.0);
+    }


I need to confirm the conditions for making it opaque.

three-vrm/packages/three-vrm-materials-mtoon/src/shaders/mtoon.frag

Lines 871 to 873 in 9e16c38

#ifdef OPAQUE

diffuseColor.a = 1.0;

#endif

https://github.com/mrdoob/three.js/blob/687a4caf8c5d633df691a90369c9c185a4acb0fc/src/renderers/webgl/WebGLPrograms.js#L250
opaque: material.transparent === false && material.blending === NormalBlending && material.alphaToCoverage === false,

0b5vr · 2024-05-24T04:14:33Z

Resolved conflict with dev-v3, it's now ready to build using esbuild.
I don't have to explicitly specify three/addons/nodes/Nodes.js, esbuild seems to handle this by only specifying three.

0b5vr added 26 commits December 25, 2023 18:10

deps: Bump three.js to r160

d4352ed

- The type parameter of `THREE.Object3D` is no longer required (used to be `THREE.Object3D<THREE.Event>` - `THREE.UniformsUtils.merge` now has more appropriate typing which breaks the existing code, workarounded with `as any`

examples: Change light intensity from 1 to PI

ac686ce

Starting from r155, the light unit is set to physically based one by default. We need to multiply the directional light intensity by PI because of this change. See: mrdoob/three.js#26392

refactor: add COMPAT comment on pre-r156 compats addressed in #1304

bdaf2c7

also removed an impossible compat code path

fix (WebGPU, MToon): Fix outline, it should scale in world space inst…

6f5f087

…ead of local space

fix (WebGPU, MToon): Fix transparentWithZWrite

ebeb237

`parameters` have `depthWrite = false` when it's transparent, we have to overwrite this

fix: Fix outline width when model normal is not normalized

62ebea8

It seems model normal was multiplied by 100 in UniGLTF 1.27, 1.28. Low confidence

fix: Ignore vertex color if it's defined

8072596

also emit a warning if the model contains vertex colors

fix: (WebGPU, MToon): Fix parametricRim formula

d15cf35

fix (WebGPU, MToon): Fix outline color calculation method

638d0ee

The outline was too bright when the outlineLightingMix is zero

fix (WebGPU, MToon): set alpha to 1 when the material is transparent

b77f355

This prevents opaque / cutout materials to go white

feature (WebGPU, MToon): support screenCoordinate width outline and o…

0d8d79a

…utlineWidthTexture

fix (WebGPU, MToon): fix matcap and parametric rim intensity when rim…

74acea5

…LightingMix factor is 1 it should be multiplied by (1.0 / PI)

fix (WebGPU, MToon): Fix outline color alpha

3037a5d

outline didn't respect alpha when outlineLightingMixFactor is not 1

improve (WebGPU, MToon): improve customProgramCacheKey

ce08f6f

outlineWidthMode is not required when it's not outline pass also add comma

improve (WebGPU, MToon): remove outlineWidthMode from customProgramCa…

962b75c

…cheKey it's included in cacheKey in either way. See: https://github.com/mrdoob/three.js/blob/r160/examples/jsm/renderers/common/RenderObject.js#L115

improve (WebGPU, MToon): Improve code around parametric rim

bc9e6ab

We should have the parametric rim result to a temporary variable

improve: Improve removeUnnecessaryJoints

33fc057

WebGPURenderer creates shader variants for each skeleton uuid so we want to unify skeletons as much as possible If there is a superset skeleton, reuse it

fix: apply patch-package for NodeMaterial.setupOutput

7bf8901

Revert "improve: Improve removeUnnecessaryJoints"

1b854e0

This reverts commit 33fc057. boneInverse might be different between skeletons and the new method does not work well with such cases.

Merge branch 'bump-three-r162' into webgpu

97d4c22

fix: update type definition

2dee4ed

The `Type of property 'cache' circularly references itself in mapped type` still persists The error disappears once I bump TypeScript to v5.1.3 or above

fix: Fix rotation order of MToonAnimatedUVNode

1311e6c

tested using VRMC_materials_mtoon_UV_Animation_Test.vrm

feature: add sameBoneCounts option to removeUnnecessaryJoints

c1cbcbe

This is an option to decrease initial rendering time when the model has many materials

0b5vr added the enhancement New feature or request label Mar 25, 2024

0b5vr self-assigned this Mar 25, 2024

0b5vr added 2 commits March 27, 2024 18:54

Merge branch 'jsdelivr' into webgpu

de795b3

chore: change cdn from unpkg to jsDelivr for WebGPU examples

0b61ba1

sequel of 4de75c1

0b5vr added 2 commits April 8, 2024 16:12

fix: Patch the ShaderNodeObject circular reference issue on `@types/t…

98dee38

…hree` See: three-types/three-ts-types#744 (comment)

Merge branch 'dev' into webgpu

21499d2

0b5vr added the Breaking Change Choose wisely label May 14, 2024

0b5vr added 3 commits May 14, 2024 17:16

Merge branch 'dev' into webgpu

48eb0dd

Merge branch 'dev' into webgpu

644f168

fix: Workaround the addAssign issue by using add and assign sep…

115e2dc

…arately We need to fix the issue from the `@types/three` side someday This change and the previous r164 bump removes all patches to `@types/three`, the commit also removes `patch-package` 🎉

0b5vr force-pushed the webgpu branch from 9385045 to 115e2dc Compare May 14, 2024 09:37

Merge branch 'dev' into webgpu

8e3bde5

0b5vr marked this pull request as ready for review May 14, 2024 09:58

0b5vr mentioned this pull request May 14, 2024

chore: remove webgpu-dnd.html #1401

Merged

0b5vr added 4 commits May 14, 2024 19:07

Merge branch 'dev' into webgpu

f3401d3

chore: reintroduce webgpu-dnd.html

95eff3e

examples: Bump Three.js version of new webgpu examples

e5a5b73

0b5vr changed the title ~~WIP: BREAKING: WebGPU support (MToonNodeMaterial)~~ BREAKING: WebGPU support (MToonNodeMaterial) May 15, 2024

ke456-png reviewed May 17, 2024

View reviewed changes

packages/three-vrm-materials-mtoon/src/nodes/MToonNodeMaterialLoaderPlugin.ts Show resolved Hide resolved

ke456-png reviewed May 17, 2024

View reviewed changes

packages/three-vrm-materials-mtoon/src/MToonMaterialLoaderPlugin.ts Show resolved Hide resolved

0b5vr added this to the v3 milestone May 20, 2024

0b5vr changed the base branch from dev to dev-v3 May 20, 2024 02:49

0b5vr added a commit that referenced this pull request May 20, 2024

fix: MToon, Fix normal mapping when other textures are not specified

a59a1ed

See: #1384 (comment)

0b5vr mentioned this pull request May 20, 2024

fix: MToon, Fix normal mapping when other textures are not specified #1410

Merged

ke456-png reviewed May 22, 2024

View reviewed changes

Merge branch 'dev-v3' into webgpu

09641a7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BREAKING: WebGPU support (MToonNodeMaterial) #1384

BREAKING: WebGPU support (MToonNodeMaterial) #1384

0b5vr commented Mar 25, 2024 •

edited

0b5vr commented May 14, 2024

0b5vr commented May 15, 2024

ke456-png commented May 17, 2024

ke456-png commented May 17, 2024

0b5vr commented May 20, 2024

ke456-png commented May 22, 2024

ke456-png May 22, 2024 •

edited

0b5vr commented May 24, 2024

BREAKING: WebGPU support (MToonNodeMaterial) #1384

Are you sure you want to change the base?

BREAKING: WebGPU support (MToonNodeMaterial) #1384

Conversation

0b5vr commented Mar 25, 2024 • edited

BREAKING

TODOs

0b5vr commented May 14, 2024

0b5vr commented May 15, 2024

ke456-png commented May 17, 2024

ke456-png commented May 17, 2024

0b5vr commented May 20, 2024

ke456-png commented May 22, 2024

ke456-png May 22, 2024 • edited

Choose a reason for hiding this comment

0b5vr commented May 24, 2024

0b5vr commented Mar 25, 2024 •

edited

ke456-png May 22, 2024 •

edited