GLTFExporter GLB compatible with Facebook viewer

[fernandojsg]

After facebook announced glTF support on their timeline I've been trying to use the GLTFExporter to generate some binary glTF (glb) to test this new feature.

But I've found some problems so far:

• GLB Chunks must be 4-byte aligned GLTFExporter: GLB chunks 4-byte aligned #13395
• Validation: Fix znear and zfar: GLTFExporter: fix znear and zfar range for cameras #13396
• Vertex Color: Facebook supports just RGBA but not RGB. As shown on the validation message:
  [msg] => Vertex COLOR_0 attributes of type RGB are (temporarily) notsupported. They must be RGBA.. Although COLOR_0 could be vec3 or vec4 and we could include an optional parameter to force the conversion of the color attribute from 3 to 4 components, I don't think we should do that hack as our current implementation is following the spec and I don't see any other use case for that conversion than just hijack the facebook validator while they're working on fixing it. <-- Update: This should be done in the following weeks, so we don't need to workaround it
• Non indexed meshes are not supported: `[msg] => Mesh primitives without indices are (temporary) unsupported.
• Export other primitive modes (Currently just TRIANGLE supported)

More info: https://developers.facebook.com/docs/sharing/3d-posts/glb-tutorials

[mrdoob]

/ping @zellski

[zellski]

Hey! Yeah, that RGBA check should be gone in two weeks. Don’t work around it. :)

[mrdoob]

I'm trying to convert WaltHead.obj to glb. I'm loading it in https://threejs.org/editor/ and exporting it to GLB from there (which should have the latest patches already).

Here's WaltHead.glb and this is what I'm getting in Facebook's validator:

Your GLB File has the following errors: The 3D model could not be posted: The JSON portion of this model file is invalid.

🤔

[zellski]

It's syntactically valid JSON, but the nulls in this snippet from WaltHead.gltf don't conform to the type schema:

    {
      "bufferView": 2,
      "componentType": 5126,
      "count": 48480,
      "max": [
        null,
        null
      ],
      "min": [
        null,
        null
      ],
      "type": "VEC2"
    }

The Khronos glTF validator tool also lists about 10,000 instances of some other error in the file, too, all on the order of:

		/accessors/2: Accessor element at index 28922 is NaN or Infinity.

So it seems maybe an accessor is being generated during glTF export, to be filled with indices, but never actually receives any?

[donmccurdy]

UVs are NaN: 🙃

[fernandojsg]

@mrdoob @donmccurdy fixed! #13400
Although we still can't show that example because of

[msg] => Mesh primitives without indices are (temporary) unsupported.

(added to the to-do list)

@zellski do you have any estimation about that feature? ;)

[zellski]

@fernandojsg This one is a little trickier. It'll get fixed, but it might take a little longer. tl;dr Maybe a month?

Longer explanation: the issue is similar to the vertex colour one above, in that it's our client implementations that are lagging behind, and my validator on the backend is just protecting them from models they can't handle yet.

Obviously we have to support non-indexed geometry, the sooner the better. Ideally on the clients, but also by then I should have my backend code up to full Death Star power, where we transform all uploaded .gltf lazily/on-demand, depending on individual clients need. At that point we can do cool things like create indexed geometry on the server for the convenience of our clients.

I assume the error above comes when three.js is trying to export primitives that are naturally non-indexed in its runtime representation?

[fernandojsg]

I assume the error above comes when three.js is trying to export primitives that are naturally non-indexed in its runtime representation?

@zellski that's it! Some primitives or objects loaded on three are non-indexed. The first use case I thought for the facebook glb loader was to include drawings from our A-Painter app (more info: https://blog.mozvr.com/tag/a-painter/) and we use non-indexed geometry there too, so it would be great to have support for that ;)
I just wanted to know if that was on the roadmap, so knowing that is it and we could have it in about a month, is more than reasonable ;) thanks for sharing that info!

[mrdoob]

We may have to add a dumb index attribute in the meantime (as in 0, 1, 2, 3, 4, 5, 6, 7, 8, ...) 😕

[fernandojsg]

@mrdoob do you mean having some method to convert non-indexed to indexed as you want, or add the hack directly on the exporter?

[mrdoob]

Yeah, add a temporal hack in the exporter...

I don't know, I just want stuff to work and not have to tell people, "oh? your model doesn't work on facebook? that's because... you know what indexed geometries are? yeah, you shouldn't but..."

[fernandojsg]

Yeah I got it! Ok I'll take a look to see if I could add some non-so-dirty hack :)

[mrdoob]

@zellski for context...

I've added a Export GLB in http://threejs.org/editor/ that uses this GLTFExporter.

[takahirox]

Video: how to export model as glTF on Three.js Editor :D

https://twitter.com/superhoge/status/966689549803053056

[zellski]

Believe me, I understand the reluctance to add hacks. It's quite a struggle to maintain a patient outlook now that we've launched...

Could you make the GLTFExporter hack in a fork that's used in http://threejs.org/editor/ but not elsewhere? I would hope we'll have fixed this flaw by the time r91 comes out, so it seems a bit pointless for you folks to write careful, responsible code for it.

[mrdoob]

Could you make the GLTFExporter hack in a fork that's used in http://threejs.org/editor/ but not elsewhere?

Yeah, no worries!

I would hope we'll have fixed this flaw by the time r91 comes out

Oh, I'm aiming to release ~March 1st. Changed release cycle to beginning of the month to have proper month releases.

Seems like we still have more features to add before promoting this anyway. I don't think we are exporting roughness, metallic, normal or alpha maps.

Latest test, using 2 diffuse maps, clouds one being a transparent png:
https://www.facebook.com/fakemrdoob/posts/950266411809572

Not sure where the apparent alphaTest: 0.5 is coming from...

[fernandojsg]

Just using the workaround for indices :)
https://www.facebook.com/fernandojsg/posts/10156405595122044

[fernandojsg]

There we go #13410

[fernandojsg]

@mrdoob do you mind adding the missing features you find and that are not related to the facebook requirements on this issue: #11951
Although I need to update the status there 😇

[mrdoob]

Sounds good. I'll test by exporting and dragging back to the editor.
Should we close this one?

[fernandojsg]

@mrdoob I would leave it open until the RGB vs RGBA for vertex colors issue will get resolved on the facebook side, so if people come here looking for help they could read about it instead of opening another issue.

[fernandojsg]

Btw I've just found this link with some useful info: https://developers.facebook.com/docs/sharing/3d-posts/glb-tutorials

[mrdoob]

Apparently textures need to be power of 2...
https://twitter.com/drupalati/status/967486854630055936

[takahirox]

Doesn't Three.js automatically convert image from non-power-of-two to power-of-two on the fly?

[mrdoob]

Okay... After #12877 and #13451 a GLB export that used to be 3.3MB is now 480KB 😊

[takahirox]

Cool! #13451 meant, the image file size will be huger if we convert jpg to png?

[mrdoob]

Cool! #13451 meant, the image file size will be huger if we convert jpg to png?

Yes. #13451 is a bit of a workaround due to the fact that the editor doesn't allow changing the format of a texture at the moment. But we do the same thing in the library anyway...

https://github.com/mrdoob/three.js/blob/dev/src/loaders/TextureLoader.js#L34

But yes, GLTFExpoter currently saves as jpg when texture.format === THREE.RGBFormat.

https://github.com/mrdoob/three.js/blob/dev/examples/js/exporters/GLTFExporter.js#L493

Which is not ideal, because we're recompressing a jpg... But better than oversized exports I guess?

[zellski]

I had to add code to FBX2glTF that actually inspects the alpha values of even RGBA images, because people (or, more precisely, people's tools) create them by default, and quite often it's utterly unnecessary to keep them as PNGs. Even after trying the most brutal GPU-enabled, non-linearly-optimising PNG crunchers out there, it seems JPEG really rules the roost... the size difference is pretty incredible!

I am a little worried about inter-channel bleed for things like the ORM texture (occlusion/roughness/metal) where each component carries data that's utterly independent of the other components' data... but in practice it seems to work fine.

[webprofusion-chrisc]

The exporter could also make quality level optional when using canvas.toDataURL( mimeType ) - my textures are generated at runtime from composite images, that would help too.

@zellski for fb's pipeline/viewer I'm guessing you could do like sketchfab and serve up a low res texture version, then stream the high/full res textures and replace them in the model as they load. Not a small piece of work but do-able.

[zellski]

@webprofusion-chrisc Yes, we may end up doing that. At the moment we've gone all out on .glb as the transmitted entity, which is hard to combine with selective LOD type streaming (since there's just a single sequential file with no random access). But I expect we'll reevaluate basic assumptions quite often, depending on how things go. :)

[Ben-Mack]

GLTFExporter can export BufferGeometry and import to Facebook fine. But any Geometry or BufferGeometry converted using method fromGeometry are not working on Facebook. I always receive this in FB validator:

[msg] => Vertex COLOR_0 attributes of type RGB are (temporarily) not supported. They must be RGBA.

Step to reproduce:

• Using misc_exporter_gltf example latest in dev, Export Sphere or Walthead working good on FB, but export Scene1 result cannot import into FB.

Is this some expected behavior and have to wait FB side?

I expected that BufferGeometry using fromGeometry should working extractly as normal BufferGeometry, please guide me some quick fixes to overcome this issue.

[fernandojsg]

@Ben-Mack this is something that should be fixed in the next weeks according to @zellski -> #13397 (comment)

[zellski]

As of build 161 and on (current App Store version is 160) of main FB app, this won’t be a crasher anymore and we’ll remove this validation check. I expect this to happen within the week.

[fernandojsg]

@zellski that's awesome! thanks :)

[mrdoob]

Makes me wonder though...

The "real" reason THREE.GLTFExporter can't export THREE.Geometry is because when we convert THREE.Geometry to THREE.BufferGeometry we're creating a color attribute which is, in most of the cases, full of zeros.

So, one "solution" (and optimisation) would be to not export the color attribute if material.vertexColors is set to THREE.NoColors?

[fernandojsg]

Ops I didn't know that :D that's for sure a must-do optimization :D

[ov]

@fernandojsg thanks for the updates you made, much appreciated. There are two more things worth adding:

1. Multi-material meshes support. The ones that have groups in their geometries and array in Mesh.material - they currently cannot be exported properly;
2. Better compatibility between the only supported MeshStandardMaterial and the result we have on Facebook. So far metallic and diffuse surfaces look quite different in three.js and on Facebook. Maybe we'll have a special "Facebook" material one day?

Thank you

[donmccurdy]

@ov I think both of those are likely to be fixed around r91:

1. multi-material export GLTFExporter: Multi-material support #13536
2. metal/roughness fixes Replace 8 with real maxMipLevel in lights_fragment_maps.glsl #13501

It's possible that Facebook's materials aren't quite right yet either, but glTF is pretty specific about how things should appear so eventually we should converge.

[donmccurdy]

Oh, we should add ability to export KHR_materials_unlit too...

EDIT: Opened #13566

[Ben-Mack]

The "real" reason THREE.GLTFExporter can't export THREE.Geometry is because when we convert THREE.Geometry to THREE.BufferGeometry we're creating a color attribute which is, in most of the cases, full of zeros.

So, one "solution" (and optimisation) would be to not export the color attribute if material.vertexColors is set to THREE.NoColors?

+1 hope this will be fixed soon. Lots of library for Three.js still performing based on THREE.Geometry.

( FB still having error ...must be RGBA with THREE.Geometry )

[looeee]

@Ben-Mack, which libraries are you using that still use Geometry? Perhaps we could work with the owners to get them updated.

[Ben-Mack]

@looeee Please take a look at this library: https://github.com/a-jie/threejs-geometry-modifiers

[zellski]

Hey folks -- as of this morning, Facebook no longer rejects RGB (VEC3) vertex colours as "invalid".

The power-of-two textures requirements remains for the moment, but I'm working on that too.

[fernandojsg]

@zellski coool! :) I'm very close to have fully support for a-painter :D Is there any plan to implement the rest of the primitive modes? Right now I believe just TRIANGLE are supported, It could be great to support the rest, eg in A-painter we're using TRIANGLE_STRIP to save some space 👼

[zellski]

It would be crazy not to implement them, right? Ideally all valid glTF should be accepted. I don't know how that work will be prioritised, though. We're a small team with a lot of strong urges. :)

[mrdoob]

I think this issue can now be closed?