NEXT-00000 - Add gltf file format support

[akf-bw]

1. Why is this change necessary?

• Currently the new "Spatial Commerce" feature only supports .glb files not the .gltf file format, which is just the non binary version of a .glb file
• The .gltf file format allows programs/developers to modify the file much more easily since it is just a JSON file, making it more attractive to use compared to a .glb file in some cases
• The storefront already uses the GLTFLoader so we just need to allow the .gltf file format for upload & usage

Shopware-TODO:

• Someone from the Shopware core team would have to create one of these amazing .svg icons for the gltf format. Currently the icons-multicolor-file-thumbnail-gltf.svg file is only a copy of the icons-multicolor-file-thumbnail-glb.svg file

2. What does this change do, exactly?

• Added detection of the gltf file format in SpatialObjectTypeDetector
• Added support for the gltf file format in allowed_extensions & private_allowed_extensions in shopware.yaml
• Added detection of the gltf file format in sw-media-base-item, sw-media-preview-v2, sw-media-quickinfo, sw-product-media-form, sw-product-detail-base, file-validation.service & media.api.service

3. Describe each step to reproduce the issue or behaviour.

• Try to upload a .gltf file

4. Please link to the relevant issues (if any).

/

5. Checklist

• I have rebased my changes to remove merge conflicts
• I have written tests and verified that they fail without my change
• I have created a changelog file with all necessary information about my changes
• I have written or adjusted the documentation according to my changes
• This change has comments for package types, values, functions, and non-obvious lines of code
• I have read the contribution requirements and fulfil them.

[github-actions]

	Warnings
⚠️	You updated the shopware.yaml, please consider to update the config-schema.json

[ffrank913]

Hey guys!

We see a couple of problems in adding .gltf format for 3D files.

As you might know 3D models can not only use materials (which are physical values, also JSON-able) but also textures. Textures are images that are baked on top of a certain model which can be defined in the GLTF as well. Since textures are images they also can have different formats. They can either be for example a Base64 representation of the image or they are just file-referenced.

Example:
{ ..., images: [ 'yadda/yadda/yadda.jpg', 'yadda/yadda/test.jpg', 'yadda/yadda/42.jpg', ], ..., }

Since three.js reviews a .gltf file and loads all it's resources (for example all textures) it is in fact not a problem to reference the image (if you have the correct file system access of course).

So what happens when a merchant upload a .gltf file with referenced textures?

Since we upload only the .gltf file and not all of the referenced images (see example above) the images are missing. Of course three.js itself can catch that and it will not crash the application. But in the end the model will be displayed incorrectly or at least different than expected since the textures are missing.

So what are our options?

Option 1: We forbid uploading .gltf files and ask the merchant to create .glb files from their .gltf files. That is not really what we want, right?

Option 2: While uploading we scan the .gltf file for external resources and ask the user to upload them separately and link them after so everything is restored again. This would either involve a good understanding of what she is doing (at least that's what I think). So it is kind of an option.

Option 3: After uploading a .gltf file we create a shadow DOM with -tags with the sources of the images used in the .gltf. In the image's onLoad-function we create a Base64 string from the loaded image and replace the linked source (yadda/yadda/yadda.png) with the actual source (data:image/png;base64,iVBORw0KGgoAAAANSUhEUgA...=). In my opinion this would be the most clean option.

How do we choose what to do?

Actually I don't know. But we have to improve this functionality to grant correctness of uploaded data. Otherwise this whole .gltf-Upload thing does not make sense.

If you have any questions please don't hesitate to contact me!

[akf-bw]

@ffrank913

Option 2: While uploading we scan the .gltf file for external resources and ask the user to upload them separately and link them after so everything is restored again. This would either involve a good understanding of what she is doing (at least that's what I think). So it is kind of an option.

I think this option would be the best.
If someone uses embedded image file paths (not embedded image files) in a GLTF, the user knows what they are doing.
I guess we can just add an info message when someone uploads a GLTF file with embedded image file paths to simply remind them that there are paths in the file that may need to be adjusted.

Uploading gltf and image files together like in option 3 should not be the approach. Multiple gltf files could point to the same image (which one would then be required to be uploaded twice?), or it could be an external source, which cannot be uploaded by the user at all.

The only valid option here is option 2 with only an information reminder.

[ffrank913]

Hey @akf-bw!

I don't think that your code covers everything.

First since the .gltf format is just a "recipe" to assemble different files it might occur that even mesh data or animations are not embedded in array buffers but stored in separated .bin files. So with the current standard we cannot even make sure that the user's mesh data is transported to Shopware properly. We would not only have to make sure that all textures are set correct but also the array buffer data is correct.

You can find the .gltf-standard here: https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#gltf-basics

In addition it is very hard for us to set up connections between files since we don't have such a system yet. For example let's say you have a tree-model with a bark texture. And let's not care for the mesh data for now, let's assume it is embedded in the .gltf-file. What happens if you delete the bark texture in your Media? The model display would not be correct anymore and you would not know why. Since not every person that manages product media or product data has a technical background they might not know what the impact of deleting a strange wooden image from their database might have.

So to warn the merchant if something breaks due to their changes we would need a system to link files to each other double-sided (to ensure that the 3D model is displayed correctly and not unnecessary data is stored on the server).

As you mentioned correctly, the .gltf-format makes it way easier for human beings to edit the assembled 3D data. But since Shopware is only capable of displaying and not in-depth editing a 3D model there is not technical need to store data in .gltf-format. For now editing has to be done in distinct software outside of Shopware (like Blender).

I can also refer you to a package that can handle transform .gltf-files to .glb-files: https://github.com/donmccurdy/glTF-Transform

If you have any other thoughts please share!

[akf-bw]

Hey @ffrank913,

So with the current standard we cannot even make sure that the user's mesh data is transported to Shopware properly. We would not only have to make sure that all textures are set correct but also the array buffer data is correct.

Shopware is not responsable for the gltf file to be valid. You have to provide a valid gltf file for this all to work.
We can't just check the full spec of gltf just to make sure the user can't upload an invalid gltf file. This is also not done with other file types in shopware so why include it for gltf?

In addition it is very hard for us to set up connections between files since we don't have such a system yet. For example let's say you have a tree-model with a bark texture. And let's not care for the mesh data for now, let's assume it is embedded in the .gltf-file. What happens if you delete the bark texture in your Media? The model display would not be correct anymore and you would not know why. Since not every person that manages product media or product data has a technical background they might not know what the impact of deleting a strange wooden image from their database might have.

So to warn the merchant if something breaks due to their changes we would need a system to link files to each other double-sided (to ensure that the 3D model is displayed correctly and not unnecessary data is stored on the server).

The same thing here, Shopware is not responsable for your gltf file to be valid. It is not possible in Shopware to link files together and would need an insame about of rework of the current media behavior just to add support for your needs here.

As you mentioned correctly, the .gltf-format makes it way easier for human beings to edit the assembled 3D data. But since Shopware is only capable of displaying and not in-depth editing a 3D model there is not technical need to store data in .gltf-format. For now editing has to be done in distinct software outside of Shopware (like Blender).

I don't mean to edit the file in the administration, but rather programmatically. For example, we wrote a plugin that reads multiple GLTF media files from the server and merges them inside of PHP to directly provide a merged GLTF JSON to the user to display it in the frontend.
With our plugin we are already extending the core to be able to upload gltf files, but this PR should integrate this directly into the core.

[ffrank913]

You are correct. We are not responsible for taking care of correctness of data. But what we have to take care of is a proper user experience.

Typically files are standalone and work for their own. Before .gltf we are just working with such files, including .glb-files. For images basically we have a quick feedback for the merchant to view the image in a sidebar to check if everything is working properly. So in fact we do have such a check, at least not programmatically but done very easy by the merchant.

We don't have such a viewer for our 3D models yet. When we have, we will probably include .gltf-files as well since we then have a proper tool to view if everything works fine.

I understand your use-case and where you are coming from. But image there are people paying a huge amount of money to have their products visualized in 3D and have no clue of how all this magic works. We need a proper way to help our merchants or at least have to properly error-catch our algorithms and not simply throw errors that are not precise enough to give a quick correct answer. At least this is my opinion on that topic.

I will discuss this with our teams and come back later. Feel free to add what you think.

[akf-bw]

We don't have such a viewer for our 3D models yet. When we have, we will probably include .gltf-files as well since we then have a proper tool to view if everything works fine.

What about including the Three.js viewer also in the administration in the "preview slot of images"?
This way the user would get instant feedback from the visual file.