Define a way to provide unique identifiers for glTF data (nodes/etc) #2337
Comments
|
Sound like a great idea |
|
This would be a huge leap forward for 3D workflows in Godot |
|
There is already a generic way of specifying metadata everywhere in a glTF. IT is a little cumbersome, but it does technically have this capability. It is via this already approved extension: https://github.com/KhronosGroup/glTF/blob/main/extensions/2.0/Khronos/KHR_xmp_json_ld/README.md What you would do is create a bunch of xml packages at the top level, each one containing metadata, and then reference them from each node or material. Would that be possible? The idea was to avoid extensions that just add metadata and unify them into this single extension. |
|
@bhouston This would be a single string per node, XML is vastly overkill for this. (Also note: I have a Godot implementation of KHR_xmp_json_ld, but I can't say that I'm a fan of the spec...) |
|
I think this may also be relevant: |
|
@hybridherbst wrote:
Unfortunately it doesn't add any new identifiers such as uuid. @aaronfranke wrote:
This doesn't involve XML, rather it uses XMP. This spec defines all metadata at the top level and then you reference it from the resource you want to use it from. This allows you to reference duplicated metadata on multiple nodes if you wanted to, but in this case you want unique metadata per node but it stills support that use case as well. This isn't that hard to use. At the top level you would define your UUIDs: "extensions": {
"KHR_xmp_json_ld": {
"packets": [
{
"@id": "",
},
{
"@id": "",
},
{
"@id": "",
}
]
}
}And then in each node, mesh, texture or material or scene you would do: "meshes": [
{
[...rest of mesh definition...]
"extensions": {
"KHR_xmp_json_ld": {
"packet": 0
}
}
},
{
[...rest of mesh definition...]
"extensions": {
"KHR_xmp_json_ld": {
"packet": 1
}
}
}
] |
|
@aaronfranke do you understand the example usage I have outlined above using the existing extension? It isn't that much work to do. I think it would work perfectly for your use case, no? I bet you could write support for it in a hour or less. EDIT: I think that maybe the "dc:identifier" may be a better top level name than "@id"? |
|
I really think a dedicated extension is probably better for this, to better encourage other 3D DCCs to implement this functionality. |
|
It MUST be integrated into Godot and i fully support this ! |
|
A unique indentifier should be present in absolutely everything that deals with transfer and storage of data, and it has to be a first-class property, not an aftertought. warning: pseudocode/python ahead. I will be 100% honest here, this is not an acceptable way to access a unique identifier, what does even mean "dc:*"? This adds unnecessary clutter, the way to grab the information is noisy and convoluted. This is not something that is reasonable in any situation. Compare with this: |
|
I also support the idea of this becoming its own extension. It would be awkward that tool A exported uids via some raw metadata field and then tool B checked precisely those fields that happen to have a very specific meaning. It would be like having a de facto extension, only not specified. |
|
Right, I mean, this is something that once defined every game engine and a lot of DCCs will be happy to implement. It is precisely why it should be standardized. |
|
Yes please! |
|
Yes! |
|
The first comment links to an issue that already contains some discussion, and it summarizes some of the discussion points, clarifications, and open questions from there. And judging from the high number of 👍 's (and comments, even though many of them do not go beyond the meaning of a 👍 ), there seems to be a high demand for this feature. But I think that it is really important to be clear about the scope, intended use, and behavior of such an extension. Therefore, I will ask "Devil's advocate" questions (again). I'm asking these with the goal of fleshing out what could eventually become the 'Introduction' section of an extension specification, and make sure that everybody is on the same page (and I hope that there will not be toooo many people accusing me of being stupid and/or dismissive for asking these questions...) Referring to the high-level use-case description:
This sounds like something that is mainly (or only?) relevant during the authoring phase of an asset. (NOTE: I'm aware that glTF - even though it is primarily intended as a 'last mile' delivery format - is used in authoring workflows. And I acknowledge that unique identifiers could be useful for supporting the authoring workflow. The following is really about the scope, "management", and intended usage of these IDs). The assumption that this is mainly relevant during the authoring seems to be confirmed by this point:
A model is imported into an engine. And this model is expected to contain a node ("MyNode") that is supposed to be identified with an ID. Who is responsible for establishing the connection between the IDs that have been assigned by the modeling application and the use of these IDs in the engine, on a technical level? In how far do the authoring application and the engine have to "know each other" (and the IDs that they are assigning and expecting)? Or to put it that way: The description sounds like this only refers to the case where the engine imports a model, and it should be possible for the engine to assume that ...
Is that correct? (If it is correct, then some constraints for a possible specification can be derived from that. For example, that no engine may assume the presence of IDs to begin with, and that authoring applications may never modify IDs that are already there. It could be emphasized that the extension/IDs are only useful within that "authoring cycle" of editing/importing models between authoring applications and engines that both support the extension in exactly this way) Another important question is: What are engines allowed to do based on these IDs? For example: An engine finds a certain mesh primitive, based on its ID. And it could assign a new material to this mesh primitive. This would mean that the appearance of that glTF asset does no longer depend on the glTF asset itself, but on the presence of a certain ID (and the specific engine that is importing it). The follow-up question would be: Shouldn't it be possible to eventually "bake" these modifications into the asset itself, after the authoring process is 'finished'? (Meaning that at the end, one could also remove all IDs?) Otherwise, glTF may lose some of its portability. Specifically: I wonder which aspects of the current portability of glTF might be endangered by ~"modifications of the asset that are based on the presence of certain IDs"... Again: I'm not opposed to introducing identifiers (for example, to support common authoring workflows). But it should be made clear which kinds of behaviors and interconnections between authoring applications and engines are expected to (or rather: "allowed to") be established with these IDs. |
|
What we want is to be able to transfer modifications made to an scene imported from a glTF to another scene imported from a newer version of the glTF. Ideally we would not create a new scene from the new version of the glTF, instead we would reuse the existing objects, something like this:
Assume the glTF is authored by a different person to who is manipulating it in engine. While this is not necessarily true, since these are different know-hows it is common that they are done by a different person. I hope this makes sense. |
|
@javagl I agree this is only useful for authoring. But reality is that GLTF is used hugely for this. It is a very fast format to export (due to its binary nature) and reimport into an engine. Most game engines do not use GLTF natively though, they only use it for opening the assets, then use their own formats to export. This is the case of Godot and pretty much any other engine I can think of. My feeling is that, if you are making a game engine and you want to actually ship the very same GLTFs, then its up to you to clean it up. GLTF already provides a lot of information that may be redundant (as example, names), so its up to the engine do do this clean up process when shipping. |
|
That description sounds reasonable, and seems to be in line with what someone (maybe even one of you, I'd have to look it up) said in the linked discussion. Roughly: The glTF itself is used only as a basis for building the "engine-specific asset". And this may even be stored in an engine-specific asset (file) format, which only refers to the glTF as its input. In this case, of course, any engine-specific additions have to know which element of the glTF they refer to. One specific example (just to get an idea of whether I got the intention right): There might be a glTF asset with some avatar/character, and it has identified elements like 'head, torso, armL, armR, legL, legR'. The engine-specific asset uses this as the basis for building some ~"physics computation structures" (say, for some ragdoll effect, or something else that cannot be modeled with glTF animations). In this case, the base glTF asset would not even be intended to be "portable" any more (not beyond showing that avatar in the T-pose in which it was originally authored). It might therefore be that some of my concerns are negligible in the "real world". But I'd still wonder how to avoid "obscure" usages of these IDs. I know, this is kind of a "worst case" scenario, but people could just throw a bunch of meshes (that are not attached to nodes) or materials (that are not associated to meshes) or just empty nodes that only contain IDs (!) into an asset, and say: Yeah, my engine knows how to assemble something that makes sense from these fragments, based on their IDs. Right now, I can drag-and-drop any GLB file into Three.js, Babylon.js, Filament, PlayCanvas, Cesium.js, the Khronos glTF Viewer, ClayGL, Hilo3d, RedCube.js, and any other glTF viewer, and they will all behave the same way, and show the same result ... mostly: Achieving portability is an important goal of glTF, ranging from the lowest level of mathematical details of the specification of PBR, up to the question of what the (visual) "ground truth" actually is (see for example https://modelviewer.dev/fidelity/ ). We should be careful to keep these goals in mind, and in the context of a possible specification of 'unique identifiers', we should clearly describe their intended use (with the focus on the "authoring cycle") and the limits (what they should not be used for, to ensure that glTF keeps its portability). |
|
I think one aspect here is that unique IDs in glTF would allow for one – and only one – workflow to be better:
in a repeatable way while renaming/moving nodes around. Going from Application B to Application C would have likely different unique IDs since internal data formats do not necessarily match the glTF data model (e.g. some engines have submeshes - aka multiple primitives per mesh - while others don't). In practice, importing and exporting glTF is almost never a perfect roundtrip due to these engine-specific differences. So IDs would be useful for importing and might change when exporting again unless you expect every application to somehow keep the IDs and have perfect roundtrips, ignoring their own data format. |
|
It is already possible to store arbitrary data in glTF. Take for example the suggested alternatives to this proposal: However, in practice, we do not see this. I believe the incentives are either non-existent or negligible. The general availability of glTF viewers would mitigate such attempts, as users expect glTF to be portable across software an platforms, and not being able to view a glTF in a general purpose glTF viewer would suggest that something is wrong with it. Beyond that, if there is a solution to prevent misuse of The idea of a roundtrip is alluring. However it is not the goal, and would not always be possible. While we would be interested in tools generating consisten Similarly, no tool should be required to preserve However, as game engines take advantage of In the topic of adoption by tools... As far as I know |
The term "roundtrip" also came up in the earlier discussion (and it was said that this was not the primary goal). But in view of the alternatives that have been mentioned here ( In how far would these IDs go beyond what an authoring application and an engine could achieve with a bilateral agreement? The application and the engine could just agree to store the ID, as a string, in the For example, the specification could require a roundtrip capability for the most simple case (that does not involve editing), by stating: "Importing (then not modifiying) and exporting an asset MUST keep the original IDs". Of course, there are many cases to consider: What if multiple assets with conflicting IDs are merged, and the result is exported again? All this could go down into very nitty-gritty details, e.g.: What if an authoring application just swaps two nodes? Do they keep their IDs (because they are still the same nodes), or do they have to receive new IDs (because they have new parents)? I think that some/many of these constraints could only sensibly refer to authoring applications in particular. For example, one could expect that Blender builds sophisticated (authoring-oriented) structures that allow to keep track of all IDs that have been read from the input. In constrast to that, a glTF loader library in a game engine might have methods to read glTF into the engine-specific 'model' object, or write such a 'model' as a glTF, but it might perform operations (e.g. optimzations, like dropping "unnecessary" data, like the IDs themself, unused materials, or nodes in node chains) that make it impossible to reconstruct the original IDs. |
|
@bhouston I understand perfectly what you mean, but I completely disagree. It's not a helpful layer of abstraction. Using
This is not always possible. For example, setting a material may use an engine-specific material type that has no way to be saved to glTF. Or, you may attach a script written in that game engine's programming language using that game engine's API, which can never be fully portable to glTF.
Yes, this is precisely the idea. Well, ideally, the glTF is a very large part of the asset, like 90% or more. This way when you need to have some data specified in your engine-specific format, you can have 90% or more of your asset be a portable glTF, instead of 0% (with all data being stored in the engine-specific format).
Not necessarily. The base asset could be portable, but it would just be missing whatever functionality you added in-engine. For example, when making an avatar for VRChat, you start with the base model with a mesh, skeleton, materials, etc, and may replace materials, add new functionality (like spring bones for dynamic hair/tails/ears/etc), etc. The base model is still fairly portable, it can still be moved to other apps, used with its skeleton etc, but it will just be missing the final last-stage tweaks (like the hair will be static relative to the head). Of course ideally we should continue building standards to allow specifying more and more of the data in the glTF file itself (for example, the VRM consortium has a glTF extension for spring bones), but there will always be application-specific needs that go beyond what's standardized. |
To quote JonathanDotCel's #1051 (comment):
Since the engine builds and shows a very close representation of the glTF structure, we have the expectation that the name carries out. If it is the name what is held unique and unchanging, we want a display name property, so we would be talking about adding a new property any way.
If they have Yes, there would be workflows. The following is what comes to mind: Ignoring the Authoring tools could also create new The authoring tools could also preserve Scrapping the However, game engines could also keep track of The following are my ideas of what to do with conflicts: Under the premise that exported
In eiher case, it seems that some id for glTFs as a whole would be useful to solve conflicts. I'm inclined to believe that authoring software could use (a hash of) the (relative) file path for this purpose. I'm suggesting a hash here as to not embed potentially confidential paths, and I'm suggesting relative paths to allow moving the files of the project as a whole without breaking these ids. |
One could consider to explicitly recommend to not use the IDs for a purpose that can be achieved with pure glTF. For example, they could be used to assign "physical material properties" to meshes, but should probably not be used to model something like parent-child relations between nodes (or anything else that can already be represented as pure glTF). It could be hard to phrase that precisely. It can hardly be a strict requirement, because the features of glTF will be extended with ... other extensions. But it might be something on the level of a hint about the scope of the extension, like a "best practice", or an "Implementation Note"...
The comment about using the
Yes, it would defeat the specific purpose of 'identifying a node regardless of its parent'. I could now ask further (overly specific) questions: Will a node receive a new ID when a child node is attached? Will it receive a new ID when a Which editing operations in an authoring application MAY/MUST cause the IDs to change, and which ones MAY/MUST NOT affect the ID? (More technically, this could be seen as a question about the concept of 'equality'. Or more philosophically, as an instance of the thought experiment of the 'Ship Of Theseus'...) For each answer, one could come up with scenarios. For example: If the ID was intended to identify a node that is expected to contain a mesh (let's say a node with a mesh for which physics computations should be performed), then removing the mesh will make the node "invalid" for that purpose. Iff something like this was an intended use case, then I'd be curious about the behavior that is expected from an authoring application and an engine in such a case. (Note: These questions are not meant to dismiss the idea. And the answers to these questions may very well be "This is not relevant", or start with the usual "That depends...". They are intended to get a clearer idea about what could (reasonably) be specified for such an extension, beyond the JSON schema that says that there is some extension object with a |
No, that would defeat the point of UIDs.
No. In editors where this is possible, the UID should not change because it's the same node. But also, note that in many apps like in Blender or Godot, creating a new mesh requires creating a new node.
Yes, but if you are looking for a mesh and that mesh is gone, there is no possible configuration that would allow mesh modifications to be re-applied. So this is expected. Also, this scenario won't occur with Blender or Godot, because the only way to add/remove meshes to nodes is to add/remove the nodes and have them be of type mesh.
No, that would defeat the point of UIDs.
Creating a node MAY give it a UID. Or, if there's an existing file without UIDs, one may wish to add them afterwards. Once a node has a UID, it MAY be stripped for the "final product", but it MUST NOT be changed or replaced with any other editing operation. There are no valid editing operations that will result in the UID property automatically changing from one UID to another UID. If a node has a UID, it must never have any other UID. |
|
Just for clarification,
You do mean: in the same file in the same application, right? "Exporting the GLB and importing it again in the same software" may already result in new UIDs due to internal format differences or collisions (things that aren't nodes are turned into nodes on export and vice versa). "Exporting the GLB and importing it elsewhere and exporting it again" may also result in new UIDs. |
That's the core of the answer that my questions aimed at. And is the strictest requirement that can be imposed here (and that could go into the specification accordingly). But I'm s stickler, and will ask a few more questions. I think that this is important for the long(er) term goal of a robust technical specification. Some of these questions might seem to be hypothetical (for example, because they are not applicable for one specific authoring tool - even though they may be applicable to another one). If someone thinks that this is not important, then these questions can be ignored. There seems to be a level where we could talk about the vocabulary:
A "node of type 'mesh'" is something that doesn't translate well to glTF. When I'm talking about a 'node', then this refers to a glTF node, as something that determines the hierarchical structure of the scene. This might be mapped to the concepts of authoring applications in different ways. Regarding the question of 'removing a mesh from a node':
One caveat here is that a 'mesh' in glTF cannot sensibly receive an ID. Of course, the The question about 'removing a mesh from a node' could therefore be phrased in a more abstract and generic way: Who is responsible for ensuring that structural modifications (that keep existing IDs) do not interfere with the purpose of the identified element? To emphasize this again: There might be 'simple' answers to that, like "This is not relevant for the specification", or "It's the responsibility of the person who edits the model and who has to know which modifications are allowed". And that's perfectly fine. On this level, one could say that I just want to know whether there is such a 'simple' answer or not...) (And an aside: If someone is supposed to actually implement the handling of glTF IDs in an authoring application, there will be some questions on a far lower technical level. For example: When you delete a node and press CTRL-Z (undo) - will it be guaranteed to receive the same ID again? It should be, right? What about the squence |
Sure, I'm just mentioning this to argue that the case you mention will not occur in Godot and Blender. But anyway, even in applications where it can occur, it's not a problem, changing the contents of a node should still keep the UID.
In Godot, this is not a problem. Mesh resources are stored in memory. If multiple nodes use the same mesh, then by default they share the same mesh resource. So a single mesh instanced multiple times is still one mesh with one object in memory and one UID (if it has a UID). I suppose this may not be the case in all apps, but most engines have the concept of instancing a mesh. What gives you the impression that glTF meshes "cannot sensibly receive" a UID? Also, the case you mention about physics - I get what you're saying in a hypothetical sense, but in this particular case it's not a valid example, because glTF physics does not work like that (in both of the competing extensions, a physics shape may use a mesh, but they do not add any data to the mesh resource itself). |
|
The point about the uniqueness and instancing of meshes was probably not stated properly. I'll try to be more specific. This attempt to be more specific bears the risk of being too specific, causing the response: "That's now how it is done in engine X". But the behavior and handling of IDs should be consistent across multiple applications, as far as reasonably possible (!), and within the intended use-cases. For an application-independent format, one should be able to either say 1. what the behavior should be, or 2. explicitly (!) say that a certain aspect of the behavior is not specified. (And again: that's fine. I'm trying to find 'the limits of what can be specified' here...). A glTF file may contain a certain mesh. And this mesh has an ID. The engine knows that it should create, say, collision detection information for this mesh, based on its ID. For example, if the mesh is attached to a node with a translation of (1,2,3), then the engine may create its collision detection information (some BVH or spatial hash) specifically for the mesh at the position (1,2,3). When the mesh is attached to a different node with a different translation (in the authoring application), then the engine will build the collision detection data structure for a different location. That's fine, And one of the explicitly intended use-cases, as far as I understood. |
|
@javagl I think you are making it more complex than it need to be. For unique IDs, to me everything should potentially be able to have it. Both instance and mesh, as well as material, texture, animation, etc. |
If you have a UID on an object directly, it:
None of the things you listed are benefits of XMP, all are achievable without XMP.
No, it's that XMP provides no benefits. Everything we desire to do with UIDs can be done without XMP.
It's even easier to read the UID on the nodes without parsing an XMP packet.
Again, you have not listed any benefits of XMP. Just evidence that using XMP will satisfy the need. But... you can also not use XMP, and that will also satisfy the need. |
A proposed gltf extension by donmccordy would be to simply enforce unique names for all resources. Advantage is that this 1. avoids uids 2. avoids xmp 3. avoids making downstream applications change their design. Now you have a primary identifier of the unique name and the secondary identifier of the animation pointer path. You can use both identifiers to handle collisions. |
|
@donmccurdy @weegeekps Seriously, this take of wanting to add unique IDs in a generic container extension is really confusing to me. Why are we making a standard otherwise? IMO these things should be used only when the exporter and importer want to send custom information, not when you want to push for something standard. Unique IDs is a feature with a lot of demand that we expect will be implemented in game engines and DCCs all over the place. Trying to "push it to generic container" IMO is not the right way to go on this, given the demand. Is this not what extensions are for, agree on a way we all want things and have a centralized place (this repository) where you can find them? |
|
As I understand it, a first level Otherwise, the solution is simple, like other third parties do, it's possible to propose a |
Major disadvantage is, if user renames asset int he DCC, then you expect it to be renamed in the GLTF too, hence point of extension is moot. UID is the only real solution to this.
To be honest discussing or speculating around this further is what should be justified, not the other way around. The use case is 100% clear and obvious here and the extension is extremely simple:
So:
IMO, then as a standard body, the steps that I expect should be taken here should be quite clear too:
If there is really something further that needs to be left to the realm of speculation, IMO at this stage could be done in the request for feedback. But saying that more should we discussed because there could be more potential use cases IMO is not the right way a standards body should work. If there are more use cases, then they should be a separate extension. There is no reason to try to cram and discuss everything that potentially and speculatively happen under the sun in a single extension. This is a standards body and IMO the purpose of creating extensions should be by focusing only on concrete, explicit use cases with real world demand, not speculation. |
|
@reduz by "other use cases" I don't mean uses cases other than DCC=>Engine I mean things like this: Let's say the UID in the extension is a STRING, so anything goes as long as it's unique. Then a given DCC exports UIDs that are INTEGERS. So how do you address that? I maintain a tool that allows transforming gltfs, merging them, etc. One of the operations the tool can do is to apply an Axis switch, which is typically done by adding a root node with a 90 degree rotation and moving everything else inside that node. Let's say someone processes a model that has UIDs.... the resulting model will have UIDs in the pre-existing nodes, but NOT in the root node. How tools like mine should handle that? Then there's the question of asset merging and collision UIDs, how do you handle them? All these questions need to be answered in the specification so all tools and pipelines behave in a consistent way
Then you have to be sensitive to other parties requirements and concerns. |
Game engines do not keep track of this currently, so IMO this is what I mean with speculation, but if they did they do and this was a problem, they either add a workaround or they can simply SHA/MD5 the string.
That's up to you. The use case here is basically for the DCC -> Importer workflow which is a very common use case. Yours is not, you can give options to users on what to do.
Again, all this is speculation to me and I don't think a standards body should act upon speculation. It should act upon concrete use cases. In this case the concrete use case is tracking your data from the DCC. Asset merging has nothing to do with it and can't be considered a concrete use case. That said, given those are text, you can simply append the filename to it and be it; like "uinqueid-file1.glb" or give the choice to users if they want a re-hashed UID between both. IMO this is dependent on the situation and what you are doing, so it should be on the merging side. Another solution would be enforcing universal unique IDs (uuids), but IMO this would make the extension far more complex because then you would have to determine in the extension how these IDs need to be generated (precise algorithm). This would make the extension far more difficult to implement and process, and for use cases that are entirely speculative.
Why? As I said if other parties have a need, they can propose their own extension and justify why they need this, not try to cram their needs on something unrelated to it. Doing speculation is free, anyone can do it, hence these opinions should not be taken into account. The only reality that matters is when you have an actual, real world need that needs a solution. This should be acted upon. Speculation means solutions in search of a problem, should not be acted upon. |
|
@vpenades Actually upon further thought, I wonder if an extension could suggest the creation of UUIDs to implementers with any library of their choice, but not require it. |
|
On the question of "What operations should and shouldn't affect UIDs?": |
|
I'll try to limit my nitpicking questions here to a minimum. It might very well be that there are "obvious" answers to some of the questions, and I'm just not aware of them, because I don't know the technical details of the interaction between authoring applications and engines for specific workflows. But...
... this sounds like much of the responsibility is moved to the person who edits the asset, and has to be aware of the concept of IDs, how they are assigned, and how they are used. (The latter may be very specific for the authoring application, the engine, and maybe even the model). I could imagine that someone picks out one node from a hierarchy, attaches it to a different parent node, adds new child nodes, assigns a different mesh to this node, and maybe makes this node part of a skeleton for vertex skinning or adds an animation that modifies the transform of this node... the meaning, context, and purpose of that node will then have changed completely, but the IDentity of that node may remain the same. However, if this is not expected to cause trouble, then that's fine... |
|
@javagl By the time this enters the game engine, it has the file context added, so if things are combined inside the game engine this is not a problem. To my make position clearer, if you workflow is this:
Which I am confident will cover well over 99% of cases. You are good. If you do this:
Then it is up to whatever the processing is doing. If, as mentioned above, you are somehow mixing GLTF files, then I can see this benefiting from UUID instead of UID (though I am pretty sure this can be worked around by just appending the file name to the UID, but whathever). If you do this:
Then you will most likely not care about UIDs anyway, since this is a feature geared for game engines. This is why I think, if you want to cover 100% all bases, the extension may have a recommendation on using UUID for the UIDs, but definitely should not be incorporated to the extension due to the high complexity on generating those. |
That is a fair point but I think the concept of a node still being the same node even if you do a lot of changes to it is a fairly intuitive one. (Someone editing the asset wouldn't need to know that this is a UID under the hood, just that a thing generally retains its identity even if they modify it) |
I see @vpenades's example as a very important workflow, as well. But I'm pretty confident that whatever approach we decide on will be OK in that regard, and that details can be worked out off-thread. Example / proposalProcessing tools adding or merging nodes SHOULD aim to generate a UID that is deterministic, and reproducible given the same input. This might mean an +Zup→+Yup transform node added to the scene root is always given UID Not a strict requirement, and obviously needs a little more detail, but I view the risk that we cannot solve this as "low". I expect that a similar approach would be fine for glTF Transform, gltfpack, and other optimization pipeline tools.
Unfortunately, I'm not aware that Blender has any unique ID associated with an object, other than the name... Do other DCC tools? Do exports to FBX or USD avoid this issue today? If there were unique IDs already sitting around in DCC tools that glTF simply can't use, that would be really useful to know. 🙂 To be clear — I'm broadly in favor of solving the issue raised here, and OK with the possibility that it requires a new extension. But let's not treat questions that aren't central to Putting my three.js hat on for a moment ... unique, human-readable names are really nice. I wish glTF names were unique per resource type, as Blender's are. That would solve some issues for three.js users. Unique integers or hash strings are, well, still useful, but I'm going to have to think about how/if we can change our implementation to use a new "type" of ID in addition to names and our existing three.js UUIDs. I'm not saying this in the interest of blocking the proposal here, but to indicate that I can't (yet!) give positive signal as a second implementor. |
We could have an extension that does nothing except enforce the constraint that node names must be unique (see also #2329 which is an extension that only enforces a constraint). Implementations could then look for this and use it as a hint that the names should be treated as unique IDs. This would solve the case of paths changing such as when nodes are moved or parents are renamed, and has the advantage of no new fields. However, this does not fully solve the problem, because some operations (renaming a modified node, or deleting a node and creating another with the same name) will have different behavior compared to UIDs. |
|
Hmm, another idea: What if an extension does both of these things at once? Meaning:
|
|
It would be best to choose one or the other approach. Either/or and specifications doesn't generally mix well and only adds to developer confusion. Between the two ideas the idea to have an extension where it's inclusion guarantees that
It's worth mentioning that a It's likely that many of the DCCs would keep UID as hidden as they do not today have mechanisms for display of these. Even if they could show them, most users can't tell the difference between two UUIDs quickly and speaking anecdotally for a moment, most artists will complain vehemently about having to deal with them. (I've worked in systems before where UUIDs were the main identifier.) UI constraints could also be an issue from the DCC standpoint, which is why we really need their representation here. I can definitely see cases where DCCs may refuse to implement this extension because it falls outside of the norm of their UX workflows. We've dealt with similar cases in the past for other metadata extensions. The UX aspect is actually another reason why I'm really a fan of enforcing uniqueness. It leaves it up to the artist to determine the name, as long as it's unique. Typically they will chose one that they can easily identify and that may be a beneficial matter. If it's made clear that the engine is using the name to link custom materials and other in-engine components to a particular node, in my experiences most artists will take note of that. This downside is outweighed by the fact that now they can also easily identify a node X as having custom Y in the engine while they're working on modifications in Blender. Many of the DCCs also essentially already enforce uniqueness themselves so this may be really easily implemented. |
I think that's absolutely not the way to go. As I said before, I think this is a terrible idea and it should not even be further discussed:
IMO unique ID as an extra property is the only way to do this properly. |
This is the biggest uphill battle I see with this proposal then. We need direct input from the DCCs at this point because my gut is telling me that most will not have interest in implementing this. Internally, most work with their own formats and do not keep track of details between glTF import/exports, and the proposal as it currently stands would require them to start doing so. I suspect this will be an onerous ask. |
|
@weegeekps You don't really need the DCCs to implement this. AFAIK all the popular DCCs can store metadata in the edited objects (Blender, Max and Maya can). Additionally, you know as well as I do that Autodesk will never support GLTF2 officially. All you need is the GLTF2 exporter plugin authors to implement this. |
|
My final thoughts; The title of this issue is misleading because it proposes a mechanism to tag glTF elements with a general purpose UID. A general purpose UID extension requires discussing issues like equality, scope, collision resolution and affected use cases, which are well known issues of unique identifiers. I don't see interest discussing these here. So I agree this issue must be restricted to This proposal asks to resolve a problem, which is to add a mechanism to identify parts of a glTF model when exported from a DCC and imported into an engine, so the engine can apply modifications to said parts. Now, whatever solution it is agreed here, it must go through the process of developers of DCC export plugins adopting that solution. That solution, would look like this in the export dialog
So if the current scene fails to meet these requirements, it can show an error and tell the artist to fix the problem before trying again. At this point any model exported with these plugins will be garanteed to have unique names. Then, as it's been stated here many times, the exported glTF goes straight away into the game engine, this will ensure all names are unique, solving the problem without any extension at all. And a very important detail here: let's say an Extension is added for this, the modifications to the plugins would need to be done anyway, the only difference is that the plugins would also include the extension in the file. In other words, such an extension would only serve as a seal of approval
Yes, and as @weegeekps and others have stated, the artist can do many other operations that would break the model anyway, like cloning what's on scene and deleting the original. Renaming a node is just another way in which the artist can break the model. Furthermore. if the artist breaks the model by renaming something in the scene, names have an advantage over internal UIDs you can rename the node back to it's original name, which would fix the problem. With UIDs, if the DCC chooses to recycle/refresh them, or the artist performs a destructive UID operation for whatever reason, you're sold , there's no way to fix the UIDs back. So I dare say that names are a superior solution over internal unique UIDs My feelings? an uid system is not going to fix a flawed workflow (I'm looking at you, Unity), no matter how common it is nowadays. End of line. |
If you are not familiar with this problem or this problem does not affect you, then I would very much appreciate you simply mind your own and don´t try to gaslight other people that their problems are not real. This is very much a real problem and it needs a real solution. We have plenty of users complaining about this. Suggesting It's not a real problem is extremely unpolite from your part. |
I never said that, I know It's a VERY important problem. I've been developing games, and even 3dsmax export plugins, since almost 30 years now, and I know the problems of exporting assets from DCCs to Engines maybe too well. That's why I offered alternatives to the problem that may probe useful. |
Thanks for taking the effort to do this, but I insist the real root of the problem is that the following to me are facts:
If you have this consensus, it seems reasonable to me that this extension can be proposed. I can ensure we get this consensus, but we must agree this is the way forward. |
That "Why?" could be moved up one indentation level. Everybody can propose an extension. For example, a (I just wanted to point that out, in view of the overall extension development workflow. Any further statements that I could make here would not be constructive (on a technical level) and therefore self-contradictory...) |
|
@reduz I've been following this discussion along and as a fellow Godot developer I fully understand our needs here, but I would like to ask you to take a step back and look at what we're asking from a point of view of 3D authoring software, it is not as simple as you believe. Yes you are absolutely right that DCCs can store meta data, but that is only the start of what is needed here. There are many, many features in DCCs that will break the ability to guarantee these unique IDs are actually unique and are properly tracked. One good example here is Blenders ability to reference multiple blend files, blend files artists often will have purchased from assets stores, assets that often begin life as a single blend file with a number of base objects, that are then duplicated to create variations on the same theme. Blend files that may then be referenced from the actual blend file the artist is using to create a level/asset/etc. At this point in time there will be a large swat of overlapping UIDs that break the system. Another example are tools like Houdini or other generative systems that often have no way of persisting UIDs as geometry is recreated, users will be even more frustrated here as changes will not propagate properly as UIDs change while to the user it's damn obvious the same object was recreated with new settings. I can easily list many more. I'm not saying these issues can't be fixed, but we are putting a requirement up as a consumer of the GLTF format that will stifle the abilities of many authoring software to a point that it is unlikely many will be willing to implement the extension, or if they do, do so with a big asterix saying that the system will likely fail in complex situations. That all said, the most important thing here is to listen, not to me, but everyone here reacting. The problem isn't that they do not understand the needs of a game engine, the problem here is that the needs of a game engine do not outweigh the larger needs of this industry. My humble opinion here is that this is an unsolvable problem because DCCs have too many features that make a simple UID solution unmanageable. I'm also leaning more to either enforcing unique names, or at the very least warn users that re-using names will lead to an inability to track changes over versions. I fully agree that there are weaknesses to this approach and that the user can easily break things by changing names. |
|
There is an issue #1713 and apparently something that is called (We could consider to explicitly ping |
|
Sorry folks for the heated debate and I apologize for being pushy with this. I appreciate all the feedback given and will try to contact the right stakeholders here to put together a better formed specification proposal to find a way to solve this, where we can better discuss on something concrete. |
Yep.
@aaronfranke this is more or less what we did (though we never used UIDs, we just used the name) We created this extension for many of the reasons mentioned here. I left Asobo at the end of last year, so I can't speak for experiences past that, but I'll write down some notes. The origin of this extension is here. I must say that, in practice, this indirection of identification does complexify a number of things. It is an effective solution for many problems, but relying on unique ids requires a certain understanding from everyone involved in the pipeline. For example, an artist can't simply destroy and recreate an object and "give it the same name". Additional tooling (and knowledge) is required for such cases. Some verifications on exports were added too. Flight simulator uses a somewhat modified version of glTF that imports directly into the game. We used this extension to be able to link up glTF hierarchies at runtime. This allowed us to export animation files and "mesh liveries" separately from the models, and allows us to change things as long as the nodes targeted for merging and the hierarchies in between them don't change (this means the hierarchy above an animation-targeted node can actually be changed without issue). For an SDK perspective, see the MSFS docs about submodel merging. Inside the simulation, if a unique id extension is marked as used in a file, the unique id of a node is either the id specified by the extension, or its name if the extension is absent. Gameplay code and scripts did not use the unique id extension, it was used exclusively for the model merging, though that is certainly also because it was created very late in the project. I can't find the json schema for the extension, but it's very simple anyway: "extensions": {
"ASOBO_unique_id": {
"id": "HIPS"
}
},We also created (at the same time) an extension called |
and others
Continuing from the discussion here: #1051 (comment)
The glTF standard does not currently endorse any particular way to define unique identifiers. There are no UIDs, nothing beyond names is provided to identify glTF objects. This can make it tricky for applications to deterministically keep track of objects in glTF files. The problem is not isolated to game engines, it also affects glXF files.
The problem:
Some options for solutions: (EDIT: Removed number 2)
Recommend using node names as unique identifiers, and do not add a UID property. This is in line with what glXF already does, it can reference nodes in a glTF scene by unique name.
KHR_unique_idthat has one property:"uid"(and for glXF, a way to refer to these).Add a
"uid"property to the base "core glTF" spec itself. This is probably not the preferred solution given that the glTF spec is pretty much frozen, but if we did add this, it wouldn't break either forward or backward compatibility because it is optional, so it could be done in a hypothetical glTF 2.1 (no need to wait for a hypothetical glTF 3.0).EDIT: Another option, currently my favorite. This is a combination of ideas 1 and 3. Basically, we have an extension that enforces the constraint that glTF files have unique identifiers. By default, use the name as the unique identifier (idea 1), but optionally we can supply a separate UID (idea 3). This allows retroactively adding UIDs to files that didn't start with one (ex: node "A", renamed to "B" with UID "A", then it can track that as the same node).
Regardless of which option is chosen:
The text was updated successfully, but these errors were encountered: