Allow usage of custom id in collections

[jeanbmar]

First, a few words about our use case: we load payload with a bunch of external data generated programmatically by a 3rd party system, and we give a few users the power to override some of the data.

Using mongodb generated ids implies a pretty bad overhead, because we have to insert all the records in the collections, and rebuild and update all the relationships between them afterward by retrieving all the documents, using alternate keys / indices, etc.

Ideally, we would like to be able to specify our own unique ids as a replacement of the standard ObjectId when submitting records. By doing so, we could load the records easily together with their relationships.

We are willing to work on a PR if you agree with this idea.

A possible way to do it could be through an explicit id field in collection definitions.
Payload could then act accordingly when:

• Creating and updating records and relationships in backend
• Creating records in admin
• Reading records (e.g. here)
• Building API bindings

What are your thoughts on this?

[jmikrut]

Hey @jeanbmar — this is a great idea and the Payload team certainly sees the value in this being a new feature to add. We've actually got a project running now that would also benefit from this 😎 .

We also like your suggestions about how the API would function, via a deliberate id field.

Looks like MongoDB supports it too. However, as we use Mongoose under the hood, we would need to ensure that Mongoose's findByID / similar APIs still function accordingly as well.

This may be a relatively complex change... And it would certainly need to be very thoroughly tested. If you and your team would like to take a stab at it, by all means, we would support you! However, we'd be happy to slate this into our own workload as well. We have a list of priorities at the moment, including some pretty exciting stuff, so this might take us a few months to tackle. But we'd be happy to do it.

What do you think?

[Ontopic]

Definitely want to help out here as well! Currently I'm use MongoDB directly for various reasons and ObjectId is getting kind of annoying...

[jeanbmar]

Thank you for the quick reply! I'm happy that you see value in this.

I'm not worried about MongoDB and Mongoose because we've used custom ids with both in the past.
Now we need to have a closer look at payload code to measure the technical impacts in details and see if we can take it.
I'll follow up here when we are done.

[Ontopic]

I mainly use it for bulk operations where I don't need the relationships, but f.e. just all the urls. Sometimes - during a build f.e. - the Payload instance might or might not be there. This way core operations can always run. Looking up relationships has to be done manually, but if types are exposed in the future that could be simplified / automated.

[richardvanbergen]

Hey, newbie mongoDB user here. Why would you not want to use the object ids? I kind of get the importing data thing but I'm interested to understand why it's an issue otherwise.

[jeanbmar]

Let's say your data come with existing relationships between logic units. For instance you have characters which have relationship with skills. And you have skills which have (different) relationships with characters.
You have to:

• Load all the characters without relationships
• Load all the skills without relationships
• Retrieve all the characters from the database (to get mongo ids), and update every single skill record with the relevant character ids to establish the relationship
• Retrieve all the skills from the database, and update every single character record with the relevant skill ids to establish the relationship

Plus anytime you want to update some data in payload after a change in the 3rd party system, you will have to retrieve the records in payload using an index instead of the ids because the 3rd party system has no knowledge of payload ids.
All of this is doable but there's just a huge overhead.

[Ontopic]

Missed your reply here. This #225 might be interesting, not sure if you saw.

[jeanbmar]

@jmikrut I'm almost done and should be able to submit a PR with a small footprint within 1-2 days.
Mongoose has options that make this easy.

[jeanbmar]

Opened #266

[DanRibbens]

@jeanbmar After messing up your PR entirely, I finished up the work on custom ID fields and it is now released in v0.10.0 🎉

I'll be using this in another project soon, thanks for the work on this one!

[jeanbmar]

Awesome! 🎉🎉
This was crucial for us to go to production, now we are almost ready!

[DanRibbens]

Glad to hear it! Let us know if you need anything else.