Favor upsert over add + use element_id to prevent duplicates

[0xjgv]

Make chroma ingest pipeline idempotent :)
@potter-potter

[potter-potter]

@0xjgv Thanks for this! And thanks for alerting me. Great idea.

I will bring it up to the team.

[ahmetmeleq]

Thanks for the PR! I do agree with the upsert.

I had doubts on assigning the element id, however based on the last state of id calculation, collisions don't seem to be likely. Looks good to me, needs to pass ingest tests.

[potter-potter]

@ahmetmeleq @0xjgv can you see any downsides to doing it this way? (seems like it is a good idea all aound)

[potter-potter]

@ryannikolaidis Think this change will affect any users of ChromaDB that expect duplicates? (not sure why they would want duplicates....)

[ryannikolaidis]

@ryannikolaidis Think this change will affect any users of ChromaDB that expect duplicates? (not sure why they would want duplicates....)

with matching ids? I think that's okay

[0xjgv]

@ahmetmeleq @0xjgv can you see any downsides to doing it this way? (seems like it is a good idea all around)

🤔 I can't think of any downsides though. Collisions are not very likely and there are no clear advantages to allowing duplicates in the DB. Idepomtecy reduces storage space, increases query performance, and simplifies data maintenance and integrity.

[potter-potter]

@0xjgv Thanks for this! I am putting it into a small pr with one other small update so that it runs through our CI/CD.

#3086

[0xjgv]

Implemented in #3086