You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Right now most vector DBs support uploading duplicate vectors as long as the ID is unique. It might be beneficial for vectorflow to 1) prevent the uploading of duplicate vectors
2) deduplicate an index/collection/class object store
But is it possible to enforce these things in a performant way
The text was updated successfully, but these errors were encountered:
dgarnitz
changed the title
Investigate a duplication prevention and deduplication mechanism
Add a duplicate prevention mechanism
Apr 4, 2024
The default deduplication field should be the source filename. We do not need to go down to the chunk level. Whoever builds this should take into account how to check this for each vector DB. If it's not possible to search qdrant/weaviate/pinecone by metadata field or ID, then we may need to use a SQL table in postgres. I don't want to add Redis just for this though
Right now most vector DBs support uploading duplicate vectors as long as the ID is unique. It might be beneficial for vectorflow to 1) prevent the uploading of duplicate vectors
2) deduplicate an index/collection/class object store
But is it possible to enforce these things in a performant way
The text was updated successfully, but these errors were encountered: