Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support grouping of vector data in HNSW #3227

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Support grouping of vector data in HNSW #3227

wants to merge 1 commit into from

Conversation

heemin32
Copy link

@heemin32 heemin32 commented Jan 30, 2024 •
edited

Add IDGrouper in SearchParameter so that it can be used to dedupe search result based on its group id.

Issue: #3087

@heemin32 heemin32 force-pushed the parent-join branch 7 times, most recently from 6e697d9 to 99dd306 Compare January 30, 2024 23:13
@heemin32 heemin32 mentioned this pull request Feb 1, 2024
Merged
5 tasks
@heemin32
Copy link
Author

heemin32 commented Feb 7, 2024 •
edited

@mdouze , would you mind taking a look?

@mdouze
Copy link
Contributor

mdouze commented Feb 9, 2024

Sorry for being unresponsive.
The reason I'm reluctant with this solution is that IMO the new ResultHandler API makes it possible to move the whole de-duplication logic outside of Faiss. In the calling code, just set up a ResultHandler class that does the deduplication and call hnsw.search on that.

@heemin32
Copy link
Author

heemin32 commented Feb 9, 2024 •
edited

I search HNSW through IndexIDMap. Do you have a plan to expand the ResultHandler to IndexIDMap? I need a way to access id_map data to dedupe result properly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CLA Signed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@heemin32 @mdouze @facebook-github-bot