New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add metadata to output encoding file #369

Open

joyceyuu opened this issue Jul 21, 2020 · 0 comments

Labels

Contributor

joyceyuu commented Jul 21, 2020

Before computing similarity scores and matching, we normally need to check the count the encodings to see if it is consistent with blocking file.

Currently we either load the whole JSON just to get the number of encodings or use ijson to iteratively count the number of encodings.

It would be better to store the count in the metadata and just read this metadata when needing the count.

joyceyuu added the enhancement label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment