Binary passage retriever (BPR) is a two-stage ranking approach that represents the passages in both binary codes and dense vectors for memory efficiency and effectiveness.
Ikuya Yamada, Akari Asai, Hannaneh Hajishirzi. Efficient Passage Retrieval with Hashing for Open-domain Question Answering. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 979-986, 2021.
We have replicated BPR's results and incorporated the model into Pyserini. To be clear, we started with model checkpoint and index releases in the official BPR repo and did not train the query and passage encoders from scratch.
This guide provides instructions to reproduce the BPR's results.
Here's how our results stack up against results reported in the paper using the BPR model (index 2.3 GB + model 0.4 GB):
| Dataset | Method | Top-20 (orig) | Top-20 (us) | Top-100 (orig) | Top-100 (us) |
|---|---|---|---|---|---|
| NQ | BPR | 77.9 | 77.9 | 85.7 | 85.7 |
| NQ | BPR w/o reranking | 76.5 | 76.0 | 84.9 | 85.0 |
BPR with brute-force index:
python -m pyserini.search.faiss \
--index wikipedia-dpr-100w.bpr-single-nq \
--topics dpr-nq-test \
--encoded-queries bpr_single_nq-nq-test \
--output runs/run.bpr.rerank.nq-test.nq.hash.trec \
--batch-size 36 --threads 12 \
--hits 100 --binary-hits 1000 \
--searcher bpr --rerankThe option --encoded-queries specifies the use of encoded queries (i.e., queries that have already been converted into dense vectors and cached).
To evaluate, first convert the TREC output format to DPR's json format:
$ python -m pyserini.eval.convert_trec_run_to_dpr_retrieval_run \
--index wikipedia-dpr-100w \
--topics dpr-nq-test \
--input runs/run.bpr.rerank.nq-test.nq.hash.trec \
--output runs/run.bpr.rerank.nq-test.nq.hash.json
$ python -m pyserini.eval.evaluate_dpr_retrieval \
--retrieval runs/run.bpr.rerank.nq-test.nq.hash.json \
--topk 20 100
Top20 accuracy: 0.7792
Top100 accuracy: 0.8573