RFC: Transparently fetch multiple documents grouped per block #2276
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
adamreichold
force-pushed
the
block-cache-control
branch
from
9ce44a3
to
d6e1ec8
Compare
adamreichold
force-pushed
the
block-cache-control
branch
from
d6e1ec8
to
6fe01c4
Compare
adamreichold
force-pushed
the
block-cache-control
branch
from
6fe01c4
to
59695d1
Compare
adamreichold
changed the title
adamreichold
commented
Dec 11, 2023
| &self, | ||
| doc_addresses: impl IntoIterator<Item = DocAddress>, | ||
| ) -> crate::Result< | ||
| impl Iterator<Item = impl Future<Output = crate::Result<Vec<(DocAddress, D)>>>> + '_, |
There was a problem hiding this comment.
The signature is admittedly somewhat unwieldy even though relatively easy to use in practice as the test shows. I wonder if I should add a no_run example showing usage with FuturesUnordered? (no_run to avoid making the example unreadable due to index setup code. Alternatively the setup code could be hidden.)
There was a problem hiding this comment.
Added a no_run example on how to use this with FuturesUnordered and block_on.
adamreichold
force-pushed
the
block-cache-control
branch
3 times, most recently
from
d00dd5a
to
a2953d6
Compare
…by group them by segment and block.
…PI if only Searcher::docs_async uses them.
adamreichold
force-pushed
the
block-cache-control
branch
from
a2953d6
to
d1177fe
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
My personal take on the discussion in #2252, i.e. expose enough information and control to schedule concurrent reads into independent block caches as to maximize cache effectiveness and then expose this as
Searcher::docs_asyncwhich yields an iterator of futures which user code can then poll concurrently or in parallel as it sees fit.