Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: use the file metadata cache in scalar indices #2330

Conversation

westonpace
Copy link
Contributor

@westonpace westonpace commented May 13, 2024

We were not using the cache before which meant every scalar index query had to load a bunch of metadata from the index files (some of which was quite expensive)

Closes #2313

@westonpace westonpace force-pushed the perf/use-file-metadata-cache-in-scalar-index branch from 8b1ac79 to 5f3b492 Compare May 13, 2024 18:50
@codecov-commenter
Copy link

codecov-commenter commented May 13, 2024

Codecov Report

Attention: Patch coverage is 93.33333% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 80.66%. Comparing base (b61850b) to head (08d6be9).

Files Patch % Lines
rust/lance-core/src/cache.rs 0.00% 0 Missing and 1 partial ⚠️
rust/lance-index/src/scalar/lance_format.rs 94.11% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2330      +/-   ##
==========================================
+ Coverage   80.64%   80.66%   +0.01%     
==========================================
  Files         192      192              
  Lines       56684    56699      +15     
  Branches    56684    56699      +15     
==========================================
+ Hits        45712    45735      +23     
+ Misses       8405     8395      -10     
- Partials     2567     2569       +2     
Flag Coverage Δ
unittests 80.66% <93.33%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@westonpace westonpace force-pushed the perf/use-file-metadata-cache-in-scalar-index branch from 5f3b492 to 08d6be9 Compare May 15, 2024 15:21
@westonpace westonpace merged commit dfb531b into lancedb:main May 15, 2024
18 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Lance scalar index search loads dataset metadata (which should be cached)
3 participants