Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invalidating cached samples #18

Open
calebwin opened this issue Aug 13, 2021 · 0 comments
Open

Invalidating cached samples #18

calebwin opened this issue Aug 13, 2021 · 0 comments
Assignees
Labels
banyan-jl Concerning Banyan.jl bug Something isn't working

Comments

@calebwin
Copy link
Contributor

When a call to a read_* or write_* (e.g., read_hdf5 or write_parquet) function is made, we are simply assigning a Location to a Future. As part of constructing a Location we take a random sample of the data at the location along with some metadata but we also cache this location locally so that we don't have to recompute it.

Naturally, on a call to a write_* function, we invalidate the cached sampled if a sample was cached for the location being written to. But we should also store the date that a sample was taken so that we can determine whether S3 data was written to more recently which would invalidate the cache.

Also, we should expose keyword arguments for Location constructors to callers of the read_* or write_* functions.

@calebwin calebwin added bug Something isn't working banyan-jl Concerning Banyan.jl labels Aug 13, 2021
@calebwin calebwin self-assigned this Aug 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
banyan-jl Concerning Banyan.jl bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant