You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When a call to a read_* or write_* (e.g., read_hdf5 or write_parquet) function is made, we are simply assigning a Location to a Future. As part of constructing a Location we take a random sample of the data at the location along with some metadata but we also cache this location locally so that we don't have to recompute it.
Naturally, on a call to a write_* function, we invalidate the cached sampled if a sample was cached for the location being written to. But we should also store the date that a sample was taken so that we can determine whether S3 data was written to more recently which would invalidate the cache.
Also, we should expose keyword arguments for Location constructors to callers of the read_* or write_* functions.
The text was updated successfully, but these errors were encountered:
When a call to a
read_*
orwrite_*
(e.g.,read_hdf5
orwrite_parquet
) function is made, we are simply assigning aLocation
to aFuture
. As part of constructing aLocation
we take a random sample of the data at the location along with some metadata but we also cache this location locally so that we don't have to recompute it.Naturally, on a call to a
write_*
function, we invalidate the cached sampled if a sample was cached for the location being written to. But we should also store the date that a sample was taken so that we can determine whether S3 data was written to more recently which would invalidate the cache.Also, we should expose keyword arguments for
Location
constructors to callers of theread_*
orwrite_*
functions.The text was updated successfully, but these errors were encountered: