The ultimate ambition is to enable folks to efficiently load and process large, multi-dimensional data as fast as modern hardware will allow.
For now, this repo is just a place for me to tinker with ideas. This code won't do anything vaguely useful for months!
Under the hood, my hope is that light-speed-io will use io_uring on Linux for local files, and use object_store for all other data I/O.
My first use-case for light-speed-io is to help to speed up reading Zarr. After that, I'm interested in helping to create fast readers for "native" geospatial file formats like GRIB2 and EUMETSAT native files. And, even further than that, I'm interested in efficient & fast computation on out-of-core, chunked, labelled, multi-dimensional data.
See planned_design.md for more info.
Light Speed IO is organised as a Cargo workspace with multiple (small) crates. The crates are organised in a flat crate structure. The flat crate structure is used by projects such as Ruff, Polars, and rust-analyser.
LSIO crates names use snake_case, following in the footsteps of the Rust Book and Ruff. (The choice of snake_case versus hyphens is, as far as I can tell, entirely arbitrary: Polars and rust-analyser both use hyphens. I just prefer the look of underscores!)