Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Documentation About Dask-Distributed Support for file types #61

Open
CSSFrancis opened this issue Nov 8, 2022 · 1 comment
Open

Comments

@CSSFrancis
Copy link
Member

Describe the functionality you would like to see.

I would like to add to the documentation information about which file loaders support the dask-distributed backend. Mostly just add an extra column here

Currently I believe that this is only the zspy and the new file loader #11 but we can think about adding in support for the hspy file format as well as any of the other binary files.

Describe the context

I have defined a function in #11 that works as a drop in replacement for np.memmap and allows for distributed loading of some data. This is particularly useful for large data sets as well as does a much better job handling the available resources.

Additional information

Using the dask-distributed scheduler is the preferred way to interact with dask in most cases. Supporting distributed schedulers at the loading level is important for larger datasets and allows for much better scalable preformance.

@ericpre
Copy link
Member

ericpre commented Nov 12, 2022

Yes, sounds good. This table is created manually and it would better to have this information available in the specification and generate table automatically as we do with hs.print_known_signal_types() for example.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants