Skip to content
This repository has been archived by the owner on May 2, 2022. It is now read-only.

optimize the sending results from worker nodes for data frames #11

Open
TaiSakuma opened this issue Sep 16, 2017 · 0 comments
Open

optimize the sending results from worker nodes for data frames #11

TaiSakuma opened this issue Sep 16, 2017 · 0 comments

Comments

@TaiSakuma
Copy link
Member

TaiSakuma commented Sep 16, 2017

currently, the package concurrently, uses pickle to send the results from the worker nodes.

pickle files with data frames tend to become large. and loading large pickles is slow.

develop an alternative implementation which is specifically optimized for data frames, for example, by using dask so that loading the results at the interactive node is fast.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant