Merge Two Deeplake datasets

[gilvikra]

Hello! I want to parallelize Deeplake dataset creation over multiple machines and then merge them together for speeding up the process. My datasets are 10+ TB in size with images. Is there any possible way to accomplish the same ?

[istranic]

Hey @gilvikra thank you for raising this question. We have partial solutions to your issue, and we also have additional improvements on our roadmap. Here's a summary.

• Currently, the best way to accelerate ingestion is via the deeplake.compute API. It only works on a single machine, but is uses multiple processes and threads.
• We're releasing speedups to the API above later this week. Here's the PR.
• In the future, we will extend the API above to work using Ray on multi-node systems.
• A very popular request from users has been to combine datasets virtually (i.e. datasets are stored separately, but can be concatenated into a single virtual dataset that can be used in the Deep Lake API). We will likely build this feature in the next month.

Would any of these options help you with your workflows?

[gilvikra]

Hello @istranic .! Thanks for the prompt response. Let me try out the deeplake compute API. It should suffice to some degree. Other than than that, integration with RAY and the ablity to combine datasets virtually would be pretty helpful too. But again if the separately stored datasets can be all put on a local storage and addressed as one dataset, then it may speed up the fetching process. Would this part be addressed in your upcoming feature too ?

[istranic]

@gilvikra Yes, this is what a virtual dataset would be. It would enable you to combine different datasets and address them as one. I wouldn't matter if they're local on in the cloud.