records: download all button

[lnielsen]

No description provided.

[lnielsen]

Can likely be achieved via AIP. Related to #34.

[fedorov]

+1

A possibility you could consider is for the user to "request download all" to initiate compression, and then send notification (by email) with a download link of the zip file when compression is done. You could make the link expire in 24 hours and then release the storage.

[slint]

Another solution might be to stream all the files inside a zip using a library such as https://github.com/SpiderOak/ZipStream. No extra temporary disk storage will be required and (hopefully, if the library works as advertised) no extra memory.

Of course maybe a combination of the mentioned methods might be best (if [total size of files] > 2GB, send email with link to async generated zip, otherwise use ZipStream).

Edit: Another way is triggering multiple downloads via JavaScript. This library seems to do it: https://github.com/sindresorhus/multi-download

[peterjc]

I just uploaded a dataset of a few hundred files, and am shocked that there is no end user "download all button" giving a ZIP or tar-ball or similar (which could be generated on the fly).

In hindsight, I should have uploaded an archive myself, but the upload interface didn't give explicit guidance, and was clearly designed to cope with multiple files.

Is the current work around to instead upload a single archive (e.g. example.zip or example.tar.gz as appropriate for the targeted community), as a new version if necessary?

[Update: Given the dataset has not been shared yet, I have used https://zenodo.org/support to ask about replacing the files]

[lnielsen]

Yes, the current workaround is to upload a ZIP (better, and previewed - tar.gz is not previewed).

The problem here is that we have TB-sized datasets, and thus making a "download all" button is not trivial if it needs to scale.

[jcolomb]

An automatic zipping for smaller dataset (at the upload or even later) would then solve most problems, right? It would also save storage space ?

[zaneselvans]

I would also really appreciate this -- we have several files in our archives so users have the option of grabbing only the data they need, but many folks want all of it, and it's tedious to have to click every single file. Or is the general expectation that folks just upload one big zip file as an archive?

[fgnievinski]

Yes, the current workaround is to upload a ZIP (better, and previewed - tar.gz is not previewed).

The problem here is that we have TB-sized datasets, and thus making a "download all" button is not trivial if it needs to scale.

may I suggest including some guidance in the upload user interface.

where it currently says:

Drag and drop files here
— or —
Choose files
(minimum 1 file required, max 50 GB per dataset - contact us for larger datasets)

There could be a note such as the following:

Note: in case the dataset contains more than a few files, please consider packing them in a zipfile, to facilitate download by the user.

[fgnievinski]

btw, there is zenodo_get, a downloader for Zenodo records:

https://doi.org/10.5281/zenodo.1261812

[ivan-paleo]

There could be a note such as the following:

Note: in case the dataset contains more than a few files, please consider packing them in a zipfile, to facilitate download by the user.

It seems to me to be a good idea as long as there is no solution to "download all".

[ivan-paleo]

btw, there is zenodo_get, a downloader for Zenodo records:

https://doi.org/10.5281/zenodo.1261812

What about integrating that tool into the Zenodo interface?

[ivan-paleo]

FYI, there is also an R package to do it (and many other things): https://github.com/eblondel/zen4R

[martinlester]

I can see why this might be quite difficult to implement on the server side.

However, I reckon (if you have a recent enough browser), a "Download All" button that generates a zip could be implemented entirely on the client side using this small JavaScript library:

https://github.com/Touffy/client-zip

If Zenodo's admins don't want to implement it officially, an enthusiastic user could probably make a bookmarklet that implements it.

[lnielsen]

Hi, it's not that we don't want to implement it. It's rather that its a task that likely takes ~2-3 weeks from start to launch and we're heavily resource constrained. As part of optimising resources, we're moving Zenodo on top of the InvenioRDM platform, so if you willing to help out, we're be more than happy to engage with you on how to implement it in InvenioRDM. We do have partners in InvenioRDM that's interested in the same feature, so any help would be much appreciated.