Optimise upload speed of multiple files

[danpalmer]

We're trying to upload ~2.5k small files, and currently this takes around 8 minutes, even though we're on a gigabit connection (and GCS being quite fast).

It would be great to get some sort of bulk or parallel upload working. I'd expect this could reduce our upload time to well under a minute. I think my preference would be for batched uploads rather than parallel uploads, as those should use far fewer system resources.

[nigesh-pugazhendhi]

@stephenashank any update on this enhancement? @danpalmer did you find a workaround for this ?

we are trying to upload ~16GB worth of multiple files and it takes ~16mins whereas s3plugin takes in seconds! would love to hear any workaround to this.

[danpalmer]

I’m afraid we never found a workaround to this, and have now moved away from Jenkins entirely.

Recently I’ve been thinking about doing this again for GCS uploads, and found https://rclone.org which looks like it could be much more optimal for large sets of small files where not all have changed (our use-case, and a classic problem that rsync is great at solving). You may find it useful to investigate this, but we haven’t done any testing with it yet so I don’t know how well it works in practice.