copy data cross object storage

[yitzhaklevi]

Is it possible to use S5CMD to transfer data from S3 to GS ? (or vise versa)
(Currently thinking about transferring from S3 to local file system to GS (after configurating properly), but this solution would probably run slower + require local disk memory.
Thanks !

[szeitlin]

Should be possible if you're logged into a machine on one, you can pull or push to the other (I'm doing this now with a couple of different filesystems, just not those).

[yitzhaklevi]

Thanks @szeitlin for the quick response !
I didn't quite understand your answer, to which machine I need to be connected ?

[szeitlin]

Having slept on it, I'm actually not sure if what I'm doing would work for what you want to do. I think probably what you need to do is have 2 profiles - one for S3, and one for GCS, and it doesn't look like s5cmd currently supports being able to use both at the same time.

I'm currently working with a system where I can log in and install my credentials for the other storage system, so I only need one set. I'm not sure if you could get that to work with e.g. EC2 or something.

[igungor]

Hello @yitzhaklevi

Unfortunately, it's not possible right now. As you mention, you'd need a buffer (such as disk, or memory) to first download, and then upload the objects you want to transfer.

If it's a one time thing, you may want to take a look at S3 to GCS transfer guide. If you'd like to synchronize occasionally, rclone tool is good at syncing between different storage providers.

Duplicate of #529