Download an entire bucket to a virtual machine

[dachosen1]

I have a storage bucket that has mp3 and Wav files, and I have a few questions.

1. is there a way to read data directly from a bucket without downloading it? I've browsed through the files and it seems that you can only download a file and then read it?
2. Is there a way to download an entire bucket to a virtual machine?
3. Can you migrate all the data from bucket to SQL?

If it helps, my goal is to train a machine learning model and deploy it using google cloud. I'm at the stage where I'm experimenting with different feature engineering methods and save the data for training.

[kurtisvg]

Hi @dachosen1,

This repo is limited in scope to our samples and issues with them. In the future, this type of question would be better fitted to something like StackOverflow.

However, let me provide some quick pointers to some of your questions:

1. It doesn't look like the python-storage library supports streaming from a bucket.
2. You can use gsutil to do this (as well as mount buckets to the filesystem).
3. There is no automatic feature to sync a GCS bucket and SQL - since the data structure here is greatly varied, you'd need to write your own pipeline that understands how to read your bucket content and convert that content into a the correct SQL format.