Storage: Capture relevant headers to blob properties during download

[tseaver]

Residual from googleapis/google-cloud-python#9003.

@william-silversmith notes that even with raw_download enabled, he is unable to detect the content_type of a downloaded blob without performing an additional reload request, which is prohibitive for his usecase at scale. E.g.:

blob = bucket.blob( key )
binary = blob.download_as_string(raw_download=True)
if blob.content_encoding == 'gzip':
    return gunzip(binary)
elif blob.content_encoding == 'br':
    return brotli.decompress(binary)
else:
    return binary

Potentially even...

if blob.content_type == 'application/json':
    return json.loads(binary.decode('utf8'))

[william-silversmith]

Thank you Tres!

[tseaver]

@william-silversmith The headers needed to do this task are buried several layers deep. I'm wondering if a better solution than trying to guess which ones to map onto the blob's properties, we might be better off designing for your usecase by supplying a response_callback argument to the Blob.download_* methods:

• The callback would take the blob instance and a requests.Response object as its arguments.
• For non-chunked downloads, the callback would be called once; chunked downloads would call it once per chunk.

An example callback implementation for your usecase might be something like:

def _update_blob_content_encoding(blob, response):
    if blob.content_encoding is None:  # latch
        blob.content_encoding = response.headers.get('Content-Encoding')

[tseaver]

If we don't use a callback, then the relevant headers to start with would seem to be:

• Cache-Control -> Blob.cache_control
• Content-Disposition -> Blob.content_disposition
• Content-Encoding -> Blob.content_encoding
• Content-Length -> Blob.size
• Content-Type -> Blob.content_type
• Etag -> Blob.etag
• X-Goog-Generation: Blob.generation
• X-Goog-Hash: Blob.crc32c
• X-Goog-Metageneration: Blob.metageneration
• X-Goog-Storage-Class: Blob.storage_class

[william-silversmith]

I almost never need more exotic headers than the ones you listed. If it's easier to only expose those headers, that's fine with me. If you want to go for a more general solution, I am fine with a (synchronous) callback. Thanks again!

[william-silversmith]

Any word on when this might drop? Thanks for your help!

[william-silversmith]

Hi Tres, I've made some changes to my own branch of this repo and they seem to work. I tried making a PR but the button was greyed out. Here's a link to the changes in case y'all are interested. In the mean time, I'll direct my repos to use the forked version.

master...seung-lab:master

[william-silversmith]

Unfortunately, it seems PyPI won't let me use a direct reference to github. It would be extremely convenient if some version of this could get merged.

[william-silversmith]

Figured out the PR submission issue.