Feature Request: Download bestref from AWS S3 buckets

[pllim]

In an R&D effort over at https://jira.stsci.edu/browse/JUSI-8 and trying to run ACS pipeline in a notebook, I see commands like:

import os

os.environ['CRDS_SERVER_URL'] = 'https://hst-crds.stsci.edu'
os.environ['CRDS_SERVER'] = 'https://hst-crds.stsci.edu'
os.environ['CRDS_PATH'] = './crds_cache'
os.environ['jref'] = './crds_cache/references/hst/acs/'

crds bestrefs --files j9l960a7q_raw.fits --sync-references=1 --update-bestrefs

While it works, when I am in a JupyterHub instance hosted on AWS and want to grab the bestrefs from existing S3 buckets, I can't think of a way to do it without bypassing CRDS interface (e.g., manually parsing FITS header and constructing URIs for each reference file).

Currently, ASB has all HST reference files uploaded at S3://stpubdata/hst/public/references/. For example:

• s3://stpubdata/hst/public/references/hst_wfc3_bpixtab_0260.rmap
• s3://stpubdata/hst/public/references/2ck1856fi_bpx.fits

I am wondering if it is a reasonable request to have CRDS have a "cloud" option like astroquery.mast does; see Python snippet in https://mast-labs.stsci.io/2018/06/hst-public-data-on-aws .

cc @stscicrawford @ivastar