S3 dataset access

[robmarkcole]

Hi
I understand the dataset can be streamed from S3, following the example in the docs I get an error, and assume access must be granted?

 > aws s3 ls s3://clay-tiles-02/02/27WXN/

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

[brunosan]

Hi Rob!!
I think the right move here is to copy a representative sample of embeddings to source.coop

I don't think if it makes sense to publicly host a copy of the whole training set publicly, when is just a cropped selection of data already available. E.g. on v1 we have 50M chips and we are anyways moving towards streaming from source COGs into the GPUs on training. https://github.com/Clay-foundation/stacchip

In the meantime I've just activated requester pays on this bucket.

[robmarkcole]

@brunosan I get an error:

⚡ ~/Clay-Foundation-Model aws s3 ls s3://clay-tiles-02/02/27WXN/ --request-payer requester

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied