allow custom configs for s3

[activesoull]

🚀 🚀 Pull Request

Impact

• Bug fix (non-breaking change which fixes expected existing functionality)
• Enhancement/New feature (adds functionality without impacting existing logic)
• Breaking change (fix or feature that would cause existing functionality to change)

for the cases if the sure has a V4 signing requirement set on the storage, with the functionalities where the user is trying to connect to the dataset or create a dataset need to provide a custom config

import deeplake
from botocore.config import Config

creds = dict(.....)
cfg = Config(
    signature_version = 's3v4'
)
creds["config"] = cfg
deeplake.empty(path, creds=creds)

[hoshimura]

Could you live with these changes? in the s3.py S3Provider class

    def _get_bytes(
        self, path, start_byte: Optional[int] = None, end_byte: Optional[int] = None
    ):
        range_kwarg = {}
        if start_byte is not None and end_byte is not None:
            if start_byte == end_byte:
                return b""
            range_kwarg["Range"] = f"bytes={start_byte}-{end_byte - 1}"
        elif start_byte is not None:
            range_kwarg["Range"]  = f"bytes={start_byte}-"
        elif end_byte is not None:
            range_kwarg["Range"]  = f"bytes=0-{end_byte - 1}"
        resp = self.client.get_object(Bucket=self.bucket, Key=path, **range_kwarg)
        return resp["Body"].read()

instead of the old

    def _get_bytes(
        self, path, start_byte: Optional[int] = None, end_byte: Optional[int] = None
    ):
        if start_byte is not None and end_byte is not None:
            if start_byte == end_byte:
                return b""
            range = f"bytes={start_byte}-{end_byte - 1}"
        elif start_byte is not None:
            range = f"bytes={start_byte}-"
        elif end_byte is not None:
            range = f"bytes=0-{end_byte - 1}"
        else:
            range = ""
        resp = self.client.get_object(Bucket=self.bucket, Key=path, Range=range)
        return resp["Body"].read()

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud