Max retries and connection timeout arguments

[lisovskey]

Sometimes I have connection issues when uploading data via load_and_copy, for example:

botocore.exceptions.ConnectionClosedError:
Connection was closed before we received a valid response from endpoint URL: "https://MY-S3-FILE".

I found a proposition to use bigger connection_timeout in this thread. Also I'm interested in using retries option (it's zero by default I guess). Both are attributes of botocore.config.Config.

Can I somehow pass Config to Redshift object?

[fdosani]

Sorry was out of the office for the holidays. Taking a look at this now.

[fdosani]

@lisovskey I'm wondering if in the Redshift's __init__ you can use the profile argument to set it via profile settings: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html?highlight=retries#using-a-configuration-file

This should flow through to the S3 object and set it for its session: https://github.com/capitalone/locopy/blob/develop/locopy/s3.py#L99

Maybe give that a try and see if it gets things working for you?

[lisovskey]

@fdosani thanks, environment variable AWS_MAX_ATTEMPTS worked for me