RecursionError: Maximum recursion depth exceeded

[MustaphaU]

Describe the bug

I need help with this recursion error maximum recursion depth exceeded from boto3. This occurs when I initialize an s3 client in my inference script to allow me read s3 objects. Your insights will be deeply appreciated! Similar issue was posted on stackoverflow 2 months ago here: https://stackoverflow.com/questions/77786275/aws-sagemaker-endpoint-maximum-recursion-depth-exceeded-error-when-calling-boto

Here is the relevant code block responsible for the error:

def get_video_bytes_from_s3(bucket_name, key):
    s3_client = boto3.client('s3')
    try:
        video_object = s3_client.get_object(Bucket= bucket_name, Key=key)
        video_bytes = video_object['Body'].read()
        return video_bytes
    except Exception as e:
        print(f"Failed to fetch video from S3: {e}")

Expected Behavior

The s3 client created to enable access to the s3 objects

Current Behavior

Here is the full error log:

Traceback (most recent call last):
  File "/sagemaker/python_service.py", line 423, in _handle_invocation_post
    res.body, res.content_type = handlers(data, context)
  File "/opt/ml/model/code/inference.py", line 156, in handler
    video_bytes = get_video_bytes_from_s3(key)
  File "/opt/ml/model/code/inference.py", line 16, in get_video_bytes_from_s3
    s3_client = boto3.client('s3')
  File "/usr/local/lib/python3.10/site-packages/boto3/__init__.py", line 92, in client
    return _get_default_session().client(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/boto3/session.py", line 299, in client
    return self._session.create_client(
  File "/usr/local/lib/python3.10/site-packages/botocore/session.py", line 997, in create_client
    client = client_creator.create_client(
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 159, in create_client
    client_args = self._get_client_args(
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 490, in _get_client_args
    return args_creator.get_client_args(
  File "/usr/local/lib/python3.10/site-packages/botocore/args.py", line 137, in get_client_args
    endpoint = endpoint_creator.create_endpoint(
  File "/usr/local/lib/python3.10/site-packages/botocore/endpoint.py", line 409, in create_endpoint
    http_session = http_session_cls(
  File "/usr/local/lib/python3.10/site-packages/botocore/httpsession.py", line 323, in __init__
    self._manager = PoolManager(**self._get_pool_manager_kwargs())
  File "/usr/local/lib/python3.10/site-packages/botocore/httpsession.py", line 341, in _get_pool_manager_kwargs
    'ssl_context': self._get_ssl_context(),
  File "/usr/local/lib/python3.10/site-packages/botocore/httpsession.py", line 350, in _get_ssl_context
    return create_urllib3_context()
  File "/usr/local/lib/python3.10/site-packages/botocore/httpsession.py", line 139, in create_urllib3_context
    context.options |= options
  File "/usr/local/lib/python3.10/ssl.py", line 620, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/usr/local/lib/python3.10/ssl.py", line 620, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/usr/local/lib/python3.10/ssl.py", line 620, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  [Previous line repeated 479 more times]

Reproduction Steps

simply initializing an s3 client within an inference script like so:

s3_client = boto3.client('s3')

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.34.55

Environment details (OS name and version, etc.)

Sagemaker endpoint for Tensorflow serving

[avishwanathan88]

I am facing this maximum recursion depth issue suddenly as well when trying to check if the object exists in the s3 bucket using

s3_client.head_object(Bucket=bucket_name, Key=key)

It used to work before but not sure if something changed suddenly.
the s3 client is created using

boto3.client(service_name='s3',
                                 use_ssl=False,
                                 region_name=region,
                                 endpoint_url=endpoint_url,
                                 aws_access_key_id=key_id,
                                 aws_secret_access_key=access_key,
                                 config=Config(
                                    s3={'addressing_style': 'path'},
                                    signature_version='s3v4'))

[RyanFitzSimmonsAK]

Hi @MustaphaU, thanks for reaching out. If you limit the script to only be initializing a client (no actual operations), do you still have this behavior? In other words, what is the minimum reproducible code snippet that produces this recursion depth error? Thanks!

[MustaphaU]

Hi @MustaphaU, thanks for reaching out. If you limit the script to only be initializing a client (no actual operations), do you still have this behavior? In other words, what is the minimum reproducible code snippet that produces this recursion depth error? Thanks!

Hi @RyanFitzSimmonsAK
Just initializing the s3 client in my inference script like below is enough to reproduce the error

s3_client = boto3.client('s3')

Thank you.

Edit: The error persists. Apologies for the back and forth. Yes, s3_client=boto3.client('s3') should produce the error. I just tested now and got the error.

[MustaphaU]

@RyanFitzSimmonsAK

Please see the attached below from cloudwatch logs:

Also, see the relevant part of the inference script:

You would observe from the log that execution failed at the point of initializing the s3 client.

Thanks.

[pmaoui]

I also have this bug. One message I got while performing some tests to fix that:

Hope it could help.

As a workaround I used the awscli already present in the container:

import subprocess
subprocess.run(["/usr/local/bin/aws", "s3", "cp", "s3://bucket/file, "/local/file"], check=True)

[shresthapradip]

I am also getting the same error. It was working fine a few weeks ago.

[shresthapradip]

So,

s3.Bucket(settings.S3_BUCKET).put_object(Key=key, Body=file_data)

works, but the following code doesn't. This is a nightmare :)

res = self.s3.put_object(Bucket=settings.S3_BUCKET,
                      Key=key,
                      Body=file_data)

[shresthapradip]

Probably same thing goes to get_object