On some platforms, parse_timestamp throws RuntimeError: Unable to calculate correct timezone offset for "-62135596800"

[ilg]

Describe the bug

Similar to #3642, but using get_object on an S3 client instance and passing IfModifiedSince=datetime.min. I'm running inside a docker container derived from python:3.11.9 with boto3==1.34.100.

When I run this container locally, I do not get the runtime error.

# uname -a
Linux dev 5.10.0-22-amd64 #1 SMP Debian 5.10.178-3 (2023-04-22) x86_64 GNU/Linux

When I run on the same container on AWS EB + EC2, I get the runtime error.

# uname -a
Linux ip-10-0-1-32.us-west-2.compute.internal 4.14.301-224.520.amzn2.x86_64 #1 SMP Fri Dec 9 09:57:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Both systems seem to have their local timezone set to UTC.

(It seems quite possible that this is ultimately a programmer error on my part for passing datetime.min, but it had been working on EB+EC2 previously—possibly broken by updating to a newer OS image—and works locally, so maybe there's some issue that could be addressed through boto3 or maybe this will just be a useful note for someone else encountering the same issue in the future.)

Expected Behavior

I'd expect the behavior of boto3 to be the same on both platforms and ideally to not raise the RuntimeError.

Current Behavior

Only when running the container on the EB+EC2 instance, not when running the container locally:

  File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 565, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 974, in _make_api_call
    request_dict = self._convert_to_request_dict(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/client.py", line 1048, in _convert_to_request_dict
    request_dict = self._serializer.serialize_to_request(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/validate.py", line 382, in serialize_to_request
    return self._serializer.serialize_to_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/serialize.py", line 481, in serialize_to_request
    self._partition_parameters(
  File "/usr/local/lib/python3.11/site-packages/botocore/serialize.py", line 624, in _partition_parameters
    value = self._convert_header_value(shape, param_value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/serialize.py", line 655, in _convert_header_value
    return self._convert_timestamp_to_str(timestamp, timestamp_format)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/serialize.py", line 155, in _convert_timestamp_to_str
    datetime_obj = parse_to_aware_datetime(value)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/utils.py", line 1038, in parse_to_aware_datetime
    datetime_obj = parse_timestamp(value)
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/botocore/utils.py", line 1000, in parse_timestamp
    raise RuntimeError(
RuntimeError: Unable to calculate correct timezone offset for "-62135596800"

Reproduction Steps

I believe that making an S3 get_object call with IfModifiedSince=datetime.min (with from datetime import datetime) should reproduce the issue, but only on some systems.

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.34.100

Environment details (OS name and version, etc.)

Linux ip-10-0-1-32.us-west-2.compute.internal 4.14.301-224.520.amzn2.x86_64 #1 SMP Fri Dec 9 09:57:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

[nateprewitt]

Hi @ilg, it looks like you might be hitting this issue in Python 3.8+. Due to a change in PEP 495, datetime.min may be producing a value with year 0 because of your systems timezone offset.

You can see this same issue reproduced with this line of just the Python datetime library:

time = datetime.fromtimestamp(-62135596800)

or

datetime.min.timestamp()

The problem is Python's moving this time out of "year 1" into "year 0" when subtracting the timezone offset and Python doesn't acknowledge a year 0 as being valid input. I'm not sure if we can realistically adjust this in our code base until it's fixed upstream. As a short term workaround, you could potentially add a timedelta to datetime.min >= 24 hours. This would look something like this:

IfModifiedSince=(datetime.min + timedelta(days=1))

While not ideal, it works around the current limitation in the standard library and shouldn't interfere with the desired behavior unless you have some very old files stored in S3.

[ilg]

@nateprewitt Thanks for the detailed response. The Python issue link in particular helped me get my head around what's going on.

I still don't understand why I'm not running into the issue on my local dev box, since datetime.min.timestamp() throws the "year 0" error there, too, but it's clear this isn't a boto3 issue.

Adding a day is basically the workaround I'd put into place, but from the Python issue, it seems like datetime.min.replace(tzinfo=timezone.utc) might be a better fix.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.