aws iot describe-ca-certificate not able to display expiration date for ca certificate

[vesajaaskelainen]

Describe the bug

I registered IEEE Secure Device Identity based CA certificate in the AWS IoT Core.

In AWS IoT Console Web UI it is shown OKish.

Expires
January 01, 10000, 01:59:59 (UTC+02:00)

Now I try to see certificate details from CLI:

$ aws --version
aws-cli/2.11.16 Python/3.11.3 Linux/5.19.0-40-generic exe/x86_64.ubuntu.22 prompt/off

$ aws iot describe-ca-certificate --profile my-profile --certificate-id $CA_CERTIFICATE_ID

date value out of range

My current time zone is Europe/Helsinki (GMT+02:00)

Expected Behavior

Command aws iot describe-ca-certificate would display details correctly.

Current Behavior

Same command with additional --debug:

$ aws iot describe-ca-certificate --profile my-profile --certificate-id $CA_CERTIFICATE_ID --debug
...
2023-04-30 18:04:02,468 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"certificateDescription":{"autoRegistrationStatus":"ENABLE","certificateArn":"arn:aws:iot:...","certificateId":"...","certificateMode":"SNI_ONLY","certificatePem":"...","creationDate":1.68286317354E9,"customerVersion":3,"generationId":"...","lastModifiedDate":1.682863774742E9,"ownedBy":"...","status":"ACTIVE","validity":{"notAfter":2.53402300799E11,"notBefore":1.682862388E9}},"registrationConfig":null}'
2023-04-30 18:04:02,469 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "awscli/clidriver.py", line 460, in main
  File "awscli/clidriver.py", line 595, in __call__
  File "awscli/clidriver.py", line 798, in __call__
  File "awscli/clidriver.py", line 929, in invoke
  File "awscli/clidriver.py", line 941, in _make_client_call
  File "awscli/botocore/client.py", line 341, in _api_call
  File "awscli/botocore/client.py", line 683, in _make_api_call
  File "awscli/botocore/client.py", line 703, in _make_request
  File "awscli/botocore/endpoint.py", line 101, in make_request
  File "awscli/botocore/endpoint.py", line 133, in _send_request
  File "awscli/botocore/endpoint.py", line 165, in _get_response
  File "awscli/botocore/endpoint.py", line 219, in _do_get_response
  File "awscli/botocore/parsers.py", line 248, in parse
  File "awscli/botocore/parsers.py", line 864, in _do_parse
  File "awscli/botocore/parsers.py", line 873, in _add_modeled_parse
  File "awscli/botocore/parsers.py", line 914, in _parse_payload
  File "awscli/botocore/parsers.py", line 322, in _parse_shape
  File "awscli/botocore/parsers.py", line 651, in _handle_structure
  File "awscli/botocore/parsers.py", line 322, in _parse_shape
  File "awscli/botocore/parsers.py", line 651, in _handle_structure
  File "awscli/botocore/parsers.py", line 322, in _parse_shape
  File "awscli/botocore/parsers.py", line 651, in _handle_structure
  File "awscli/botocore/parsers.py", line 322, in _parse_shape
  File "awscli/botocore/parsers.py", line 670, in _handle_timestamp
  File "awscli/customizations/timestampformat.py", line 44, in iso_format
  File "awscli/botocore/utils.py", line 749, in parse_timestamp
  File "awscli/botocore/utils.py", line 720, in _parse_timestamp_with_tzinfo
  File "dateutil/tz/_common.py", line 140, in fromutc
  File "dateutil/tz/_common.py", line 254, in fromutc
  File "dateutil/tz/_common.py", line 231, in _fromutc
OverflowError: date value out of range

date value out of range

Reproduction Steps

Invoke AWS CLI method:

1. Create Root CA with expiration date 9999-12-31T23:59:59Z.

in example with the following scripts:

$ git clone https://github.com/dachaac/device-certificate-scripts.git
$ cd device-certificate-scripts/
$ ( cd manufacturer-ca && ./create-manufacturer-ca.sh )
$ openssl x509 -noout -text -in manufacturer-ca/manufacturer-root-ca/manufacturer-root-ca.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: C = FI, O = Manufacturer, CN = Products Root CA
        Validity
            Not Before: Apr 30 14:50:20 2023 GMT
            Not After : Dec 31 23:59:59 9999 GMT
        Subject: C = FI, O = Manufacturer, CN = Products Root CA
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (384 bit)
                pub:
                    04:63:0c:e2:a0:ca:42:df:7b:38:ab:88:0a:94:e4:
                    3e:d2:0b:7d:8b:b2:8b:59:20:75:b3:99:69:a2:16:
                    dc:dc:b1:94:ef:59:73:b6:ac:ef:f2:48:d0:3e:75:
                    30:47:54:a0:fa:95:ca:3a:b0:82:d1:14:9f:35:61:
                    0e:b0:ed:d1:ba:58:bd:e5:b1:52:a3:47:82:04:a3:
                    a1:60:96:1c:62:63:da:4b:d3:09:bf:ba:a1:c8:9c:
                    f1:a1:a3:3e:31:01:45
                ASN1 OID: secp384r1
                NIST CURVE: P-384
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                14:BD:82:9B:D6:BC:BF:11:B8:FE:AB:1D:9B:D1:14:7E:3C:2C:EC:99
            X509v3 Authority Key Identifier: 
                14:BD:82:9B:D6:BC:BF:11:B8:FE:AB:1D:9B:D1:14:7E:3C:2C:EC:99
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
    Signature Algorithm: ecdsa-with-SHA384
    Signature Value:
        30:64:02:30:4f:29:f0:69:51:02:11:3c:19:f3:2e:40:da:ba:
        da:ce:0e:08:e1:30:9f:8e:76:35:e9:95:74:98:ff:d6:1c:18:
        e7:17:ff:d7:1c:64:0f:d1:3e:ee:0f:34:a3:44:10:da:02:30:
        44:db:9b:71:15:ce:67:85:51:8a:21:56:8f:f6:ae:16:f4:27:
        57:4d:55:7a:7b:e6:09:0b:91:a5:2c:79:6a:83:99:fa:e2:ac:
        57:90:fb:9b:74:92:ff:70:dc:12:46:b2

2. Register the CA certificate into AWS IoT core in Multi-account mode (SNI_ONLY).
3. Get CA certificate ID for the registered CA certificate
4. Use AWS CLI to observe details of CA certificate:

$ aws iot describe-ca-certificate --profile my-profile --certificate-id $CA_CERTIFICATE_ID

---

Alternatively just invoke the parse_timestamp function with 2.53402300799E11:

$ git clone https://github.com/boto/botocore.git
Cloning into 'botocore'...
remote: Enumerating objects: 93439, done.
remote: Counting objects: 100% (700/700), done.
remote: Compressing objects: 100% (280/280), done.
remote: Total 93439 (delta 354), reused 645 (delta 325), pack-reused 92739
Receiving objects: 100% (93439/93439), 208.03 MiB | 11.94 MiB/s, done.
Resolving deltas: 100% (58867/58867), done.
Updating files: 100% (4470/4470), done.
$ cd botocore/
botocore (develop)$ python3 -m venv venv
botocore (develop)$ . venv/bin/activate
(venv) botocore (develop)$ pip install -r requirements.txt
...
(venv) botocore (develop)$ pip install -e .
....
Successfully built botocore
Installing collected packages: urllib3, python-dateutil, jmespath, botocore
Successfully installed botocore-1.29.123 jmespath-1.0.1 python-dateutil-2.8.2 urllib3-1.26.15
(venv) botocore (develop)$ python3
Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from botocore.utils import parse_timestamp
>>> notAfter=2.53402300799E11
>>> parse_timestamp(notAfter)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/boto-test/botocore/botocore/utils.py", line 941, in parse_timestamp
    return _parse_timestamp_with_tzinfo(value, tzinfo)
  File "/tmp/boto-test/botocore/botocore/utils.py", line 912, in _parse_timestamp_with_tzinfo
    return datetime.datetime.fromtimestamp(value, tzinfo())
  File "/tmp/boto-test/botocore/venv/lib/python3.10/site-packages/dateutil/tz/_common.py", line 144, in fromutc
    return f(self, dt)
  File "/tmp/boto-test/botocore/venv/lib/python3.10/site-packages/dateutil/tz/_common.py", line 258, in fromutc
    dt_wall = self._fromutc(dt)
  File "/tmp/boto-test/botocore/venv/lib/python3.10/site-packages/dateutil/tz/_common.py", line 235, in _fromutc
    dt += delta
OverflowError: date value out of range

Possible Solution

Treat the time as UTC always:

>>> notAfter=2.53402300799E11
>>> notAfter
253402300799.0
>>> datetime.datetime.fromtimestamp(notAfter)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: year 10000 is out of range
>>> datetime.datetime.fromtimestamp(notAfter, datetime.timezone.utc)
datetime.datetime(9999, 12, 31, 23, 59, 59, tzinfo=datetime.timezone.utc)
>>> datetime.datetime.utcfromtimestamp(notAfter)
datetime.datetime(9999, 12, 31, 23, 59, 59)
>>> datetime.datetime.fromtimestamp(notAfter, datetime.timezone.utc).isoformat()
'9999-12-31T23:59:59+00:00'

and then trying it out:

(venv) vej@yocto-vm:/tmp/boto-test/botocore (develop)$ git diff
diff --git a/botocore/compat.py b/botocore/compat.py
index c9d5821ce..cf123bcfe 100644
--- a/botocore/compat.py
+++ b/botocore/compat.py
@@ -28,7 +28,7 @@ from math import floor
 
 from botocore.vendored import six
 from botocore.exceptions import MD5UnavailableError
-from dateutil.tz import tzlocal
+from dateutil.tz import tzlocal, tzutc
 from urllib3 import exceptions
 
 logger = logging.getLogger(__name__)
@@ -285,7 +285,7 @@ def get_tzinfo_options():
 
         return (tzlocal, tzwinlocal)
     else:
-        return (tzlocal,)
+        return (tzutc, tzlocal)
 
 
 # Detect if CRT is available for use
diff --git a/botocore/utils.py b/botocore/utils.py
index fbc4963cc..8304087f2 100644
--- a/botocore/utils.py
+++ b/botocore/utils.py
@@ -939,6 +939,8 @@ def parse_timestamp(value):
     for tzinfo in get_tzinfo_options():
         try:
             return _parse_timestamp_with_tzinfo(value, tzinfo)
+        except (ValueError, OverflowError):
+            pass
         except OSError as e:
             logger.debug(
                 'Unable to parse timestamp with "%s" timezone info.',

(venv) botocore (develop)$ python3
Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from botocore.utils import parse_timestamp
>>> notAfter=2.53402300799E11
>>> parse_timestamp(notAfter)
datetime.datetime(9999, 12, 31, 23, 59, 59, tzinfo=tzutc())

To get consistent output I would probably just always use tzutc() for parsing the date time and if for some reason this does not work then use local timezone.

Testing the change:

(venv) botocore (develop)$ pip install awscliv2
...
(venv) botocore (develop)$ ./venv/bin/awsv2 iot describe-ca-certificate --profile my-profile --certificate-id $CA_CERTIFICATE_ID
{
    "certificateDescription": {
        "certificateArn": "arn:aws:iot:...",
        "certificateId": "...",
        "status": "ACTIVE",
        "certificatePem": "...",
        "ownedBy": "...",
        "creationDate": "2023-04-30T13:59:33.540000+00:00",
        "autoRegistrationStatus": "ENABLE",
        "lastModifiedDate": "2023-04-30T14:09:34.742000+00:00",
        "customerVersion": 3,
        "generationId": "...",
        "validity": {
            "notBefore": "2023-04-30T13:46:28+00:00",
            "notAfter": "9999-12-31T23:59:59+00:00"
        },
        "certificateMode": "SNI_ONLY"
    }
}

Personally I would prefer to have timestamp in ISO format eg. 9999-12-31T23:59:59Z but also that +00:00 form is understandable.

Additional Information/Context

No response

SDK version used

1.29.123

Environment details (OS name and version, etc.)

Ubuntu 22.04

[vesajaaskelainen]

Same problem is also present in aws iot describe-certificate and the proposed change also fixes that.

[aBurmeseDev]

Hi @vesajaaskelainen - thanks for reaching out.

I've attempted a couple times to reproduce this but was unable to. Then I continue to look into the Overflow Error you're seeing and it seems to be a generic Python lib error rather than botocore/AWS CLI.
Also I noticed that you installed awscliv2 via pip which we do not support. Please use the installation methods mentioned in this docs and retry the command?

[vesajaaskelainen]

Hi @vesajaaskelainen - thanks for reaching out.

I've attempted a couple times to reproduce this but was unable to. Then I continue to look into the Overflow Error you're seeing and it seems to be a generic Python lib error rather than botocore/AWS CLI. Also I noticed that you installed awscliv2 via pip which we do not support. Please use the installation methods mentioned in this docs and retry the command?

Please set you time zone somewhere that is positive like +02:00 then the python's time sanity checker goes haywire.

I installed awscliv2 with as instructed but with that you cannot debug it as you encapsulate the libraries. The problem description command was executed with the pristine release installed with installation instructions.

Only way to debug the problem was to install the components in python's virtual environment.

Here is example session when playing out with time zone from command line (with release aws cli):

$ aws --version
aws-cli/2.11.16 Python/3.11.3 Linux/5.19.0-41-generic exe/x86_64.ubuntu.22 prompt/off

$ aws iot describe-ca-certificate --profile my-profile --certificate-id xxxxxx

date value out of range
$ TZ='America/New_York' aws iot describe-ca-certificate --profile my-profile --certificate-id xxxxxx
{
    "certificateDescription": {....
    }
}
$ TZ='Europe/Helsinki' aws iot describe-ca-certificate --profile my-profile --certificate-id xxxxxx
date value out of range