s3-to-dc yaml return datetime is not json serializable

[pindge]

cmd : s3-to-dc --no-sign-request --skip-lineage s3://dea-public-data-dev/usgs_c2l1_ard_test/ga_ls_fc_3/**/*.odc-metadata.yaml ga_ls_fc_c2_3 returned error

[2021-11-05 22:02:19,175] {pod_launcher.py:149} INFO - Traceback (most recent call last):
[2021-11-05 22:02:19,175] {pod_launcher.py:149} INFO -   File "/usr/local/bin/s3-to-dc", line 8, in <module>
[2021-11-05 22:02:19,175] {pod_launcher.py:149} INFO -     sys.exit(cli())
[2021-11-05 22:02:19,176] {pod_launcher.py:149} INFO -   File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1128, in __call__
[2021-11-05 22:02:19,176] {pod_launcher.py:149} INFO -     return self.main(*args, **kwargs)
[2021-11-05 22:02:19,176] {pod_launcher.py:149} INFO -   File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1053, in main
[2021-11-05 22:02:19,176] {pod_launcher.py:149} INFO -     rv = self.invoke(ctx)
[2021-11-05 22:02:19,176] {pod_launcher.py:149} INFO -   File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1395, in invoke
[2021-11-05 22:02:19,176] {pod_launcher.py:149} INFO -     return ctx.invoke(self.callback, **ctx.params)
[2021-11-05 22:02:19,176] {pod_launcher.py:149} INFO -   File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 754, in invoke
[2021-11-05 22:02:19,176] {pod_launcher.py:149} INFO -     return __callback(*args, **kwargs)
[2021-11-05 22:02:19,176] {pod_launcher.py:149} INFO -   File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/s3_to_dc.py", line 129, in cli
[2021-11-05 22:02:19,177] {pod_launcher.py:149} INFO -     added, failed = dump_to_odc(
[2021-11-05 22:02:19,177] {pod_launcher.py:149} INFO -   File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/s3_to_dc.py", line 58, in dump_to_odc
[2021-11-05 22:02:19,177] {pod_launcher.py:149} INFO -     index_update_dataset(metadata, uri, dc, doc2ds, update, update_if_exists, allow_unsafe)
[2021-11-05 22:02:19,177] {pod_launcher.py:149} INFO -   File "/usr/local/lib/python3.8/dist-packages/odc/apps/dc_tools/utils.py", line 146, in index_update_dataset
[2021-11-05 22:02:19,177] {pod_launcher.py:149} INFO -     ds, err = doc2ds(metadata, uri)
[2021-11-05 22:02:19,177] {pod_launcher.py:149} INFO -   File "/usr/local/lib/python3.8/dist-packages/datacube/index/hl.py", line 281, in __call__
[2021-11-05 22:02:19,177] {pod_launcher.py:149} INFO -     dataset, err = self._ds_resolve(doc, uri)
[2021-11-05 22:02:19,177] {pod_launcher.py:149} INFO -   File "/usr/local/lib/python3.8/dist-packages/datacube/index/hl.py", line 142, in resolve_no_lineage
[2021-11-05 22:02:19,177] {pod_launcher.py:149} INFO -     product = match_product(doc)
[2021-11-05 22:02:19,177] {pod_launcher.py:149} INFO -   File "/usr/local/lib/python3.8/dist-packages/datacube/index/hl.py", line 63, in match
[2021-11-05 22:02:19,178] {pod_launcher.py:149} INFO -     % (json.dumps(relevant_doc, indent=4),
[2021-11-05 22:02:19,178] {pod_launcher.py:149} INFO -   File "/usr/lib/python3.8/json/__init__.py", line 234, in dumps
[2021-11-05 22:02:19,178] {pod_launcher.py:149} INFO -     return cls(
[2021-11-05 22:02:19,178] {pod_launcher.py:149} INFO -   File "/usr/lib/python3.8/json/encoder.py", line 201, in encode
[2021-11-05 22:02:19,178] {pod_launcher.py:149} INFO -     chunks = list(chunks)
[2021-11-05 22:02:19,178] {pod_launcher.py:149} INFO -   File "/usr/lib/python3.8/json/encoder.py", line 431, in _iterencode
[2021-11-05 22:02:19,178] {pod_launcher.py:149} INFO -     yield from _iterencode_dict(o, _current_indent_level)
[2021-11-05 22:02:19,178] {pod_launcher.py:149} INFO -   File "/usr/lib/python3.8/json/encoder.py", line 405, in _iterencode_dict
[2021-11-05 22:02:19,178] {pod_launcher.py:149} INFO -     yield from chunks
[2021-11-05 22:02:19,178] {pod_launcher.py:149} INFO -   File "/usr/lib/python3.8/json/encoder.py", line 405, in _iterencode_dict
[2021-11-05 22:02:19,179] {pod_launcher.py:149} INFO -     yield from chunks
[2021-11-05 22:02:19,179] {pod_launcher.py:149} INFO -   File "/usr/lib/python3.8/json/encoder.py", line 438, in _iterencode
[2021-11-05 22:02:19,179] {pod_launcher.py:149} INFO -     o = _default(o)
[2021-11-05 22:02:19,179] {pod_launcher.py:149} INFO -   File "/usr/lib/python3.8/json/encoder.py", line 179, in default
[2021-11-05 22:02:19,179] {pod_launcher.py:149} INFO -     raise TypeError(f'Object of type {o.__class__.__name__} '
[2021-11-05 22:02:19,179] {pod_launcher.py:149} INFO - TypeError: Object of type datetime is not JSON serializable

[woodcockr]

Just encountered this - the problem is in the error output using json.dumps. What is actually happening is a BadMatch error on the metadata but you can't see the error (and s3-to-dc will hang) because it can't finish raising the BadMatch exception...

    def single_product_matcher(rule):
        def match(doc):
            if matches(doc, rule):
                return rule.product
            relevant_doc = {k: v for k, v in doc.items() if k in rule.signature}
            raise BadMatch('Dataset metadata did not match product signature.'
                           '\nDataset definition:\n %s\n'
                           '\nProduct signature:\n %s\n'
                           % (json.dumps(relevant_doc, indent=4),
                              json.dumps(rule.signature, indent=4)))