[Bug]: TypeError: Object of type date is not JSON serializable (issue for RedisNoSqlTarget & ParquetSource)

[george0st]

MLRun Version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of MLRun CE.

Reproducible Example

import mlrun
import mlrun.feature_store as fstore
from mlrun.datastore.targets import RedisNoSqlTarget
from mlrun.datastore.sources import ParquetSource
from mlrun.data_types.data_types import ValueType


def redis_parquet(project_name):
    mlrun.set_env_from_file("mlrun-nonprod.env")
    project = mlrun.get_or_create_project(project_name, context='./', user_project=False)

    # use ParquetSource
    feature_name = "test"
    feature_set=create_featureset(feature_name)
    feature_set.set_targets(targets=[RedisNoSqlTarget(path="redis://localhost:6379")],
                            with_defaults=False)

    try:
        # it generates the issue
        feature_set.ingest(ParquetSource(name="tst2", path="./01-basic-party.parquet"),
                      return_df=False,
                      infer_options=mlrun.data_types.data_types.InferOptions.default())
    except Exception as ex:
        print(ex)

if __name__ == '__main__':
    redis_parquet(project)

Issue Description

See the exception (known solution see https://pynative.com/python-serialize-datetime-into-json/). Input file used in sample see attachment (only remove suffix '.txt').
01-basic-party.parquet.txt

Traceback (most recent call last):
  File "C:\Python\test\.venv\lib\site-packages\storey\table.py", line 526, in _safe_process_events
    raise ex
  File "C:\Python\test\.venv\lib\site-packages\storey\table.py", line 519, in _safe_process_events
    return await self._internal_persist_key(job.key, job.data, job.aggr_by_key, job.additional_data_persist)
  File "C:\Python\test\.venv\lib\site-packages\storey\table.py", line 161, in _internal_persist_key
    await self._storage._save_key(
  File "C:\Python\test\.venv\lib\site-packages\storey\redis_driver.py", line 367, in _save_key
    ) = self._build_feature_store_lua_update_script(
  File "C:\Python\test\.venv\lib\site-packages\storey\redis_driver.py", line 243, in _build_feature_store_lua_update_script
    expression_value = self._convert_python_obj_to_lua_value(value)
  File "C:\Python\test\.venv\lib\site-packages\storey\redis_driver.py", line 178, in _convert_python_obj_to_lua_value
    return json.dumps(value)
  File "C:\Users\jiri.steuer\AppData\Local\Programs\Python\Python39\lib\json\__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "C:\Users\jiri.steuer\AppData\Local\Programs\Python\Python39\lib\json\encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "C:\Users\jiri.steuer\AppData\Local\Programs\Python\Python39\lib\json\encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "C:\Users\jiri.steuer\AppData\Local\Programs\Python\Python39\lib\json\encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type date is not JSON serializable
> 2024-04-19 15:16:44,989 [error] Pushing error to error stream: Object of type date is not JSON serializable
Traceback (most recent call last):
  File "C:\Python\test\.venv\lib\site-packages\storey\flow.py", line 212, in _do_and_recover
    return await self._do(event)
  File "C:\Python\test\.venv\lib\site-packages\storey\targets.py", line 1154, in _do
    await self._table._persist(_PersistJob(key, data_to_persist, self._handle_completed, event))
  File "C:\Python\test\.venv\lib\site-packages\storey\table.py", line 495, in _persist
    await self._worker_awaitable
  File "C:\Python\test\.venv\lib\site-packages\storey\table.py", line 455, in _persist_worker
    raise ex
  File "C:\Python\test\.venv\lib\site-packages\storey\table.py", line 430, in _persist_worker
    completed = await task[1]
  File "C:\Python\test\.venv\lib\site-packages\storey\table.py", line 526, in _safe_process_events
    raise ex
  File "C:\Python\test\.venv\lib\site-packages\storey\table.py", line 519, in _safe_process_events
    return await self._internal_persist_key(job.key, job.data, job.aggr_by_key, job.additional_data_persist)
  File "C:\Python\test\.venv\lib\site-packages\storey\table.py", line 161, in _internal_persist_key
    await self._storage._save_key(
  File "C:\Python\test\.venv\lib\site-packages\storey\redis_driver.py", line 367, in _save_key
    ) = self._build_feature_store_lua_update_script(
  File "C:\Python\test\.venv\lib\site-packages\storey\redis_driver.py", line 243, in _build_feature_store_lua_update_script
    expression_value = self._convert_python_obj_to_lua_value(value)
  File "C:\Python\test\.venv\lib\site-packages\storey\redis_driver.py", line 178, in _convert_python_obj_to_lua_value
    return json.dumps(value)
  File "C:\Users\jiri.steuer\AppData\Local\Programs\Python\Python39\lib\json\__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "C:\Users\jiri.steuer\AppData\Local\Programs\Python\Python39\lib\json\encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "C:\Users\jiri.steuer\AppData\Local\Programs\Python\Python39\lib\json\encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "C:\Users\jiri.steuer\AppData\Local\Programs\Python\Python39\lib\json\encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type date is not JSON serializable

Expected Behavior

It will be possible to use date type in Redis target

Installation OS

Windows

Installation Method

Docker

Python Version

3.9.10

MLRun Version

1.6.2

Additional Information

No response

[george0st]

[xsqian]

@george0st Thank you for the comprehensive tests on the redis target. We will be looking into your test cases.

[george0st]

Hi @xsqian , are you able to reproduce the issue?