Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

get_description date_range truncates nanoseconds #1460

Open
poodlewars opened this issue Mar 25, 2024 · 0 comments · May be fixed by #1461
Open

get_description date_range truncates nanoseconds #1460

poodlewars opened this issue Mar 25, 2024 · 0 comments · May be fixed by #1461
Assignees
Labels
Bug fixing drive Q1 2024 bug Something isn't working

Comments

@poodlewars
Copy link
Collaborator

Describe the bug

Write a dataframe with a nanosecond precision index.

Call lib.get_description(sym).date_range.

You receive a tuple of datetime.datetime which is only microsecond precision - nanoseconds are truncated off.

This is because we are calling:

def _from_tz_timestamp(ts, tz):
    # type: (int, Optional[str])->(datetime.datetime)
    return pd.Timestamp(ts).tz_localize(tz).to_pydatetime(warn=False)

Steps/Code to Reproduce

lib can be any ArcticDB Library.

In [24]:     sym = "test_filter_datetime_nanoseconds"
    ...: 
    ...:     # Dataframe has three rows and a single column containing timestamps 1 nanosecond apart
    ...:     timestamp_1 = pd.Timestamp("2023-03-15 10:30:00")
    ...:     timestamp_0 = timestamp_1 - pd.Timedelta(1, unit="ns")
    ...:     timestamp_2 = timestamp_1 + pd.Timedelta(1, unit="ns")
    ...:     df = pd.DataFrame(index=[timestamp_0, timestamp_1, timestamp_2], data={"a": [1,2,3]})

In [25]: lib.write("sym", df)
Out[25]: VersionedItem(symbol='sym', library='one', data=n/a, version=1, metadata=None, host='LMDB(path=/home/alex/tmp/tst)', timestamp=1711387582004575048)

In [26]: lib.get_description("sym").date_range
Out[26]: 
(datetime.datetime(2023, 3, 15, 10, 29, 59, 999999, tzinfo=datetime.timezone.utc),
 datetime.datetime(2023, 3, 15, 10, 30, tzinfo=datetime.timezone.utc))

Expected Results

Date range should not drop precision.

The difficulty here will be that changing the return type is a breaking API change.

OS, Python Version and ArcticDB Version

Backend storage used

Any

Additional Context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug fixing drive Q1 2024 bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants