Filtering dask dataframe by a datetime object

[frbelotto]

Hello guys,
I am starting to convert some of my codes from pandas to dask because the dataframe become bigger than my memory. One of my firt issues is to handle how to filter a dask dataframe by a datetime/timestamp value.

My question may seems dumb but I coulnd´t find any solution yet.

Creating a sample padas dataframe and converting to dask daframe

import pandas as pd
from datetime import datetime, date
import dask.dataframe as ddf
data = pd.DataFrame({
    'status' :  ['pending', 'pending','pending', 'canceled','canceled','canceled', 'confirmed', 'confirmed','confirmed'],
    'product' : ['afiliates', 'pre-paid', 'giftcard','afiliates', 'pre-paid', 'giftcard','afiliates', 'pre-paid', 'giftcard'],
    'brand' : ['brand_1', 'brand_2', 'brand_3','brand_1', 'brand_2', 'brand_3','brand_1', 'brand_2', 'brand_3'],
    'date' : [date(2022,1,1),date(2023,6,1),date(2020,1,12),date(2025,1,1),date(2024,1,1),date(2023,1,1),date(2021,1,1),date(2022,1,1),date(2022,1,1)],
    'datetime' : [datetime(2022,1,1),datetime(2023,6,1),datetime(2020,1,12),datetime(2025,1,1),datetime(2024,1,1),datetime(2023,1,1),datetime(2021,1,1),datetime(2022,1,1),datetime(2022,1,1)],
    'gmv' : [100,100,100,100,100,100,100,100,100]})
data['date'] = data.date.astype('datetime64[ns]')
df = ddf.from_pandas(data,npartitions=1)

Now I try to filter it.
df.loc[df['date'].dt.date >= date(2023,6,1)].compute()

But it fail

ArrowNotImplementedError                  Traceback (most recent call last)
File [c:\Users\fabio\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\core\arrays\arrow\array.py:630](file:///C:/Users/fabio/AppData/Local/Programs/Python/Python311/Lib/site-packages/pandas/core/arrays/arrow/array.py:630), in ArrowExtensionArray._cmp_method(self, other, op)
    629 try:
--> 630     result = pc_func(self._pa_array, self._box_pa(other))
    631 except (pa.lib.ArrowNotImplementedError, pa.lib.ArrowInvalid):

File [c:\Users\fabio\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyarrow\compute.py:245](file:///C:/Users/fabio/AppData/Local/Programs/Python/Python311/Lib/site-packages/pyarrow/compute.py:245), in _make_generic_wrapper.<locals>.wrapper(memory_pool, *args)
    244     return Expression._call(func_name, list(args))
--> 245 return func.call(args, None, memory_pool)

File [c:\Users\fabio\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyarrow\_compute.pyx:367](file:///C:/Users/fabio/AppData/Local/Programs/Python/Python311/Lib/site-packages/pyarrow/_compute.pyx:367), in pyarrow._compute.Function.call()

File [c:\Users\fabio\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyarrow\error.pxi:144](file:///C:/Users/fabio/AppData/Local/Programs/Python/Python311/Lib/site-packages/pyarrow/error.pxi:144), in pyarrow.lib.pyarrow_internal_check_status()

File [c:\Users\fabio\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyarrow\error.pxi:121](file:///C:/Users/fabio/AppData/Local/Programs/Python/Python311/Lib/site-packages/pyarrow/error.pxi:121), in pyarrow.lib.check_status()

ArrowNotImplementedError: Function 'greater_equal' has no kernel matching input types (string, date32[day])

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
File [c:\Users\fabio\AppData\Local\Programs\Python\Python311\Lib\site-packages\dask\dataframe\utils.py:194](file:///C:/Users/fabio/AppData/Local/Programs/Python/Python311/Lib/site-packages/dask/dataframe/utils.py:194), in raise_on_meta_error(funcname, udf)
    193 try:
--> 194     yield
...
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\fabio\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\core\arrays\arrow\array.py", line 636, in _cmp_method
    result[valid] = op(np.array(self)[valid], other)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I does work if I try to filter by the datetime values.

df.loc[df['datetime'] >= datetime(2023,6,1)].compute()