Porting more related tidyverse packages

[GitHunter0]

I believe your great initiative could be stronger if you join forces with siuba, which has the same philosophy, instead of building a new project from the ground up. What do you think about that?

[pwwang]

Thanks for the suggestion.

I talked to @machow a couple of weeks ago. These two packages are focusing on different aspects. suiba is more powerful with SQL-generating support; while datar is more stringent on following the original dplyr and family's API design. And datar is now pretty complete on porting all of the APIs from dplyr, tidyr, and tibble.

I have kept an eye on suiba's development, too. You may check out some of my comments on issues of suiba.

[GitHunter0]

I get it, thank you for the clarification. I just wish users could use the same package for both purposes.

By the way, what other packages do you plan to port? forcats and purrr would be great additions

[pwwang]

Great mentions about forcats, purrr and even dbplyr, a dplyr backend for databases, which functions similarly as suiba does. And maybe other related packages.

Maybe I would make it independent of datar, or optional to datar (i.e. to be installed via pip install datar[dbplyr]). Since the backend support (https://github.com/pwwang/pipda) for the piping syntax is independent, they don't have to be bundled with datar.

I need to block out a period on my calendar to implement them, but obviously not in a near future.

Of course, contributions and PRs are welcome for those ideas.

[GitHunter0]

Yes, dbplyr would be fantastic too. Nice to know you are open to it. Thank you for the feedback

[pwwang]

You are absolutely welcome!

Leaving this open for future discussions and people who may be interested in this topic.

[pwwang]

#51

[willros]

Hmm, I have a problem with the import:

ImportError: cannot import name 'append' from 'datar.base' (/opt/conda/lib/python3.8/site-packages/datar/base/init.py)

Do you know what may be causing this?

[pwwang]

@willros Have you done a full upgrade using: pip install -U datar?

[willros]

Yes, I have. Thanks for the fast update!!!!

[willros]

Works when I just tried it fast in colabs. Did not work in my current docker container. Will try to restart that and try again.

[pwwang]

The reference map is also added:

https://pwwang.github.io/datar/reference-maps/forcats/

[GitHunter0]

I have not been able to import forcats module, did you succeeded @willros ?

import datar.forcats returns

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
d:\...\my_file.py in 
----> 6 import datar.forcats

~\miniconda3\envs\py38\lib\site-packages\datar\forcats\__init__.py in <module>
      1 """Port of forcats"""
----> 2 from .lvl_order import (
      3     as_factor,
      4     fct_relevel,
      5     fct_inorder,

~\miniconda3\envs\py38\lib\site-packages\datar\forcats\lvl_order.py in <module>
     32 )
     33 from ..core.utils import get_option, logger
---> 34 from .lvls import lvls_reorder, lvls_seq
     35 from .utils import check_factor
     36 

~\miniconda3\envs\py38\lib\site-packages\datar\forcats\lvls.py in <module>
     25 )
     26 
---> 27 from ..dplyr import recode_factor
     28 
     29 from .utils import check_factor

~\miniconda3\envs\py38\lib\site-packages\datar\dplyr\__init__.py in <module>
      4 from .across import across, c_across, if_all, if_any
      5 from .arrange import arrange
----> 6 from .bind import bind_cols, bind_rows
      7 from .context import (
      8     cur_column,

~\miniconda3\envs\py38\lib\site-packages\datar\dplyr\bind.py in <module>
     15 from ..core.names import repair_names
     16 from ..core.grouped import DataFrameGroupBy
---> 17 from ..tibble import tibble
     18 
     19 

~\miniconda3\envs\py38\lib\site-packages\datar\tibble\__init__.py in <module>
      1 """APIs for R-tibble"""
----> 2 from .tibble import tibble, tibble_row, tribble, zibble
      3 from .verbs import (
      4     enframe,
      5     deframe,

~\miniconda3\envs\py38\lib\site-packages\datar\tibble\tibble.py in <module>
      4 
      5 from pandas import DataFrame
----> 6 from varname import argname, varname, VarnameException
      7 
      8 import pipda

ImportError: cannot import name 'VarnameException' from 'varname' (C:\Users\user_name\miniconda3\envs\py38\lib\site-packages\varname\__init__.py)

Basic system info:

windows 10 pro
python 3.8.8
miniconda 3
datar 0.5.0

[GitHunter0]

Thanks @pwwang , that solved the problem.

I'm just having an issue with multi-line execution of datar code in VScode.

If a run this line by line, it works smoothly.

from datar.all import (f, mutate, tibble, fct_infreq, fct_inorder, pull)
df = tibble(var=['b','b','b','c','a','a'])
df = df >> mutate(fct_var = f['var'].astype("category"))

However, if I select all the lines and execute them, it returns:

C:\Users\user_name\miniconda3\envs\py38\lib\site-packages\pipda\utils.py:161: UserWarning: Failed to fetch the node calling the function, call it with the original function.

>>> df = df >> mutate(fct_var = f['var'].astype("category"))

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\user_name\miniconda3\envs\py38\lib\site-packages\pipda\register.py", line 396, in wrapper
    return calling_rule(generic, args, kwargs, envdata)
  File "C:\Users\user_name\miniconda3\envs\py38\lib\site-packages\pipda\_calling.py", line 93, in verb_calling_rule3
    return generic(*args, **kwargs)
  File "C:\Users\user_name\miniconda3\envs\py38\lib\functools.py", line 872, in wrapper
    raise TypeError(f'{funcname} requires at least '
TypeError: _not_implemented requires at least 1 positional argument

[pwwang]

Do you mean you send the code to terminal to run in vscode?

[GitHunter0]

Do you mean you send the code to terminal to run in vscode?

Yes, Interactive Window or Terminal, both return the same error

[pwwang]

Could you submit an issue instead, maybe with some screenshots? This could be a common issue.

[GitHunter0]

Could you submit an issue instead, maybe with some screenshots? This could be a common issue.

Yes, I saw you already opened an issue, so I will respond there

[GitHunter0]

You are pretty close my friend, once purrr and dbplyr get incorporated, datar will be the best tool to break free from pandas and build a clear, smooth and intuitive data wrangling workflow.

I have a suggestion for dbplyr. Instead of creating it from scratch, I believe it is probably better/faster to map dbplyr to SQLAlchemy and let it do the heavy lifting, since SQLAlchemy already supports SQLite, Postgresql, MySQL, Oracle, MS-SQL, Firebird, Sybase, etc.

What do you think @pwwang ?

[GitHunter0]

I see your point @pwwang and I agree speed can be a big issue.
Considering what we have now, what would you recommend to do in Python to replace this R code in one line?

df |> mutate(new_df_var = list( purrr::map_df(…) ) )

By the way, incorporating other backends like polars (in addition to dbplyr support) is just fantastic, I really love what datar is becoming.

[pwwang]

Could you give a runnable example?

[GitHunter0]

Sure, something like this for example:

library(tidyverse)
df <- mtcars |>
  select(cyl, mpg) |>
  group_by(cyl) |>
  summarize(new_var_df =
    unique(mpg) |>
    map_dfr(~tibble(mpg = .x, economy_car = ifelse(.x > 20, TRUE, FALSE))) |>
    list()
  )
df
df[[2]][[1]]
df[[2]][[2]]
df[[2]][[3]]

I know in this case map_dfr() is not necessary since tibble() would suffice but it is just a quick example to keep it simple.

[pwwang]

This one is a little bit tricky:

>>> from pandas import Series
>>> from pipda import register_func
>>> 
>>> 
>>> @register_func
... def map_dfr(x, fn):
...     # apply to each element of the series
...     # but inorder to keep the tibble (not being broadcasted by pandas)
...     # we need to convert the series to frame first.
...     return x.to_frame().transform(fn, 1).mpg
... 
>>> gf >> summarize(
...     new_var_df= map_dfr(
...         # datar.base.unique does not support grouped data yet, so we use pandas
...         f.mpg.agg("unique"),
...         lambda x: [
...             tibble(
...                 mpg=list(x)[0], # x is a series, so with value [1,2,3], list(x) will become [[1,2,3]]
...                 economy_car = if_else(f.mpg > 20, TRUE, FALSE)
...             )
...         ]
...     )
... )
      cyl new_var_df
  <int64>   <object>
0       6   <DF 6x2>
1       4   <DF 9x2>
2       8  <DF 12x2>
>>> _.new_var_df[0]
        mpg economy_car
  <float64>    <object>
0      21.0        True
1      21.4        True
2      18.1       False
3      19.2       False
4      17.8       False
5      19.7       False

[GitHunter0]

It is indeed tricky (that's where R shines) but thank you for providing an alternative code.