Chaining base operations via pipe?

[GG-Delta]

Hey there,
I am still amazed by your wonderful and extremely powerful package! Using it on a daily basis. Please allow me a short question on chaining base operations using pipes. Let me shortly use one of your own (slightly modified) examples to quickly see my question:
df4 = tibble.tibble(A = ["apple","cherry","orange","banana"], B = [25,np.nan,15,28])

Quickly checking for the presence of NaN in the data frame using datar works fine (as expected):
base.sum(base.any_na(df4)) results in the correct response:
1

Question:
Is it somehow possible to re-formulate the above operations by chaining them together via pipe? To give you an impression: df4 >> base.any_na() >> base.sum()
This code doesn't work right now.
Do you you have any hint for me whether I can make it work or if I am missing something? Or alternatively, this kind of chaining up basic operations is not supported (similar to some R functions)?

Thanks a lot for your help!
All the best,
Gernot

[pwwang]

Hi Gernot,

It's all about Verb vs Function, which are registered by register_verb() and register_func() from pipda.

You can simply check whether a function is registered as a Verb or Function by:

>>> from datar import base
>>> from pipda import Function, Verb
>>> base.any_na
<datar.core.factory.DatarAggFunction object at 0x7f730eff25b0>
>>> from pipda import Function
>>> isinstance(base.any_na, Function)
True
>>> base.dim
<pipda.verb.Verb object at 0x7f730ef78b50>
>>> isinstance(base.dim, Verb)
True

A verb is a function that is able to accept the arguments except the first, which is the data, and wait for it to be piped in. For example, df >> base.dim(). However, Function just literally accepts its full argument list, which means, when the interpreter sees this: base.sum(), it just thinks that the arguments are complete.

Why just register all functions as Verbs, so that we can do piping with any registered functions? Is it feasible?

Theoretically, yes. But would cause confusion in this scenario:

>>> from datar import f
>>> from datar.base import as_character
>>> from datar.dplyr import everything, mutate, across
>>> from datar.datasets import table3
>>> table3
       country    year               rate
      <object> <int64>           <object>
0  Afghanistan    1999       745/19987071
1  Afghanistan    2000      2666/20595360
2       Brazil    1999    37737/172006362
3       Brazil    2000    80488/174504898
4        China    1999  212258/1272915272
5        China    2000  213766/1280428583
>>> table3 >> mutate(across(everything(), as_character))
       country     year               rate
      <object> <object>           <object>
0  Afghanistan     1999       745/19987071
1  Afghanistan     2000      2666/20595360
2       Brazil     1999    37737/172006362
3       Brazil     2000    80488/174504898
4        China     1999  212258/1272915272
5        China     2000  213766/1280428583

It worked just fine, as as_character was registered as a Function. Imagine under the hood how it works, should we pass table3 here to as_character? If the function is a Verb, we need to pass table3 internally to it to make the function work; otherwise, we don't pass table3, as Functions or plain functions don't accept data argument. In such a case, all the Verbs used in this scenario must accept a DataFrame as the first argument. In such a case, we need to register as_character as below to make it work:

@register_verb(DataFrame)
def as_character(_data, x):
  return ...

Even though _data will not be used. However, this won't work if we want to do: as_character(table3.country), because it's awaiting the data to be piped in.

There is a way to work around it. For example, check if DataFrame is a registered type for the Verb, if so, pass table3, otherwise don't. But it still could be arbitrary when x in the above case is expected to be a DataFrame.

Ideas are welcome for solving this issue and making all functions Verbs.

[GG-Delta]

Thanks a lot, pwwang! Your systematic and in-depth answers really explained the fundamentals behind my question. As you explained, one can check how a specific operations is registered (either as verb or function) - and this type defines the actual use. I think in principle this is a clear-cut and logical distinction I can work with. Thanks again - also for your hint about checking the status of a given operations via pipda!

Regarding your overall discussion question: are there ideas about making all functions Verbs? You made it clear in your response that there are challenges, in particular about the expectation about the first argument (data) in case of a function. We might overcome this issue if every function would have a kind of "switch" (i.e. an additional argument set as TRUE or FALSE, for example) telling the function if the first argument should be data input or not. The default for a function could be FALSE so that a function behaves as it does now (depending on the DataFrame input as first argument). But if a given function has this switch set to be TRUE it would behave like a Verb, i.e. expecting the feeding of data via Pipe.

In short, the potential idea is that every function can be turned explicitly by the user input into a verb by including an additional argument. The default could be the same behaviour of functions as we have it now. However, a user could turn each function explicitly into a 'Verb'. I am curious what you think? Please tell me if I can help in any way! Thank you, Gernot

[pwwang]

Did you mean something like this:

[1, 2, 3] >> base.sum(__as_verb=True)

?

[GG-Delta]

Yes, exactly! Your code line captures the idea!

[GitHunter0]

@GG-Delta , @pwwang , in some cases, you could use sspipe:

from sspipe import p, px
from datar import base, tibble
import numpy as np

[1,2,4] | p(base.sum)

df4 = tibble.tibble(A = ["apple","cherry","orange","banana"], 
                    B = [25,np.nan,15,28])

base.sum(base.any_na(df4))
# or
df4 | p(base.any_na) | p(base.sum)

[pwwang]

I've gone through the cases and found that across is probably the only edge case.
And I just realized that with R, when a verb is used in across, the data has to be passed explicitly:

r$> df = tibble(x=1:3, y=4:6)

r$> df = tibble(y=df)

r$> df
# A tibble: 3 × 1
    y$x    $y
  <int> <int>
1     1     4
2     2     5
3     3     6

r$> df |> mutate(across(everything(), ~ select(.x, "x")))  # need to pass .x proun to refer the data, it is explicitly passed.
# A tibble: 3 × 1
    y$x
  <int>
1     1
2     2
3     3

That means, we just need to pass the arguments literally to the function used in across, no matter if it's a Verb or a Function, we don't need to pass the data implicitly. This avoids ambiguity.

So we should be able to register all functions as Verbs theoretically, but we need to see if other edge cases pop up.

[GG-Delta]

That sounds awesome! Thanks for your in-depth explanations!