double groupby results

[samukweku]

Let's have a look at the dataframe below:

DT = dt.Frame(G=[1, 1, 1, 2, 2, 2],
                  V=[None, None, None, None, 3, 5],
                  N=[None, None, None, None, None, None])

DT
   |     G      V     N
   | int32  int32  void
-- + -----  -----  ----
 0 |     1     NA    NA
 1 |     1     NA    NA
 2 |     1     NA    NA
 3 |     2     NA    NA
 4 |     2      3    NA
 5 |     2      5    NA
[6 rows x 3 columns]

DT[:, dt.mean(f.V), f.G]
   |     G        V
   | int32  float64
-- + -----  -------
 0 |     1       NA
 1 |     2        4
[2 rows x 2 columns]

DT[:, dt.sum(f.V), f.G]
   |     G      V
   | int32  int64
-- + -----  -----
 0 |     1      0
 1 |     2      8
[2 rows x 2 columns]

DT[:, dt.mean(dt.sum(f.V)), f.G]
   |     G            V
   | int32      float64
-- + -----  -----------
 0 |     1  2.66667    
 1 |     2  9.32677e+13
[2 rows x 2 columns]

DT[:, dt.sum(dt.mean(f.V)), f.G]
   |     G             V
   | int32       float64
-- + -----  ------------
 0 |     1  4           
 1 |     2  1.38241e-309
[2 rows x 2 columns]

The outputs for the double aggregation in a groupby yields wrong results. Any ideas @oleksiyskononenko ?

[samukweku]

i noticed something similar in 1.0.0 :

In [1]: from datatable import dt, f, by

In [2]: dt.__version__
Out[2]: '1.0.0'

In [3]: DT = dt.Frame(G=[1, 1, 1, 2, 2, 2],
   ...:                   V=[None, None, None, None, 3, 5],
   ...:                   N=[None, None, None, None, None, None])

In [4]: DT
Out[4]: 
   |     G      V     N
   | int32  int32  void
-- + -----  -----  ----
 0 |     1     NA    NA
 1 |     1     NA    NA
 2 |     1     NA    NA
 3 |     2     NA    NA
 4 |     2      3    NA
 5 |     2      5    NA
[6 rows x 3 columns]

In [5]: DT[:, dt.sum(dt.mean(f.V)), f.G]
Out[5]: 
   |     G             V
   | int32       float64
-- + -----  ------------
 0 |     1  0           
 1 |     2  7.26276e-322
[2 rows x 2 columns]

[oleksiyskononenko]

Side note: when creating examples, it is nice to create an MRE (minimum required example) avoiding unnecessary complexities. In your case the column N is not really needed to demonstrate the issue and the number of rows could be reduced to improve readability.

Preparing an MRE also gives you additional information on the bug. You start with a simple one row frame

>>> DT = dt.Frame([[0], [1]])
>>> DT[:, dt.sum(dt.mean(f.C1)), f.C0]
   |   C0       C1
   | int8  float64
-- + ----  -------
 0 |    0        1
[1 row x 2 columns]
>>> DT[:, dt.mean(dt.sum(f.C1)), f.C0]
   |   C0       C1
   | int8  float64
-- + ----  -------
 0 |    0        1
[1 row x 2 columns]

and see there is no issue. Adding one more row, the bug appears

>>> DT = dt.Frame([[0, 0], [1, 2]])
>>> DT[:, dt.mean(dt.sum(f.C1)), f.C0]
   |   C0            C1
   | int8       float64
-- + ----  ------------
 0 |    0  -3.45876e+18
[1 row x 2 columns]
>>> DT[:, dt.sum(dt.mean(f.C1)), f.C0]
   |   C0       C1
   | int8  float64
-- + ----  -------
 0 |    0      1.5
[1 row x 2 columns]

Meaning that the issue is somehow related to multiple number of rows per group. Running the above MRE in the debug dt, you will notice an exception being thrown

AssertionError: Assertion 'i < nrows()' failed in src/core/column.cc, line 251

So we're actually accessing non-existent data in the case of >=2 rows in the group by context.

[samukweku]

Thanks for the guide @oleksiyskononenko

[oleksiyskononenko]

You're welcome. I've updated the message above, as the issue is actually with the multiple number of rows per a group and not the multiple number of groups.

[oleksiyskononenko]

It seems that MRE could be simplified even more

>>> DT[:, dt.mean(dt.sum(f.C1))]
   |         C1
   |    float64
-- + ----------
 0 | 2.8823e+18
[1 row x 1 column]

So the groupby context is not really necessary to face the bug.

[oleksiyskononenko]

I propose you open a new issue with the following MRE

>>>from datatable import dt, f
>>>DT = dt.Frame([1, 2])
>>> DT[:, dt.mean(dt.sum(f.C0))]
   |         C0
   |    float64
-- + ----------
 0 | 2.8823e+18
[1 row x 1 column]
>>> DT[:, dt.sum(dt.mean(f.C0))]
   |          C0
   |     float64
-- + -----------
 0 | 2.31584e+77
[1 row x 1 column]

[oleksiyskononenko]

Yeah, we seems to introduce a bug during refactoring, but a similar bug exists in 1.0.0 and we need to take a look. It is interesting though that we don't have any tests that could catch such a simple bug.

[oleksiyskononenko]

Nice catch!