Master list: Feature completion for complex-valued columns

[MichaelChirico]

As a follow-up to #1444 and #3689, I think it's useful to keep a list of all the features missing for users who want to do work involving complex-valued columns z:

• 1. Sorting by z (Extend forder support to 'complex' type. #1703); by extension frank(z), rowid(z), dcast, setkey, uniqueN
• 2. Grouping by z
• 3. z I/O (fread (without colClasses) & fwrite; overlap with Master task for fread bugs / proposals #2247)
• 4. CJ(z)
• 5. Assign z by group
• 6. GForce on z
• 7. Joins on z wait until a user requests it with justification (that part of bmerge.c is complex)
• 8. coalesce(z)
• 9. shift(z)
• 10. unique(z)
• 11. := Wrong values in j with complex values if "by" has more than 1 group. #3639

v1.12.4 NEWS contains :

---

Examples

DT = data.table(a = 1:5, z = 6:10 + (5:1)*1i)

1.

DT[order(z)]

Error in forderv(ans, cols, sort = TRUE, retGrp = FALSE, order = if (decreasing) -order else order, :
Column 1 of by= (1) is type 'complex', not yet supported. Please use the by= argument to specify columns with types that are supported. See NEWS item in v1.12.2 for more information.

2.

DT[ , .N, by = z]

Error in [.data.table(DT, , .N, by = z) :
column or expression 1 of 'by' or 'keyby' is type complex. Do not quote column names. Usage: DT[,sum(colC),by=list(colA,month(colB))]

3.

fwrite(DT)

Error in fwrite(DT) :
Column 2's type is 'complex' - not yet implemented in fwrite.

fread('a\n1+i')
#         a
#    <char>
# 1:    1+i

4.

DT[CJ(a, z)]

Error: Column 1 of by= (1) is type 'complex', not yet supported. Please use the by= argument to specify columns with types that are supported. See NEWS item in v1.12.2 for more information.

Coming from forderv so first this is a repeat of 1

DT[CJ(a, z, sorted = FALSE)]

Error: Type 'complex' not supported by CJ.

5.

DT[ , total := base::sum(z), by = .(a %% 2)]

DT[ , total := base::sum(z), by = .(a %% 2)]
Error in [.data.table(DT, , :=(total_rad, base::sum(z)), by = .(a%%2)) :
Internal error: writeNA passed a vector of type 'complex'

6.

DT[ , sum(z), by = .(a %% 2)]

Error in gsum(z) :
Type 'complex' not supported by GForce sum (gsum). Either add the prefix base::sum(.) or turn off GForce optimization using options(datatable.optimize=1)

7.

DT[DT, on = 'z']

Error in bmerge(i, x, leftcols, rightcols, roll, rollends, nomatch, mult, :
x.z is type complex which is not supported by data.table join

8.

DT[ , coalesce(`is.na<-`(z, 1), 0+0i)]

Error in coalesce(is.na<-(z, 1), 0 + (0+0i)) :
Unsupported type: complex

9.

DT[ , shift(z)]

Error in shift(z) : Unsupported type 'complex'

[MichaelChirico]

I think we could get away with hacking sorting immediately by morphing order(z) into order(Re(z), Im(z)); is there any immediacy to doing this, or is worth wait until going through and doing it properly in C on the CPLXSXP?

[MichaelChirico]

I don't really know where to start on fread... but Hugh's colTypes generalization already works:

class(fread('a\n0+1i', colClasses = c(a='complex'))[[1]])
# [1] "complex"

I'd say this is good enough unless there is some specific demand to speed this up. At a glance it would take significant effort to teach fread to detect & read complex values automatically.

[mattdowle]

Fantastic! Can this be closed now @MichaelChirico ?

[MichaelChirico]

I'd say so! maybe just open a follow-up issue for complex joins? or we could leave something in the NEWS asking if there's any real use case for it while also suggesting Re=Re & Im=Im as a workaround?

…

On Sat, Jul 20, 2019, 5:50 AM Matt Dowle ***@***.***> wrote: Fantastic! Can this be closed now @MichaelChirico <https://github.com/MichaelChirico> ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3690?email_source=notifications&email_token=AB2BA5KWMTOODDUNTY6XM7DQAIZL3A5CNFSM4H7RF6L2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2M3MII#issuecomment-513390113>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB2BA5PTXLN6VEVUT4VJRATQAIZL3ANCNFSM4H7RF6LQ> .

[mattdowle]

Great. I edited the revised news item into the main comment at the top of this issue. It asks for a use case. Is that ok? I'd prefer not to create a new issue for join-on-complex until someone asks for it.

[MichaelChirico]

Yes that's perfect. Thanks for all your help getting the rest of this issue over the finish line!