fread/fwrite *base data types* directly for efficiency

[jangorecki]

There could be an option for fwrite and fread to dump/read the unclassed representation of objects. In such a case whole IO process can be much faster, because we don't have to format POSIXct, [I]Date and other classes for writing to file, we just unclass them, and write numbers (or any other base type they use) to a file. Later when reading with fread we provide colClasses (or yaml), and those numbers (or another base types) are properly recognized as their original class.

library(data.table)
dt = data.table(a=as.Date("2015-01-01"), b=as.POSIXct("2015-01-01 15:30:20"))
fwrite(dt, "fwrite-tests.csv", AsIs=TRUE)
system("cat fwrite-tests.csv")
#a, b
#16436, 1420106420
fread("fwrite-tests.csv", colClasses = c(a="Date", b="POSIXct"))
#            a                   b
#       <Date>              <POSc>
#1: 2015-01-01 2015-01-01 15:30:20

[clarkdk]

It would be awesome if fwrite by default wrote Date and POSIXct objects in standard formats (ie ISO 8601, or Lubridate) that fread would automatically recognize and read in as Date or POSIXct, without requiring colClasses to be specified for each datetime column. A default automatic approach would be really welcome since I spend a lot of programming time converting datetime character strings back to POSIXct when importing time series data, even when I have written the file myself with write.csv. It would presumably also require an assumption (default value) for tz on import, unless the default format included the time zone offset on every value. Perhaps a POSIXct.format="format-string" parameter could be included to specify the format string to use when writing and reading POSIXct values, since colClasses = c(a="Date", b="POSIXct") doesn't allow a specific format and/or timezone to be specified.

[jangorecki]

@clarkdk You discuss quite a different issue, as it requires parsing strings that stores date/datetime. This issue is about date/datetime stored as numerics, and those can be effectively optimized for IO speed if fwrite and fread would be able to write and read those types as their numerics transparently (or using just colClasses). So no parsing is required. If you would like to speedup parsing strings then fasttime can help.

[clarkdk]

@jangorecki OK, now I understand. I will make a separate FR. In implementing datetimes as numerics in fread/fwrite, please don't rule out a possible future feature involving writing and parsing of datetimes as formatted strings.

[jangorecki]

now when have yaml support this feature is even more useful as we can maximise IO without trade-offs

[PavoDive]

Not sure if this belongs to this issue or to 3391 or to the master task 2247, but I could reproduce an unexpected behavior with fread and fwrite when yaml = TRUE:

dt = data.table(date = as.POSIXct(c("2006-05-01", "2006-05-02")), 
                 b = as.factor(c(1,2)), 
                 c = c(3,4))
print(class(dt$date))
fwrite(dt, "dt.csv", yaml = TRUE)

dt2 = fread("dt.csv", yaml = TRUE)
print(class(dt2$date))

See: https://stackoverflow.com/questions/58493926/fread-with-yaml-true-read-type-date-as-character

[MichaelChirico]

@PavoDive could you file a new issue for that? please & thank you