fread - OOM issue

[gautam-ergo]

Issue:

+1 for the awesome package.

Encountering Out Of Memory issue when reading a LARGE csv file (~200 million rows) using fread.
Datatable code is being run on an aks-cluster with 20Gi of RAM and its still tripping

Code:

csv file:

Expected behavior:
Being able to load the file and perform manipulations.

Environment:
datatable version - 1.0.0
python version - 3.8.0
operating system - linux

Any pointers on how to do the aforementioned task in a memory efficient way would be of great help.

Thanks.

[Mathanraj-Sharma]

@gautam-ergo try converting your data to Jay format and load it

#2860 (comment)

https://stackoverflow.com/questions/57653983/is-the-jay-file-format-specific-to-python-datatable