Roadmap comments from DAI perspective

[mmalohlava]

This is the list of issues annotated by the priority from DAI perspective:

🚨 Top Priority

Goal: optimize computation speed, minimize data conversions

• Optimize MLI code to adopt datatable (DAI: TOP, time needed: ?)
• Arrow support/conversion (TODO issue) (DAI: TOP, time needed: 2-3 weeks)
  • Arrow -> datatable (Implement converting an Arrow Table into a Frame #2711)
  • datatable -> Arrow (Implement converting a Frame into an Arrow Table #2717) (parallelized, there is still copy - see to_arrow docs)
  • 📝 Note: the GPU version of XGB will benefit from it
  • ➡️ Action: @oleksiyskononenko @st-pasha needs to clarify usage (how to be memory efficient)
  • 📝 Note: Arrow -> datatable -> some limited ops (avoiding materialization) -> Arrow - will avoid copy
• Date/time columns (Implement datetime columns #1646) (DAI: TOP, when available: End of February first phase - rough estimate)
  • Implement date32 column #2858: date32 column (date only);
  • Implement time64 column #2911: time64 column (timestamp).
• Interpretable model (Implement an Interpretable linear model with stochastic gradient descent learning #2358) (DAI: TOP, when available: most of the features are already available docs)
• Upgrade of datatable does not work with the latest DAI

High Priority

• User-defined aggregator / map functions (Support user-defined aggregation and mapping functions #1960) (DAI: HIGH, time needed: ?)
• Tighter integration with LGBM (maybe the arrow support is the way): (DAI: TOP, time needed: ?) [clarify]
  • 📝 Note: LGBM can accept native datatable frame for computation (no conversion to pandas/numpy)
  • 📝 Note: This will need change in LGBM - since it needs numpy input (dups of memory, slow serial creation of data)
  • 📝 Note: It seems like LGBM does not support Arrow yet: Feature Requests & Voting Hub microsoft/LightGBM#2302
• Performance on multi-core systems/cache trashing (TODO issue) (DAI: HIGH, time needed: ?) [clarify]
• Fread improvements:
  • Implement streaming reading (Streaming approach for reading data in fread #1958, Eliminate "Rereading" pass in fread? #1843, streaming dt.fread(path).to_jay() #1750) (DAI: HIGH, time needed: ?)
    • @arnocandel we can pass total memory limitation to fread to enable chunk-based reading.

Other

• Full support of datasets with >2B rows (Implement sorting of columns with >2B rows #2336) (DAI: MEDIUM, time needed: ?)

  • Note: ability to read and write is good enough as the first step
  • Note: efficient version needs lot of effort

• Inner/outer joins (Implement inner/outer joins for non-keyed frames #1080) (DAI: LOW)

• Ability to score dataframe using MOJO in efficient way (f.score(mojo)) (DAI: MEDIUM, time needed: ?) [clarify]

• Fread improvements:

  • Implement streaming reading (Streaming approach for reading data in fread #1958, Eliminate "Rereading" pass in fread? #1843, streaming dt.fread(path).to_jay() #1750) (DAI: LOW, time needed: ?)
  • Implement reading + filtering (xread: an "extract+read" #1950) (DAI: LOW, time needed: ?)
  • Better auto-detection of parse settings (fread may sometimes detect incorrect newline character #1343, Unable to parse attached dataset. #1045, File containing a single unescaped " out-of-sample is read incorrectly #1036, Improve headers detection logic when all columns are of "string" type #946, If last field has unclosed quote, then it will not be parsed properly #934, fread should not detect sep within quoted fields #922, fread erroneously guesses sep=' ' #518) (DAI: LOW, time needed: ?)

• Functions for string manipulations (DAI: MEDIUM, time needed: ?)

  • Apply slice to a string (FR: Apply a slice to each string in a string column #1667)
  • Convert to upper/lower case
  • Repeat strings (*)
  • Strip whitespace / other characters

• Functions for date/time manipulations (DAI: LOW, time needed: ?)

• Rolling windows support (Rolling aggregate support based on windows within a DT #1500) (DAI: LOW, time needed: ?)

New proposals

• Data compression (DAI: MEDIUM, time needed: ?)
• Data encryption in Jay file (DAI: MEDIUM, time needed: ?)
• Per column Jay file, support for efficient sporadic data updates (DAI: LOW, time needed: ?)
• Distributed mode of datatable (via Dask) (DAI: LOW, time needed: ?)

[mmalohlava]

Any news on time-estimates?

[oleksiyskononenko]

Yes, see some updates above. We also added a tag [clarify] for some items that are not fully clear from dt perspective at the moment.

[bramverbeke2]

Is there any idea when basic string manipulations would be possible (lower/upper)?

[oleksiyskononenko]

@bramverbeke2 This discussion was related to datatable improvements from the DAI perspective. Therefore, the basic string manipulations were not a top priority. If you're interested in having the lower/upper case function, please open the corresponding issue.

[mmalohlava]

regarding:

Tighter integration with LGBM (maybe the arrow support is the way):

My understanding: Right now, we load data in DTBL format, do munging, but before pushing into LGBM we need to convert data into format expected by LGBM (=copy of data). @arnocandel was proposing to integrate DTBL directly to LGBM to be able to accept data in DTBL format.

Performance on multi-core systems/cache trashing

@arnocandel ?

Ability to score MOJO in efficient way

Able to use MOJO in "map" call on data frame in a similar way how H2O-3/Spark are scoring with MOJO: split data to partitions and score partitions in parallel.

[mmalohlava]

Actions

• ➡️ @oleksiyskononenko @st-pasha needs to clarify usage (how to be memory efficient)
• ➡️ are there any PPC64le related issues (@surenH2oai asked)

[mmalohlava]

Current priority from H2O:

• linear model - needs categorical support
  • action: needs meeting with @arnocandel @pseudotensor to figure out next steps
• time column
• flag for fread to disable datetime column
• functionality of stream freading

From community perspective:

• google summer of code - reshape/melt function
• function apply