K-Means UDTF & Intel oneDAL initial integration #669

averbukh · 2021-07-30T12:51:58Z

Contains:

K-Means clustering algorithm UDTF
Initial integration with Intel oneDAL (ML library)

There are some workaround to make it possible to use Intel oneDAL which requires modern TBB and OmniSciDB together. It makes sense to get rid of them after OmniSci will move to latest TBB.

Dependency script modification was made for Ubuntu only. Wider OS support to be added later. I would like to get comments from maintainers before that.

… k-means

This reverts commit 9bfba2e.

This reverts commit 9625a89.

…SQL measurements

This reverts commit 52fe343.

This reverts commit 79a33e2.

… in OmniSQL measurements" This reverts commit baa629c.

# Conflicts: # CMakeLists.txt # QueryEngine/TableFunctions/TableFunctions.hpp

…he env

… variable

CLAassistant · 2021-07-30T12:52:04Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

Michael Averbukh seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

Michael Averbukh added 30 commits November 30, 2020 17:05

Adding DAAL k-means as UDTF

f2da94a

DAAL paths fix

5fbc3ff

Include fixed

d6a5ce9

DAAL part moved into top of CMakeLists.txt

78e2c73

Compilation fix

f7e0c65

Compilation fix

2280c5c

CMakeLists.txt modification to suport DAAL in UdfTest

d44fd77

CMakeLists.txt modification to suport DAAL in TableUpdateDeleteBenchmark

b1f0ae7

CMakeLists.txt modification to suport DAAL in initDB

5694a1c

CMakeLists.txt modification to suport DAAL in ImportExportTest

8ac2eda

CMakeLists.txt modification to suport DAAL in test is another way

2e90cf7

CMakeLists.txt modification to suport DAAL in PersistentStorageTest

11ca191

Compilation fix

f67a914

CMakeLists.txt modification to suport DAAL in ForeignStorageCacheTest

a8491cc

DOUBLE replaced by FLOAT in Calcite factory registration function for…

b8407e0

… k-means

Object creation for Calcite function registration

2b213a6

Experiment to exclude NUMERIC from list of operand types

9625a89

Experimental code to provide workaround

9bfba2e

Revert "Experimental code to provide workaround"

8fedf0c

This reverts commit 9bfba2e.

Revert "Experiment to exclude NUMERIC from list of operand types"

39df235

This reverts commit 9625a89.

Sizer arg pos fixed, signature for Calcite fixed

88ef2a8

Zer-copy approach for k-means assignments

d36b8fa

Time measurement added to UDTF k-means

7da03c5

Todd's suggestion of code midifcation to improve performance of UDFs

4a66dba

Experimental code to understand total time and execution time in Omni…

baa629c

…SQL measurements

Time measurement added to sql_execute_impl

79a33e2

Additional time measurement added to sql_execute_impl

52fe343

Revert "Additional time measurement added to sql_execute_impl"

3d5ecd8

This reverts commit 52fe343.

Revert "Time measurement added to sql_execute_impl"

67c0288

This reverts commit 79a33e2.

Revert "Experimental code to understand total time and execution time…

435b14e

… in OmniSQL measurements" This reverts commit baa629c.

Michael Averbukh added 29 commits July 5, 2021 17:35

cmake file fixed according to @pearu suggestion

fd0ad33

Calcite registration removal fixed

8d1e6f1

Column::value_type added

8617ce6

Code cleanup

8d336ff

Merge branch 'master' of https://github.com/omnisci/omniscidb

0a9321b

# Conflicts: # CMakeLists.txt # QueryEngine/TableFunctions/TableFunctions.hpp

Code cleanup

c9c8224

Java compilation fix

89e3371

New cmake integration with oneDAL

d981edf

mapd-deps.sh generation fixed

86ba155

Cmake package name fixed for oneDAL

e27fcc6

oneAPI vars scripts is used instead of oneDAL script to have TBB in t…

f1f00d6

…he env

Compilation fix

d037b17

cmake file code cleanup

ddd9bba

cmake file improvement: oneDAL include directory assighment via cmake…

84f2b12

… variable

Get rid of output from setvars script

fa52d62

Cmake file improvement experiment

4e341e6

Cmake file improvement experiment partially reverted

cf81045

Compilation fix

17e513f

oneDAL cmake config settings fixed

ef1efc7

Static linking for oneDAL instead of dynamic

b284fe0

Compilation fix

39eb9f6

LATEST_TBB_LIBRARIES added to cmake variables

87c2491

LATEST_TBB_LIBRARIES usage for tests added

3ccc0f5

Merge branch 'master' of https://github.com/omnisci/omniscidb

e5ee331

TBB full path experiment

6d7ecad

Latest TBB for omnisci_server

733e410

TBB full path replaced by environment variable usage

cfd44bc

Some formatting issues fixed (that introduced during previous merges)

2a77ec6

ENABLE_ONEDAL set to off by default

1cdf997

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

K-Means UDTF & Intel oneDAL initial integration #669

K-Means UDTF & Intel oneDAL initial integration #669

averbukh commented Jul 30, 2021

CLAassistant commented Jul 30, 2021 •

edited

K-Means UDTF & Intel oneDAL initial integration #669

Are you sure you want to change the base?

K-Means UDTF & Intel oneDAL initial integration #669

Conversation

averbukh commented Jul 30, 2021

CLAassistant commented Jul 30, 2021 • edited

CLAassistant commented Jul 30, 2021 •

edited