3.2.0

What's Changed

• [MINOR] Update next dev version in the project by @j143 in #1789
• [MINOR] Cleanups in cleaning pipelines by @Shafaq-Siddiqi in #1792
• [MINOR] Windows action try out by @Baunsgaard in #1795
• Bump jackson-databind from 2.13.4.1 to 2.13.4.2 by @dependabot in #1799
• [MINOR] MacOS guide update by @Baunsgaard in #1823
• [SYSTEMDS-3490] Compressed Transformencode Binning by @Baunsgaard in #1824
• [MINOR] Federated Tutorial by @Baunsgaard in #1851
• [MINOR] Split balanced minor optimization by @Baunsgaard in #1872
• [MINOR] Timeout test by @Baunsgaard in #1878
• Bump actions/checkout from 3 to 4 by @dependabot in #1892
• [MINOR] Refine internal sparse printing by @Baunsgaard in #1893
• [MINOR] Frame Float detection refinement by @Baunsgaard in #1896
• Bump docker/metadata-action from 4 to 5 by @dependabot in #1906
• Bump docker/setup-buildx-action from 2 to 3 by @dependabot in #1907
• Bump docker/build-push-action from 4 to 5 by @dependabot in #1908
• Bump docker/login-action from 2 to 3 by @dependabot in #1909
• Bump actions/setup-node from 3 to 4 by @dependabot in #1931
• Bump actions/setup-java from 3 to 4 by @dependabot in #1950
• [SYSTEMDS-3652] Adding support for removing duplicate rows to the unique() builtin function by @BACtaki in #1949
• Bump actions/upload-artifact from 3 to 4 by @dependabot in #1962
• Bump actions/download-artifact from 3 to 4 by @dependabot in #1963
• Bump actions/setup-python from 4 to 5 by @dependabot in #1954
• Bump actions/cache from 3 to 4 by @dependabot in #1984
• [SYSTEMDS-3670] Added early-stopping mechanism to tSNE by @ramesesz in #1990
• [MINOR] copy shaded jar instead of moving by @j143 in #1996
• [MINOR] Update generated documentation by @j143 in #1997
• [SYSTEMDS-3672] GIO Test Cases false positive by @fathollahzadeh in #1999
• [SYSTEMDS-2926] AWS scripts update for EMR-7.0.0 by @lachezar-n in #2003

New Contributors

• @BACtaki made their first contribution in #1949
• @ramesesz made their first contribution in #1990
• @lachezar-n made their first contribution in #2003

Full Changelog: 3.1.0-rc1...3.2.0-rc2

3.1.0

What's Changed

Sub-task

[SYSTEMDS-2411] - Performance codegen kmeans mnist80m w/ compression
[SYSTEMDS-3088] - Prefetch instruction
[SYSTEMDS-3098] - Broadcast instruction
[SYSTEMDS-3256] - Create apply functions for cleaning primitives
[SYSTEMDS-3286] - LogicalEnumerator change with transitions concept and cleanups
[SYSTEMDS-3347] - Flatten the nested loop for parallel pipelines execution
[SYSTEMDS-3376] - Adding apply_pipeline() builtin for cleaning pipelines API
[SYSTEMDS-3401] - Release docker images with GitHub actions
[SYSTEMDS-3422] - Add monitoring tool testing workflows
[SYSTEMDS-3443] - Asynchronous Execution and Persist Spark Transformations
[SYSTEMDS-3466] - Future-based asynchronous execution of Spark actions
[SYSTEMDS-3469] - New operator linearization order to maximize inter-operator parallelism
[SYSTEMDS-3470] - Lineage-based reuse of Spark actions
[SYSTEMDS-3473] - Push down rmvar instructions for asynchronous instructions
[SYSTEMDS-3474] - Lineage-based reuse of asynchronous operators
[SYSTEMDS-3479] - Persist and reuse Spark RDDs
[SYSTEMDS-3497] - Refactor to add LOP rewrite step in compilation

Bug

[SYSTEMDS-1026] - Fix memory configuration in sparkDML.sh
[SYSTEMDS-1281] - OOM Error On Binary Write
[SYSTEMDS-1283] - Out of memory error
[SYSTEMDS-2948] - CLA Improved Run estimation
[SYSTEMDS-3045] - AttributeError: Function definition not found
[SYSTEMDS-3272] - applySchema built-in to set the schema of frame from DML
[SYSTEMDS-3339] - CSR TSMM left with filled rows bug
[SYSTEMDS-3353] - Sparse TSMM dense row blocks CSR
[SYSTEMDS-3354] - py4j.Py4JException: Method exceptionString([class org.apache.spark.SparkConf]) does not exist
[SYSTEMDS-3355] - MatrixBlock size using CSR when allowed
[SYSTEMDS-3379] - Federated Nan Values
[SYSTEMDS-3390] - countDistinctApprox() operation in AggregateUnaryCPInstruction is inefficient for row/col aggregations
[SYSTEMDS-3391] - Correct the release artifact generation date
[SYSTEMDS-3394] - Log4j incompatible dependencies
[SYSTEMDS-3396] - ConcurrentModificationException in federated execution
[SYSTEMDS-3398] - Jackson Core missing for json writing and reading in reduced binary
[SYSTEMDS-3400] - Fix Java doc warnings
[SYSTEMDS-3408] - Enque output not UTF-8 python
[SYSTEMDS-3409] - Read CSV directly without mtd python
[SYSTEMDS-3411] - Python configuration not loading defaults
[SYSTEMDS-3412] - Matrix Multiplication crash in Spark
[SYSTEMDS-3414] - Pipelines failing in Hybrid execution
[SYSTEMDS-3415] - Built-in tests failure in Git actions
[SYSTEMDS-3416] - Cleaning Pipelines failed with No space left on device
[SYSTEMDS-3417] - IndexOutOfBounds due to int overflow on replace
[SYSTEMDS-3418] - Cleaning Pipelines: Replace function failure in hybrid execution
[SYSTEMDS-3419] - Cleaning Pipelines: Block Sizes mismatch
[SYSTEMDS-3420] - Cleaning Pipelines in hybrid mode: Invalid block dimensions error
[SYSTEMDS-3424] - Federated Statistics print in non federated scenario
[SYSTEMDS-3425] - Spark Aggregate Binary operations parse to Fed instruction
[SYSTEMDS-3432] - FederationUtils.bindResponses causes out of memory because of sparse matrices.
[SYSTEMDS-3433] - Python IDE test Docs fail
[SYSTEMDS-3435] - MSVM robustness for non-existing classes
[SYSTEMDS-3436] - CLA ArrayOutOfBounds in sample
[SYSTEMDS-3437] - CLA Invalid Unique estimate DDC
[SYSTEMDS-3439] - Federated read cache cannot be disabled
[SYSTEMDS-3442] - Monitoring Heavy hitters not always correct list
[SYSTEMDS-3451] - Slow Federated Mlogreg on Criteo (dummy-coded)
[SYSTEMDS-3452] - Incorrect warning when reading scalars
[SYSTEMDS-3476] - Spark with default settings
[SYSTEMDS-3477] - Cleaning Pipelines: Task Parallel Experiments failing in spark mode
[SYSTEMDS-3498] - Unique() crashes with iterator EOF on vectors with >1K distinct items
[SYSTEMDS-3500] - Perftest: Mlogreg on 1M_1k_dense w/ unnecessary spark jobs
[SYSTEMDS-3501] - Perftest: lmDS on 1M_1k_dense with unnecessary spark tsmm
[SYSTEMDS-3503] - Java doc warnings

Epic

[SYSTEMDS-450] - Extended spark interfaces
[SYSTEMDS-3445] - Combining compression schemes together
[SYSTEMDS-3459] - Reorganization and cleanup of the internal representation of FrameBlocks

New Feature

[SYSTEMDS-2551] - Federated Compression Instruction
[SYSTEMDS-2699] - CLA IO Compressed Matrices
[SYSTEMDS-2754] - Compressed Max/Min Index support.
[SYSTEMDS-2830] - Functional Compression
[SYSTEMDS-3280] - Homomorphic Encryption for Federated Parameter Servers
[SYSTEMDS-3303] - NN Builtin: Attention Layer
[SYSTEMDS-3325] - Multi-threaded tokenization
[SYSTEMDS-3337] - CLA TSMM direct multiplication
[SYSTEMDS-3360] - Federated async compression
[SYSTEMDS-3361] - Federated Workload-aware Compression
[SYSTEMDS-3369] - Timout setting for all federated tests
[SYSTEMDS-3374] - Federated primitive for transferring a local data object to a federated representation
[SYSTEMDS-3404] - Synchronous with backup workers mode for Parameter Servers
[SYSTEMDS-3405] - Federated Write at site
[SYSTEMDS-3438] - CLA RowSlice compressed return
[SYSTEMDS-3478] - Bitset array for Frames
[SYSTEMDS-3481] - Frame from MatrixBlock improvement
[SYSTEMDS-3493] - Python windows Install
[SYSTEMDS-3494] - Python 3.9 support
[[SYSTEMDS-3495](https://issues.ap...

3.0.0

What's Changed

• [SYSTEMDS-3193] Promote svn repo dev/systemds/2.x-rc to release/systemds/2.x by @j143 in #1427
• [SYSTEMDS-3190] Do not run workflow for empty checks by @j143 in #1426
• [SYSTEMDS-2599] Development branch name change to main by @j143 in #1429
• [MINOR] Update java language level by @Baunsgaard in #1430
• [MINOR] Update docs to 2.3.0-SNAPSHOT by @j143 in #1433
• [SYSTEMDS-3196] Update org.apache parent pom version to 24 by @j143 in #1436
• [SYSTEMDS-3212] Move Prefetch threadpool to CommonThreadPool by @phaniarnab in #1450
• [MINOR] Upgrade junit version to 4.13.2 by @j143 in #1451
• [MINOR] Silence mvn package download info by @j143 in #1465
• [MINOR] Fix imports in BuiltinUnaryGPUInstructionTest by @j143 in #1467
• [MINOR][PYTHON-DOC] Update Multiple Federated Environments Example Result by @fathollahzadeh in #1469
• [SYSTEMDS-3236] Cache-friendly Apply phase for sparse target matrix by @phaniarnab in #1473
• [SYSTEMDS-3239] Train and predict in different Contexts Python by @Baunsgaard in #1474
• [SYSTEMDS-3240] Fix IOGEN test path if the iogen directory doesn't exist by @fathollahzadeh in #1475
• [SYSTEMDS-3187] Add documentation for the release scripts by @j143 in #1478
• [MINOR] Cleaning Pipelines cleanups by @Shafaq-Siddiqi in #1493
• Action for building docker images automatically by @j143 in #1441
• [SYSTEMDS-3268] Publish Docker images on schedule by @j143 in #1500
• [SYSTEMDS-3270][DOCS] Docker usage documentation by @j143 in #1501
• [SYSTEMDS-3271] Publish docker test image when requested by @Baunsgaard in #1502
• [SYSTEMDS-3273] Federated Timeout by @Baunsgaard in #1508
• Bump actions/setup-python from 1 to 2.3.1 by @dependabot in #1512
• [MINOR][DOC] xgboost function y parameter correct usage by @j143 in #1532
• Bump actions/setup-python from 2.3.1 to 2.3.2 by @dependabot in #1538
• [MINOR] Refactoring input and output parameters of dbscanApply and db… by @Shafaq-Siddiqi in #1541
• [SYSTEMDS-3288] CLA SDC isolated DefaultTuple by @Baunsgaard in #1533
• [SYSTEMDS-3184] Builtin for computing information gain using entropy and gini by @morf1us in #1520
• [DOCS] Add missing data files in docs by @j143 in #1548
• Bump actions/setup-python from 2.3.2 to 3 by @dependabot in #1552
• Bump actions/checkout from 2 to 3 by @dependabot in #1553
• [MINOR] Performance improvements in cleaning pipelines by @Shafaq-Siddiqi in #1556
• Bump actions/cache from 2 to 3 by @dependabot in #1570
• [Cleaning Pipelines] MINOR improvements in allocation of resources by @Shafaq-Siddiqi in #1576
• keep release scripts concise by @j143 in #1549
• Bump actions/setup-java from 2 to 3 by @dependabot in #1582
• Create CITATION.cff by @j143 in #1449
• [SYSTEMDS-3345] Add support for as.boolean(X) by @Baunsgaard in #1579
• [SYSTEMDS-3022] Avoid cudaMemset() where possible by @corepointer in #1572
• Bump jackson-databind from 2.12.3 to 2.12.6.1 by @dependabot in #1577
• [MINOR] Github actions use Java OpenJDK adopt-hotspot by @Baunsgaard in #1598
• Bump docker/login-action from 1 to 2 by @dependabot in #1606
• Bump docker/build-push-action from 2 to 3 by @dependabot in #1605
• Bump docker/setup-buildx-action from 1 to 2 by @dependabot in #1604
• Bump docker/metadata-action from 3 to 4 by @dependabot in #1607
• generate proto file by @j143 in #1620
• [MINOR] Add protobuf parameter for consistency by @j143 in #1621
• Bump actions/setup-python from 3 to 4 by @dependabot in #1633
• [MINOR] Minor fixes i.e., validation checks, formatting e.t.c. by @Shafaq-Siddiqi in #1636

New Contributors

• @fathollahzadeh made their first contribution in #1469
• @morf1us made their first contribution in #1520

Full Changelog: 2.2.0-rc1...3.0.0-rc2

2.2.2

2.2.2-rc1

[maven-release-plugin] copy for tag 2.2.2-rc1

Release 2.2.1

2.2.1-rc3

[maven-release-plugin] copy for tag 2.2.1-rc3

Release 2.2.0

2.2.0-rc1

[maven-release-plugin] copy for tag 2.2.0-rc1

2.1.0-rc3

[maven-release-plugin] copy for tag 2.1.0-rc3

v2.0.0-rc3

[SYSTEMDS-2667] Fix extra/NOTICE

v2.0.0-rc2

[MINOR][DOC] Update header links and image files

Fix relative path when inside site

closes #1072