* only active if kernel == 'poly'
* adapt the metadata to reflect this

* black checker

* Simplified

* add examples to black format check

Co-authored-by: Matthias Feurer <feurerm@informatik.uni-freiburg.de>

* re-structure manual and use 'collapse'

* ADD link to auto-sklearn-talks

* unifying titles

* Clarify default memory and cpu usage

* FIX sphinx_gallery to <=0.10.0

0.10.1 would raise an error for '-D plot_gallery=0'

* Re-structure faq

* FIX comments by mfeurer

* boldface items

* merge manual into FAQ

* FIX minor

* FIX typo

* Update doc/faq.rst

Co-authored-by: Eddie Bergman <eddiebergmanhs@gmail.com>

* Update doc/faq.rst

Co-authored-by: Eddie Bergman <eddiebergmanhs@gmail.com>

* Update doc/faq.rst

Co-authored-by: Eddie Bergman <eddiebergmanhs@gmail.com>

* Update doc/faq.rst

Co-authored-by: Eddie Bergman <eddiebergmanhs@gmail.com>

* Update doc/manual.rst

Co-authored-by: Eddie Bergman <eddiebergmanhs@gmail.com>

* Update doc/manual.rst

Co-authored-by: Eddie Bergman <eddiebergmanhs@gmail.com>

* Update doc/faq.rst

Co-authored-by: Eddie Bergman <eddiebergmanhs@gmail.com>

* FIX link

Co-authored-by: Eddie Bergman <eddiebergmanhs@gmail.com>

If you're only exposure to using... -> If your only exposure to using...

* np.bool deprecation

* Invalid escape sequence \_

* Series specify dtype

* drop na requires keyword args deprecation

* unspecified np.int size deprecated, use int instead

* deprecated unspeicifed np.int precision

* Element wise comparison failed, will raise error in the future

* Specify explicit dtype for empty series

* metric warnings for mismatch between y_pred and y_true label count

* Quantile transformer n_quantiles larger than n_samples warning ignored

* Silenced convergence warnings

* pass sklearn args as keywords

* np.bool deprecation

* Invalid escape sequence \_

* Series specify dtype

* drop na requires keyword args deprecation

* unspecified np.int size deprecated, use int instead

* deprecated unspeicifed np.int precision

* Element wise comparison failed, will raise error in the future

* Specify explicit dtype for empty series

* metric warnings for mismatch between y_pred and y_true label count

* Quantile transformer n_quantiles larger than n_samples warning ignored

* Silenced convergence warnings

* pass sklearn args as keywords

* flake8'd

* flake8'd

* Fixed CategoricalImputation not accounting for sparse matrices

* Updated to use distro for linux distribution

* Ignore convergence warnings for gaussian process regressor

* Averaging metrics now use zero_division parameter

* Readded scorers to module scope

* flake8'd

* Fix

* Fixed dtype for metalearner no run

* Catch gaussian process iterative fit warning

* Moved ignored warnings to tests

* Correctly type pd.Series

* Revert back to usual iterative fit

* Readded missing iteration increment

* Removed odd backslash

* Fixed imputer for sparse matrices

* Ignore warnings we are aware about in tests

* Flake'd:

* Revert "Fixed imputer for sparse matrices"

This reverts commit 05675ad.

* Revert "Revert "Fixed imputer for sparse matrices""

This reverts commit d031b0d.

* Back to default values

* Reverted to default behaviour with comment

* Added xfail test to document

* flaked

* Fixed test, moved to np.testing for assertion

* Update autosklearn/pipeline/components/data_preprocessing/categorical_encoding/encoding.py

Co-authored-by: Matthias Feurer <feurerm@informatik.uni-freiburg.de>

Co-authored-by: Matthias Feurer <feurerm@informatik.uni-freiburg.de>

* Added manual dispatch to tests

* Removed parameters to manual dispatch

…tors (#1332)

* Update docstrings and types

* doc typo fix

* flake'd

* added python 3.10 to versions

* Added quotes around versions

* Trigger tests

* Add submodule

* Port to abstract_ensemble, backend from automl_common

* Updated workflow files

* Update imports

* Trigger actions

* Another import fix

* update import

* m

* Backend fixes

* Backend parameter update

* fixture fix for backend

* Fix tests

* readd old abstract ensemble for now

* flake8'd

* Added install from source to readme

* Moved installation w.r.t submodules to the docs

* Temporarily remove submodule

* Readded submodule

* Updated to use automl_common under autosklearn

* Updated MANIFEST

* Removed uneeded statements from MANIFEST

* Fixed import

* Fixed comment line in MANIFEST.in

* Added automl_common/setup.py to MANIFEST

* Added prefix to script

* Re-added removed title #

* Added note for submodule for CONTRIBUTING

* Made the submodule step a bit more clear for contributing.md

* CONTRIBUTING fixes

* Added versioning for sphinx, docutils - introduced by sphinxtoolbox

* Fixed bug with config value for `plot_gallery` in doc makefile

* Update linkcheck command as well

* Added ignored_warnings file

* Use ignored_warnings file

* Test regressors with 1d, 1d as 2d and 2d targets

* Flake'd

* Fix broken relative imports to ignore_warnings

* Removed print and updated parameter type for tests

* Type import fix

* Added random state to classifiers

* Added some doc strings

* Removed random_state again

* flake'd

* Fix some test issues

* Re-added seed to test

* Updated test doc for unknown test

* flake'd

* Added ignored_warnings file

* Use ignored_warnings file

* Test regressors with 1d, 1d as 2d and 2d targets

* Flake'd

* Fix broken relative imports to ignore_warnings

* Removed print and updated parameter type for tests

* Added warning catches to fit methods in tests

* Added more warning catches

* Flake'd

* Created top-level module to allow relativei imports

* Deleted blank line in __init__

* Remove uneeded ignore warnings from tests

* Fix bad indent

* Fix github merge conflict editor whitespaces and indents

* update workflow files

* typo fix

* Update pytest

* remove bad semi-colon

* Fix test runner command

* Remove explicit steps required from older version

* Explicitly add Conda python to path for subprocess command in test

* Fix the mypy compliance check

* Added PEP 561 compliance

* Add py.typed to MANIFEST for dist

* Remove py.typed from setup.py

* rename OSX -> macOS as it is the new name

rename OSX -> macOS as it is the new name for the operating system. e.g. see https://www.apple.com/macos

* Update doc/installation.rst

Co-authored-by: Matthias Feurer <lists@matthiasfeurer.de>

* Update doc/installation.rst

Co-authored-by: Matthias Feurer <lists@matthiasfeurer.de>

Co-authored-by: Matthias Feurer <feurerm@informatik.uni-freiburg.de>
Co-authored-by: Matthias Feurer <lists@matthiasfeurer.de>

…semble (#1321)

* Changed show_models() function to return a dictionary of models in the ensemble instead of a string

* Remove flaky dep

* Remove unused pytest import

* Fix: MLPRegressor tests

* Fix: Ordering of statements in test

* Fix: MLP n_calls

* Fix: Raises errors with the config

* Add: Skip error for kernal_pca

Seems kernel_pca emits the error:
* `"zero-size array to reduction operation maximum which has no identity"`

This is gotten on the line `max_eig = lambdas.max()` which makes me
assume it emits a matrix with no real eigen values, not something we
can really control for

…#1316)

…ures (#1250)

* Moved to new splitter, moved to util file

* flake8'd

* Fixed errors, added test specifically for CustomStratifiedShuffleSplit

* flake8'd

* Updated docstring

* Updated types in docstring

* reduce_dataset_size_if_too_large supports more types

* flake8'd

* flake8'd

* Updated docstring

* Seperated out the data subsampling into individual functions

* Improved typing from Automl.fit to reduce_dataset_size_if_too_large

* flak8'd

* subsample tested

* Finished testing and flake8'd

* Cleaned up transform function that was touched

* ^

* Removed double typing

* Cleaned up typing of convert_if_sparse

* Cleaned up splitters and added size test

* Cleanup doc in data

* rogue line added was removed

* Test fix

* flake8'd

* Typo fix

* Fixed ordering of things

* Fixed typing and tests of target_validator fit, transform, inv_transform

* Updated doc

* Updated Type return

* Removed elif gaurd

* removed extraneuous overload

* Updated return type of feature validator

* Type fixes for target validator fit

* flake8'd

* Moved to new splitter, moved to util file

* flake8'd

* Fixed errors, added test specifically for CustomStratifiedShuffleSplit

* flake8'd

* Updated docstring

* Updated types in docstring

* reduce_dataset_size_if_too_large supports more types

* flake8'd

* flake8'd

* Updated docstring

* Seperated out the data subsampling into individual functions

* Improved typing from Automl.fit to reduce_dataset_size_if_too_large

* flak8'd

* subsample tested

* Finished testing and flake8'd

* Cleaned up transform function that was touched

* ^

* Removed double typing

* Cleaned up typing of convert_if_sparse

* Cleaned up splitters and added size test

* Cleanup doc in data

* rogue line added was removed

* Test fix

* flake8'd

* Typo fix

* Fixed ordering of things

* Fixed typing and tests of target_validator fit, transform, inv_transform

* Updated doc

* Updated Type return

* Removed elif gaurd

* removed extraneuous overload

* Updated return type of feature validator

* Type fixes for target validator fit

* flake8'd

* Fixed err message str and automl sparse y tests

* Flak8'd

* Fix sort indices

* list type to List

* Remove uneeded comment

* Updated comment to make it more clear

* Comment update

* Fixed warning message for reduce_dataset_if_too_large

* Fix test

* Added check for error message in tests

* Test Updates

* Fix error msg

* reinclude csr y to test

* Reintroduced explicit subsample values test

* flaked

* Missed an uncomment

* Update the comment for test of splitters

* Updated warning message in CustomSplitter

* Update comment in test

* Update tests

* Removed overloads

* Narrowed type of subsample

* Removed overload import

* Fix `todense` giving np.matrix, using `toarray`

* Made subsampling a little less aggresive

* Changed multiplier back to 10

* Allow argument to specfiy how auto-sklearn handles compressing dataset size  (#1341)

* Added dataset_compression parameter and validation

* Fix docstring

* Updated docstring for `resampling_strategy`

* Updated param def and memory_allocation can now be absolute

* insert newline

* Fix params into one line

* fix indentation in docs

* fix import breaks

* Allow absolute memory_allocation

* Tests

* Update test on for precision omitted from methods

* Update test for akslearn2 with same args

* Update to use TypedDict for better Mypy parsing

* Added arg to asklearn2

* Updated tests to remove some warnings

* flaked

* Fix broken link?

* Remove TypedDict as it's not supported in Python3.7

* Missing import

* Review changes

* Fix magic mock for python < 3.9

* Fixed bad merge

* commit meta learning data bases

* commit changed files

* commit new files

* fixed experimental settings

* implemented last comments on old PR

* adapted metalearning to last commit

* add a text preprocessing example

* intigrated feedback

* new changes on *.csv files

* reset changes

* add changes for merging

* add changes for merging

* add changes for merging

* try to merge

* fixed string representation for metalearning (some sort of hot fix, maybe this needs to be fixed in a bigger scale)

* fixed string representation for metalearning (some sort of hot fix, maybe this needs to be fixed in a bigger scale)

* fixed string representation for metalearning (some sort of hot fix, maybe this needs to be fixed in a bigger scale)

* init

* init

* commit changes for text preprocessing

* text prepreprocessing commit

* fix metalearning

* fix metalearning

* adapted test to new text feature

* fix style guide issues

* integrate PR comments

* integrate PR comments

* implemented the comments to the last PR

* fitted operation is not in place therefore we have to assgin the fitted self.preprocessor again to it self

* add first text processing tests

* add first text processing tests

* including comments from 01.25.

* including comments from 01.28.

* including comments from 01.28.

* including comments from 01.28.

* including comments from 01.31.

… and #1250 (#1386)

* Add: Doc for `dataset_compression`

* Fix: Shorten line

* Doc: Make more clear that the argument None still provides defaults

…#1387)

* Fix: ignore for certain configuration

* Fix: Extend timeout duration for tests

* Draft tidy of workflows

* Fix: mypy should not ignore missing imports

* Added pydocstyle to checkers

* Added check and format make options

* Fix: mutiple entries in same line setup.py

* Change: black line length to 88

* Fix: make check to only perform checks

* Update: Flake8 ignores style (handled by black/isort)

* Add: pydocstyle, disabled in pre-commit

* Add: Makefile `make pre-commit`

* Fix: Ignores for mypy on untyped modules

* Limit scope of pre-commit steps

* Fix: flake8 no longer concerned about line length

* Add: Flake8 to `make check`

* Fix: reduce scope of black and isort

* Fix: Pydocstyle now uses numpy convention

* Fix: workaround for test imports of `automl_common`

* Fix: `mypy` ignores `automl_common` now

* Fix: Limit scope of `black` and `isort` formatting

* Fix: pre-commit performs no file changes now

* Add: `make pre-commit` to `make help`

* Fix: `make help` docstring for `make pre-commit`

* Fix: isort update sections autosklearn, types

* Fix: warnings by flake8 for line length

* Fix: Types section for isort

* Fix: reenable `flake8` formatting checking

* Update: flake8 to use black's line length of 88

* add: ignore D205 pydocstyle

* Fix: Import order for futures

* Fix: flake8 ignore E203

* Fix: Formatting and fixed long lines

* Del: black/isort checker, checked with pre-commit

* Fix: test dummy prediction error msg

* Add: `coverage` to `pyproject.yaml`

* Add: coverage ignore for `if TYPE_CHECKING`

* Fix: missing coma

* Fix: `toml` dependency for pydoctyle in pre-commit

* Fix: isort src path

* Add: `make test`

* Fix: Add name of module to check coverage of

* Maint: isort and black most recent dev

* Fix: import typo

* Change: format now performs individually on each directory

…red (#1393)

* Update: Contributing guide with respect to #1358

* Fix: Add line on manually running pre-commit

* Add: Line for `isort` in contributing guide

* Fix: uncomment `make examples` in overview

* Fix the ensemble == 0 error in fit_ensemble and show_models function by adding a valueError to the former and giving a warning and returning empty dictionary in the latter

* Update automl.py

* Two tests for ensemble_size == 0 cases

Added two tests to check if the automl.fit_ensemble() raises error when ensemble_size == 0 and if show_models() returns empty dictionary when ensemble_size == 0

* Update automl.py

* Update test_automl.py

Test for checking if the show_models() functions raise an error if models are not fitted.

* Update automl.py

Add a function __sklearn_is_fitted__() which returns the boolean value of self.fitted(). And add the check for model fitting in show_models() function.

* Update autosklearn/automl.py

* Formatting changes to clear all the pre-commit tests

Co-authored-by: Eddie Bergman <eddiebergmanhs@gmail.com>

* Fix: Implement hotfix from #1407, aimed at master

* Add: Argument to ExecuteTaFuncWithQueue

* Add: multi_objectives arg to tests

* Fix: Two more locations of `multi_objectives`

* rename "ngram_range" to "ngram_upper_bound" this includes renaming it in all *csv and *json files for metalearning

* rename "ngram_range" to "ngram_upper_bound" this includes renaming it in all *csv and *json files for metalearning

* handle the following issue
#1373 (comment)

this commit fixes the first 3 bullet points on the to do list.

1. rename hyperparameter "ngram_range" --> "ngram_upper_bound"
   this includes changing all *csv and *json files

2. Create a new textpreprocessing example_text_preprocessing.py, this new example features the 20Newsgroups dataset

import in example_text_preprocessing.py to long, but i can not come up with a good solution

* handle the following issue
#1373 (comment)

this commit fixes the first 3 bullet points on the to do list.

1. rename hyperparameter "ngram_range" --> "ngram_upper_bound"
   this includes changing all *csv and *json files

2. Create a new textpreprocessing example_text_preprocessing.py, this new example features the 20Newsgroups dataset

import in example_text_preprocessing.py to long, but i can not come up with a good solution

include feedback from 02.24.

* handle the following issue
#1373 (comment)

this commit fixes the first 3 bullet points on the to do list.

1. rename hyperparameter "ngram_range" --> "ngram_upper_bound"
   this includes changing all *csv and *json files

2. Create a new textpreprocessing example_text_preprocessing.py, this new example features the 20Newsgroups dataset

import in example_text_preprocessing.py to long, but i can not come up with a good solution

include feedback from 02.24.

* handle the following issue
#1373 (comment)

this commit fixes the first 3 bullet points on the to do list.

1. rename hyperparameter "ngram_range" --> "ngram_upper_bound"
   this includes changing all *csv and *json files

2. Create a new textpreprocessing example_text_preprocessing.py, this new example features the 20Newsgroups dataset

import in example_text_preprocessing.py to long, but i can not come up with a good solution

include feedback from 02.24.

* handle the following issue
#1373 (comment)

this commit fixes the first 3 bullet points on the to do list.

1. rename hyperparameter "ngram_range" --> "ngram_upper_bound"
   this includes changing all *csv and *json files

2. Create a new textpreprocessing example_text_preprocessing.py, this new example features the 20Newsgroups dataset

import in example_text_preprocessing.py to long, but i can not come up with a good solution

include feedback from 02.24.

* handle the following issue
#1373 (comment)

this commit fixes the first 3 bullet points on the to do list.

1. rename hyperparameter "ngram_range" --> "ngram_upper_bound"
   this includes changing all *csv and *json files

2. Create a new textpreprocessing example_text_preprocessing.py, this new example features the 20Newsgroups dataset

import in example_text_preprocessing.py to long, but i can not come up with a good solution

include feedback from 02.24.

* handle the following issue
#1373 (comment)

this commit fixes the first 3 bullet points on the to do list.

1. rename hyperparameter "ngram_range" --> "ngram_upper_bound"
   this includes changing all *csv and *json files

2. Create a new textpreprocessing example_text_preprocessing.py, this new example features the 20Newsgroups dataset

import in example_text_preprocessing.py to long, but i can not come up with a good solution

include feedback from 02.24.

* limit 20NG to 5 labels. automl.leaderboard has problems if the ensamble contains only one model. Therefore we reduced the problem complexity

* limit 20NG to 5 labels. automl.leaderboard has problems if the ensamble contains only one model. Therefore we reduced the problem complexity

* limit 20NG to 2 labels. automl.leaderboard has problems if the ensamble contains only one model. Therefore we reduced the problem complexity

* limit 20NG to 2 labels. automl.leaderboard has problems if the ensamble contains only one model. Therefore we reduced the problem complexity

* rename `auto-sklearn/autosklearn/pipeline/components/data_preprocessing/feature_reduction` to `auto-sklearn/autosklearn/pipeline/components/data_preprocessing/text_feature_reduction`.

also rename corresponding feature reduction class FeatureReduction to TextFeatureReduction.
`auto-sklearn/autosklearn/pipeline/components/data_preprocessing/text_feature_reduction/truncated_svd.py:TextFeatureReduction`

This includes adapting all *csv and *json participating in metalearning

The "real" changes are limited to
  1. truncated_svd.py
  2. feature_type_text.py

* rename `auto-sklearn/autosklearn/pipeline/components/data_preprocessing/feature_reduction` to `auto-sklearn/autosklearn/pipeline/components/data_preprocessing/text_feature_reduction`.

also rename corresponding feature reduction class FeatureReduction to TextFeatureReduction.
`auto-sklearn/autosklearn/pipeline/components/data_preprocessing/text_feature_reduction/truncated_svd.py:TextFeatureReduction`

This includes adapting all *csv and *json participating in metalearning

The "real" changes are limited to
  1. truncated_svd.py
  2. feature_type_text.py

* rename `auto-sklearn/autosklearn/pipeline/components/data_preprocessing/feature_reduction` to `auto-sklearn/autosklearn/pipeline/components/data_preprocessing/text_feature_reduction`.

also rename corresponding feature reduction class FeatureReduction to TextFeatureReduction.
`auto-sklearn/autosklearn/pipeline/components/data_preprocessing/text_feature_reduction/truncated_svd.py:TextFeatureReduction`

This includes adapting all *csv and *json participating in metalearning

The "real" changes are limited to
  1. truncated_svd.py
  2. feature_type_text.py

* rename `auto-sklearn/autosklearn/pipeline/components/data_preprocessing/feature_reduction` to `auto-sklearn/autosklearn/pipeline/components/data_preprocessing/text_feature_reduction`.

also rename corresponding feature reduction class FeatureReduction to TextFeatureReduction.
`auto-sklearn/autosklearn/pipeline/components/data_preprocessing/text_feature_reduction/truncated_svd.py:TextFeatureReduction`

This includes adapting all *csv and *json participating in metalearning

The "real" changes are limited to
  1. truncated_svd.py
  2. feature_type_text.py

* change treatment of generic column dtype `object` for pandas dataframes.
The `object` type will be treated as `string` in the future.

* change treatment of generic column dtype `object` for pandas dataframes.
The `object` type will be treated as `string` in the future.

* change treatment of generic column dtype `object` for pandas dataframes.
The `object` type will be treated as `string` in the future.

* change treatment of generic column dtype `object` for pandas dataframes.
The `object` type will be treated as `string` in the future.

add new test case to `test_feature_validator.py`

* change treatment of generic column dtype `object` for pandas dataframes.
The `object` type will be treated as `string` in the future.

add new test case to `test_feature_validator.py`

* change treatment of generic column dtype `object` for pandas dataframes.
The `object` type will be treated as `string` in the future.

add new test case to `test_feature_validator.py`

* Add: Dedicated fixtures location

* Add: Typing to conftest

* Fix: More typing

* Move: Dask fixtures to own location

* Move: backend to it's own fixture location

* Move: Automlstub thing to it's own fixture

* Fix: docstring warning in conftest

* Fix: Remove dead code

* Add: Functionality to skip marked `slow` tests

* Add: Fixture for getting datasets

* Update: Tests for automl.py w/ fixtures, cases, mocks

This commit (too much) focues on extending our pytest capabilities to
make things more modular. Notably, making a fixtures and mocks folders
for testing so there is a unified source of mocks and fixture locations.

Split the tests for automl.py into seperate sub files to allow for an
easier time discerning what is being tested where.

Introduces `pytest_cases` to allow for easier future
testing of properties.

Introduces cached automl models to allow for a quicker time testing
attributes while keeping tests seperate and distinct.

This also rearragnes `unittest` import unfortunatly which causes many
files to be touched.

* Doc: Add documentation about our testing to conftest.py

* Doc: Add a bit more on documenting of tests

* Fix: Add pytest-cases to test dependancies

* Fix: Broken tests

* Fix: Failing test

* Fix: Ordering of check in `_includes`

* Remove `_includes` and use sets instead

* Fix: Black

* Update: test workflow knows about `.pytest_cache`

* change treatment of string features. In file `smbo.py` string columns are marked as categorical. Previously string columns are treated as not categorical which transfers to numerical. We decided that encoded string columns are more similar to OHE as they are to numerical transformations. This change effects the metalearning part of `autosklearn` exclusively.

* change treatment of string features. In file `smbo.py` string columns are marked as categorical. Previously string columns are treated as not categorical which transfers to numerical. We decided that encoded string columns are more similar to OHE as they are to numerical transformations. This change effects the metalearning part of `autosklearn` exclusively.

For models that are in the ensemble but have no corresponding entry in
model_runs (built from run_history), we fill in NA values to prevent
crashing. Why this occurs, I don't know but this should at least prevent
total failure when we can still provide other useful information back to
the user.

…1425)

* Cleanup: Stopwatch

* Uddate: Stopwatch with context manager and use it

* Fix: Don't call property

* Add: Ignore automl_common

* Fix: wrong variable name `self.time_for_task`

* Remove: Old test file

* Better error message for stopwatch `time_since`

* Update: `time_since` can now optionally raise

* Use the new `time_since` stopwatch method

* Trigger workflow

* Add extra time for cross-val automl cases

* Add comment in code to explain why

Co-authored-by: Rattko <radovan.haluska1@gmail.com>

* Use newer --strict arg for twine

* Update precommit config

* change treatment of string features. In file `smbo.py` string columns are marked as categorical. Previously string columns are treated as not categorical which transfers to numerical. We decided that encoded string columns are more similar to OHE as they are to numerical transformations. This change effects the metalearning part of `autosklearn` exclusively.

* change treatment of string features. In file `smbo.py` string columns are marked as categorical. Previously string columns are treated as not categorical which transfers to numerical. We decided that encoded string columns are more similar to OHE as they are to numerical transformations. This change effects the metalearning part of `autosklearn` exclusively.

* made categorical dictionary more cohesive. Use the previously create feat_type dictionary which stores more information and reduce it to categorical where it is really needed.

* change test files since they now get the feat type dictionary and no longer the categorical one

* change test files since they now get the feat type dictionary and no longer the categorical one

* moved feat_type to categorical conv. to the helper functions

* fixed minor issues

* fixed minor issues

* fixed minor issues

* fixed bug in metalearning tests

* First draft of multi-objective optimization

Co-authored-by: Katharina Eggensperger <eggenspk@informatik.uni-freiburg.de>

* Feedback from Eddie

* Make metric internally always a list

* Fix most examples

* Take further feedback into account

* Fix unit tests

* Fix one more example

* Add multi-objective example

* Simplify internal interface

* Act on further feedback

* Fix bug

* Update cv_results_ for multi-objective sklearn compliance

* Update leaderboard for multi-objective optimization

* Include Feedback from Katharina

* Take offline feedback into account

* Take offline feedback into account

* Eddie's feedback

* Fix metadata generation unit test

* Test for metrics with the same name

* Fix?

* Test CV results

* Test leaderboard for multi-objective optimization

* Last batch of unit tests added

* Include Eddie's feedback

Co-authored-by: Katharina Eggensperger <eggenspk@informatik.uni-freiburg.de>

* Make test caching opt-in `--cached`

* Remove unused param

* Move ensemble_bulder test data to named folder

* Update backend to take a temlate to copy from

* Update tests to use new cases system

* Update tests to be documented and cleaned up

* Switch to using cached automl backends

* Readd missing file which failed test for `case_3_models`

* Seperate out tests that rely on old toy data and those that don't

* Setup test framework for ensemble builder on real situations

* Formatting

* Remove `unit_test` arg

* Remove SAVE2DISC

* Split builder and manager into seperate files

* Tidy up init of EnsembleBuilder

* Moved to cached properties

* Change List to list

* Move to solely using cached properties

* Add disk util file with `sizeof`

* Update tests to use cached mechanism

* Switch `sizeof` for disk consumption

* Remove disk consumption

* Remove unneeded function

* Add type hints and documenation

* Simplyify _read_np_fn

* Update get_valid_test_preds to use Pathlib

* Add intersection to functional

* Make functional take *args

* Further simplifications

* Add a dataclass to represent run information for builder

* Rename to Run

* Change to Run objects

* Formatting

* Reduce side effects of `compute_loss_per_model`

To make testing easier and changes easier, the targets are now passed to
the method. This also reduces it's complexity by removing the checking
from the method as we can assume the parameters coming in are correct.

* Change Tuple to tuple

* Forcibly add data files for tests

* Fix: Can now load pickled numpy arrays w/ test

* Add test for checking ensemble builder output

* Fix bug with using list instead of set

* Making deubgging message a little clearer

* Fix typing and case name

* Rename test file to reflect what it tests

* Make pynisher context optional

* Fix loaded models test

* Updates to Run dataclass

* Add method to `Run` to allow recording of last modified

* Change Run mtimes to dictionary

* Change `compute_loss_per_model` to use new Run dataclass

* Factor out run loss into main loop

* Simplyify get_nbest and compute_losses

* Major rewrite of ensemble builder main loop

* Change to simpler hashing

* Start value split

* Add `value_split`

* Reworked Builder

* Add some docstring

* Formatting

* Fix type signature

* Fix typing for `loss`

* Removed Literal

* Mypy fixes for ensemble builder

* Mypy fixes

* Tests for `Runs`

* Move `make_run` to fixtures

* Fix run deletion

* Test candidates

* Made delete it's own function

* Further simplifications

* Fixup test with simplification

* Test: `max_models` for `requires_deletion`

* Test: `memory_limit` for `requires_deletion`

* Test: Loss of runs

* Test: Delete runs

* Test: `fit_ensemble` of ensemble builder

* Add test for run time parameter

* Remove parameter `return_predictions`

* Add note about pickled arrays should not be supported

* Make cached automl instances copy backend

* Add valid static method to run

* Remove old test data

* Add filter for bad run dirs

* Made `main` args optional

* Fix check for updated runs

* Make `main` raise errors

* Fix default value for ensemble builder `main`

* Test valid ensemble with real runs

* Rename parameter for manager

* Add defaults and reorder parameters for EnsembleBuilderManager

* Fixup parameters in `fit_and_return_ensemble`

* Typing fixes

* Make `fit_and_return_ensemble` a staticmethod

* Add: `make_ensemble_builder_manager`

* Add: Test files for manager

* Add atomic rmtree

* Add: atomic rmtree now accepts where mv should go

* Make builder use atomic rmtree

* Fix import bugs, remove valid preds in builder

* Remove `np.inf` as valid arg for `read_at_most`

* Possible reproducible num_run, no predictions error

* Make automl caching robust to `pytest-xdist`

* Test fixes

* Extend interval for test on run caching

* Use pickle for reseting cache

* Fix test for caching mechanism to not rely on `stat`

* Move run deletion to the end of the builder `main`

* Remove `getattr` version of tae.client

* Remove `normalize`

* Extend not for `Run`

* Fix `__init__` of `Run`

* Parameter and comment fixes from feedback

* Change to `min(...)` instead of `sorted(...)[0]`

* Make default time `np.inf`

* Add test for safe deletion in builder

* Update docstring of `loss` for a run

* Remove stray print

* Minor feedback fixes

* Fix `_metric` to `_metrics`

* Fix `make_ensemble_builder`

* One more fix for multiple metrics

* Load single best model as fallback

* Update estimators.py

* Improve comment in code.

* Fix meta-data generation test, potentially improve model loading

* FIX minor

* update submodule

* update submodule

* ADD pass xdata to metric

* update submodule

* Fix tests

* update submodule

* ADD example

* UPDATE example

* ADD extra method to concat data

* RM note

* Fix minor

* ADD more types for xdata

* FIX unittests

* FIX variable naming bug

* change varible name; fix docstring

* Rename variable

* FIX example

* Update examples/40_advanced/example_metrics.py

Co-authored-by: Matthias Feurer <feurerm@informatik.uni-freiburg.de>

Co-authored-by: Matthias Feurer <feurerm@informatik.uni-freiburg.de>

* Multi-objective ensemble API

Co-authored-by: eddiebergman <eddiebergmanhs@gmail.com>

* update for rebase, add loading of X_data in ensemble builder

* Add unit tests

* Fix unittest?, increase coverage (hopefully)

* Rename methods to be Pareto set methods

Co-authored-by: eddiebergman <eddiebergmanhs@gmail.com>

lso removed a loop that wasn't doing anything. DataFrame.append doesn't operate in place so that second loop wasn't causing any side effects, and it was also just adding empty series anyway. If anyone has any insight about why that was there in the first place that would be nice.

…side _init_ (#1473)

* Encapsulated the selector training within a function and called it inside __init__

* Made changes in the test script that trains selectors before testing

* Moved global variables inside the askl2 class, changes in the test script supporting that

* Committing after running pre-commit

* Fixed issue resulting into failure of case 2

* Changed metrics to selector_metrics

* Added additional tests for askl2

* Added a boolean to the class to check for re-training of selectors

* Using tmp_path instead of tmpdir

* Removed tests for all metrics

* Removed test for custom metric

* Update workflows

* Add dependabot to check workflows

* Update FAQ with text stuff

* Take suggestions into account

* Push

* `fit_ensemble` now has priority for kwargs to take

* Change ordering of prefernce for ensemble params

* Add TODO note for metrics

* Add `metrics` arg to `fit_ensemble`

* Add test for pareto front sizes

* Remove uneeded file

* Re-added tests to `test_pareto_front`

* Add descriptions to test files

* Add test to ensure argument priority

* Add test to make sure X_data only loaded when required

* Remove part of test required for performance history

* Default to `self._metrics` if `metrics` not available

* Create simple example and doc for naive early stopping

* Fix doc, pass through SMAC callbacks directly

* Fix `isinstance` check

* Add test for early stopping

* Fix signature of early stopping example/test

* Fix doc build

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 3 to 4.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](actions/setup-python@v3...v4)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 2 to 3.
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](actions/download-artifact@v2...v3)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 2 to 3.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](codecov/codecov-action@v2...v3)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 2 to 3.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@v2...v3)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix logging server cleanup

* Add comment relating to the `try: finally:`

* Remove nested try: except: from `fit`

Bumps [peter-evans/find-comment](https://github.com/peter-evans/find-comment) from 1 to 2.
- [Release notes](https://github.com/peter-evans/find-comment/releases)
- [Commits](peter-evans/find-comment@v1...v2)

---
updated-dependencies:
- dependency-name: peter-evans/find-comment
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [actions/stale](https://github.com/actions/stale) from 4 to 5.
- [Release notes](https://github.com/actions/stale/releases)
- [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md)
- [Commits](actions/stale@v4...v5)

---
updated-dependencies:
- dependency-name: actions/stale
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Init commit

* Fix logging server cleanup (#1503)

* Fix logging server cleanup

* Add comment relating to the `try: finally:`

* Remove nested try: except: from `fit`

* Bump peter-evans/find-comment from 1 to 2 (#1520)

Bumps [peter-evans/find-comment](https://github.com/peter-evans/find-comment) from 1 to 2.
- [Release notes](https://github.com/peter-evans/find-comment/releases)
- [Commits](peter-evans/find-comment@v1...v2)

---
updated-dependencies:
- dependency-name: peter-evans/find-comment
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump actions/stale from 4 to 5 (#1521)

Bumps [actions/stale](https://github.com/actions/stale) from 4 to 5.
- [Release notes](https://github.com/actions/stale/releases)
- [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md)
- [Commits](actions/stale@v4...v5)

---
updated-dependencies:
- dependency-name: actions/stale
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Init commit

* Update evaluation module

* Clean up other occurences of the word validation

* Re-add test for test predictions

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add debug statements and 30s timeouts

* Fix formatting

* Update internal timeout param

* +timeout, use allocated tmpdir

* +timeout, use allocated tmpdir

* Remove another occurence of explicit `tmp`

* Increase timelimits once again

* Remove incomplete comment

* Init commit

* Fix DummyClassifiers in _load_pareto_set

* Add test for dummy only in classifiers

* Update no ensemble docstring

* Add automl case where automl only has dummy

* Remove tmp file

* Fix `include` statement to be regressor

* Create PR

* Update MLP regressor values

* Make docker file install from `setup.py`

* Add pytest cache to gitignore

* Up timeouts on test_metadata_generation

* Create PR

* Fix test fixture

* Bump docker/build-push-action from 1 to 3

Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 1 to 3.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](docker/build-push-action@v1...v3)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update docker-publish.yml

Replace password by token

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matthias Feurer <feurerm@informatik.uni-freiburg.de>

* Create PR

* Abstract out dask client types

* Fix _ issue

* Extend scope of dask_client in automl.py

* Add docstring to dask module

* Indent result addition

* Add basic tests for Dask wrappers