Add 30, 10, and 5 minute constraints

[Innixma]

This PR is a proposal to add shorter time constraint options. This is based on my own findings that AutoML systems are capable of achieving meaningful results with less than one hour of runtime on many datasets. For example, AutoGluon is able to succeed on all 104 datasets with 10 folds on best_quality using a 10 minute constraint, and almost all datasets with a 5 minute constraint.

These runs would be much cheaper from a compute cost standpoint and could be feasible to run on local compute without clusters in some cases. For example, 5 minute runs with 1 fold could finish 104 datasets in under 10 hours on a single machine.

Some additional ideas on how smaller time budgets could be leveraged:

• As AMLB gets more frameworks added to it, it would be good to have a filtering mechanism to only run expensive benchmarks on frameworks that are competitive. By running most frameworks with low time budget, this could hopefully be a predictive indicator of which systems would also perform well on a large time budget.
• By combining smaller time budgets with a subset of the datasets/folds, we could enable developers to test their systems on a local machine using a "mini-benchmark" for fast iteration and to avoid compute budget concerns.

[PGijsbers]

TL;DR: I am in favour of having constraints that reduce computational costs, but I am unsure what the best way to go about this would be.

Sorry for the very slow reply on this one. I absolutely agree with the fact that we should (also) benchmark with smaller time constraints for all the benefits you listed. Even for scientific evaluations, the current settings may be prohibitively expensive to some, which may lead them to use non-standardized subsets or budgets. Additionally, including shorter time constraints may also lead to additional effort in developing "faster" methods which may have very beneficial impacts in the overall energy usage of AutoML (also outside of scientific evaluations). I also have the slight suspicion that (especially with the current set of datasets), the relative performance of AutoML frameworks is somewhat consistent across time budgets - so there may not even be a big downside to focusing more on smaller budgets.

That said, I am not yet entirely sure what the best way to go about this would be. People are already somewhat easily able to define their own custom constraints (which in my opinion largely at least tackles the AutoML developer concern). The out-of-the-box constraints seem more "official" and my worry is that if there are too many options, we will fragment the literature into works that evaluate at 5 minute budgets, vs 10 minute budgets, and so on, and thus again making it hard to compare results across papers. Additionally, for the largest datasets a 5 minute runtime may be somewhat problematic - though perhaps that is something the AutoML framework should cope with.

On a related note, there are various AutoML frameworks that are currently configured to always use the full time budget while they have an early stopping mode (which is often the default!). Here too, I find that we may want to slightly alter the design of (some) settings so that this early stopping default can be benchmarked - it's more realistic to the use of those frameworks in practice, should lower the overall computational costs, and reduce benchmark runtimes.

That's why I've been holding of a merge so far. I would love to discuss this a bit more (I sent you a message on OpenML slack).