Skip to content

optuna/kurobako

Repository files navigation

kurobako

kurobako Documentation Actions Status Coverage Status License: MIT Docker Hub

kurobako is a command-line tool to benchmark black-box optimization algorithms.

Features:

  • It has the capability to support wide range of optimization problems:
    • various search sapce:
      • Continuous numerical, discrete numerical and categorical
      • Uniform distribution and log uniform distribution
      • Conditional
    • Constrainted problems
    • Multi-objective problems
  • Generating a markdown report and PNG plots from benchmarking results
  • Easy to add user-defined optimization problems and solvers
  • Simulating a concurrent environment in which an optimization process is executed by multiple workers simultaneously
  • Reproducible

Installation

Precompiled binaries

Precompiled binaries for Linux are available in the releases page.

$ curl -L https://github.com/optuna/kurobako/releases/download/${VERSION}/kurobako-${VERSION}.linux-amd64 -o kurobako
$ chmod +x kurobako
$ ./kurobako -h

Using Cargo

If you have already installed Cargo, you can install kurobako by executing the following command:

$ cargo install kurobako

Dependencies

If you want to use kurobako plot command, you need to have installed gnuplot package on your environment.

Usage Example

// Define solver.
$ kurobako solver random | tee solver.json
{"random":{}}

// Define problem.
$ curl -OL http://ml4aad.org/wp-content/uploads/2019/01/fcnet_tabular_benchmarks.tar.gz
$ tar xf fcnet_tabular_benchmarks.tar.gz && cd fcnet_tabular_benchmarks/
$ kurobako problem hpobench fcnet_protein_structure_data.hdf5 | tee problem.json
{"hpobench":{"dataset":"fcnet_protein_structure_data.hdf5"}}

// Run benchmark.
$ kurobako studies --solvers $(cat solver.json) --problems $(cat problem.json) | kurobako run > result.json
(ALL) [00:00:01] [STUDIES     10/10 100%] [ETA  0s] done

// Report the benchmark result.
$ cat result.json | kurobako report
...abbrev...

// Plot the benchmark result.
$ cat result.json | kurobako plot curve
(PLOT) [00:00:01] [1/1 100%] [ETA  0s] done (dir="images/curve/")

Build-in Solvers and Problems

Solvers:

Problems:

Where does the name come from?

"kurobako" is a Japanese translation of "black box".

References