Feature: listOMLHyperparams for list of run.ids

[DanielKuehn87]

It would be nice to have a function to extract the hyper parameters for several run ids. The current way is to use getOMLRun() for each single run id, which is really slow.

[giuseppec]

Do we have an API for this now?
If not, you could request this in the main issue tracker. However, you could use getOMLRun(x, only.xml=TRUE) from #347 this should be faster and is the best I could do for now.

[berndbischl]

what we want is this:

listOMLSetups(setup.ids) --> data frame of hyperparams for the setup ids

listOMLRunsEvaluations:
new arg "expand.setup" which turn setup.ids in hyperpar cols.

we need this very soon. can this be done pls?

@giuseppec
@jakob-r

[berndbischl]

@joaquinvanschoren

[giuseppec]

The API is not working, see openml/OpenML#411 (comment) . Will have a look when this is fixed.

[giuseppec]

I started to work on this here #378

[giuseppec]

Done in #378, we now have the listOMLSetup function where you can provide a vector of setup.ids (e.g. from the listOMLRunEvaluations) and get their hyperparameter settings of each setup.

[DanielKuehn87]

Ok, thanks.
This is a client side adjustment and not really supported by the OML API, right?

[giuseppec]

it's supported by the REST API but not 100% documented in https://www.openml.org/api_docs/#!/setup/get_setup_id

[DanielKuehn87]

Ok, but you can send only one run.ID per request, right?
So when we want to get 5 million settings, we still need to send 5 million requests to the API.

[giuseppec]

Nope, you can also ask for all parameter configurations of a specific flow. This is only one request and it already gives you more than one run (but a maximum of 10000 results if I remember correctly).
Of course this list will also contain runs of other users that have used the same flow as you.

For now, you could get your own runs using listOMLRunEvaluations and join the result of listOMLRunEvaluations with the result of listOMLSetup by the setup.id to focus on your own runs.

But maybe @janvanrijn or @joaquinvanschoren will extend this list/setup so that you can also filter for tags, uploader.id and task.id.

[berndbischl]

Nope, you can also ask for all parameter configurations of a specific flow.

can we please add a practical example to the vignette / online tutorial? this is an important point