Making sense of best_trial_evaluation from HPO

[MSBradshaw]

I am running a hyperparameter optimization pipeline for rotatE on the Hetionet data set using NSSA loss. The best pipeline reported in study_name/best_pipeline/pipeline_config.json lists the best_trial_evaluation as 0.1697545349597931. I assume this corresponds to the specified loss function and that this is the same as the value column of 'study_name/trials.tsv'. But the confusing part is that trials.tsv lists numerous trials that had lower and higher values than that reported as best_trial_evaluation, as low as 0.0018553906319438273 and as high as 0.2494133854768465.

I hope I am just misunderstanding something.

Exactly what value is reported in study_name/best_pipeline/pipeline_config.json : best_trial_evaluation?

How is that used to choose the best pipeline?

My code for running the pipeline:

import os
os.environ['OMP_NUM_THREADS'] = '16' 
# this must be set before pykeen in imported so that it can take effect before things get started

from pykeen.pipeline import pipeline
from pykeen.datasets.base import PathDataset
from pykeen.pipeline import pipeline
from pykeen.hpo import hpo_pipeline
from optuna.samplers import RandomSampler
import torch

torch.manual_seed(42)

study_name = "study_name"  # Unique identifier of the study.
storage_name = "sqlite:///{}.db".format(study_name) # this for a sqlite database in the current working directory

hpo_pipeline_result = hpo_pipeline(
    n_trials=30,
    sampler=RandomSampler,

    model_kwargs={
            "random_seed": 42,
        },

    model_kwargs_ranges=dict(
            embedding_dim=dict(
                type=int,
                low=64,
                high=512,
                q=16,
            ),
        ),

    training_kwargs_ranges=dict(
            num_epochs=dict(
                type=int,
                low=100,
                high=1000,
                q=100,
            ),
        ),
    
    optimizer_kwargs_ranges=dict(
            lr=dict(
                type=float,
                low=0.001,
                high=0.1,
                log=True,
            ),
        ),
    
    negative_sampler_kwargs_ranges=dict(
            num_negs_per_pos=dict(
                type=int,
                low=1,
                high=100,
                q=10,
            ),
        ),
    dataset="Hetionet",
    model='rotate',
    device='cuda',
    storage='sqlite:///{}.db'.format(study_name),
    load_if_exists=True,
    study_name=study_name,
    stopper='early',
    stopper_kwargs={'frequency':10, 'patience':2, 'relative_delta':0.002},
    loss="NSSA",
)

hpo_pipeline_result.save_to_directory('PyKeenOut/{}'.format(study_name))

Pykeen version info:

Key	Value
OS	posix
Platform	Linux
Release	3.10.0-1160.80.1.el7.x86_64
Time	Mon Nov 27 09:30:24 2023
Python	3.9.16
PyKEEN	1.10.1
PyKEEN Hash	UNHASHED
PyKEEN Branch
PyTorch	2.0.1+cu117
CUDA Available?	false
CUDA Version	11.7
cuDNN Version	8500

[mberr]

best_trial_evaluation comes directly from optuna's Study object:

pykeen/src/pykeen/hpo/hpo.py

Lines 342 to 345 in c94213c

	metadata = {
	"best_trial_number": self.study.best_trial.number,
	"best_trial_evaluation": self.study.best_value,
	}

It corresponds to the optimization metric you have selected. If you do not explicitly select a metric, it will default to mean reciprocal rank (MRR).

[MSBradshaw]

That makes sense, thank you!

[MSBradshaw]

Actually, can you elaborate on how that explains best_trial_evaluation still not being the trial with the highest MRR?

[mberr]

best_trial_evaluation should correspond to the best value in the MRR column, or, more precisely, both.realistic.inverse_harmonic_mean_rank; there may be more columns of the form <SIDE>.<RANK_TYPE>.inverse_harmonic_mean_rank that correspond to the MRR evaluated on either head or tail prediction only (SIDE), and using a different tie-breaking policy (RANK_TYPE).