Using RankBasedEvaluator directly on a ranking

[romy-vos]

Hi!

I am using pykeen for a study where I compare multiple datasets with several link prediction methods. Some of these are pykeen models, but some are not. I want to use the same evaluation for all methods for consistency. However, it seems that to use the pykeen evaluator (RankBasedEvaluator), it is required to input a pykeen model.
Is there a way to use the evaluator (or separately hits@k, MRR, MAR) when I already have ranked/scored triples, resulting from any model (potentially not implemented in pykeen)?

Thank you!

[mberr]

Yes, you can directly use the RankBasedEvaluator instance to process scores, e.g.:

import torch

from pykeen.evaluation import RankBasedEvaluator
from pykeen.datasets import get_dataset
from pykeen.typing import LABEL_TAIL

# instantiate an evaluator
evaluator = RankBasedEvaluator()

# process the scores
# todo: replace this by your way how to calculate scores
dataset = get_dataset(dataset="nations")
for hrt_batch in dataset.training.mapped_triples.split(16, dim=0):
    scores = torch.rand(size=(hrt_batch.shape[0], dataset.num_entities))
    true_score = torch.gather(scores, dim=-1, index=hrt_batch[:, 2:3])
    evaluator.process_scores_(hrt_batch=hrt_batch, target=LABEL_TAIL, scores=scores, true_scores=true_score)

# calculate aggregation metrics
result = evaluator.finalize()
print(result.to_df())

[romy-vos]

Thank you for your quick answer, this is very helpful! Much appreciated :)

[yaaani85]

Hii,

Could you @mberr please explain how this works for the filtered setting? Because the dataset is not part of the evaluator class and RankBasedEvaluator.process_scores does not use the dense positive mask? Hope you can help!

Best,

Yannick

[mberr]

@yaaani85 , you would need to filter the scores (e.g, using pykeen.evaluation.evaluator.filter_scores_) before passing them into evaluator.process_scores_.