Error when Calculating Evaluation Metrics in using Pytroch_geometric UPFD Dataset

[victordaniel]

I am trying to calculate evaluation metrics (AUC, accuracy, and F1 score) for DOMINANT model using the UPFD dataset. However, I am encountering an error related to inconsistent sample sizes when attempting to calculate these metrics.

I am using PyTorch Geometric's DataLoader to load the UPFD dataset and have made corrections to my code as follows:

eval_roc_auc(data.y.numpy(), scores)

Here is my complete code:

import argparse
import os.path as osp
import numpy as np
from pygod.detector import DOMINANT
from torch_geometric.data import Data
from torch_geometric.datasets import UPFD
from torch_geometric.loader import DataLoader
from torch_geometric.transforms import ToUndirected
from sklearn.metrics import roc_auc_score
from pygod.metric.metric import eval_roc_auc

model = DOMINANT(num_layers=4, epoch=20)

for batch in train_loader:
data = batch
print(data)
model.fit(data)

scores = model.decision_function(data)
y_true = data.y.numpy()
y_pred = scores.detach().numpy()  # Convert PyTorch tensor to NumPy array
roc_auc = roc_auc_score(y_true, y_pred)
print(f"ROC AUC: {roc_auc:.4f}")

I am still encountering the following error message:

Traceback (most recent call last):
File "new_sub.py", line 48, in
eval_roc_auc(data.y.numpy(), scores)
File "/home/victor/victor/lib/python3.8/site-packages/pygod/metric/metric.py", line 33, in eval_roc_auc
roc_auc = roc_auc_score(y_true=label, y_score=score)
File "/home/victor/victor/lib/python3.8/site-packages/sklearn/metrics/_ranking.py", line 567, in roc_auc_score
return _average_binary_score(
File "/home/victor/victor/lib/python3.8/site-packages/sklearn/metrics/_base.py", line 75, in _average_binary_score
return binary_metric(y_true, y_score, sample_weight=sample_weight)
File "/home/victor/victor/lib/python3.8/site-packages/sklearn/metrics/_ranking.py", line 342, in _binary_roc_auc_score
fpr, tpr, _ = roc_curve(y_true, y_score, sample_weight=sample_weight)
File "/home/victor/victor/lib/python3.8/site-packages/sklearn/metrics/_ranking.py", line 962, in roc_curve
fps, tps, thresholds = _binary_clf_curve(
File "/home/victor/victor/lib/python3.8/site-packages/sklearn/metrics/_ranking.py", line 733, in _binary_clf_curve
check_consistent_length(y_true, y_score, sample_weight)
File "/home/victor/victor/lib/python3.8/site-packages/sklearn/utils/validation.py", line 332, in check_consistent_length
raise ValueError(
ValueError: Found input variables with inconsistent numbers of samples: [16, 1124]

Please tell me how to print the accuracy, AUC (Area Under the ROC Curve), and F1 score.Thanks.

[kayzliu]

I think UPFD dataset is for graph level classification, while DOMINANT is for node level outlier detection. You may try to_graph_score to convert node score to graph score.