the ndcg&map score are too small when using torch_geometric.metrics

[xiche0525]

According to the example of torch_geometric/metrics/link_pred.py and test/metrics/test_link_pred_metric.py, I write the code below

@torch.no_grad()
def test(data):
    model.eval()
    z = model(data.x, data.edge_index)
    out = (z[data.edge_label_index[0]] * z[data.edge_label_index[1]]).sum(-1).view(-1).sigmoid()
    AUC_score = roc_auc_score(data.edge_label.cpu().numpy(), out.cpu().numpy())
    out1 = z @ z.t()
    out_top_k = out1.topk(2, dim=1)[1]
    ground_truth_len = int(0.5 * data.edge_label.size(0))
    postive_edge_index = data.edge_label_index[:, :ground_truth_len]
    source_node, target_node = postive_edge_index
    arange = torch.empty(data.num_nodes,dtype=source_node.dtype)
    arange[source_node] = torch.arange(source_node.numel())
    y_source = arange[source_node]
    metric_MAP = LinkPredMAP(k=2)
    metric_MAP.update(out_top_k[source_node], (y_source, target_node))
    MAP_score = metric_MAP.compute()
    metric_NDCG = LinkPredNDCG(k=2)
    metric_NDCG.update(out_top_k[source_node], (y_source, target_node))
    NDCG_score = metric_NDCG.compute()
    return AUC_score,MAP_score,NDCG_score

The question is why two score too small.
next is supplementary information, the data is divided by

linkspilt = RandomLinkSplit(num_val=0.1, num_test=0.1, is_undirected=True,add_negative_train_samples=False)
print("data",data)
data = T.NormalizeFeatures()(data)
train_data, val_data, test_data = linkspilt(data)

The statistics about the data are

And I use this function in

best_val_auc = final_test_auc = 0
for epoch in range(1, 101):
    loss = train()
    val_auc, val_map, val_ndcg = test(val_data)
    test_auc, test_map, test_ndcg = test(test_data)
    if val_auc > best_val_auc:
        best_val_auc = val_auc
        final_test_auc = test_auc
    print(f'Epoch: {epoch:03d}, Loss: {loss:.4f}, Val_AUC: {val_auc:.4f},Val_MAP: {val_map:.4f}, Val_NDCG: {val_ndcg:.4f},'
          f'Test_AUC: {test_auc:.4f},Test_MAP: {test_map:.4f}, Test_NDCG: {test_ndcg:.4f}')

The console output is

The questions are that I do not know whether I write wrong or not and what can I do to enhance both score.
But when I use the methods in sklearn to calculate ndcg and map, it works fine.

@torch.no_grad()
def test(data, epoch):
    model.eval()
    z = model(data.x, data.edge_index)
    out = (z[data.edge_label_index[0]] * z[data.edge_label_index[1]]).sum(-1).view(-1).sigmoid()
    AUC_score = roc_auc_score(data.edge_label.cpu().numpy(), out.cpu().numpy())

    NDCG_score = ndcg_score([data.edge_label.numpy()], [out.numpy()], k=5)

    MAP_score = average_precision_score(data.edge_label.numpy(), out.numpy())

    return AUC_score,MAP_score,NDCG_score

[rusty1s]

The two metrics are not necessarily comparable. In the sklearn example, you have a fixed postive and negative set, while in the LinkPredMetrics example you are comparing against all possible negative edges.

[xiche0525]

Thank you for your reply！Can I ask more detailed explanation?
I'm sorry for bothering you, but I already make sure that the second parameter of the '''update()''' function only include positive edges, which implement by this:
'''
ground_truth_len = int(0.5 * data.edge_label.size(0))
postive_edge_index = data.edge_label_index[:, :ground_truth_len]
source_node, target_node = postive_edge_index
'''

[rusty1s]

Yes, this part of the code looks totally fine.