Converted PLSC to hierarchical

[x-tabdeveloping]

Checklist for adding MMTEB dataset

Reason for dataset addition:
Converted both PLSC tasks (S2S, P2P) to hierarchical clustering. #702

• I have tested that the dataset runs with the mteb package.
• I have run the following models on the task (adding the results to the pr). These can be run using the mteb run -m {model_name} -t {task_name} command.
  • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
  • intfloat/multilingual-e5-small
• I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
• If the dataset is too big (e.g. >2048 examples), considering using self.stratified_subsampling() under dataset_transform()
• I have filled out the metadata object in the dataset file (find documentation on it here).
• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.
• I have added points for my submission to the points folder using the PR number as the filename (e.g. 438.jsonl).

[x-tabdeveloping]

The later levels seem very hard. Maybe we should limit the levels to two?

[x-tabdeveloping]

I'm not sure whether the way I formulated the task makes sense.
@rafalposwiata You added the dataset initially, therefore you might know: Is the "disciplines" column hierarchically ordered or just multilabel? Or could "scientific_fields" be used as the first level and "disciplines" as the second? What's your take on this?

[KennethEnevoldsen]

Looks good

[KennethEnevoldsen]

I would cut from here and down

[rafalposwiata]

Is the "disciplines" column hierarchically ordered or just multilabel?

Disciplines are multilabel but for the added clustering tasks I chose only those cases where there is one discipline.

Or could "scientific_fields" be used as the first level and "disciplines" as the second?

Yes, "scientific_fields" could be used as the first level and "disciplines" as the second.

The entire dataset is available at https://huggingface.co/datasets/rafalposwiata/plsc

[KennethEnevoldsen]

@x-tabdeveloping will you add points for this then I believe it is ready to merge

[x-tabdeveloping]

I'm not sure though. The task formulation might be wrong. I think doing "scientific_fields" as first level and "disciplines" as the second might be the way to go.
From what I've gathered it seems that this is just multilabel, not hierarchical the way I formulated it, right @rafalposwiata ?

[KennethEnevoldsen]

@x-tabdeveloping but the current approach is fine with that right? As I understand it is just does the clustering at each level?

[x-tabdeveloping]

Yes, unless the order is not fixed, and I don't know if it is (we have to check)

[KennethEnevoldsen]

Right. Once checked we can either close or merge

[x-tabdeveloping]

Nope, it's not hierarchical at all. We can maybe rephrase it as multilabel classification if we really want to, otherwise fine to leave it as flat clustering.

[KennethEnevoldsen]

Let us leave it as flat clustering