PSCDB models have poor performance and data seems to be wrong

[Attornado]

Hello Graphein community, I've recently tried to develop some models to classify the protein structures from the PSCDB dataset version provided in this repository, and I've encountered some issues:

• the baselines provided in the documentation use outdated and deprecated functions, and once fixed, appear to have very poor performance (slightly less than 24/25% accuracy, micro-weighted, much less if macro weighted). I also checked out the Graphein paper and apparently the performance of the baselines is similar to the one I found with my experiments. Can anyone confirm what I have said or give me some advice to improve the models on this dataset?

• it seems that some proteins are indeed duplicates in the "structural_rearrangement_data.csv" file, in particolar all the ones having "other_motion" as "motion_type" appear to be duplicated "buried_ligand_motion" proteins, since all the 104 proteins from each of the two classes overlap in every field except the label. For double-checking, I also downloaded the original unprocessed PSCDB data from the official site and found out that the proteins marked as "buried_ligand_motion" are indeed correctly labeled, but all the real 35 proteins from the "other_motion" are missing. Also, I tried to re-execute the notebook that was likely used to generate the "structural_rearrangement_data.csv" file to see if I could maybe fix the issue with the duplicate structures, but it contains outdated and now-taken-down links, so it's completely impossible to execute it. Can anyone give me some clue about this?

[a-r-j]

Hi @Attornado, thanks for the heads up about the deprecations.

We've not played with this dataset a huge amount but poor performance has been consistent. You could try including the per-residue RMSD as node features between bound & unbound (or try to classify pairs of proteins).

Re the data itself: I didn't realise the resource has gone down. That is unfortunate (and all too common :( ) . You might have some luck getting a copy from here

[a-r-j]

It's a small dataset so it might be quite challenging to get good performance. Good sanity checks are overfitting to a batch and the whole dataset.

1. Yep. You could simply get the embeddings for both and concat/sum/aggregate in some way.
2. It should be straightforward but you'll need to align them first. (Kabsch assumes the proteins are of the same length).
3. RMSD can also be computed in biopandas but it should also be straightforward to do so in torch (see this gist)

[Attornado]

@a-r-j thank you very much. Your help is much appreciated. Just two more things:

• what do you exactly mean by "overfitting to a batch"? I mean, I know what overfitting is, but I didn't completely understand what do you mean I should do as sanity check in relation to this kind of overfitting.

• do you know any other graph classification dataset where I can experiment with graphs derived from PDBs (similar to what it is straightforward to do with PSCDB I mean)? I'm asking since for most of the standard benchmark datasets like D&D and ENZYMES it seems difficult to access the actual protein 3D structures as PDB/CIF file, since most of the versions I was able to find online contain indeed the graphs and respective lables, but with no reference to the protein they are derived from, let alone the pdb code.

[a-r-j]

• The sanity check is just to make sure everything is wired up properly wrt data pipeline, model and training loop.

• Yep, you can use the EC, GO and Fold classification dataset. If you search around you can find them, they're quite commonly used. As a starting point: https://github.com/phermosilla/IEConv_proteins/tree/master/Datasets/data

[Attornado]

@a-r-j thank you again, very much! Sorry if I'm going to repeat myself, but I do really appreciate when an author is so dedicated to his project to almost instantly reply to any issue a random guy posts on its repo.

As for the sanity check, I think I got what you meant. If I understand correctly, you mean that, in order to evaluate if everything's ok with the model/data pipeline I should check if my model is able to overfit the dataset, which is a common practice in my ML projects actually, but initially I didn't get what you meant exactly. Thank you again a thousand times.

[a-r-j]

Happy to help 😀

Do reach out if you run into any issues :)