Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Analysis - Adjust generalization and specificity metrics #2212

Open
jadie1 opened this issue Mar 5, 2024 · 0 comments
Open

Analysis - Adjust generalization and specificity metrics #2212

jadie1 opened this issue Mar 5, 2024 · 0 comments

Comments

@jadie1
Copy link
Contributor

jadie1 commented Mar 5, 2024

Currently, the generalization is measured using the distance between true and reconstructed unseen particles, and specificity is measured using the distance between sampled particles and closest real particles.

The problem with this is it is dependent on the accuracy of correspondence. When the SSM has miscorrespondences they may be incorrectly captured in PCA as variations that are actually present in the training data. This can lead to a model that, when generating new samples, produces shapes that are closer to these "noisy" versions of the training shapes, yielding good specificity when the sampled shapes do not actually resemble the shape population.

To remedy this, generalization and specificity should not be quantified as distance between particles, but rather as distance between particles and the ground truth shape/surface. Meaning generalization is the distance between reconstructed particles and the true shape surface and specificity is the distance between sampled particles and the surface. This would provide more consistent metrics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants