New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Gaussian Mixture based adaptive threshold #1051
Open
yujiepan-work
wants to merge
7
commits into
openvinotoolkit:main
Choose a base branch
from
WenjingKangIntel:adaptive-threshold
base: main
Could not load branches
Branch not found: {{ refName }}
Could not load tags
Nothing to show
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Add Gaussian Mixture based adaptive threshold #1051
yujiepan-work
wants to merge
7
commits into
openvinotoolkit:main
from
WenjingKangIntel:adaptive-threshold
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yujiepan-work
changed the title
GMM estimator based adaptive threshold
Gaussian Mixture estimator based adaptive threshold
Apr 26, 2023
yujiepan-work
force-pushed
the
adaptive-threshold
branch
from
April 27, 2023 16:16
6c3b47d
to
d75524f
Compare
yujiepan-work
requested review from
samet-akcay,
ashwinvaidya17 and
djdameln
as code owners
April 27, 2023 16:49
yujiepan-work
force-pushed
the
adaptive-threshold
branch
2 times, most recently
from
April 27, 2023 17:01
16e5732
to
d75524f
Compare
Signed-off-by: Pan, Yujie <yujie.pan@intel.com>
Signed-off-by: Pan, Yujie <yujie.pan@intel.com>
Signed-off-by: Pan, Yujie <yujie.pan@intel.com>
Signed-off-by: Pan, Yujie <yujie.pan@intel.com>
Signed-off-by: Pan, Yujie <yujie.pan@intel.com>
yujiepan-work
force-pushed
the
adaptive-threshold
branch
from
April 27, 2023 17:02
d75524f
to
3d93965
Compare
Hi all, this PR by anomalib_Team3 implements another threshold estimator, which serves as a solution to #1027 |
Signed-off-by: Pan, Yujie <yujie.pan@intel.com>
yujiepan-work
changed the title
Gaussian Mixture estimator based adaptive threshold
Gaussian Mixture based adaptive threshold
Apr 27, 2023
yujiepan-work
changed the title
Gaussian Mixture based adaptive threshold
Add Gaussian Mixture based adaptive threshold
Apr 27, 2023
mvtec result comparisonImage f1 when using "same_as_test" for validation"Adaptive threshold" can be regarded as the "upper bound", and we see GMM is close to that.
Image f1 when using "synthetic" data for validationUses random Perlin noise masks already in Anomalib.
|
Signed-off-by: Pan, Yujie <yujie.pan@intel.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR implements a new adaptive threshold based on Gaussian Mixture Model (GMM).
The problems of existing
AnomalyScoreThreshold
and our corresponding solutions:AnomalyScoreThreshold
can only propose limited number of candidate thresholds. For example, when the validation scores are [1,2,3] for normal and [4,5,6] for anomalous, it can only propose 1,2,3,4,5,6 as thresholds. However, it is intuitive to say that 3.5 might be a better choice. To solve this, we use GMM to estimate the full distribution of normal/abnormal scores. As such, we can calculate f1 at any threshold t by cumulative distribution function (CDF), and then the optimal threshold can be found:FP = (1 - CDF_normal(t) ) * normal_rate
TP = (1 - CDF_anomalous(t) ) * anomalous_rate
FN = CDF_anomalous(t) * anomalous_rate
f1_scores = (TP * 2) / (TP * 2 + FP + FN)
AnomalyScoreThreshold
cannot handle real anomalous rate. We might want to generate more anomalous data to help estimate the anomalous score distribution, even though in real cases the anomalous rate cannot be that high. As shown in the above formula, anomalous_rate can affect the f1 score. To simplify this, we allow users to decide an anticipated "anomalous rate" when initializingGMMthreshold
. Then, regardless of how many anomalous data are generated, it will decide the optimal threshold with pre-defined "anomalous rate".In summary, this new method computes threshold by:
Notes
sklearn.neighbous.KernelDensity
) internally instead of GMM, but it is slower than GMM. We can discuss if other density estimators are helpful.AnomalyScoreThreshold
needs to support other threshold methods. I notice other PR dealing with that. For demo purpose, I have implemented a workaround in this PR.Changes
Checklist