Wanted to get the basic idea of how scoring works in analyzer

[dristysrivastava]

Wanted to get the basic idea of how scoring works in analyzer, like how based on regex pattern score is given,
let us talk about an example of us_passport_recognizer

It has different regex, and for each regex, different scores are there

Pattern("Passport (very weak)", r"(\b[0-9]{9}\b)", 0.05),
Pattern("Passport Next Generation (very weak)", r"(\b[A-Z][0-9]{8}\b)", 0.1),

So basically wanted to get an idea of how these scoring is decided.

[dristysrivastava]

@omri374 Can you help me with this

[omri374]

Hi, the first thing I would suggest is it to call presidio with return_decision_process=True. This gives a detailed response of the scores at each step.
In high level, this is the process, but it can change depending on the recognizer and applies only for pattern based (regex) recognizers:

1. A substring is matched with a regex pattern, the regex pattern score is assigned. Very weak regex patterns usually have a score of 0.01.
2. In case there is a validation method, then the score increases to the validation score, defined in EntityRecognizer.MAX_SCORE=1.0. Invalidation reduces the score to 0.
3. Then, the context process starts and updates the score based on the context logic. By default (with the LemmaContextAwareEnhancer), the score increases by context_similarity_factor=0.35, but can be changed by the user), and is bounded to a minimum of min_score_with_context_similarity=0.4 which can also be changed by the user.

For non-pattern based recognizers, the score is given by the model/logic within the recognizer, and can be boosted by context similar to the context mechanism in the pattern based recognizers.

[dristysrivastava]

@omri374 So my question is:

Pattern("Passport (very weak)", r"(\b[0-9]{9}\b)", 0.05),
Pattern("Passport Next Generation (very weak)", r"(\b[A-Z][0-9]{8}\b)", 0.1),

How we have defined 0.05 and 0.1 as a score, is there any logic behind it, or we are predicting it randomly

[omri374]

The best practice is to test this pattern on a dataset, and evaluate false positives/negatives. We have some documentation on this here: https://microsoft.github.io/presidio/analyzer/developing_recognizers/#accuracy