Welcome to CleverCSV Discussions!

[GjjvdBurg]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about in the Q&A section
• Share ideas about improvements to CleverCSV in the Ideas section
• Tell us about use cases where CleverCSV was helpful to you in the Show & Tell section
• Engage with other community members: if you know the answer to a question posted by someone else, please help them out!
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪. Please keep the code of conduct in mind.

[BoboOpenSource]

Dear Gertjan van den Burg,After looking through the CleverCSV code, I have some points that I do not quite understand and hope you can help answer. In the detect_pattern.py pattern_score() method, I understand the meanings of k, K, Nk, and Lk but do not understand the formula for calculating P. For example, what is the reason for using (Lk - 1)/ Lk? Why do we sum the score for each pattern and then divide by K?

[GjjvdBurg]

Hi @BoboOpenSource, thanks for your question.. We use (Lk - 1)/Lk because it weighs longer row patterns more than shorter row patterns, while remaining bounded to [0, 1]. This way, it's a weight for the frequency of a row pattern (Nk). Summing the score for each pattern and then dividing by the number of patterns (K) is simply taking the average. So we can think about the pattern score for a dialect as the average of the "pattern weights" that it results in, where a pattern weight is a combination of both the pattern's frequency (Nk) and it's length ((Lk - 1)/Lk).

Ultimately the formula is a heuristic, you can find more details in Section 4.1 of our paper.

[BoboOpenSource]

So that's it. I understand. Thank you very much for your answer