Study fingerprint hash collisions

[denis-stepanov]

advent=> select count(hash) from fingerprints;
 count  
--------
 760255
(1 row)

advent=> select count(distinct(hash)) from fingerprints;
 count  
--------
 610168
(1 row)

That's about 20% of hash collisions. This looks high and might contribute to false positives. The task would be to study if these are true collisions or induced ones coming from the fact that Dejavu truncates the hashes before storing. If the latter, maybe the length of the hashes needs to be increased. Note that this would lead to increased CPU consumption and would require re-fingerprinting the entire data set.