Prevent duplicate audits

[jeffpaul]

Description

There appears to be an issue with the Firestore integration locks as we're seeing instances of duplicate audits of the same plugin checksum (in some cases more than two audits are run), so we'll want to review the locking to ensure its catching all edge cases to eliminate this waste of resources. Some examples...
Reply Comment to Email:
https://wptide.org/api/tide/v1/audit/51026
https://wptide.org/api/tide/v1/audit/51027
Feedburner Right Now Stats:
https://wptide.org/api/tide/v1/audit/51020
https://wptide.org/api/tide/v1/audit/51021
https://wptide.org/api/tide/v1/audit/51022

Steps to Reproduce

1. Query for plugin results via checksum
2. See multiple results
3. 💩

Expected behavior: [What you expect to happen]
Only a single audit results for each plugin checksum

Actual behavior: [What actually happens]
Multiple cases of duplicate (or more) audit results for a single plugin checksum

Reproduces how often: [What percentage of the time does it reproduce?]
I haven't run a full population analysis of audit results, but its significant enough to be a concern.

Additional info:
It seems like an issue in Firestore not locking the record fast enough, we might have to switch to Pub/Sub.

[jeffpaul]

Per bugscrub today, we're keeping this in the 1.0.0 release.

[jeffpaul]

Note that this issue was referenced in Slack.

[jeffpaul]

Per discussion in Slack, we're punting this to Future Release as this is technically challenging and may prove to be more effort than we can commit to for 1.0.0.

[valendesigns]

This one is super frustrating and hard to test locally. It's really obvious in a cloud environment where the Kubernetes pods are hitting the message queue so fast that it seems like 1-3 pods pick it up at the near same time.

[jeffpaul]

@rheinardkorf any chance you'd be able to work on a PR for this?

[jeffpaul]

Punting this to Future Release per today's Tidechat discussion.