Add new index for the unique constraint on imapuid

[drewler]

Queries such as this one and this one have a high impact on IO, related to the elevated number of messages returned plus the data per row (as all the columns are fetched).

This situation could be mitigated by using a covering index on ("account_id", "folder_id", "msg_uid"). Thing is, there's an index there already because of the UniqueContraint which creates an index behind the scenes... just in the wrong order. By reordering it and leaving msg_uid as the last column on the index, the unique constraint should double as a covering index resulting in a significative reduction in IO (around 75%).

This first migrations creates a new unique constraint (plus related index) using the new ordering. After that we'll use a second one to drop the old constraint.

[squeaky-pl]

This is on hold for now, commenting so the bot does not remind me every day.

[wojcikstefan]

Is there a particular action item or some other thing that needs to happen before we can revisit this PR @drewler @squeaky-pl?

[drewler]

I don't have the screenshot showing the difference for the cluster where these changes had been applied, but comparing that cluster with another one with a similar load today, the number of rows read dropped by ~33%. One MAX query (before, the third worst performing query) is now instantaneous thanks to the index. As a result, IO usage drops and query latency improves. We didn't complete the upgrade because:

• There was another query that should have seen its performance improved because of this index, but we couldn't notice any difference and didn't understand why.
• Either for the query that did improve or for the one that didn't improve, we couldn't show any meaningful impact. In the end, we had started investigating this because of an alert related to CPU usage. While looking into that, we found about this IO performance issue. While performance improved, we couldn't demonstrate a meaningful impact so this has been kept in the freezer until we could investigate some more.

Guess we should either revert to the old index (we put the new one in place in one of the clusters) or update it for all clusters.

Query that improved:

SELECT MAX(imapuid.msg_uid)
FROM imapuid
WHERE imapuid.account_id =...
    AND imapuid.folder_id =...

Query that didn't improve:

-- we also tried FORCE INDEX on this one
SELECT imapuid.msg_uid
FROM imapuid
WHERE imapuid.account_id =...
    AND imapuid.folder_id =...

@squeaky-pl feel free to expand my takes (or correct them if there's something wrong).