Sharded compaction

[mdisibio]

What this PR does:
This is the largest change to compaction in years. It is a new sharding compactor that splits blocks up by trace ID divisions. For example: instead of having 2 blocks with the entire range of trace IDs 00..FF, the sharding compactor can combine and split them into 2 blocks with the ranges 00..80 and 80..FF where each block contains half of the range it did before.

There are several benefits from this approach:

• The read path is more efficient. Trace lookup can filter out more blocks faster by checking the Min/Max ID in the meta without having to resort to the bloom filter. Metrics queries inspect fewer blocks to find the traces in the job shard.
• More effective deduplication of data in the backend. Because the sharding compactor clusters traces with similar values together, it is more effective at finding and deduplicating the copies of a trace due to replication factor. A recent internal analysis measured that compactors were only bringing the replication factor from 3 down to ~2.5. The sharding compactor should achieve ~1.5 or better.

The shard count is configurable globally and per-tenant. Setting to 2 or higher activates the new sharding compactor. 0 or 1 will fallback to the existing compactor. Expecting useful values to be 2 through 8. Values higher than 8 could be interesting, but at the cost of greatly increasing the size of the blocklist.

Description
How the sharding compactor works can be considered as 3 phases.

1. The first phase is taking fresh blocks from the ingesters and splitting them. The ingesters haven't been affected in this PR, and continue to flush blocks with the full range 00..FF. The sharding compactor identifies them by looking for blocks with CompactionLevel=0 and will split them. This is a combination of combine/split like described at the time, so that up to 4 (maxInputBlocks) ingester blocks will be rewritten as N sharded blocks. This step feeds work to the next step:
2. The second phase is the typical compaction/reduction of sharded blocks with other blocks in the same shard. This is same as the existing compaction, but shard-aware. The compactor identifies them by CompactionLevel>0 and the block min/max trace IDs fall within the same shard (i.e. the block is "well-sharded")
3. A third phase exists that will upgrade/resplit older blocks too, but it is the lowest priority and only when the first 2 phases have no outstanding work.

Which issue(s) this PR fixes:
Fixes n/a

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[mdisibio]

This fixes a "memory leak" where the buffer wasn't returned to the pool. This actually has a significant effect on memory because this isn't happening once at the end, but on every loop of the multiblock iterator. Once 1 block is exhausted it begins leaking a pooled buffer on every call. The better fix is to make the multiblock iterator stop calling Next() once an iter is exhausted, but it was going to be a lot more involved.

[mapno]

Nice work!

[knylander-grafana]

Thank you for adding some information to the configuration docs. Will we need additional information in the docs that talks about when to use this ocnfiguration?

[joe-elliott]

long overdue work. looks great. added some Qs but fine with things as is.

if you need my approval please ask. the only reason i'm not approving is b/c you and @mapno have been handling these PRs

[joe-elliott]

won't this adjust the original "currentBlock" since they point to the same thing?

[mdisibio]

Hmm this block was moved from the bottom of the loop to the top so that the CutBlock callback can inspect the next trace (lowestID). You're right it does seem to be out of date and the copy is no longer needed. I'll take a look.

[mdisibio]

Hi everyone: Putting this back to draft because it is not yet ready for large scale. Testing it out in a large cluster is showing it difficult to achieve the balance between splitting and combining blocks, even with tuning. Would like to review the switch statement and priorities of each group of blocks. When there is an imbalance the result is that compactors are less effective and the blocklist increases more than acceptable: i.e. If the compactor spends too much time splitting blocks then the backend is composed of (too) many small blocks, and if the compactor spends too much combining, then ingester level-0 blocks are neglected.