**WIP** feat(cass): add shardKey to partKey table #1361
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alextheimer
force-pushed
the
ns-mig
branch
2 times, most recently
from
5aa7797
to
69ab230
Compare
alextheimer
commented
Apr 9, 2022
Comment on lines
+551
to
+562
| def records(ds: Dataset, readerSeq: Seq[RowReader]): SomeData = { | ||
| val builder = new RecordBuilder(MemFactory.onHeapFactory) | ||
| readerSeq.foreach { row => builder.addFromReader(row, ds.schema) } | ||
| builder.allContainers.zipWithIndex.map { case (container, i) => SomeData(container, i) }.head | ||
| } | ||
|
|
||
| def partKeyFromRecords(ds: Dataset, records: SomeData, builder: Option[RecordBuilder] = None): Seq[Long] = { | ||
| val partKeyBuilder = builder.getOrElse(new RecordBuilder(TestData.nativeMem)) | ||
| records.records.map { case (base, offset) => | ||
| ds.comparator.buildPartKeyFromIngest(base, offset, partKeyBuilder) | ||
| }.toVector | ||
| } |
There was a problem hiding this comment.
(Almost) mirror the same methods in GdeltTestData.
alextheimer
commented
Apr 9, 2022
Comment on lines
86
to
+91
| val partKeyTablesInit = Observable.fromIterable(0.until(numShards)).map { s => | ||
| getOrCreatePartitionKeysTable(dataset, s) | ||
| }.mapEval(t => Task.fromFuture(t.initialize())).toListL | ||
| val shardKeyToPartKeyTableInit = Observable.fromIterable(0.until(numShards)).map { s => | ||
| getOrCreateShardKeyToPartKeyTable(dataset, s) | ||
| }.mapEval(t => Task.fromFuture(t.initialize())).toListL |
There was a problem hiding this comment.
Every PartKeyTable init/insert/delete has a counterpart ShardKeyToPartKeyTable call.
alextheimer
commented
Apr 9, 2022
Comment on lines
+576
to
+577
| val shardKeyToPartKeyTableCache = concurrentCache[DatasetRef, | ||
| ConcurrentLinkedHashMap[Int, ShardKeyToPartKeyTable]](tableCacheSize) |
There was a problem hiding this comment.
Should there be a table per shard (like PartitionKeysTable)?
alextheimer
commented
Apr 11, 2022
| def scanPartKeysByShardKey(ref: DatasetRef, shard: Int, shardKey: Array[Byte]): Observable[Array[Byte]] = { | ||
| val table = getOrCreateShardKeyToPartKeyTable(ref, shard) | ||
| table.scanPartKeys(shardKey) | ||
| // TODO(a_theimer): figure out if token ranges apply here |
There was a problem hiding this comment.
These will only apply if we add extra values to the PK to prevent hot-spotting, right?
alextheimer
commented
Apr 11, 2022
|
|
||
| val suffix = s"shardKeyToPartKey_$shard" | ||
|
|
||
| // TODO(a_theimer): compression settings okay? |
There was a problem hiding this comment.
Left the same as PartitionKeyTable
alextheimer
commented
Apr 11, 2022
Comment on lines
+22
to
+23
| // TODO(a_theimer): probably need to prevent hot-spotting on a single node | ||
| // (i.e. add more fields to [Cassandra's] partition key) |
There was a problem hiding this comment.
I won't do this until benchmark tests indicate we need it (or one of you already knows we'll need it).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds a Cassandra table that maps shard keys to partition keys.
Now, all time-series partition keys for a given shard key can be efficiently scanned.