[admin-tool] Allow admin tool to dump kafka topics have no store vers… #929
Conversation
…ion in Venice Today KafkaTopicDumper does not allow dump kafka topics that don't have a corresponding venice store version. For debugging purpose, dumping such kafka topics might still be useful. This change relaxes such condition and report a warning message when it happens.
| this.isChunkingEnabled = storeInfo.getVersion(version).get().isChunkingEnabled(); | ||
| } else { | ||
| // Use the default chunking setting from store if the version does not exist. | ||
| this.isChunkingEnabled = storeInfo.isChunkingEnabled(); |
There was a problem hiding this comment.
This could be wrong as the chunking flag in store level can be different from the version level.
I wonder whether we can dynamically infer whether the chunking is enabled or not as when chunking is enabled, when chunking is enabled, regular key will have the NON_CHUNK_KEY_SUFFIX, so I guess we can try to compare the tailing part of key is same as the serialized bytes of NON_CHUNK_KEY_SUFFIX or not.
Another more complex way is to write the chunking flag into SOP message, which would need to evolve KME schema, and we don't have to do that for this debugging purpose.
There was a problem hiding this comment.
Or, I am wondering can we make every store/version chunking + rmd chunking enabled? For <1MB data, that's not going to change anything, is that right?
There was a problem hiding this comment.
chunking has some overhead as it will append some chunking suffix into the key, and in theory, we don't want to enable chunking by default as the large value is not expected in the regular stores as it would slow down the read path significantly.
…ion in Venice
Today KafkaTopicDumper does not allow dump kafka topics that don't have a corresponding venice store version. For debugging purpose, dumping such kafka topics might still be useful. This change relaxes such condition and report a warning message when it happens.
How was this PR tested?
Does this PR introduce any user-facing changes?