[Enhancement] Support for upgrading metadata in json to rocksdb

[LetLetMe]

Before Creating the Enhancement Request

• I have confirmed that this should be classified as an enhancement rather than a bug/feature.

Summary

我在看目前开源版rocksdb方案topic和group元数据相关的代码时，发现了一个问题：目前开源版Rocketmq是不支持json版本升级到rocksdb版本的，这会导致存量用户无法享受到rocksdb带来的额外好处，为了支持这个升级，我们准备分俩步来做，首先是元数据的升级：

• 为rocksdb版本的元数据增加独立的DataVersion，记录数据版本的同时也能方便数据清洗时的进度记录
• 增加json到rocksdb的数据清理逻辑，支持json版本原地升级到rocksdb版本

When we were looking at the code related to topic and group metadata in the current open-source version of RocksDB solution, we found an issue: the current open-source version of Rocketmq does not support upgrading from JSON version to RocksDB version. This will result in existing users not being able to enjoy the additional benefits brought by RocksDB. To support this upgrade, we will do it in two parts. First is the upgrade of metadata:

• Add a separate DataVersion for the RocksDB version of metadata, which not only records the data version but also facilitates progress tracking during data cleaning.
• Add logic for cleaning data from JSON to RocksDB, supporting an in-place upgrade from JSON version to RocksDB version.

Motivation

现如今老版本json方案升级到rocksdb版本时候，缺乏数据清洗逻辑和数据版本号记录

Nowadays, when upgrading from the old JSON scheme to the RocksDB version, there is a lack of data cleaning logic and data version number recording during data migration.

Describe the Solution You'd Like

在现在的rocksdb方案中，topic和group分别使用了独立的目录，数据存储在相应目录的Default列簇中，我为他们每个额外添加了一个列簇，名字为kvDataVersion，里边分别只会有一个键值对，key为kvDataVersion，value为json序列化以后得DataVersion对象，并且我们在rocksdb版本topic和group的configManager中增加了数据清洗逻辑

对于这块的测试，我们提供了独立的单测类：

• RocksdbTopicConfigTransferTest
• RocksdbGroupConfigTransferTest
  对于测试场景我们需要满足如下要求：
• 新部署后 DataVersion为0，即使存在着初始化的一些topic和group资源，这些初始化动作并不会增加DataVersion
• 添加资源并且重启后，可以读到被持久化的资源
• json版本升级到Rocksdb版本，会且仅会触发一次marge，且保证DataVersion正确

In the current rocksdb solution, the topic and group each use separate directories. Data is stored in the Default column family of the respective directory. I have added an additional column family for each of them, named kvDataVersion. Inside, there will only be one key-value pair, where the key is kvDataVersion and the value is the DataVersion object after JSON serialization. Additionally, we have added data cleansing logic in the configManager of the rocksdb version for topic and group.

For this test, we have provided separate unit test classes:

• RocksdbTopicConfigTransferTest
• RocksdbGroupConfigTransferTest
  For the test scenarios, we need to meet the following requirements:
• After a new deployment, DataVersion is 0, even if there are some initialized topic and group resources, these initialization actions will not increase DataVersion
• Adding resources and restarting should allow reading of persisted resources
• Upgrading from json version to Rocksdb version will trigger a merge only once, ensuring DataVersion is correct

Describe Alternatives You've Considered

无
nothing

Additional Context

关联的issue #7064

Associated issue #7064

[lizhimins]

rocksdb 为我们提供了性能更好的元数据管理能力，从旧实现平滑迁移到新实现的过渡也是很重要的。