Gravitino 0.5.0 Latest

Gravitino Release 0.5.0

We are glad to announce the release of Gravitino 0.5.0. This release is a major milestone for Gravitino and includes over 240 issues that cover several new features, improvements, and bug fixes.

This release introduces several core features like Apache Spark connector support, messaging catalog support, general user and authority management system, event listener system, and Python client support. In the meantime, we have made a lot of improvements and bug fixes to the existing features.

Core Features

1. Apache Spark connector support: Gravitino now supports Spark connector. You can use Spark to read and write catalog metadata through Gravitino. #1286, for more, refer to spark-connector

2. Messaging catalog support: Gravitino now supports messaging catalogs such as Apache Kafka or Kafka-compatible streaming systems. You can use Messaging Catalog to manage your messaging catalog. #2369, for more, please refer to kafka-catalog

3. General user and access management: Gravitino now supports general user and access management. #2232. This feature is currently in an alpha phase and is not available for production usage.

4. Event listener system: Gravitino now supports an event listener system. You can use it to manage all operation events or use the hook mechanism for your own events, such as operation history auditing, operation monitoring, etc. etc. #2233, refer to event-listener-configuration for more information

5. Python client support: Gravitino now supports a Python client. Users can use Python to connect to Gravitino and operate the catalog directly. #2229. Currently, we only support fileset type catalogs with Python clients.

6. Doris catalog support: Gravitino now supports Apache Doris catalogs. #1339, for more information, you can refer to jdbc-doris-catalog

7. Support JDBC backend store: Gravitino now supports using a JDBC backend store besides RocksDB. If you want to use MySQL or PostgreSQL as the entity store, you can use the JDBC entity store. #1811, for more, refer to storage-configuration

8. Support fileset catalog: Gravitino now supports Fileset catalogs. with it, you can manage non-tabular data on HDFS, S3, or other Hadoop-compatible filesystems. #1241, for more, refer to fileset catalog

9. Introduce capability framework: This framework shows the capability of different catalogs such as whether the name is case sensitive, name specification, whether null values are supported or not, and so on. #2952

Gravitino core

• Handling multi-thread issues in Gravitino and introducing tree lock. #407
• Introduces the user system. #2232
• Support multiple kinds of entity types in the same namespace with the same name. #2697
• Improve KV GC collector. #1276, #2888
• Improve client API #1292, #2628, #839, #1635, #1793, #1759, #1758
• Separate Java client. #2478
• Make class loaders of catalog able to be GC. #2706
• Support UnparedType to handle an unresolvable type from the catalog. #2117

Catalog related

MySQL & PostgreSQL

• Add PostgreSQL support for array type conversion. #947
• Obtain MySQL table meta information from JDBC metadata. #2934
• Avoid using system tables for MySQL catalogs. #2085

Kafka

• Please see the section Core Features

Fileset

• Please see the section Core Features

Doris

• Please see the section Core Features

Trino connector

• Add datatype test cases for the Trino connector. #2034
• Optimize varchar/char mapping between Gravitino catalogs and the Trino server. #2356
• Support the system table catalog. #2416
• Support update catalog operations in the Trino connector. #2417
• Make the Gravitino Trino connector compatible with Trino 435. #2376

Spark connector

• Please see the section Core Features

Build, test, and CI

• Introduce the error-prone plugin to check the code quality. #2225
• Increase the retry interval of the container status check. #2365
• Isolate catalog class path in IT. #2397
• Extend sleep time in the testInternalCache unit test. #2745
• Add web UI support for the fileset catalog. #2883
• Add check mechanism in CI to validate gradle publish. #2655
• More tests added in the Gravitino web E2E test framework. #1503
• Separate output log of test containers. #2839
• Upload process logs of IT container. #2832
• Merging embedded and deploy test mode for frontend integration test. #2798

Web UI

• Verify whether the catalog exists before creating it. #2324
• Add web UI support for the Kafka catalog. #2614
• Add web UI support for the Fileset catalog. #2292

Documents

• Separate metadata operations into different docs. #2750
• Add a document about how to debug the Gravitino Trino connector locally. #2446

Limitation and known issues

• The Doris catalog does not support features like sort order, distribution, and partitioning, and these features are under development.

Credits

@bknbkn @caican00 @charliecheng630 @ch3yne @coolderli @Clearvive @danhuawang @diqiu50 @FANNG1 @hiirrxnn @ichuniq @jerryshao @justinmclean @lw-yang @Lanznx @LauraXia123 @mchades @MohitKambli @nk1506 @qqqttt123 @shaofengshi @SteNicholas @TEOTEO520 @unknowntpo @xiacongling @xiaozcy @xloya @xunliu @yijhenlin @yuqi1129 @Yangxuhao123 @YxAc @zhaoyongjie @zhoukangcn @zivali

Gravitino 0.4.0

Gravitino Release 0.4.0

Gravitino 0.4.0 is the fourth release since its inception and the second release since becoming open-source. With substantial contributions from the open-source community, this release has addressed over 280 issues.

This release is a stable release that mainly focuses on bug fixes. In the meantime, this release also introduces several features, like the public preview of Gravitino web UI, the support of unified partition management, the column default value and indexing support for table, the Kerberos support for client-server communication, the user impersonation support, the Kerberos support of hive catalog, the multiple Java version support, etc.

To download Gravitino 0.4.0, please visit the releases pages, For detailed changes, you can visit Gravitino milestone 0.4.0 for more details. We have also curated a list of high-level changes here, grouped by major modules.

Highlights

• Add the UI support for Gravitino #605
• Add partition support for Gravitino #1575
• Support default value for column #896
• Support auto increment property for column #1277
• Support index for table #1398 #1707 #1736
• Support Kerberos for client-server communication #1185 #1618
• Support user impersonation #596
• Support Kerberized Hive metastore #1415
• Support Java 8, 11 and 17 for Gravitino #1098
• Introduce fine-grained lock support in Gravitino #778
• Support partitioning, bucketing, sorting for Trino Gravitino connector #1510
• Support query optimization in Trino connector #1491

Gravitino core

• Improve the ConfigEntry to support configuration without default value #950
• Add CORS filter for Jetty server Support Java 8, 11 and 17 for Gravitino #42
• Support auto increment property for column #1277
• Ensure the correct semantics of dropTable and purgeTable #1436
• Support index for table #1398
• Support default value for column #896
• Add partition support for Gravitino #1575
• Support Kerberos for client-server communication #1185 #1618
• Introduce fine-grained lock support in Gravitino #778
• Refine Literal APIs #1379

Catalogs

Hive

• Add partition support for Hive #1575
• Add user impersonation support for Hive catalog #596
• Support Kerberized Hive metastore #1451

Iceberg

• Add the reigster IcebergCatalogConfig abstract class #874
• Support Distribution for Iceberg #1460
• Add the basic framework to store Iceberg metrics #1187
• Support struct column for Iceberg #1313 #1471

MySQL & PostgreSQL

• Support auto increment property for column #1388 #1279
• Support PostgreSQL index #1736
• Support MySQL index #1707
• Refactor to get metadata from JDBC #1737 #1468
• Add column default value support for MySQL and PostgreSQL #1531

Trino connector

• Add the Trino TPC-H and TPC-DS test sets to the Trino connector integration test #1291
• Optimize Trino connector's property mapping system #1455
• Create docker-compose container for Trino test #919
• Support partitioning, bucketing, sorting for Trino Gravitino connector #1512
• Support query optimization in Trino connector #1491

UI

• Add the UI support for Gravitino #605

Build and others

• Remove sequence dependencies in test with annotation Order improvement #995
• Support Java 8, 11 and 17 for Gravitino #1098
• Add multiple MySQL and PostgreSQL version support in integration test #1367 #1361
• Upgrade Jersey version to 2.41 #1756
• Upgrade Guava version 32.1.3-jre #1755
• Add test framework for web UI #1694
• Support query optimization in Trino connector #1491
• Pass the Trino-connector attributes when creating a Graviton catalog #1688

Fixed known CVEs

• Upgrade Jersey version to 2.41 #1756
• Upgrade Guava version 32.1.3-jre #1755

Know issues

• Varchar type of PG catalog is not handled correctly by Gravitino #2035
• OAuth mode, authentication will expire after a few minutes #2099
• Gravitino web UI has issues on Safari #2100

Limitation:

• Audit information is empty for MySQL schema #1767

Credits

@abant07 @amananand20 @austin362667 @brandboat @BWbwchen @ch3yne @Clearvive @coolderli @cottage14 @cxzl25 @diqiu50 @FANNG1 @FrankYang0529 @hiirrxnn @jerryshao @justinmclean @mchades @NIRMALraki @pan3793 @qqqttt123 @SophieTech88 @SteNicholas @TEOTEO520 @xnge @xunliu @yuqi1129 @YxAc @724thomas

Gravitino 0.3.1

Gravitino 0.3.1 is a maintenance release on branch-0.3, which fixes a bunch of UI, documentation and JDBC catalog issues.

For the details, please see the detailed change list below.

What's Changed

• [Minor] Minor formating and changes to fit with the style guide. by @github-actions in #1127
• [MINOR] fix(docs): Fix the wrong link in the README.md by @github-actions in #1130
• [MINOR] docs(lakehouse-iceberg): improvement lakehouse iceberg document by @github-actions in #1138
• [#1136] improvement(docs): Add the md suffix for every link in the markdown by @github-actions in #1140
• [Minor] improvement(docs): Optimized Trino related docs. by @github-actions in #1144
• [MINOR] fix(docs): Fix link of REST API page by @github-actions in #1148
• [MINOR] fix(docs): fix ./gradlew spotlessApply of Coding standards by @SteNicholas in #1153
• [Minor] improvement(docs): Optimize Trino related docs and fix some minor mistakes in docs by @github-actions in #1159
• [#1166] fix(docs): Fix typos graviton and syntax problems in docs by @github-actions in #1176
• [MINOR] fix(docs): Fix a typo by @github-actions in #1192
• [#1154] fix(trino-connector): Fix the issue with joins causing errors in PostgreSQL. by @github-actions in #1186
• [#1047] improvement(core): Fix the issue of StringIdentifier in metalake property by @github-actions in #1188
• [#1171] bugfix(postgresql): Fix bug where the same table name can be loaded under multiple schemas. by @github-actions in #1195
• [MINOR] fix(docs): Modify the required value for the document security.md by @github-actions in #1223
• [MINOR] fix(doc): Fix a JSON format error in doc manage-metadata-using-gravitino.md by @github-actions in #1229
• [#1112] fix(jdbc): deleteColumn throws exception for table with ifExisting = true. by @github-actions in #1230
• [#879][#925] Docker network Pool overlaps with other one on this address space by @github-actions in #1234
• [Minor] docs: Add more details about dropping schema and tables by @github-actions in #1240
• [MINOR] fix(docs): Modify the document according to the README of playground by @github-actions in #1250
• [#1231] Fix bugs in drop MySQL schemas by @github-actions in #1251
• [#1253] bugfix(jdbc): fix jdbc catalog store empty audit info issue by @github-actions in #1261
• [Minor] Improve English, use better names and data, simplify SQL by @github-actions in #1272
• [#1469] improvement(doc): Add doc about webUI operation by @github-actions in #1281
• [MINOR] improve(doc): Improve the index to reorganize the order of the docs by @github-actions in #1288
• Happy new year! by @github-actions in #1289
• [#1135] improvement(docs): Add docs about tables advanced feature like partitioning by @github-actions in #1296
• [#1299] Fix the copyright year by @github-actions in #1309
• [MINOR] doc: Improve English and add warning that web UI only works in Chrome by @github-actions in #1314
• [#1311] bugfix(docker): Unable to run datastrato/trino Docker in AWS by @github-actions in #1315
• Fix minor grammar errors and English mistakes in documentation. by @github-actions in #1320
• [MINOR] fix typo in URL by @github-actions in #1324
• [#1151] fix(UI): Fix status cache after modified a Metalake by @github-actions in #1325
• [MINOR] Add note on supported build systems by @github-actions in #1326
• [#1113][#1224] improvement(UI): TS support and improve global request handing by @github-actions in #1332
• [#1237] fix(trino): Fix dropping iceberg schema failure by @github-actions in #1333
• [#1150] fix(UI): Fix details comment error and improve components check by @github-actions in #1334
• [#1282] bugfix(CI) Automatically trigger CI for auto-cherry-pick workflow by @jerryshao in #1342
• [#1328] fix(trino-connector): Fix the failed 'drop catalog' test on Trino image gravitino-ci-trino:0.1.3. by @jerryshao in #1347
• [#1295] improvement(CI): separate trino image for CI and users by @jerryshao in #1344
• [#1267] fix: Add missing headers and license by @jerryshao in #1360
• [#1317] fix(trino-connector): Support Fixed type conversion by @jerryshao in #1376
• [#1001] fix(UI): Fix null requests in the web UI by @jerryshao in #1378
• [#1350] fix(UI): Fix fields validation when create a catalog in the web UI by @jerryshao in #1387
• [#1273] fix(UI): Fix cache values in the dialog when create a catalog by @jerryshao in #1382
• [#1391] fix(UI): Fix update metalake error in the web UI by @jerryshao in #1400
• [#1403] fix(UI): Remove unnecessary fields when creating a catalog in the web UI by @jerryshao in #1407
• [#1428] improvement(UI): Filtering empty keys when create or update by @jerryshao in #1430

Full Changelog: v0.3.0...v0.3.1

Gravitino 0.3.0

Release notes

Gravitino server

• Authorization support for Gravitino.
• Metrics system for Gravitino.
• Optimize and finalize the v1 storage layout.

Core metadata

• Add a unified type system.
• Add a unified expression system.

Catalog support

• Add new JDBC Catalog support (with MySQL and PostgreSQL support).

Trino connector

• Add Iceberg and JDBC connector support.
• Add catalog/schema/table level property support.
• Add full type system support.

Web UI

• Gravitino UI support.

Gravitino 0.2.0

This is the Gravitino 0.2.0 release. You can verify the release using the signature (.asc files) or hashes (.sha256 files).