How to store Hive configurations in Gravitino, such as core-site.xml and hdfs-site.xml?

[diqiu50]

When using the Gravitino-Trino connector to connect to Hive with Trino, users sometimes need to configure core-site.xml and hdfs-site.xml. Currently, users are required to store these configuration files on the Trino nodes, and specify the file paths when Gravitino creates a catalog. We hope that Gravitino can manage these configuration files, so that when the Gravitino-Trino connector loads a catalog, it automatically retrieves and uses these configurations.

Currently, there are two ideas:

1. Users upload the configuration files to the Gravitino server for storage, and the Gravitino-Trino connector obtains them through the Gravitino-client API according to the catalog configuration.
2. Users upload the configuration files to an object storage service, and the Gravitino-Trino connector obtains them through a REST request according to the catalog configuration.

What do you all think about this issue?

[shaofengshi]

In my opinion, both are not the best option, because these configuration files may change as time goes on (for example, the administrator made some configuration update). If there is no an automatic way to syn them, they may become obsolete. Furthermore, the files may contain sensitive data, distributing to other places will increase the risk of data leakage.

Is it possible that Gravitino only keep the real location URL, not the real file?

[diqiu50]

I hope to use a remote URI, if a local address is used, it would require users to upload the configuration files on the Trino cluster, which would lack flexibility.

We now support catalog reload, and when the catalog is reloaded, the configuration will be reloaded as well.

[yuqi1129]

Is the object storage service a third-party service or a service managed by us? If it's a third-party service, I suggest using the former one.

[diqiu50]

We don't have managed the services. We only using an URI

[mchades]

Which configuration items does Trino rely on in these two configuration files?(core-site.xml and hdfs-site.xml)? Can you find the smallest set of dependencies?

[zhoukangcn]

+1, Could we find the smallest set of properties, and Gravitino can return it to Trino by catalog properites. Then maybe we do not need real config file (includes both core-site.xml and hdfs-site.xml)

[coolderli]

I think it's meaningful. If we can load the configuration and session token of s3( for HDFS, it will be a delegation token) at runtime. We can simplify the usage of clients. The client only needs to load the runtime jar and get the credentials for authentication.

I think it's not necessary to get the real file. We can put the common configuration to the metalake properties. The catalog properties can inherit and override the metalake properties, so does the schema and the table.

And we do not need to find the smallest set of properties, we can support the feature that loads the configuration from the properties. It's up to users whether to use it. For more users, they may like to use the configuration that is loaded from the engine side.