GetTablesWithQuery equivalent parameter for kyuubi-hive-jdbc-shaded.jar

[mahalingamagesthian]

We are using the kyuubi-hive-jdbc-shaded.jar with Tableau. When we list all the tables in the Database/Schema, GetTables API is invoked by spark driver.

With Simba Spark ODBC we have two options, We can either use GetTables API or SHOW TABLES. This is configured using the GetTablesWithQuery a. set to 0 for GetTables API b. set to 1 for SHOW TABLES.

Do we have a similar option available with kyuubi-hive-jdbc-shaded.jar. I tried using;
jdbc:kyuubi://hostname:10009/;GetTablesWithQuery=1;
it didn't work as it worked for Simba Spark ODBC.
Could you suggest if we could use SHOW TABLES in the place of GetTables API for JDBC driver - kyuubi-hive-jdbc-shaded.jar ?

[pan3793]

There are no such options now.

May I know why you want to use SHOW TABLES instead of metadata API?

[mahalingamagesthian]

We have a scenario with more than 7000tables, when we use GetTables API, it takes about 70ms to fetch metadata for one table, This is the time taken to fetch the details from Hive RDBMS(Oracle). GetTables API issues 22 SQL statements to HiveMetastore's RDBMS.. Show Tables performed well., Show Tables was issuing 1 SQL statement to fetch all table names.

[pan3793]

Maybe we can optimize the performance of metadata API, we did such a thing before, try setting kyuubi.operation.getTables.ignoreTableProperties=true

[pan3793]

UPDATE:

I checked the org.apache.kyuubi.engine.spark.util.SparkCatalogUtils#getCatalogTablesOrViews, there is still room to change. PRs are welcome.

[pan3793]

also cc @liaoyt

[mahalingamagesthian]

We tried with the kyuubi.operation.getTables.ignoreTableProperties=true, The duration of GetTables API remained the same.

[pan3793]

FYI: kyuubi.operation.getTables.ignoreTableProperties=true was enhanced to support session catalog cases in #6018

[mahalingamagesthian]

Hi @pan3793 - Thanks for this approach. It Looks promising. I will have this tested and keep you posted.

[mahalingamagesthian]

@pan3793 - I have tested this fix , it works for us. Thanks very much.

[pan3793]

thanks for your feedback :)