Unity Catalog Support

[dhirschfeld]

I'm opening this to discuss the state of Unity Catalog support in delta-rs.

In my ideal world I'd be able get a DeltaTable object from Unity Catalog, with permissions controlled by UC access policies.

[dhirschfeld]

xref:

• fix: remove the get_data_catalog() function #1941

[dhirschfeld]

Extract the session credentials from your cluster

• https://kb.databricks.com/en_US/notebooks/access-s3-temp-session-token

[rtyler]

I have had a number of discussions with Databricks on the topic. The 🦀 code has support for table lookup but it's kind of useless IMHO without credential vending support from the Unity APIs. Basically a user today would need to have an access token for Unity and their own separate storage credentials for accessing the table in object storage.

What is missing is a public API from Databricks which would allow the actor running delta-rs to retrieve credentials for use in accessing storage.

So yes, I would like to see this too, but I'm marking this on-hold until Databricks provides us the necessary REST APIs for Unity Catalog

[dhirschfeld]

Thanks for the info @rtyler. I'll continue to monitor this thread in the hope that something changes on the Databricks side...