What would it take to port the dbt-dremio connector over to SQLMesh?

[usbrandon]

Hi there.

I have been using the DBT connector for Dremio Cloud over here https://github.com/dremio/dbt-dremio
What would it take to port it to SQL Mesh?

If there is not too much hair on it, I might try to help with the effort and send a pull request but I wanted to make sure I understood how the SQL Mesh project operates.

Warm regards,

Brandon

[eakmanrq]

Hey @usbrandon. 👋

I have not used Dremio myself but my high level understanding is that it is similar to Trino and users would likely be storing data in Iceberg. It can be run locally in a docker container and then also use the cloud offering. I will be making that assumption for the remained of my advice.

Here is a more recent-ish PR that shows the high level work involved in adding an engine: #1579

I would focus on adding for Dremio open source first since that is easier than adding a remote cloud connection:

• Add a new connection config for Dremio open source
• Add a new class for your connector that implemented the base engine adapter class
• Update test_integration.py engine tests to include your new engine: https://github.com/TobikoData/sqlmesh/blob/c1ce829b00043ef2f8d46f31eec035ba4fc2ee6b/tests/core/engine_adapter/test_integration.py
• Add Dremio to the existing engine stack: https://github.com/TobikoData/sqlmesh/blob/c1ce829b00043ef2f8d46f31eec035ba4fc2ee6b/tests/core/engine_adapter/docker-compose.yaml
• Run those tests and fix errors that come up
• Documentation

Once this is done then I would repeat but for Demio Cloud. Adding the cloud would likely easy since you have added the foundational support for Dremio.

It is not a trivial amount of work. If you did add the cloud support then we would also ask you about how to setup an account with that cloud so we could run our tests against it to make sure we don't break anything.