Databricks SQL support

[samuell]

Based on recent work for clients, I for one would think support for Databricks SQL would be a huge deal. Seems to be used a little bit of everywhere.

I recently learned Databricks has teamed up with dbt to enable using their SQL templating tech for simplifying writing transformation logic, but Logica would be on a completely different level in my opinion.

[EvgSkv]

Thanks for filing the issue!

I didn't work with Databricks before. I briefly looked at documentation.

The good news is that Databricks has structs and arrays, so adding this as a Logica engine appears doable.

Unfortunately it looks like Databricks doesn't have a free tier. The free trial is limited to 14 days. So to implement it we'd need somebody who already pays for Databricks.

Drop a note in this issue if you are interested in contributing to this.

[rvalek]

Hi, @samuell and @EvgSkv

My team and I also looked into DatabricksSQL support, and we're doing some work right now to target Databricks at least partially. Some of it might be a bit crude, but we'd be happy to raise a PR to see if it could be useful to others once we're done.

I'll follow up on this within several weeks.

[EvgSkv]

Hi @rvalek ,
that's great to hear!
Adding nuances of Databricks to dialects (under compile folder) should be straightforward and safe.
If deeper changes are needed (e.g. in universe.py or expr_translate.py), I'm happy to work with you to get them done.
Looking forward!

[rvalek]

Hi!
We are a little behind on this promise due to some unrelated technical issues, but we are close to resolving them.
Just wanted update here that we are still interested in sharing our work on the Databricks dialect. We're planning to raise the PR in early-mid September.

[KantorSerhiy]

hi @EvgSkv!
can you suggest in which branch it is best to open a PR for this?

[EvgSkv]

Hi @KantorSerhiy !
Sorry, missed the comment. Sending PR to main is just fine! Thank you!

[EvgSkv]

@KantorSerhiy , btw, I did respond in the PR, it looks great! I left just a couple of requests to comment and it's good to be merged after that.

[KantorSerhiy]

hey @EvgSkv!
yes, I saw your comments and will definitely deal with them.
but it takes for me 1-2 weeks=(
Thank you!

[EvgSkv]

@KantorSerhiy great! No worries! Just wanted to make sure your progress isn't blocked on me :-)

[rvalek]

Hi @samuell , we merged the baseline databricks dialect a couple of weeks ago.
Feel free to try it out!