Add github actions job for the creating/terminating databricks cluster

[Lee-W]

Description

What is the current behavior?

run databricks testing on an existing cluster

What is the new behavior?

• bring up a databricks cluster
• run the databricks tests
• terminate cluster

In this PR, I also unify the yaml style in .github. The main changes live in the first and the third commit

Does this introduce a breaking change?

no

Checklist

• Created tests which fail without the change (if possible)
• Extended the README / documentation, if necessary

[Lee-W]

@phanikumv @tatiana We're now testing bringing up the databricks cluster and then terminating it after the tests. Could you please help us set up DATABRICKS_HOST in secret? Thanks!

[tatiana]

I just added the secret, @Lee-W !

[Lee-W]

Thanks @tatiana ! After a few tests, I found out we might need an extra secret here. Already DM you the secret. Could you please help us set it up? Thanks

[tatiana]

Hi @Lee-W , I just added it - thank you very much for working on this!

[tatiana]

@Lee-W this ticket is hard work 😬
Please ping us when the tests are passing so we can merge it!

[Lee-W]

@Lee-W this ticket is hard work 😬 Please ping us when the tests are passing so we can merge it!

The next thing I'll try is make the secrets small strings instead of the whole json.

It seems structured data as json is not recommended by github https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions#naming-your-secrets