[ISSUE] Issue with databricks_sql_table resource

[ollimandoliini]

Configuration

resource "databricks_sql_table" "thing" {
  name               = "table_name
  catalog_name       = "catalog_name"
  schema_name        = "schema_name"
  table_type         = "EXTERNAL"
  data_source_format = "DELTA"
  storage_location   = var.s3_location
  warehouse_id       = "warehouseid"

  column {
    name = "field"
    type = "STRING"
  }

  column {
    name = "another_field"
    type = "BIGINT"
  }
}

Expected Behavior

A previously created SQL table should work normally when running plan/apply/destroy.

Actual Behavior

When running plan/apply/destroy with unchanged configuration for a previously created table, the command will fail with an error:

Error: changing the 'type' of an existing column is not supported

  with module.published_tables["ollis_test_table_delta"].databricks_sql_table.thing,
  on ../modules/published_table/main.tf line 91, in resource "databricks_sql_table" "thing":
  91: resource "databricks_sql_table" "thing" {

Steps to Reproduce

Terraform and provider versions

Terraform v1.8.1
on linux_amd64

• provider registry.terraform.io/databricks/databricks v1.40.0

Is it a regression?

I don't know

Debug Output

Important Factoids

The first and only mention of the error in debug logs:

2024-04-19T12:40:52.270+0300 [ERROR] provider.terraform-provider-databricks_v1.40.0: Response contains error diagnostic: diagnostic_severity=ERROR diagnostic_summary="changing the 'type' of an existing column is not supported" tf_proto_version=5.4 @module=sdk.proto diagnostic_detail="" tf_rpc=PlanResourceChange tf_provider_addr=registry.terraform.io/databricks/databricks tf_resource_type=databricks_sql_table @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/diag/diagnostics.go:58 tf_req_id=5c2e925f-cc4b-7414-2cf5-065b8e22430e timestamp="2024-04-19T12:40:52.270+0300"

Would you like to implement a fix?

No.

[Dedvall]

@ollimandoliini, if you write type = "string" and type = "bigint" (all lower case) it will pass the plan without error.

However in my project we now have the issue of type definitions like this :
type = "bigint GENERATED BY DEFAULT AS IDENTITY" since we didn't find any other ways to create Identity columns. Either there needs to be some other way of setting Identity columns or we need to know how to pass this in as a proper string.

It looks like the provider is getting column definitions from the backend without the additional information regarding Identity settings.

Passing the type-string in as above with GENERATED BY DEFAULT AS IDENTITY appended after the type definitions works as the provider GO code just takes everything is as strings concatenates everything as a CREATE/ALTER string where
CREATE TABLE foo (id bigint GENERATED BY DEFAULT AS IDENTITY) is a valid statment

[ollimandoliini]

Thanks, that made the plan succeed!

I was thinking it must be something like this. There also seems to be an issue with type synonyms. If you define a column with type long, Databricks will change it into bigint and next time you plan it will fail.

[Dedvall]

@ollimandoliini, yes from my understanding Databricks translates some types in "popular naming" into internal types. Same with type = integer specification in Terraform, which is read as a column of type int on the next plan and plan failing because string comparison integer != int