[FEATURE] New feature request for databricks_node_type data source

[orolega]

Use-cases

Parameterise selecting node types from a set of criteria

Attempted Solutions

data "databricks_node_type" "general_purpose_graviton_large" {
  category  = "General Purpose"
  graviton  = true
  min_cores = 2
}

This returns m6g.large however there is a newer node type that also fits the criteria m7g.large. There is no way in the current data source to get the m7g instead

Proposal

The data source should either return the newest node type fitting the criteria (the above example should return m7g instead of m6g), or perhaps have a flag that enables the user to choose if the returned node type is the newest possible one.

[orolega]

# Get a list of all node types
data "http" "node_types" {
  url = "${var.databricks_host}/api/2.0/clusters/list-node-types"

  # Optional request headers
  request_headers = {
    Accept        = "application/json"
    Authorization = "Bearer ${var.databricks_token}"
  }
}

locals {
  all_node_types          = [for node_type in jsondecode(data.http.node_types.response_body)["node_types"] : node_type]
  all_graviton_node_types = [for node_type in local.all_node_types : node_type if node_type.is_graviton]
  graviton_large_node     = reverse([for node_type in local.all_graviton_node_types : node_type.node_type_id if node_type.num_cores == 2 && node_type.category == "General Purpose" && !node_type.is_io_cache_enabled])[0]
}

Got around the issue by using the above code snippet. Tried to see if I could implement it myself, but I cannot get your repo to work locally