Cannot create sources & destinations because workspace resource is not found (Airbyte OSS)

[willi-mueller]

I tried to use this Airbyte Terraform provider to manage my open-source Airbyte installation.

However, when I try to add sources or resources then I always get the error that a resource cannot be found for my workspace.

I would appreciate very much your help in getting this working.

Example:

Error: unexpected response from API. Got an unexpected response code 400
│ 
│   with airbyte_source_rki_covid.rkicovid,
│   on airbyte_connections.tf line 10, in resource "airbyte_source_rki_covid" "rkicovid":
│   10: resource "airbyte_source_rki_covid" "rkicovid" {
│ 
│ **Request**:
│ POST /v1/sources HTTP/1.1
│ Host: localhost:8006
│ Accept: application/json
│ Authorization: Basic removed-my-secret==
│ Content-Type: application/json
│ User-Agent: speakeasy-sdk/go 0.3.5 2.195.2 1.0.0 airbyte
│ 
│ {"configuration":{"sourceType":"rki-covid","start_date":"2023-11-28"},"definitionId":"060c0bfb-06e0-634b-473f-682e9381b1f3","name":"RKI
│ Covid","workspaceId":"829914dc-025c-4d8a-989b-80717a7d6b4d"}
│ **Response**:
│ HTTP/1.1 400 Bad Request
│ Content-Length: 197
│ Connection: keep-alive
│ Content-Type: application/problem+json
│ Date: Thu, 30 Nov 2023 13:12:16 GMT
│ Server: nginx/1.25.3
│ 
│ {"type":"https://reference.airbyte.com/reference/errors#resource-not-found","title":"resource-not-found","status":400,"detail":"Could not find a
│ resource for: 829914dc-025c-4d8a-989b-80717a7d6b4d"}

However, the API lists this very workspace:

 curl --request GET --url 'http://localhost:8006/v1/workspaces' --header 'accept: application/json'  -u 'username:password' | python -mjson.tool

Returns:

{
    "data": [
        {
            "workspaceId": "829914dc-025c-4d8a-989b-80717a7d6b4d",
            "name": "829914dc-025c-4d8a-989b-80717a7d6b4d",
            "dataResidency": "auto"
        }
    ]
}

Environment

Tested on Airbyte OSS versions: 0.50.33 and 0.50.34
Provider version: 0.3.5

Airbyte is running on GPC behind an IAM proxy. I have port-forwarding to access the API and can successfully connect to the UI as well as make API calls as demonstrated.

How to reproduce

terraform {
  required_providers {
    airbyte = {
      source  = "airbytehq/airbyte"
      version = "0.3.5"
    }
  }
}

provider "airbyte" {
  username   = "username"
  password   = "password"
  # password   = data.google_secret_manager_secret_version.airbyte_basic_auth_password.secret_data
  server_url = "http://localhost:8006/v1"
}

variable "workspace_id" {
  type    = string
  default = "829914dc-025c-4d8a-989b-80717a7d6b4d"
}

resource "random_uuid" "rki" {}

resource "airbyte_source_rki_covid" "rkicovid" {
  configuration = {
    start_date = "2023-11-28"
  }
  name          = "RKI Covid"
  definition_id = random_uuid.rki_definition.result
  workspace_id  = "829914dc-025c-4d8a-989b-80717a7d6b4d"
}

However, this is exactly the workspace ID that I see in both my web-browser as well as in the API response as shown above

Additional trials

I tried also the MySQL source as well as the BigQuery destination:

resource "airbyte_source_mysql" "my_source_mysql" {
  configuration = {
    database = "my_database"
    host     = "my_host"
    username = "username"
    replication_method = {
      read_changes_using_binary_log_cdc = {
        initial_waiting_seconds = 7
        server_time_zone        = "UTC"
      }
    }
  }
  definition_id = "aba25784-141a-421c-8938-ad6fcbb78bed"
  name          = "Mr. Ross Cole"
  workspace_id  = var.workspace_id
}

resource "airbyte_destination_bigquery" "my_destination_bigquery" {
  configuration = {
    big_query_client_buffer_size_mb = 15
    credentials_json                = "{\"foo\": \"bar\"}"
    dataset_id                      = "raw"
    dataset_location                = "EU"
    disable_type_dedupe             = true
    loading_method = {
      gcs_staging = {
        credential = {
          hmac_key = {
            hmac_key_access_id = "1234567890abcdefghij1234"
            hmac_key_secret    = "1234567890abcdefghij1234567890ABCDEFGHIJ"
          }
        }
        gcs_bucket_name          = "my-bucket"
        gcs_bucket_path          = "tmp"
        keep_files_in_gcs_bucket = "Delete all tmp files from GCS"
      }
    }
    project_id              = var.project_id
    transformation_priority = "batch"
  }
  definition_id = "2d142842-c5e9-475e-80d1-1a3c6d933cc0"
  name          = "Miss Celia Moore"
  workspace_id  = var.workspace_id
}

However, it results in the same error:

│ Error: unexpected response from API. Got an unexpected response code 400
│ 
│   with airbyte_source_mysql.my_source_mysql,
│   on airbyte_connections.tf line 19, in resource "airbyte_source_mysql" "my_source_mysql":
│   19: resource "airbyte_source_mysql" "my_source_mysql" {
│ 
│ **Request**:
│ POST /v1/sources HTTP/1.1
│ Host: localhost:8006
│ Accept: application/json
│ Authorization: Basic removed-my-secret==
│ Content-Type: application/json
│ User-Agent: speakeasy-sdk/go 0.3.5 2.195.2 1.0.0 airbyte
│ 
│ {"configuration":{"database":"my_database","host":"my_host","port":3306,"replication_method":{"initial_waiting_seconds":7,"method":"CDC","server_time_zone":"UTC"},"sourceType":"mysql","username":"username"},"definitionId":"aba25784-141a-421c-8938-ad6fcbb78bed","name":"Mr.
│ Ross Cole","workspaceId":"829914dc-025c-4d8a-989b-80717a7d6b4d"}
│ **Response**:
│ HTTP/1.1 400 Bad Request
│ Content-Length: 197
│ Connection: keep-alive
│ Content-Type: application/problem+json
│ Date: Thu, 30 Nov 2023 13:12:16 GMT
│ Server: nginx/1.25.3
│ 
│ {"type":"https://reference.airbyte.com/reference/errors#resource-not-found","title":"resource-not-found","status":400,"detail":"Could not find a
│ resource for: 829914dc-025c-4d8a-989b-80717a7d6b4d"} 

Then, I thought that maybe if I import or create the workspace using Terraform it will work. However, same error:

resource "airbyte_workspace" "oss_workspace" {
  name = "OSS Workspace"  
}

resource "airbyte_source_rki_covid" "rkicovid" {
  configuration = {
    start_date = "2023-11-28"
  }
  name          = "RKI Covid"
  definition_id = random_uuid.test.result
  workspace_id  = airbyte_workspace.oss_workspace.workspace_id
}

Error:

airbyte_source_rki_covid.rkicovid: Creating...
╷
│ Error: unexpected response from API. Got an unexpected response code 400
│ 
│   with airbyte_source_rki_covid.rkicovid,
│   on airbyte_connections.tf line 9, in resource "airbyte_source_rki_covid" "rkicovid":
│    9: resource "airbyte_source_rki_covid" "rkicovid" {
│ 
│ **Request**:
│ POST /v1/sources HTTP/1.1
│ Host: localhost:8006
│ Accept: application/json
│ Authorization: Basic removed-my-secret==
│ Content-Type: application/json
│ User-Agent: speakeasy-sdk/go 0.3.5 2.195.2 1.0.0 airbyte
│ 
│ {"configuration":{"sourceType":"rki-covid","start_date":"2023-11-28"},"definitionId":"060c0bfb-06e0-634b-473f-682e9381b1f3","name":"RKI
│ Covid","workspaceId":"ffce1282-d125-4c49-9cc6-92f3133ebda6"}
│ **Response**:
│ HTTP/1.1 400 Bad Request
│ Content-Length: 197
│ Connection: keep-alive
│ Content-Type: application/problem+json
│ Date: Thu, 30 Nov 2023 13:28:43 GMT
│ Server: nginx/1.25.3
│ 
│ {"type":"https://reference.airbyte.com/reference/errors#resource-not-found","title":"resource-not-found","status":400,"detail":"Could not find a
│ resource for: ffce1282-d125-4c49-9cc6-92f3133ebda6"}

Again, the API lists it:

curl --request GET --url 'http://localhost:8006/v1/workspaces' --header 'accept: application/json'  -u 'username:password' | python -mjson.tool

{
    "data": [
        {
            "workspaceId": "829914dc-025c-4d8a-989b-80717a7d6b4d",
            "name": "829914dc-025c-4d8a-989b-80717a7d6b4d",
            "dataResidency": "auto"
        },
        {
            "workspaceId": "ffce1282-d125-4c49-9cc6-92f3133ebda6",
            "name": "OSS Workspace",
            "dataResidency": "auto"
        }
    ]
}

What if I give the workspace's name instead of ID?

resource "airbyte_source_rki_covid" "rkicovid" {
  configuration = {
    start_date = "2023-11-28"
  }
  name          = "RKI Covid"
  definition_id = random_uuid.test.result
  workspace_id  = airbyte_workspace.oss_workspace.name
}

Error:

│ Error: unexpected response from API. Got an unexpected response code 400
│ 
│   with airbyte_source_rki_covid.rkicovid,
│   on airbyte_connections.tf line 9, in resource "airbyte_source_rki_covid" "rkicovid":
│    9: resource "airbyte_source_rki_covid" "rkicovid" {
│ 
│ **Request**:
│ POST /v1/sources HTTP/1.1
│ Host: localhost:8006
... as before ...
│ {"type":"https://reference.airbyte.com/reference/errors#bad-request","title":"bad-request","status":400,"detail":"Invalid value for field
│ 'workspaceId': 'OSS Workspace' is not a valid UUID type"}

What if I import the existing workspace?

resource "airbyte_workspace" "default_workspace" {
  name = "829914dc-025c-4d8a-989b-80717a7d6b4d"
}

terraform import airbyte_workspace.default_workspace "829914dc-025c-4d8a-989b-80717a7d6b4d"

airbyte_workspace.default_workspace: Importing from ID "829914dc-025c-4d8a-989b-80717a7d6b4d"...
airbyte_workspace.default_workspace: Import prepared!
  Prepared airbyte_workspace for import
airbyte_workspace.default_workspace: Refreshing state...

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

Trying to reference the imported workspace:

resource "airbyte_source_rki_covid" "rkicovid" {
  configuration = {
    start_date = "2023-11-28"
  }
  name          = "RKI Covid"
  definition_id = random_uuid.test.result
  workspace_id  = airbyte_workspace.default_workspace.workspace_id
}

Then, terraform apply fails, but this seems to be a different problem:

airbyte_workspace.default_workspace: Refreshing state... [name=829914dc-025c-4d8a-989b-80717a7d6b4d]
airbyte_workspace.oss_workspace: Refreshing state... [name=OSS Workspace]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform planned the following actions, but then encountered a problem:

  # airbyte_source_rki_covid.rkicovid will be created
  + resource "airbyte_source_rki_covid" "rkicovid" {
      + configuration = {
          + start_date = "2023-11-28"
        }
      + definition_id = "060c0bfb-06e0-634b-473f-682e9381b1f3"
      + name          = "RKI Covid"
      + source_id     = (known after apply)
      + source_type   = (known after apply)
      + workspace_id  = "829914dc-025c-4d8a-989b-80717a7d6b4d"
    }

Plan: 1 to add, 0 to change, 0 to destroy.
╷
│ Error: failure to invoke API
│ 
│   with airbyte_workspace.oss_workspace,
│   on airbyte_connections.tf line 82, in resource "airbyte_workspace" "oss_workspace":
│   82: resource "airbyte_workspace" "oss_workspace" {
│ 
│ error sending request: Get "http://localhost:8006/v1/workspaces/ffce1282-d125-4c49-9cc6-92f3133ebda6": read tcp [::1]:59507->[::1]:8006: read:
│ connection reset by peer

I would be very glad if you could take a look at this issue and help me find the problem so that I can create sources, destinations, and connections in my workspace.

Thank you!

[willi-mueller]

The problem is the specification of the workspace_id in the variable.

When I remove the default value and paste the workspace_id without the quotes upon Terraform prompting it works!

[willi-mueller]

It seems that this problem is due to a race condition and waiting for the user input prevents it in most but not all cases for me. I'll update the issue later and elaborate on how to reproduce it.

[antonio-one]

Hi, is there any movement in this at all. It would be super nice if we could all be able to codify all the sources. destination and connections.

[jbfbell]

@willi-mueller sorry this has been so frustrating - could you try removing definition_id and seeing if that works? I'm suspicious that the error message you're seeing is misleading about the workspace not existing

[antonio-one]

The provider does not like this. The argument "definition_id" is required, but no definition was found.

[jbfbell]

Thanks for testing that @antonio-one - will circle back with the team

[thedannywilcox]

I am running into the same problem except I'm using an S3 source/destination. Same workaround works for me too (leaving value blank and populating the value in after terraform runs. I also tried this to no avail:
echo <workspace_id> | terraform plan

[thedannywilcox]

I was able to workaround this further by using:

variable "airbyte_workspace_id" {
  type = string
  default = "MY_ID"
}

data "airbyte_workspace" "main" {
  workspace_id = var.airbyte_workspace_id
}

And then use data.airbyte_workspace.main.workspace_id for the source/destination.