consul_acl_token_secret_id data source errors with an access denied (403)

[ferrarimarco]

Terraform Version

Terraform v0.13.5
+ provider registry.terraform.io/hashicorp/consul v2.10.1

Affected Resource(s)

Please list the resources as a list, for example:

• data.consul_acl_token_secret_id

Terraform Configuration Files

provider "consul" {
  address    = "consul.url.abcdef"
  datacenter = "configuration-datacenter"
  scheme     = "https"
  token      = data.kubernetes_secret.consul-bootstrap-acl-token.data["token"]
}

resource "consul_acl_token" "consul-template" {
  description = "consul-template token"
  policies    = [consul_acl_policy.consul-template.name]
  local       = true
}

data "consul_acl_token_secret_id" "consul-template-secret-id" {
  accessor_id = consul_acl_token.consul-template.accessor_id
}

Debug Output

2020-11-16T16:01:32.158+0100 [DEBUG] plugin.terraform-provider-consul_v2.10.1_x4: 2020/11/16 16:01:32 [INFO] Initializing Consul client
2020-11-16T16:01:32.158+0100 [DEBUG] plugin.terraform-provider-consul_v2.10.1_x4: 2020/11/16 16:01:32 [INFO] Consul Client configured with address: 'consul.url.abcdef', scheme: 'https', datacenter: 'configuration-datacenter', insecure_https: 'false'
2020/11/16 16:01:32 [DEBUG] ReferenceTransformer: "module.configuration.consul_acl_policy.consul-template" references: []
2020/11/16 16:01:32 [DEBUG] ReferenceTransformer: "module.configuration.consul_acl_token.consul-template" references: []
2020/11/16 16:01:32 [DEBUG] ReferenceTransformer: "module.configuration.data.consul_acl_token_secret_id.consul-template-secret-id" references: []
2020/11/16 16:01:32 [ERROR] eval: *terraform.evalReadDataPlan, err: Unexpected response code: 403 (rpc error making call: Permission denied)
2020/11/16 16:01:32 [ERROR] eval: *terraform.EvalSequence, err: Unexpected response code: 403 (rpc error making call: Permission denied)
2020/11/16 16:01:32 [INFO] backend/local: plan operation completed
2020-11-16T16:01:32.793+0100 [DEBUG] plugin: plugin process exited: path=.terraform/plugins/registry.terraform.io/hashicorp/consul/2.10.1/darwin_amd64/terraform-provider-consul_v2.10.1_x4 pid=26892
2020-11-16T16:01:32.794+0100 [DEBUG] plugin: plugin exited

On the Consul side, I see a corresponding error:

[ERROR] agent.http: Request error: method=GET url=/v1/acl/token/TOKEN-ID?dc=configuration-datacenter from=1.2.3.4:54478 error="rpc error making call: Permission denied"

where TOKEN-ID is the ID of the token, that I double-checked.

Expected Behavior

Terraform should be able to present a suitable plan.

Actual Behavior

Terraform outputs the following error:

Error: Unexpected response code: 403 (rpc error making call: Permission denied)

  on ../../modules/configuration/consul-template.tf line 22, in data "consul_acl_token_secret_id" "consul-template-secret-id":
  22: data "consul_acl_token_secret_id" "consul-template-secret-id" {

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform plan

Important Factoids

I'm using the bootstrap ACL token (loaded from a Kubernetes secret). Other resources and data sources of the consul provider are working fine with the same token.

[ferrarimarco]

A couple of updates:

• Consul version: 1.8.2

I'm now checking if using the bootstrap ACL token directly instead of loading it from the K8S secret has any impact.

[ferrarimarco]

Is there a way to have a listing of the consul client configuration? I'm looking forward to the token that the client is using. Thanks!

[remilapeyre]

Hi @ferrarimarco, sorry to not have answered sooner. I had no Kubernetes cluster to reproduce your setup and needed to setup one to reproduce your setup. I have not yet succeeded in reproducing the bug. Do you have multiple Consul clusters?

I'm looking forward to the token that the client is using.
I think the simplest method to get this would be to use mitmproxy that will make it possible to inspect all requests made by the Consul provider.

[ferrarimarco]

After upgrading to the latest consul, it didn't occur anymore. I'll reopen if this appens again!

Thanks!

[ferrarimarco]

Sorry to reopen this, but started happening again. Apparently, updating consul didn't fix it in the end :(

I do have only one Consul cluster, that I deployed with the 0.24.1 chart.

[ferrarimarco]

The debug output looks like the one I appended in my first message here :)

[ferrarimarco]

Might it be a race condition of some kind?

[remilapeyre]

If you have a single datacenter I don't think a race condition should be possible. I've tried but could not reproduce the issue, could you post a complete Terraform configuration so that I can try to reproduce the bug?

[ferrarimarco]

I don't have access to that environment anymore. Feel free to close, or keep open as reference :)

[tdgeery]

I'm now experiencing this issue
Provider Config

provider "consul" {
  address    = local.consul_url
  datacenter = local.consul_datacenter
  token      = data.kubernetes_secret.consul_bootstrap_acl_token.data["token"]
}

Resources

data "kubernetes_secret" "consul_bootstrap_acl_token" {
  metadata {
    name      = "consul-bootstrap-acl-token"
    namespace = "default"
  }

  depends_on = [
    helm_release.consul
  ]
}

resource "consul_acl_policy" "vault" {
  name  = "vault"
  rules = <<-RULE
    {
      "key_prefix": {
        "vault/": {
          "policy": "write"
        }
      },
      "node_prefix": {
        "": {
          "policy": "write"
        }
      },
      "service": {
        "vault": {
          "policy": "write"
        }
      },
      "agent_prefix": {
        "": {
          "policy": "write"
        }
      },
      "session_prefix": {
        "": {
          "policy": "write"
        }
      }
    }
    RULE

  depends_on = [
    helm_release.consul
  ]
}

Running Consul 1.9.5 on GKE

Provider Versions

terraform version
Terraform v0.15.4
on darwin_amd64
+ provider registry.terraform.io/hashicorp/consul v2.12.0
+ provider registry.terraform.io/hashicorp/external v2.1.0
+ provider registry.terraform.io/hashicorp/google v3.69.0
+ provider registry.terraform.io/hashicorp/google-beta v3.69.0
+ provider registry.terraform.io/hashicorp/helm v2.1.2
+ provider registry.terraform.io/hashicorp/kubernetes v2.2.0

[manobi]

Hi @tdgeery, using the same approach to get token from kubernetes secret and having the same issue while reading consul policies. Did you figure it out how to solve?

[tdgeery]

@manobi Since I'm creating Consul and Vault in the same repository, I imported the Consul bootstrap token into terraform and am passing that to the provider:

resource "random_uuid" "consul_bootstrap_acl_token" {}

resource "kubernetes_secret" "consul_bootstrap_acl_token" {
  metadata {
    name      = "consul-bootstrap-acl-token"
    namespace = "default"
  }

  data = {
    "token" = random_uuid.consul_bootstrap_acl_token.result
  }
}

Under the Consul Helm chart:

global:
  acls:
    manageSystemACLs: true
    bootstrapToken:
      secretName: ${bootstrap_secret_name}
      secretKey: ${bootstrap_secret_key}

and then for the consul provider:

provider "consul" {
  address        = local.consul_url
  token          = random_uuid.consul_bootstrap_acl_token.result
...
}