[WIP] Ephemeral Values prototype

[apparentlymart]

This is another attempt at introducing to Terraform the idea of objects and values being "ephemeral", which means something like "lives only for the duration of one Terraform phase".

Terraform already has at least two concepts that meet this definition, despite us not previously naming it:

• Provider configurations (provider blocks): Terraform re-evaluates the arguments in a provider block separately during the plan and apply phases, and doesn't mind if the configuration is different between the two as long as the apply-time configuration allows performing the actions that were proposed during the plan phase.
• Provisioners (provisioner and connection blocks): Terraform fully evaluates these only during the apply phase, so they aren't really considered during the plan phase at all, aside from basic static validation.

However, because the idea of "ephemeral" is not available in the rest of the language, it's tough to actually benefit from this ephemerality. This prototype aims to introduce "ephemeral" as a cross-cutting concern supported broadly across the language.

Ephemeral Values

The most fundamental idea is that values used in expressions can either be ephemeral or non-ephemeral. This is an idea similar to "sensitive" in that Terraform will perform dynamic analysis such that any value derived from an ephemeral value is itself ephemeral. Ephemeral values can then be used only in parts of the language which would not require persisting the value either between the plan phase and the apply phase, or from one plan/apply round to the next.

Considering only pre-existing language features, ephemeral values can be freely used in provider blocks, provisioner blocks, connection blocks, and in local values. The following sections describe some new additions that either accept or produce ephemeral values.

resource blocks (aside from special nested parts like the aforementioned provisioner blocks) do not accept ephemeral values, because preserving resource configuration unchanged between the plan and apply phases is a fundamental part of how Terraform works to keep its promise of either doing what the plan described or returning an error explaining why that's not possible.

Because ephemeral values are not expected to persist from plan to apply or between plan/apply rounds, there is no need to save them in saved plan files or state snapshots, thus finally giving a plausible answer for what to do about #516, which has been on my mind since long before I worked at HashiCorp.

Ephemeral Input Variables

An ephemeral input variable is, in the most general terms, just an input variable that is declared as accepting ephemeral values. A non-ephemeral input variable cannot accept ephemeral values, while an ephemeral value will accept both ephemeral and non-ephemeral values but the value will always be treated as ephemeral when used inside the declaring module.

The main interesting case is when a root module declares an ephemeral input variable. In that case, Terraform will no longer remember the value for the variable provided during planning and will instead expect any ephemeral variable set during the plan step to be provided again -- possibly with a different value -- during the apply step.

The primary goal of this is to be able to use input variables to set arguments in ephemeral contexts. For example, an input variable that's both ephemeral and sensitive could provide a JSON Web Token to be used when configuring a specific provider, and then automation around Terraform could provide separate JSON Web Tokens across the plan and apply phases so that the apply phase isn't subject to the expiration time for the plan-time JWT, and so that the plan-time JWT doesn't get persisted to disk as part of a saved plan.

Ephemeral Output Values

An ephemeral output value is essentially the opposite of an ephemeral input variable, allowing a module to expose an ephemeral value to its caller. As with input variables, a non-ephemeral output value will reject having an ephemeral value assigned to it. An ephemeral output value can have both ephemeral and non-ephemeral values assigned to it, but the calling module will always see it as ephemeral.

To start the utility of this is limited just to echoing back values derived from ephemeral input variables, since nothing else I've described so far actually produces ephemeral values. However, allowing this is important to ensure that ephemeral values are supported symmetrically and will cooperate well with all other language features.

Ephemeral Resources

The final idea in this prototype -- one which this prototype probably won't explore fully just yet, and introduce only just enough to validate that it fits in well with everything else -- is a new resource mode for representing remote objects that are ephemeral themselves.

Terraform currently has two "resource modes": managed resources (resource blocks) describe objects that Terraform is directly managing, while data resources (data blocks) describe objects that are managed elsewhere that the current configuration depends on. But in both cases the assumption is that those objects persist in some sense from plan to apply and from one plan/apply round to the next, and that Terraform is supposed to detect and react to any changes to those objects and therefore needs to persist information about them itself.

Ephemeral resources, (ephemeral blocks) on the other hand, represent objects that -- at least, as far as Terraform is concerned -- exist only briefly during a single Terraform phase, and then get cleaned up once the phase is complete. This idea is an evolution of some much earlier design work I did before I even worked at HashiCorp 😀 in relation to #8367, which was about establishing temporary SSH tunnels, and the HashiCorp Vault provider I wrote in #9158 (which evolved into today's official hashicorp/vault).

The general idea of ephemeral resources, then, is that their lifecycle includes three events:

• OpenEphemeral: Prepares the object for use. For some kinds of objects this would represent a "create" action, but for others it might just open a temporary session to something that already exists, such as in the SSH tunnel use-case.

  This operation is the one that establishes the result attributes that can be accessed from other parts of the module where the resource is declared. All of these results would be ephemeral values, so that they can vary from plan to apply. For example, opening an SSH tunnel is likely to cause a different local TCP port number to be allocated each time, and so consistency between plan and apply phases is not expected.

• RenewEphemeral: Some ephemeral remote objects need to be periodically refreshed in order to stay "live", such as leases for Vault secrets.

  This optional operation is therefore opted into by the provider's OpenEphemeral response, by providing a private set of data that should be sent back to the provider's RenewEphemeral implementation and a deadline before which Terraform must renew it. The provider can then do whatever is needed to keep the object from expiring, and optionally return another renew request with a new deadline in order to repeat this renewal process.

• CloseEphemeral: Once Terraform has completed work for all objects that refer to the ephemeral resource, this operation gives the provider an explicit signal that the object is not longer required so that it can be promptly destroyed or invalidated.

  This detail is particularly helpful for the Vault provider and fixes a limitation I ran into immediately back in 2016: a dynamic secret fetched using a data block can never have its lease explicitly terminated, because data resources were intended only to read information about an object someone else is managing, not to directly manage an object (a Vault lease).

Because the results from ephemeral resources are ephemeral values, they're primarily useful in configuration for other ephemeral objects: provider blocks, provisioner/connection blocks, and of course other ephemeral blocks.

Actually changing the provider protocol and implementing real providers is not in scope for my initial prototyping work here, and so I intend to prototype this in a more limited way that just emulates how this mechanism might behave, so we can see how well it interacts with the rest of the language and the other ephemeral values discussed here.

I've also been considering a mechanism to allow managed resource types to declare individual arguments as being "write-only", such as for an RDS database password that only needs to be provided during creation and should not be provided again unless the operator actually intends to reset it. I don't intend to prototype that in here, but I intend to lay the foundations for it by having a convention that ephemeral input values and write-only arguments both treat null as meaning "don't set or change" and non-null as "set or change", thereby creating a small imperative-shaped niche in the otherwise-declarative Terraform Language to allow for using Terraform to manage objects that have write-only (typically, sensitive) arguments without needing to persist them in plan and state.

---

I'm still working on this, so not everything described above is in here yet, but the foundations for ephemeral values themselves are already in. I've opened this draft largely just because I need to put this work down for a while for a team offsite and don't want to lose the context.

[apparentlymart]

As of this comment there's enough in this branch to demonstrate using ephemeral input variables to meet the use-case of certain values needing to change between plan and apply, as is often necessary when using time-limited transient credentials.

In today's Terraform we avoid dealing with that by encouraging providers to look for such information "ambiently", such as by using environment variables or configuration files discovered in the user's home directory. That remains a good, pragmatic answer for simple cases -- and has the potentially benefit of integrating well with other software for the same platform running on the same computer -- but it doesn't work so well when things get more complicated, such as when there are two configurations for the same provider that each need different credentials.

Here's a configuration I used to exercise this:

terraform {
  experiments = [ephemeral_values]

  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}

variable "aws_assume_role" {
  type      = object({
    role_arn       = string
    session_name   = optional(string)
    external_id    = optional(string)
    identity_token = optional(string)
  })
  ephemeral = true
  sensitive = true

  validation {
    condition = var.aws_assume_role.identity_token != null || var.aws_assume_role.external_id != null
    error_message = "Must set either identity_token or external_id."
  }

  validation {
    condition = var.aws_assume_role.identity_token == null || var.aws_assume_role.external_id == null
    error_message = "Must set only one of identity_token or external_id."
  }
}

variable "aws_region" {
  type = string
}

provider "aws" {
  region = var.aws_region

  dynamic "assume_role" {
    for_each = [
      for ar in var.aws_assume_role[*] : ar
      if ar.external_id != null
    ]
    content {
      role_arn     = assume_role.value.role_arn
      session_name = assume_role.value.session_name
      external_id  = assume_role.value.external_id
    }
  }

  dynamic "assume_role_with_web_identity" {
    for_each = [
      for ar in var.aws_assume_role[*] : ar
      if ar.identity_token != null
    ]
    content {
      role_arn           = assume_role.value.role_arn
      session_name       = assume_role.value.session_name
      web_identity_token = assume_role.value.identity_token
    }
  }
}

data "aws_caller_identity" "current" {
}

output "caller_identity_arn" {
  value = data.aws_caller_identity.current.arn
}

The aws_assume_role variable is declared as ephemeral = true, which means:

• Its value is not persisted as part of the plan.
• If it's set during the planning phase then it must be provided again during the apply phase, possibly with a different value.
• The expression var.aws_assume_role in this module produces a value that's marked as being ephemeral, which blocks it from being used in non-ephemeral situations such as resource configurations. A provider configuration (like the provider "aws" block in this example) is ephemeral -- it's re-configured separately from scratch for each plan or apply -- and so it's valid to use var.aws_assume_role in there.

I exercised this using the following sequence of commands, where plan.tfvars and apply.tfvars both contain different values for aws_assume_role where the former has only read access while the latter has full access:

• terraform plan -var-file=plan.tfvars -out=tfplan
• terraform apply -var-file=apply.tfvars tfplan

In current Terraform the second command would be invalid because we disallow using -var-file in conjunction with a saved plan file. That rule is weakened in this branch so that it can be the local backend which deals with that option, and it does so using a more nuanced set of rules that makes sure that non-ephemeral variables stay set to what they were planned to be, while also making sure that ephemeral variables are provided again during the apply phase.

---

This example also illustrates an interesting design wrinkle, which isn't a totally new problem but perhaps "feels" more sinister in a world with explicit support for ephemeral values:

The aws_caller_identity data source in hashicorp/aws returns a result that varies based on how the provider is configured. Since I'm intentionally switching to a role with different permissions in the apply step, that data source would return a different result depending on whether it were read during the plan phase or during the apply phase.

That is already true today if someone achieves this same result by resetting the environment variables the AWS provider responds to, but I think it's still worth noting that this interesting seam exists, and might prove confusing for some uses.

This situation doesn't break Terraform, because if Terraform reads the data source during the planning phase then it will use that same result during the apply phase without checking it again. Terraform effectively "locks in" a particular data source result for each plan/apply round.

But it does seem like if we move forward with a change like this, which would promote intentionally varying credentials between plan and apply, that we explain the implications carefully in all relevant documentation. Although I was just using two different roles in the same AWS account here, it's technically possible for the credentials provided during apply to be for an entirely different AWS account, which would cause a very confusing result.

(I wonder if there's something in letting providers return some data from the ConfigureProvider call made during planning that Terraform Core sends back to the provider for ConfigureProvider during the apply phase, so that the provider can "remember" how it was configured and return an error if the configuration has changed to such an extent that the plan would've been invalidated. But that's a separate idea for another time.)

[apparentlymart]

As of this comment there's enough to use an ephemeral resource instance to open an SSH tunnel, though for now I did it just using a temporary ephemeral resource type I bolted into the built-in Terraform provider because there's no plugin protocol support for ephemeral resources yet. (In practice I would expect this to belong to a separate provider plugin focused on SSH.)

For example, I just applied the following configuration that configures the hashicorp/consul provider to access a Consul server using an SSH tunnel:

terraform {
  experiments = [ ephemeral_values ]

  required_providers {
    terraform = {
      source = "terraform.io/builtin/terraform"
    }
    consul = {
      source = "hashicorp/consul"
    }
  }
}

ephemeral "terraform_ssh_tunnels" "foo" {
  server = "127.0.0.1:2222"
  username = "not really my username"

  auth_methods = [
    { password = "not really my password" },
  ]

  tcp_local_to_remote "consul" {
    remote = "127.0.0.1:8500"
  }
}

provider "consul" {
  address = ephemeral.terraform_ssh_tunnels.foo.tcp_to_remote["consul"].local
}

data "consul_keys" "app" {
  key {
    name    = "thingy"
    path    = "thingy"
  }
}

output "result" {
  value     = data.consul_keys.app.var.thingy
  sensitive = true
}

With this configuration:

• Terraform notices that provider "consul" depends on ephemeral.terraform_ssh_tunnels.foo, and so it opens the SSH tunnel before configuring the provider and then waits until the provider has done all of its work before closing the SSH tunnel.
• The Consul provider's address argument is set to an ephemeral string containing something like 127.0.0.1:1234 where 1234 is a randomly-selected port number representing the local end of the tunnel.
• When Consul connects to 127.0.0.1:1234, the terraform_ssh_tunnels implementation accepts the connection, then connects to 127.0.0.1:8500, and then forwards bytes back and forth between the two as a proxy.

Here's the plan graph for this configuration, showing the above diagrammatically:

(You should read this graph bottom-to-top, because it's using Terraform's old-style graph rendering with the "root" -- the final node visited -- at the top.)

This achieves the use-case described in #8367.