Idempotency checks

[bastiandg]

It would be useful to have an idempotency check in kitchen-terraform.

What does it mean? A configuration gets rolled out, kitchen-terraform runs a plan afterwards and throws an error in case anything will be changed.

Benefits

Detection of:

1. "competing" resources which overwrite each other
2. bugs in terraform providers (see example below)
3. configuration that gets ignored or overwritten by the provider

Issue example

One example is an odd behavior of the google_container_cluster resource, where terraform in some cases wants to remove the master_authorized_networks_config block on the second apply. Reference: terraform-provider-google #3098

$ terraform apply
[…]
Terraform will perform the following actions:

  # module.k8s.google_container_cluster.cluster[0] will be updated in-place
  ~ resource "google_container_cluster" "cluster" {
[…]
      - master_authorized_networks_config {
        }
[…]
Plan: 0 to add, 1 to change, 0 to destroy.

Minimal test configurations

not idempotent

data "null_data_source" "random" {
}

resource "null_resource" "not_idempotent" {
  triggers = { rand = data.null_data_source.random.random }
}

In this example the second apply wants to recreate the resource:

$ terraform apply
[…]
Terraform will perform the following actions:

  # null_resource.not_idempotent must be replaced
-/+ resource "null_resource" "not_idempotent" {
[…]
Plan: 1 to add, 0 to change, 1 to destroy.

idempotent

resource "random_string" "random" {
  length = 8
}

resource "null_resource" "idempotent" {
  triggers = { rand = random_string.random.result }
}

Here the resource is idempotent and nothing changes on the second apply:

$ terraform apply
random_string.random: Refreshing state... [id=_qp74a2Rt>F+[(Lh]
null_resource.test: Refreshing state... [id=5376417678559308255]

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Workaround

Detection for this in terraform 0.11 was hard. Luckily terraform 0.12 makes it a lot easier to work with plans. So a simple detection can be done by parsing the terraform plan.

#!/bin/bash
terraform plan -out=plan.tfplan > /dev/null
terraform show -json plan.tfplan | \
jq -r '.resource_changes[] |
        select(
                (.change.actions | contains(["create"])) or
                (.change.actions | contains(["destroy"])) or
                (.change.actions | contains(["update"]))
        ) | .address'

The script filters out all resources which will be created, destroyed or updated. There might be other actions which imply a configuration change.

An integration of the workaround:

kitchen converge
bash idempotency-check.sh
kitchen verify
kitchen destroy

Further considerations

In some cases it is necessary to have a constantly changing resource. For example a kubernetes configuration could be written to a file with a templating mechanism and the idempotency will be checked by kubernetes itself.

So it would also be useful to have a configurable exclusion filter for the idempotency check.

[luckeyca]

It will be great to have this idempotency check feature as it's being around for long time for kitchen ansible and terratest. Definitely need an option to opt-out as some test cases may need to disable it. This option is also available with kitchen ansible and terratest.

[luckeyca]

The following workaround works too and it would be great to integrate this into the code so that we don't have to write the test case for each test module as we may also have to either parse the tfvars file and extra variables defined in the kitchen.yml, or write a lots of the duplicate terraform plan for each fixture directory.

describe command("terraform -chdir=test/fixtures/deploy_test plan -var-file=test.tfvars") do
its(:exit_status) { should eq 0 }
its(:stderr) { should eq '' }
its(:stdout) { should match /found no differences, so no changes are needed/}
end