terraform

Terraform

These docs are verbose because this is technology with which developers will rarely interact. I suggest you read them entirely before attempting to run any listed commands.

Prior Art

Persistent vs Ephemeral Infrastructure

Adapted from: https://github.com/HHS/Head-Start-TTADP#persistent-vs-ephemeral-infrastructure

The infrastructure used to run this application can be categorized into two distinct types: ephemeral and persistent

• Ephemeral infrastructure is all the infrastructure that is recreated each time the application deploys. Ephemeral infrastructure includes the "application(s)" (as defined in Cloud.gov), the EC2 instances the application runs on, and the routes that application utilizes. Our CircleCI configuration describes this infrastructure and deploys it to Cloud.gov.
• Persistent infrastructure is all the infrastructure that remains constant and unchanged despite application deployments. Persistent infrastructure includes the database used in each development environment. Our Terraform configuration files describe this infrastructure and instantiates it on Cloud.gov.

This concept is often referred to as mutable vs immutable infrastructure.

Infrastructure as Code

A high-level configuration syntax, called Terraform language, describes our infrastructure. This allows a blueprint of our system to be versioned and treated as we do any other code. This configuration can be acted on locally by a developer if deployments need to be created manually, but it is mostly and ideally executed by CircleCI.

Terraform workflow

Terraform integrates into our CircleCI pipeline via the Terraform orb, and is formally described in our deploy-infrastructure CircleCI job. Upon validating its configuration, Terraform reads the current state of any already-existing remote objects to make sure that the Terraform state is up-to-date, and compares the current configuration to the prior state, noting all differences. Terraform creates a "plan" and proposes a set of change actions that should, if applied, make the remote objects match the configuration – this is the essence of infrastructure as code.

Terraform state

We use an S3 bucket created by Cloud Foundry in Cloud.gov as our remote backend for Terraform. This backend maintains the "state" of Terraform and makes it possible for us to make automated deployments based on changes to the Terraform configuration files. This is the only part of our infrastructure that must be manually configured.

Note that a single S3 bucket maintains the Terraform State for both the development and staging environments, and this instance is deployed in the development space.

	development	staging	production
S3 Key	terraform.tfstate.dev	terraform.tfstate.staging	terraform.tfstate.prod
Service Space	tanf-dev	tanf-dev	tanf-prod

Local Set Up For Manual Deployments

Sometimes a developer will need to run Terraform locally to perform manual operations. Perhaps a new TF State S3 bucket needs to be created in another environment, or there are new services or other major configuration changes that need to be tested first.

1. Install terraform

   • On macOS: brew install terraform
   • On other platforms: Download and install terraform

2. Install Cloud Foundry CLI

   • On macOS: brew install cloudfoundry/tap/cf-cli
   • On other platforms: Download and install cf

3. Install CircleCI local CLI

   • On macOS: brew install circleci
   • On other platforms: Download and install circleci

4. Install jq CLI

   • On macOS: brew install jq
   • On other platforms: Download and install jq

5. Login to Cloud Foundry

      # login
      cf login -a api.fr.cloud.gov --sso
      # Follow temporary authorization code prompt.
      
      # Select the target org (probably `hhs-acf-prototyping`), 
      # and the space within which you want to provision infrastructure.
      
      # Spaces:
      # dev = tanf-dev
      # staging = tanf-staging
      # prod = tanf-prod

6. Set up Terraform environment variables

   In the /terraform directory, you can run the create_tf_vars.sh script which can be modified with details of your current environment, and will yield a variables.tfvars file which must be later passed in to Terraform. For more on this, check out terraform variable definitions.

   ./create_tf_vars.sh
   
   # Should generate a file `variables.tfvars` in the `/terraform/dev` directory.
   # Your file should look something like this:
   #
   # cf_user = "some-dev-user"
   # cf_password = "some-dev-password"
   # cf_space_name = "tanf-dev"

Test Deployment in Development

1. Follow the instructions above and ensure the variables.tfvars file has been generated with proper values.

2. cd into /terraform/dev

3. Prepare terraform backend:

   Remote vs. Local Backend:

   If you merely wish to test some new changes without regards to the currently deployed state stored in the remote TF state S3 bucket, you may want to use a "local" backend with Terraform.

   terraform {
    backend "local" {}
   }

   With this change to main.tf, you should be able to run terraform init successfully.

   Get Remote S3 Credentials:

   In the /terraform directory, you can run the create_backend_vars.sh script which can be modified with details of your current environment, and will yield a backend_config.tfvars file which must be later passed in to Terraform. For more on this, check out terraform variable definitions.

   ./create_backend_vars.sh
   
   # Should generate a file `backend_config.tfvars` in the current directory.
   # Your file should look something like this:
   #
   # access_key = "some-access-key"
   # secret_key = "some-secret-key"
   # region = "us-gov-west-1"

   You can now run terraform init -backend-config backend_config.tfvars and load the remote state stored in S3 into your local Terraform config.

4. Run terraform init if using a local backend, or terraform init -backend-config backend_config.tfvars with the remote backend.

5. Run terraform destroy -var-file variables.tfvars to clear the current deployment (if there is one).

   • If the current deployment isn't destroyed, terraform apply will fail later because the unique service instance names are already taken.
   • Be cautious and weary of your target environment when destroying infrastructure.

6. Run terraform plan -out tfapply -var-file variables.tfvars to create a new execution plan.

7. Run terraform apply "tfapply" to create the new infrastructure.

A similar test deployment can also be executed from the /scripts/deploy-infrastructure-dev.sh script, albeit without the destroy step.

Terraform State S3 Bucket

These instructions describe the creation of a new S3 bucket to hold Terraform's state. This need only be done once per environment (note that currently development and staging environments share a single S3 bucket that exists in the development space). This is the only true manual steps that needs to be taken upon the initial application deployment in new environments. This should only need to be done at the beginning of a deployed app's lifetime.

1. Create S3 Bucket for Terraform State

    cf create-service s3 basic-sandbox tdp-tf-states

2. Create service key

   Now we need a new service key with which to authenticate to our Cloud.gov S3 bucket from CircleCI.

   cf create-service-key tdp-tf-states tdp-tf-key

   The service key details provide you with the credentials that are used with common file transfer programs by humans or configured in external systems. Typically, you would create a unique service key for each external client of the bucket to make it easy to rotate credentials in case they are leaked.

   To later revoke access (e.g. when no longer required, or when compromised), you can run cf delete-service-key tdp-tf-states tdp-tf-key.

3. Get the credentials from the service key

   cf service-key tdp-tf-states tdp-tf-key

Update terraform remote state with the changes done outside of terraform

If there are changes that are done directly in cloud.gov or using cf commands, then the remote config will be different from both config file and from the state config file. Below, we will use an example change that has been done on cloud.gov UI. Assume we have created a new elastic service in dev environment called "es-dev". To be able to sync everything with the remote changes follow the blow steps:

1. update the config file with the resource/changes. E.g: add the following lines to config file: ``` data "cloudfoundry_service" "elasticsearch" { name = "aws-elasticsearch" }

     resource "cloudfoundry_service_instance" "elasticsearch" {
        name             = "es-dev"
        space            = data.cloudfoundry_space.space.id
        service_plan     = data.cloudfoundry_service.elasticsearch.service_plans["es-dev"]
     }
   ```
   

If we try to run plan or deploy at this point, then it will fail since the state doesn't have new "es-dev" elastic search service, so it assumes this is a new deployment and tries to deploy the new instance, which will fail since the name is already taken.

2. grab the id of remote change (in this case elastic service) by running cf commands. for the case of our example, we can run cf services, and then run cf service es-dev --guid which will show guid of newly created elasticsearch service instance, which is required for updating state with ES instance.

3. run this command to update state: terraform import cloudfoundry_service_instance.elasticsearch <guid from previous step>

You should change cloudfoundry_service_instance.elasticsearch to your instance/service you added and trying to update the state file with.

Security

The Terraform State S3 instance is set to be encrypted (see main.tf#backend). Amazon S3 protects data at rest using 256-bit Advanced Encryption Standard.

Rotating credentials:

The S3 service creates unique IAM credentials for each application binding or service key. To rotate credentials associated with an application binding, unbind and rebind the service instance to the application. To rotate credentials associated with a service key, delete and recreate the service key.