feat: Add step 5-app-infra (#382)

* Adds environment base module to create gce instance

* Adds development environment for business unit 1 example

* Adds empty README for step 5

* Adds non-prodcution environment for bu1 example

* Adds production environment for bu1 example

* Adds common.tfvars example and symbolic links

* Adds fixture for 5-app-infra

* Adds outputs for test, adds initial validation of gce instance

* Updates README

* fix linting on readme

* Adds step on build, adds number of instances as variable

* Adds tmp_plan folder on gitignore, pins version of modules

* Changes impersionate configuration to native

* Adds name of instances in outputs

* Fixes symbolic links

* Fixes fixture for test

* Adds example tfvars

* Adds initial version of README

* Fixes type of regioni

* Fixes type

* Fixes integration test

* Adds repository for step 5, adds instructions on README

* Apply suggestions from code review

Co-authored-by: Bharath KKB <bharathkrishnakb@gmail.com>

* Undo app infra repo

* Adds instructions for app infra pipeline

* Removes static_ips variable, adds environment tfvars

* Fixes service_account variable for test

* Undo change on bootstrap test

* Adds instructions for deploy

* Renames variable for service account used to create infra

* Apply suggestions from code review to step 5 README

Co-authored-by: Bharath KKB <bharathkrishnakb@gmail.com>

* fix lint issues

Co-authored-by: Daniel da Silva Andrade <dandrade@ciandt.com>
Co-authored-by: Bharath KKB <bharathkrishnakb@gmail.com>

.gitignore

@@ -57,6 +57,8 @@ override.tf.json
 **/.kitchen
 **/.kitchen.local.yml
 **/Gemfile.lock
+# Plan files
+**/tmp_plan
 
 test/fixtures/shared/terraform.tfvars
 

.kitchen.yml

@@ -113,3 +113,13 @@ suites:
           backend: local
           controls:
             - gcloud-projects
+  - name: app-infra
+    driver:
+      root_module_directory: test/fixtures/app-infra/
+    verifier:
+      color: false
+      systems:
+        - name: app-infra
+          backend: gcp
+          controls:
+            - gcp-app-infra

5-app-infra/README.md

@@ -0,0 +1,72 @@
+# 5-app-infra
+
+The purpose of this step is to deploy a simple [Compute Engine](https://cloud.google.com/compute/) instance in one of the business unit projects using the infra pipeline setup in 4-projects.
+The infra pipeline is created in step 4-projects within the shared env and has a [Cloudbuild](https://cloud.google.com/build/docs) pipeline configured to manage infrastructure within projects. To enable deployment via this pipeline, the projects deployed should [enable](https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/4-projects/business_unit_1/development/example_base_shared_vpc_project.tf#L31-L32) `enable_cloudbuild_deploy` flag and provide the Cloud Build service account value via`cloudbuild_sa`.
+
+This enables the Cloud Build service account to impersonate the project service account and use it to deploy infrastructure. The roles required for project SA can also be [managed](https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/4-projects/business_unit_1/development/example_base_shared_vpc_project.tf#L30) via `sa_roles`. (Note: This requires per project SA impersonation, if you would like to have a single SA managing an environment and all associated projects, that is also possible by [granting](https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/4-projects/modules/single_project/main.tf#L62-L68) `roles/iam.serviceAccountTokenCreator` to an SA with the right roles in `4-projects/env`.
+
+There is also a [Source Repository](https://cloud.google.com/source-repositories) configured with build triggers similar to [foundation pipeline](https://github.com/terraform-google-modules/terraform-example-foundation#0-bootstrap) setup in `0-bootstrap`.
+This Compute Engine instance will be created using the base network created during step 3-networks to access private services.
+
+## Prerequisites
+
+1. 0-bootstrap executed successfully.
+1. 1-org executed successfully.
+1. 2-environments executed successfully.
+1. 3-networks executed successfully.
+1. 4-projects executed successfully.
+
+## Usage
+
+### Setup to run via Cloud Build
+
+1. Clone repo `gcloud source repos clone bu1-example-app --project=prj-bu1-c-infra-pipeline-<random>`. (this is from the terraform output from the previous section, run `terraform output cloudbuild_project_id` in the `4-projects/business_unit_1/shared` folder)
+1. Navigate into the repo `cd bu1-example-app`.
+1. Change freshly cloned repo and change to non master branch `git checkout -b plan`.
+1. Copy contents of foundation to new repo `cp -RT ../terraform-example-foundation/5-app-infra/ .` (modify accordingly based on your current directory).
+1. Copy cloud build configuration files for terraform `cp ../terraform-example-foundation/build/cloudbuild-tf-* . ` (modify accordingly based on your current directory).
+1. Copy terraform wrapper script `cp ../terraform-example-foundation/build/tf-wrapper.sh . ` to the root of your new repository (modify accordingly based on your current directory).
+1. Ensure wrapper script can be executed `chmod 755 ./tf-wrapper.sh`.
+1. Rename `common.auto.example.tfvars` to `common.auto.tfvars` and update the file with values from your environment.
+1. Rename `bu1-development.auto.example.tfvars` to `bu1-development.auto.tfvars` and update the file with values from your environment.
+1. Rename `bu1-non-production.auto.example.tfvars` to `bu1-non-production.auto.tfvars` and update the file with values from your environment.
+1. Rename `bu1-production.auto.example.tfvars` to `bu1-production.auto.tfvars` and update the file with values from your environment.
+1. Commit changes with `git add .` and `git commit -m 'Your message'`.
+1. Push your plan branch to trigger a plan for all environments `git push --set-upstream origin plan` (the branch `plan` is not a special one. Any branch which name is different from `development`, `non-production` or `production` will trigger a terraform plan).
+    1. Review the plan output in your cloud build project https://console.cloud.google.com/cloud-build/builds?project=YOUR_CLOUD_BUILD_PROJECT_ID
+1. Merge changes to development with `git checkout -b development` and `git push origin development`.
+    1. Review the apply output in your cloud build project https://console.cloud.google.com/cloud-build/builds?project=YOUR_CLOUD_BUILD_PROJECT_ID
+1. Merge changes to non-production with `git checkout -b non-production` and `git push origin non-production`.
+    1. Review the apply output in your cloud build project https://console.cloud.google.com/cloud-build/builds?project=YOUR_CLOUD_BUILD_PROJECT_ID
+1. Merge changes to production branch with `git checkout -b production` and `git push origin production`.
+    1. Review the apply output in your cloud build project https://console.cloud.google.com/cloud-build/builds?project=YOUR_CLOUD_BUILD_PROJECT_ID
+
+### Run terraform locally
+
+1. Change into 5-app-infra folder.
+1. Run `cp ../build/tf-wrapper.sh .`
+1. Run `chmod 755 ./tf-wrapper.sh`.
+1. Rename `common.auto.example.tfvars` to `common.auto.tfvars` and update the file with values from your environment and bootstrap.
+1. Update backend.tf with your bucket from infra pipeline example. You can run
+```for i in `find -name 'backend.tf'`; do sed -i 's/UPDATE_ME/<YOUR-BUCKET-NAME>/' $i; done```.
+
+We will now deploy each of our environments(development/production/non-production) using this script.
+When using Cloud Build or Jenkins as your CI/CD tool each environment corresponds to a branch is the repository for 5-app-infra step and only the corresponding environment is applied.
+
+To use the `validate` option of the `tf-wrapper.sh` script, the latest version of `terraform-validator` must be [installed](https://github.com/forseti-security/policy-library/blob/master/docs/user_guide.md#how-to-use-terraform-validator) in your system and in you `PATH`.
+
+1. Run `./tf-wrapper.sh init production`.
+1. Run `./tf-wrapper.sh plan production` and review output.
+1. Run `./tf-wrapper.sh validate production $(pwd)/../policy-library <YOUR_INFRA_PIPELINE_PROJECT>` and check for violations.
+1. Run `./tf-wrapper.sh apply production`.
+1. Run `./tf-wrapper.sh init non-production`.
+1. Run `./tf-wrapper.sh plan non-production` and review output.
+1. Run `./tf-wrapper.sh plan non-production` and review output.
+1. Run `./tf-wrapper.sh validate non-production $(pwd)/../policy-library <YOUR_INFRA_PIPELINE_PROJECT>` and check for violations.
+1. Run `./tf-wrapper.sh apply non-production`.
+1. Run `./tf-wrapper.sh init development`.
+1. Run `./tf-wrapper.sh plan development` and review output.
+1. Run `./tf-wrapper.sh validate development $(pwd)/../policy-library <YOUR_INFRA_PIPELINE_PROJECT>` and check for violations.
+1. Run `./tf-wrapper.sh apply development`.
+
+If you received any errors or made any changes to the Terraform config or `terraform.tfvars` you must re-run `./tf-wrapper.sh plan <env>` before run `./tf-wrapper.sh apply <env>`.

5-app-infra/bu1-development.auto.example.tfvars

@@ -0,0 +1,17 @@
+/**
+ * Copyright 2021 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+project_service_account = "project-service-account@prj-bu1-d-sample-base-<random>.iam.gserviceaccount.com"

5-app-infra/bu1-non-production.auto.example.tfvars

@@ -0,0 +1,17 @@
+/**
+ * Copyright 2021 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+project_service_account = "project-service-account@prj-bu1-n-sample-base-<random>.iam.gserviceaccount.com"

5-app-infra/bu1-production.auto.example.tfvars

@@ -0,0 +1,17 @@
+/**
+ * Copyright 2021 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+project_service_account = "project-service-account@prj-bu1-p-sample-base-<random>.iam.gserviceaccount.com"

5-app-infra/business_unit_1/development/README.md

@@ -0,0 +1,24 @@
+<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| folder\_prefix | Name prefix to use for folders created. | `string` | `"fldr"` | no |
+| instance\_region | The region where compute instance will be created. A subnetwork must exists in the instance region. | `string` | n/a | yes |
+| org\_id | The organization id for the associated services | `string` | n/a | yes |
+| parent\_folder | Optional - if using a folder for testing. | `string` | `""` | no |
+| project\_service\_account | Service account email of the account created on step 4-project for the project where the GCE will be created | `string` | n/a | yes |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| available\_zones | List of available zones in region |
+| instances\_details | List of details for compute instances |
+| instances\_names | List of names for compute instances |
+| instances\_self\_links | List of self-links for compute instances |
+| instances\_zones | List of zone for compute instances |
+| project\_id | Project where compute instance was created |
+| region | Region where compute instance was created |
+
+<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

5-app-infra/business_unit_1/development/backend.tf

@@ -0,0 +1,22 @@
+/**
+ * Copyright 2021 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+terraform {
+  backend "gcs" {
+    bucket = "UPDATE_ME"
+    prefix = "terraform/app-infra/business_unit_1/development"
+  }
+}

5-app-infra/business_unit_1/development/bu1-development.auto.tfvars

@@ -0,0 +1 @@
+../../bu1-development.auto.tfvars

5-app-infra/business_unit_1/development/common.auto.tfvars

@@ -0,0 +1 @@
+../../common.auto.tfvars

5-app-infra/business_unit_1/development/main.tf

@@ -0,0 +1,33 @@
+/**
+ * Copyright 2021 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+
+data "google_active_folder" "env" {
+  display_name = "${var.folder_prefix}-development"
+  parent       = var.parent_folder != "" ? "folders/${var.parent_folder}" : "organizations/${var.org_id}"
+}
+
+module "base_shared_gce_instance" {
+  source         = "../../modules/env_base"
+  environment    = "development"
+  vpc_type       = "base"
+  num_instances  = 1
+  folder_id      = data.google_active_folder.env.name
+  business_code  = "bu1"
+  project_suffix = "sample-base"
+  region         = var.instance_region
+}

5-app-infra/business_unit_1/development/outputs.tf

@@ -0,0 +1,50 @@
+/**
+ * Copyright 2021 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+output "instances_self_links" {
+  description = "List of self-links for compute instances"
+  value       = module.base_shared_gce_instance.instances_self_links
+}
+
+output "instances_names" {
+  description = "List of names for compute instances"
+  value       = [for u in module.base_shared_gce_instance.instances_details : u.name]
+}
+
+output "instances_zones" {
+  description = "List of zone for compute instances"
+  value       = [for u in module.base_shared_gce_instance.instances_details : u.zone]
+}
+
+output "instances_details" {
+  description = "List of details for compute instances"
+  value       = module.base_shared_gce_instance.instances_details
+}
+
+output "available_zones" {
+  description = "List of available zones in region"
+  value       = module.base_shared_gce_instance.available_zones
+}
+
+output "project_id" {
+  description = "Project where compute instance was created"
+  value       = module.base_shared_gce_instance.project_id
+}
+
+output "region" {
+  description = "Region where compute instance was created"
+  value       = module.base_shared_gce_instance.region
+}

5-app-infra/business_unit_1/development/providers.tf

@@ -0,0 +1,30 @@
+/**
+ * Copyright 2021 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+locals {
+  tf_sa = var.project_service_account
+}
+
+/******************************************
+  Provider credential configuration
+ *****************************************/
+provider "google" {
+  impersonate_service_account = local.tf_sa
+}
+
+provider "google-beta" {
+  impersonate_service_account = local.tf_sa
+}

5-app-infra/business_unit_1/development/variables.tf

@@ -0,0 +1,42 @@
+/**
+ * Copyright 2021 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+variable "project_service_account" {
+  description = "Service account email of the account created on step 4-project for the project where the GCE will be created"
+  type        = string
+}
+
+variable "org_id" {
+  description = "The organization id for the associated services"
+  type        = string
+}
+
+variable "instance_region" {
+  description = "The region where compute instance will be created. A subnetwork must exists in the instance region."
+  type        = string
+}
+
+variable "folder_prefix" {
+  description = "Name prefix to use for folders created."
+  type        = string
+  default     = "fldr"
+}
+
+variable "parent_folder" {
+  description = "Optional - if using a folder for testing."
+  type        = string
+  default     = ""
+}

5-app-infra/business_unit_1/non-production/README.md

@@ -0,0 +1,24 @@
+<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| folder\_prefix | Name prefix to use for folders created. | `string` | `"fldr"` | no |
+| instance\_region | The region where compute instance will be created. A subnetwork must exists in the instance region. | `string` | n/a | yes |
+| org\_id | The organization id for the associated services | `string` | n/a | yes |
+| parent\_folder | Optional - if using a folder for testing. | `string` | `""` | no |
+| project\_service\_account | Service account email of the account created on step 4-project for the project where the GCE will be created | `string` | n/a | yes |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| available\_zones | List of available zones in region |
+| instances\_details | List of details for compute instances |
+| instances\_names | List of names for compute instances |
+| instances\_self\_links | List of self-links for compute instances |
+| instances\_zones | List of zone for compute instances |
+| project\_id | Project where compute instance was created |
+| region | Region where compute instance was created |
+
+<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->