Terraform sync tool #290
Draft
Terraform sync tool #290
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
danieldeleo
commented
Jun 8, 2022
danieldeleo
commented
Jun 8, 2022
danieldeleo
commented
Jun 8, 2022
danieldeleo
commented
Jun 8, 2022
danieldeleo
commented
Jun 8, 2022
danieldeleo
commented
Jun 8, 2022
danieldeleo
commented
Jun 8, 2022
danieldeleo
commented
Jun 8, 2022
danieldeleo
commented
Jun 8, 2022
danieldeleo
commented
Jun 8, 2022
danieldeleo
commented
Jun 8, 2022
danieldeleo
commented
Jun 8, 2022
Co-authored-by: Daniel De Leo <danieldeleo@users.noreply.github.com>
Co-authored-by: Daniel De Leo <danieldeleo@users.noreply.github.com>
danieldeleo
commented
Jul 6, 2022
danieldeleo
commented
Jul 6, 2022
danieldeleo
commented
Jul 6, 2022
danieldeleo
commented
Jul 6, 2022
danieldeleo
commented
Jul 6, 2022
danieldeleo
commented
Jul 27, 2022
Comment on lines
+77
to
+87
| ├── modules # Terraform modules directory | ||
| │ ├── bigquery # Example Terraform BigQuery Setup | ||
| │ └── ... # Other modules setup you have | ||
| ├── qa # qa environment directory | ||
| │ ├── terragrunt.hcl | ||
| │ └── terraform-sync-tool # Tool terraform-sync-tool | ||
| │ ├── json_schemas # Terraform schema files | ||
| │ ├── terragrunt.hcl | ||
| │ └── ... | ||
| ├── cloudbuild.yaml # Cloud Build configuration file | ||
| ├── deploy.sh # Build Step 0 - contains terragrunt commands |
There was a problem hiding this comment.
All of these should go inside an example/ directory
danieldeleo
commented
Jul 27, 2022
tools/terraform_sync_tool/README.md
Outdated
Comment on lines
3
to
6
| This directory contains the setup for the Terraform Sync Tool. Terraform Sync Tool was designed to address the schema drifts in BigQuery tables and keep the | ||
| Terraform schemas up-to-date with the BigQuery table schemas in production environment. Schema drifts occurred when BigQuery Table schemas are updated by newly | ||
| ingested data while Terraform schema files contain the outdated schemas. Therefore, this tool will detect the schema drifts, trace the origins of the drifts, and alert | ||
| developers/data engineers. |
There was a problem hiding this comment.
Suggested change
| This directory contains the setup for the Terraform Sync Tool. Terraform Sync Tool was designed to address the schema drifts in BigQuery tables and keep the | |
| Terraform schemas up-to-date with the BigQuery table schemas in production environment. Schema drifts occurred when BigQuery Table schemas are updated by newly | |
| ingested data while Terraform schema files contain the outdated schemas. Therefore, this tool will detect the schema drifts, trace the origins of the drifts, and alert | |
| developers/data engineers. | |
| This directory contains the Terraform Sync Tool. This tool intentionally fails your CI/CD pipeline when schema drifts occur between what your BigQuery Terraform resources declare and what's actually present in your BigQuery environment. Theses schema drifts happen when BigQuery tables are updated by processes outside of Terraform (ETL process may dynamically add new columns when loading data into BigQuery). When drifts occur, you end up with outdated BigQuery Terraform resource files. This tool detects the schema drifts, traces the origins of the drifts, and alerts developers/data engineers (by failing the CI/CD pipeline) so they can patch the Terraform in their current commit. |
danieldeleo
commented
Jul 27, 2022
tools/terraform_sync_tool/README.md
Outdated
| ingested data while Terraform schema files contain the outdated schemas. Therefore, this tool will detect the schema drifts, trace the origins of the drifts, and alert | ||
| developers/data engineers. | ||
|
|
||
| The Terraform Schema Sync Tool fails the build attemps if resource drifts are detected and notifies the latest resource information. Developers and data engineers should be able to update the Terraform resources accordingly. |
There was a problem hiding this comment.
Suggested change
| The Terraform Schema Sync Tool fails the build attemps if resource drifts are detected and notifies the latest resource information. Developers and data engineers should be able to update the Terraform resources accordingly. |
danieldeleo
commented
Jul 27, 2022
tools/terraform_sync_tool/README.md
Outdated
| The Terraform Schema Sync Tool fails the build attemps if resource drifts are detected and notifies the latest resource information. Developers and data engineers should be able to update the Terraform resources accordingly. | ||
|
|
||
| Terraform Sync Tool can be integrated into your CI/CD pipeline. You'll need to add two steps to CI/CD pipeline. | ||
| - Step 0: Use Terraform/erragrunt command to detect resource drifts and write output into a JSON file |
There was a problem hiding this comment.
Suggested change
| - Step 0: Use Terraform/erragrunt command to detect resource drifts and write output into a JSON file | |
| - Step 0: Run the Terraform plan command (using either Terraform/Terragrunt) with the `-json` option and write the output into a JSON file using the caret operator `> output.json` |
danieldeleo
commented
Jul 27, 2022
tools/terraform_sync_tool/README.md
Outdated
|
|
||
| ## How to run Terraform Schema Sync Tool | ||
|
|
||
| #### Use Terraform/Terragrunt commands to test if any resources drifts existed |
There was a problem hiding this comment.
All these should be 3rd level headers (###), not 4 (####)
danieldeleo
commented
Jul 27, 2022
| - Step 0: Use Terraform/erragrunt command to detect resource drifts and write output into a JSON file | ||
| - Step 1: Use Python scripts to identify and investigate the drifts | ||
|
|
||
| ## How to run Terraform Schema Sync Tool |
There was a problem hiding this comment.
Suggested change
| ## How to run Terraform Schema Sync Tool | |
| ## How to run Terraform Schema Sync Tool | |
| ```bash | |
| ############### | |
| # Using Terragrunt | |
| ############### | |
| terragrunt run-all plan -json --terragrunt-non-interactive > plan_output.json | |
| python3 terraform_sync.py plan_output.json | |
| ############## | |
| # Using Terraform | |
| ############## | |
| terraform plan -json > plan_output.json | |
| python3 terraform_sync.py plan_output.json |
danieldeleo
commented
Jul 27, 2022
tools/terraform_sync_tool/README.md
Outdated
Comment on lines
18
to
46
| Terragrunt/Terraform commands: | ||
| ``` | ||
| terragrunt run-all plan -json --terragrunt-non-interactive | ||
|
|
||
| # Terraform Command | ||
| terraform plan -json | ||
| ``` | ||
|
|
||
| After running the Terrform plan command, **the event type "resource_drift"("type": "resource_drift") indicates a drift has occurred**. | ||
| If drifts detected, please update your terraform configurations and address the resource drifts based on the event outputs. | ||
|
|
||
|
|
||
| #### Add Could Build Steps to your configuration file | ||
|
|
||
| Please check cloud build steps in `cloudbuild.yaml` file, and add these steps to your Cloud Build Configuration File. | ||
|
|
||
| - step 0: run terraform commands in `deploy.sh` to detects drifts | ||
|
|
||
| Add `deploy.sh` to your project directory. | ||
|
|
||
| - step 1: run python scripts to investigate terraform output | ||
|
|
||
| Add `requirements.txt` and `terraform_sync.py` to your project directory. | ||
|
|
||
| #### (Optional if you haven't created Cloud Build Trigger) Create and configure a new Trigger in Cloud Build | ||
| Make sure to indicate your cloud configuration file location correctly. | ||
|
|
||
| #### That's all you need! Let's commit and test in CLoud Build! | ||
|
|
There was a problem hiding this comment.
Suggested change
| Terragrunt/Terraform commands: | |
| ``` | |
| terragrunt run-all plan -json --terragrunt-non-interactive | |
| # Terraform Command | |
| terraform plan -json | |
| ``` | |
| After running the Terrform plan command, **the event type "resource_drift"("type": "resource_drift") indicates a drift has occurred**. | |
| If drifts detected, please update your terraform configurations and address the resource drifts based on the event outputs. | |
| #### Add Could Build Steps to your configuration file | |
| Please check cloud build steps in `cloudbuild.yaml` file, and add these steps to your Cloud Build Configuration File. | |
| - step 0: run terraform commands in `deploy.sh` to detects drifts | |
| Add `deploy.sh` to your project directory. | |
| - step 1: run python scripts to investigate terraform output | |
| Add `requirements.txt` and `terraform_sync.py` to your project directory. | |
| #### (Optional if you haven't created Cloud Build Trigger) Create and configure a new Trigger in Cloud Build | |
| Make sure to indicate your cloud configuration file location correctly. | |
| #### That's all you need! Let's commit and test in CLoud Build! |
There was a problem hiding this comment.
These are not required to run this tool. This is more related to running the example you provide with this tool.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.