feat: Support Terraform remote state when generating GCP resources #39
+248
−18
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Based on #33. We need to optionally support Terraform remote state configuration via a
backend.tf
file that's auto-generated. It contains the following:This allows users to store a file on GCS representing a record of the GCP resources that Terraform keeps track of for their pipelines.
It's now possible to do this via the
--tf-state-bucket
and--tf-state-prefix
arguments forscripts/generate_terraform.py
.Because backend states differ per user per environment, we only create the
backend.tf
file in the env folder specified, e.g..dev/datasets/ml_datasets/_terraform/backend.tf
. That is, we don't check in Terraform remote state files in the repo to keep it optional for everyone.Checklist
README
accordingly.