🚧 WARNING

This repository is under construction and contains untested, experimental and unfinished code. This code is not production-ready, and while it could be useful for others, should not be copied over without appropriate scrutiny.

Use at your own risk.

Dagster-infra

This repository contains code to set up infrastructure for Dagster open source on GCP.

Setup

Initial setup

You'll need to come up with a short 5-character name for your project (e.g. 'dagi'). This will be used to name resources in GCP.

• Create a project on GCP
• Log into your own GCP account gcloud auth login
• Set the project gcloud config set project <project-name>
• Create the GitHub Actions service account used to deploy the infra: ./scripts/create_cicd_sa.sh <project-name>
• Create the TF backend state bucket: ./scripts/create_tf_state_bucket.sh <project-short-name>
• Enable the GCP APIs: ./scripts/enable_apis.sh

Devcontainer setup

This repository contains a dev container.

Once set up, execute:

gcloud auth application-default login

To authenticate with GCP.

Next, execute:

just setup

To install dependencies.

To set the default project, open the .justfile, and change 'GCP_PROJECT_NAME' to the name of your project.

Then, execute:

just set_project

Adding secrets

Secrets that are generated outside of the configuration can be added to the GCP secret manager by adding the secret to GH actions secrets. The environment variable should be prefixed with 'DAGSTER_SECRET', and you should add it to '.github/add_secrets.py', which you can also use as a reference.

All dagster secrets are automatically picked up by the terraform deployment and stored as Kubernetes secrets.

This means that, after manually triggering the 'Add secrets' GitHub actions pipeline, you should also trigger the Terraform deployment to ensure that your secrets are available in the cluster / dagster deployment.

Migrating dagster minor version

See this guide

Uncomment lines 112-115 to get values.yaml and do tfapply

Get migration values

helm template dagster-prd dagster/dagster \
    --set "migrate.enabled=true" \
    --show-only templates/job-instance-migrate.yaml \
    --values values.yaml \
    -n dagster-prd > migration.yml

In migration.yml, change value of the postgres secret

Deploy job kubectl apply -f migration.yml -n dagster-prd

Describe pod kubectl describe pod dagster-prd-dagster-webserver-instance-migrate -n dagster-prd

View logs kubectl logs dagster-prd-dagster-webserver-instance-migrate-4lpcs -n dagster-prd