Skip to content

TEAMSchools/teamster

teamster

kipptaf kippnewark kippcamden kippmiami

pdm-managed Trunk License: GPL v3 Contributor Covenant

Photograph taken in 1960. Upload from http://www.fortepan.hu/?lang=en&img=20566, part of Commons:Batch_uploading/Fortepan.HU

Next-gen data orchestration

Features

Dagster

Dagster is our data orchestrator. Every ETL step takes place here.

Dagster Cloud is a hosted front-end for our Dagster servers where you can observe and run integration jobs.

Dagster hosts multiple "code locations", one for each of our business units, including a separate one for our CMO:

  • kippnewark
  • kippcamden
  • kippmiami
  • kipptaf

Each code location hosts and runs the code and configurations for each respective business unit. Behind-the-scenes, these are containers run on Google Cloud Kubernetes. Each code location has it's own respective jobs, schedules, sensors, and assets.

dbt & Github

Before you merge:

  1. Ensure dbt build runs successfully on your branch
  2. Format your SQL changes in dbt
  3. Ensure the Dagster build action runs successfully

Google Cloud Platform

Fivetran & Airbyte