Strava ELT

Demo Project

The goal of this project is to implement the skeleton of a robust ELT pipeline. Things to consider are:

• version control
• development flow
• file project structure
• unit testing
• logging
• documentation
• virtual enviornments/dependency management
• orchestration
• general best practices for data engineering
• containerization
• supporting downstream analytics/ML

Overall ELT pipeline

Strava API --> Python --> BigQuery + dbt --> Tableau/ML in Jupyter Notebook

• light data transformation with Pandas
• orchestration through Google Cloud Services
• data storage through BigQuery
• final data transformations (dimensional modeling + OBT) for downstream analytics through dbt
  • dbt strava project
• Containerization via Docker
• ELT job notifications sent through Slack
• Downstream analytics supported by this pipeline
  • dashboard via Tableau
  • cycling ML model via Python/Sklearn

Deployment

• Python application is containerized and pushed to Google Cloud Artifact Registry
• Container is then deployed on Cloud Run Jobs at a set schedule
• Every midnight, the ELT pipeline is ran, checking for new data to upload to BigQuery
• At job completion, a Slack notification with job meta data and success status is sent

Folder structure

• configs : .yml file with API tokens, db user/password, ELT params
• src : source code
• tests : unit tests