Skip to content

🌥️ A lightweight data pipeline based on Google Cloud's Storage, Functions, Tasks and Scheduler.

Notifications You must be signed in to change notification settings

mymindwentblvnk/vgn-departure-extractor

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

23 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

VGN Departure Extractor

The VGN Departure Extractor is a data pipeline that extracts departure information from Nuremberg's public transport company VGN and saves it to Google Cloud Storage which is then read by an external BigQuery table. The whole pipeline can easily be deployed to Google Cloud Platform (GCP) with Terraform.

Architecture

GCP resources used in this are Cloud Scheduler, Cloud Functions, Cloud Tasks, Cloud Storage, and BigQuery.

Data Pipeline Architecture

Deployment

You need a GCP project with billing enabled. Also create a Service Account with the role Owner. Export a Key as JSON file and refer it in the following environment variables.

export TF_VAR_credentials_file=path/to/service/account.json
export TF_VAR_project_id=my-gcp-project
terraform init
terraform validate
terraform plan
terraform apply

After terraform apply was executed successfully, the pipeline is executing automatically as defined in the Cloud Scheduler.

About

🌥️ A lightweight data pipeline based on Google Cloud's Storage, Functions, Tasks and Scheduler.

Topics

Resources

Stars

Watchers

Forks