Data Ingestion pipeline workflow orchestration with Prefect

Info

• A data ingestion pipeline workflow orchestration using Prefect
• The following page contains info regarding the NY_Taxi_Dataset_Info
• The dataset can be found in NY_Taxi_Dataset

Prefect

• To start the prefect server, run the following command

prefect server start

• The flows can be visualized using the UI

Instructions

• The setup, usage and deployment instructions are documented in the blog post on Prefect
• Either run the script just like any normal python script or use prefect workpools and deployments to schedule runs which is recommended