- A data ingestion pipeline workflow orchestration using Prefect
- The following page contains info regarding the NY_Taxi_Dataset_Info
- The dataset can be found in NY_Taxi_Dataset
- To start the prefect server, run the following command
prefect server start
- The flows can be visualized using the UI
- The setup, usage and deployment instructions are documented in the blog post on Prefect
- Either run the script just like any normal python script or use prefect workpools and deployments to schedule runs which is recommended