Cookiecutter ETL Data Engineer

A project structure for doing and sharing data engineer work.

Installation guide

Please install the files in requirements.txt for the project execution.

Note: To run the main program it is necessary to install all the required packages. This can be done with the command:

pip3 install -r requirements.txt 

Create a new project

In a folder where you want your project generated:

cookiecutter https://github.com/th3edger/DataEngineer_ETL

Resulting directory structure

├── data
│   ├── processed      <- The final, canonical data sets for modeling.
│   └── raw            <- The original, immutable data dump.
│
├── extract            <- Folder with the necessary files files for data extraction.
|
├── load               <- folder with the necessary files files for data upload.
│
├── transform          <- Folder with the necessary files files for data wrangling.
│
├── pipeline.py        <- Pipeline for successful execution.
│
│        
├── requirements.txt   <- The requirements for the project execution.
│
├── .gitignore         <- Files to ignore by `git`.
│
│
└── README.md          <- The top-level README for developers using this project.