Data Engineering Pet Projects This repository contains a collection of data engineering pet projects that demonstrate various concepts and technologies in the data engineering space. Each project is contained within its own subdirectory and includes a README file with a brief description and instructions on how to set up and run the project.

Table of Contents Project 1: Spotify ETL job in Apache Airflow #Project 2: Data Pipeline with Apache Airflow and PostgreSQL #Project 3: Real-time Streaming Data Pipeline with Apache Kafka and Spark #Project 4: Data Warehousing with Amazon Redshift and AWS Glue

Technologies Used Apache Airflow PostgreSQL Apache Kafka Apache Spark Amazon Redshift AWS Glue

Setup To run any of the projects in this repository, you will need to have the required technologies installed and configured on your machine. Please refer to the README file in each project's subdirectory for detailed setup instructions.

Contributing If you would like to contribute to this repository by adding your own data engineering pet project, please feel free to submit a pull request. We welcome contributions that showcase new technologies or techniques in the data engineering space.

License This repository is licensed under the MIT License. See the LICENSE file for details.

Contact If you have any questions or feedback about this repository, please feel free to contact the author at jj.jamesjijo@gmail.com.