Data Engineering Portfolio

Welcome to my Data Engineering Portfolio! Here, I showcase various projects that highlight my passion for using data and technology to solve real-world problems. Each project represents my dedication to learning and innovation in data engineering.

In this portfolio, you'll find projects covering different technologies and areas, with detailed guides and step-by-step instructions. Whether it's integrating APIs, building cloud-based solutions, or automating data workflows, these projects demonstrate my skills and curiosity in turning data into valuable insights.

I mainly use popular cloud providers like AWS, Google Cloud Platform, and Azure, which offer free-tier accounts. This means you can replicate my projects and explore the technologies I've used at no cost.

Feel free to explore the projects, check out the technologies I've used, and see how I've tackled various challenges in data engineering. With each project, I continue to grow and expand my capabilities in this exciting field.

---

Projects Overview

Weather Data and Text Message Notification

This project obtains weather data and sends them by text message. The project seamlessly integrates Weather APIs, Twilio's SMS service, and AWS EC2 instances, all while adhering to cost-effectiveness within the AWS Free Tier.

Key Highlights

• Weather API Integration: Harness the power of a Weather API to obtain up-to-the-minute weather data.
• Twilio SMS Service: Explore the integration of Twilio for seamless delivery of concise and informative weather updates directly to mobile devices.
• AWS EC2 Instances: Dive into the reliability and scalability of AWS EC2 instances, ensuring periodic execution and cost-effective operation within the AWS Free Tier.

Extract Daily Statistics YouTube

Automating YouTube channel metric extraction, this project integrates Google Cloud for YouTube API access and AWS Lambda for serverless processing with S3 storage. Scheduled updates via AWS EventBridge ensure timely data refresh. AWS Athena and Glue facilitate thorough analysis, offering SQL-like queries and comprehensive table creation for insightful metrics exploration.

Key Highlights

• Google Cloud Setup

  • Enable YouTube API v3 service.
  • Create a secure API key and control access by IP for enhanced security.

• YouTube API Analysis

  • Utilize the YouTube API for daily channel statistics analysis.

• AWS Lambda Function

  • Implement a serverless solution with AWS Lambda.
  • Leverage S3 for data storage and AWS SDK Pandas Layer for efficient data processing.

• AWS EventBridge Integration

  • Schedule daily execution with EventBridge for reliable and automated updates.

• AWS Athena and AWS Glue

  • Set up Athena for SQL-like queries and Glue for crawling and table creation, enabling comprehensive data analysis.

---

Gradually I will be adding more projects with different challenges and technologies.

Contributions

Any contributions you make are greatly appreciated.

• Fork the Project
• Create your Feature Branch (git checkout -b feature/new-feature)
• Commit your Changes (git commit -m 'Add some new-feature')
• Push to the Branch (git push origin feature/new-feature)
• Open a Pull Request

License

This project is licensed under the GPL-3.0 License.

Contact

Please feel free to contact me if you have any questions.

LinkedIn