Ayush @ Data Engineering Portfolio

One framework to develop, deploy and operate data workflows with Python and SQL.

💜🌈📊 A Data Engineering Project that implements an ETL data pipeline using Dagster, Apache Spark, Streamlit, MinIO, Metabase, Dbt, Polars, Docker 🌺

Pipeline de dados automatizado para extração e armazenamento de previsões meteorológicas para o setor de turismo.

Let your pipe lines flow thru the Python code in xonsh.

The Security Reference Architecture (SRA) implements typical security features as Terraform Templates that are deployed by most high-security organizations, and enforces controls for the largest risks that customers ask about most often.

In this project, I have created an end to end solution for analyzing the bing latest news data. I have used the microsoft fabric for all the tools.

Pipeline to automate the collection of board game and expansion data from BoardGameGeek's XML API2. Data is stored in Google Cloud Storage and BigQuery. Data is modelled using DBT in a star schema. (Terraform, GCP, Mage, Python, dbt)

Zillow Data Pipeline: Extracts data from Zillow, transfers it through AWS services, and performs analytics. Utilizes Python scripts, AWS Lambda, S3, Amazon RedShift, and QuickSight. Explore docs/images for architecture visuals.

Data Engineering 🛠️ is like the backbone of data processing 📊, managing data pipelines 🚀, warehouses 🏢, and lakes 🌊. It's the bridge 🌉 between raw data and actionable insights, powering businesses 🚀 with efficient data management and analytics 📈.

The NHANES Data 'API' is a Python tool that simplifies access to the National Health and Nutrition Examination Survey (NHANES) dataset. This project provides an easy-to-use API to retrieve NHANES data, helping researchers, data scientists, health professionals, and other stakeholders access these valuable datasets.

This project repo 📺 offers a robust solution meticulously crafted to efficiently manage, process, and analyze YouTube video data leveraging the power of AWS services. Whether you're diving into structured statistics or exploring the nuances of trending key metrics, this pipeline is engineered to handle it all with finesse.

Building a fully automated data Pipeline with Google Cloud Services

Docker powered starter for geospatial analysis of lightning atmospheric data.

Scooter-sharing system use case: This project demonstrates a local and cloud execution of automated data collection and cleaning pipelines.