ETL pipeline tailored for Olympics data
-
Updated
Jun 12, 2024 - Python
ETL pipeline tailored for Olympics data
Tokyo-olympic-azure-data-engineering-end-to-end-project
This project builds an End-to-End Azure Data Engineering Pipeline, performing ETL and Analytics Reporting on the AdventureWorks2017LT Database.
A stand-alone test framework that allows to write unit tests for Data Factory pipelines on Microsoft Fabric and Azure Data Factory.
Config files for my GitHub profile.
In this repository, you will find varies demo and presentations I have delivered throughout the year. This includes the link to the video, the source codes and the data files.
Open-source Repository of Useful Scripts and Solutions for Microsoft Data Engineers
In this project we are going to create an end-to-end data platform right from Data Ingestion, Data Transformation, Data Loading and Reporting.
A cutting-edge data project leverages Azure's suite of services to seamlessly transform raw data from GitHub into actionable insights. Using Azure Data Factory for data ingestion, Databricks for PySpark transformations, Synapse Analytics for advanced analysis, and Power BI for intuitive visualization, this project navigates complex data workflows..
Databricks ETL Pipeline for retrieving and processing NI TestStand test results, featuring a well-documented notebook for ETL operations, Data Lake for storage, Spark SQL+Python for transformations, and Power BI as the final visualization of factory metrics.
This repository contains code for an end-to-end IoT data pipeline using Azure services. It ingests, processes, and stores IoT device data from AWS S3 to Azure Data Lake Storage and Azure SQL Database, leveraging Azure Data Factory and Azure Functions for seamless integration and automation.
Created a movie recommendation system on Azure utilizing Spark SQL by analyzing the MovieLens dataset.
A comprehensive ETL pipeline and sales analysis project leveraging Microsoft Azure and PySpark, designed to optimize e-commerce sales by providing actionable insights through detailed data analysis.
This project demonstrates an ETL pipeline using Microsoft Azure for IMDb Movie Rating Dataset analysis. It covers data extraction from Azure Blob Storage, transformation with Azure Databricks, and loading into Azure SQL using Azure Data Factory. The pipeline automates insights generation and is a practical example of cloud-based data engineering.
This repository contains azure data factory related notes, configurations, best practices & all deliverables such as Pipelies, DataSets, Linked-Services and so on.
💻 Migration of On-Prem database to Azure cloud, then transformation and reporting of data
Data Engineering & Software Blog
Tools for deploying Data Factory (v2) in Microsoft Azure
Add a description, image, and links to the azure-data-factory topic page so that developers can more easily learn about it.
To associate your repository with the azure-data-factory topic, visit your repo's landing page and select "manage topics."