In this project we are going to create an end-to-end data platform right from Data Ingestion, Data Transformation, Data Loading and Reporting.

A cutting-edge data project leverages Azure's suite of services to seamlessly transform raw data from GitHub into actionable insights. Using Azure Data Factory for data ingestion, Databricks for PySpark transformations, Synapse Analytics for advanced analysis, and Power BI for intuitive visualization, this project navigates complex data workflows..

Databricks ETL Pipeline for retrieving and processing NI TestStand test results, featuring a well-documented notebook for ETL operations, Data Lake for storage, Spark SQL+Python for transformations, and Power BI as the final visualization of factory metrics.

A stand-alone test framework that allows to write unit tests for Data Factory pipelines on Microsoft Fabric and Azure Data Factory.

Open-source Repository of Useful Scripts and Solutions for Microsoft Data Engineers

This repository contains code for an end-to-end IoT data pipeline using Azure services. It ingests, processes, and stores IoT device data from AWS S3 to Azure Data Lake Storage and Azure SQL Database, leveraging Azure Data Factory and Azure Functions for seamless integration and automation.

Created a movie recommendation system on Azure utilizing Spark SQL by analyzing the MovieLens dataset.

A comprehensive ETL pipeline and sales analysis project leveraging Microsoft Azure and PySpark, designed to optimize e-commerce sales by providing actionable insights through detailed data analysis.

This project demonstrates an ETL pipeline using Microsoft Azure for IMDb Movie Rating Dataset analysis. It covers data extraction from Azure Blob Storage, transformation with Azure Databricks, and loading into Azure SQL using Azure Data Factory. The pipeline automates insights generation and is a practical example of cloud-based data engineering.

This repository contains azure data factory related notes, configurations, best practices & all deliverables such as Pipelies, DataSets, Linked-Services and so on.

💻 Migration of On-Prem database to Azure cloud, then transformation and reporting of data

Data Engineering & Software Blog

Tools for deploying Data Factory (v2) in Microsoft Azure

Pandemic Analytics with Data Factory

We have dataset of IPL from 2008 to 2020 and we have to visualize analytics on Power BI dashboard. We have to upload that dataset into data lake. After that we have to process that data through pipeline and produce modeled data in warehouse. So, that we will be able to analyze the data in Power BI through pre-defined dashboards.

Terraform module to create data factory resource on AZURE. Azure Data Factory is a cloud-based data integration service provided by Microsoft Azure.

Your guide to navigating the world of Business Intelligence (BI)

Pipeline de dados no Azure para base de imóveis, com estrutura em três camadas (unbound, silver, gold) e trigger automática a cada hora para atualização consistente.