Skip to content

ekote/Build-Your-First-End-to-End-Lakehouse-Solution

Repository files navigation

Build Your First End to End Lakehouse Solution

Join the workshop to master building end-to-end data solutions with Microsoft Fabric. Learn to integrate, transform, and manage data in a lakehouse, utilizing Fabric pipelines, dataflows, notebooks, and Spark. Understand how BI analysts and data scientists utilize lakehouse data to enhance decision-making.

Workshop Goals

  • Master Fabric Data Integration and Data Engineering and Data Science.
  • Develop a complete data workflow: ingestion, preparation, serving, and operationalization.

Project Context: Urban Mobility Transformation

  • Use Microsoft Fabric to analyze New York City's taxi data for improved urban planning and transportation safety.
  • Aim: Better traffic forecasting, route management, and safety measures, leading to enhanced urban transport services and infrastructure.

Lakehouse Solution Benefits

  • Unified urban transport data analysis.
  • Enhanced fleet management and traffic forecasting.
  • Data-driven urban development insights.
  • Improved transportation reliability and safety.
  • Adaptable data infrastructure for future urban mobility trends.

Build Your First End to End Lakehouse Solution

Agenda

Tip

You can progress through these exercises at your own pace. While we have structured logical breaks within the session, these are merely suggestions. You are not required to stop if you prefer to continue working. These breaks are provided to accommodate those who may need them. Feel free to continue through the material as fits your learning style and needs.

Important

9:00 am - 9:30 am - Introduction, Set Up and Overview of Fabric Data Platform

9:30 am - 10:00 am - Exercise 1 - Ingest data with data pipelines and shortcuts

10:00 am - 10:15 am - Coffee break 15 minutes

10:15 am - 10:45 am - Exercise 1 - Ingest data with data pipelines and shortcuts

10:45 am - 11:45 pm - Exercise 2 - Transform data using Notebooks and Spark clusters

11:45 am - 12:45 pm - Lunch 60 minutes

12:45 pm - 01:15 pm - Exercise 3 - Collaborate inside Notebooks and share Lakehouse. Use SQL Endpoint and SSMS

01:15 pm - 01:25 pm - Break 10 minutes

01:25 pm - 02:25 pm - Exercise 4 - Serve and consume data using Power BI and Data Science

02:25 pm - 02:35 pm - Break 10 minutes

02:35 pm - 03:05 pm - Exercise 5 - Latest Fabric Features

03:05 pm - 04:00 pm - Buffer, Recap and Extra exercises