README.md

Course Information
for
Spring Quarter 2024

Santa Clara University

MSIS 2527: Big Data Modeling & Analytics

---

Course Description & Objectives

• Understand the fundamentals of big data
• Understand the fundamentals of MapReduce Paradigm
• Use PySpark (Python API for Apache Spark) to solve big data problems
• Use SQL for NoSQL data (DataFrames in Spark and Amazon Athena)
• Understand Amazon Athena & Google BigQuery: Access & Analyze Big Data by SQL

---

Course Objectives

At the completion of this course, students will be able to understand:

• Elements of Big Data:

  • Cluster Comouting
  • Persistence, Queries, Analytics
  • Data Replication
  • Distributed File System and Fault Tolerance
  • Scale-out Architecture vs. Scale-up Architecture

• What is MapReduce paradigm?

  • Data partitioning and partitions
  • Mapper function: map()
  • Reducer function: reduce()
  • Combiner function: combine()
  • Sort & Shuffle: SQL's GROUP BY
  • Classic MapReduce Algorithms
  • Data Design Patterns

• Fundamentals of Spark and PySpark:

  • Spark Architecture
  • Spark: engine for large-scale data analytics
  • Data Abstractions in Spark and PySpark
  • RDDs and DataFrames
  • Transformations and Actions
  • Running simple programs in PySpark

• NoSQL Databases & Serverless Architectures

  • SQL for NoSQL data & Relational Algebra
  • Amazon Athena and SQL
  • Google BigQuery and SQL

---

Required books (all resources are online):

• 1. Data-Intensive Text Processing with MapReduce by Jimmy Lin and Chris Dyer

  • for the first 3 weeks of class

• 2. Data Algorithms with Spark by Mahmoud Parsian

  • for the last 7 weeks of class

Extra books (all resources are online):

• 1. PySpark Algorithms by Mahmoud Parsian

• 2. Mining of Massive Datasets by Jure Leskovec, Anand Rajaraman, Jeffrey D. Ullman

---

Required Software, API, and Documentation

• Apache Spark (main site)
• PySpark API and documentation
• RDD Programming Guide
• DataFrame Programming Guide

---

Tentative Course Outline

The weekly coverage might change as it 
depends on the progress of the class. 
However, you must keep up with the reading 
and programming assignments.

Main Subjects

• Classic MapReduce (Jimmy Lin's Book)

  • Solve Big Data problems using map(), combine(), and reduce() functions
  • up to 25%

• PySpark and Spark (Mahmoud Parsian's book: Data Algorithms with Spark)

  • up to 65%

• Data Partitioning and SQL Queries and Amazon Athena and Google BigQuery

  • up to 10%

---

Session-1: Wednesday, April 3, 2024

TOPIC: Introduction to Big Data and MapReduce

• REQUIRED:
  • MapReduce: Simplified Data Processing on Large Clusters by Jeffrey Dean and Sanjay Ghemawat
  • Chapter 1 of Data-Intensive Text Processing with MapReduce
  • A Very Brief Introduction to MapReduce
• OPTIONAL:
  • Introduction to Big Data
  • Introduction to MapReduce

---

Session-2: Monday, April 8, 2024

TOPIC: Introduction to Big Data and MapReduce

• REQUIRED:

  • MapReduce Tutorial Slides by Jimmy Lin
  • Chapter 2 of Data-Intensive Text Processing with MapReduce
  • Introduction to MapReduce by Mahmoud Parsian
  • MapReduce: Simplified Data Processing on Large Clusters by Jeffrey Dean and Sanjay Ghemawat

• OPTIONAL:

  • MapReduce wikipedia
  • A Comprehensive Study on MapReduce Applications
  • Introduction to MapReduce and Hadoop by Matei Zaharia

---

Session-3: Wednesday, April 10, 2024

TOPIC: Introduction to MapReduce

• REQUIRED:

  • Introduction to MapReduce
  • Chapter 3 of Data-Intensive Text Processing with MapReduce
  • Chapter 4 of Data-Intensive Text Processing with MapReduce

• OPTIONAL:

  • Introduction to MapReduce: Watch a Video
  • The Future of Big Data by Matei Zaharia -- Video
  • Introduction to MapReduce and Hadoop by Matei Zaharia

---

Session-4: Monday, April 15, 2024

TOPIC: Introduction to MapReduce

• REQUIRED:

  • Chapter 3 of Data-Intensive Text Processing with MapReduce

  • Chapter 4 of Data-Intensive Text Processing with MapReduce

  • Chapter 5 of Data-Intensive Text Processing with MapReduce

  • MapReduce Algorithms

  • Classic Join in MapReduce (inner join)

  • Join Algorithms Using MapReduce

• OPTIONAL:

  • Simplifying Big Data Applications with Apache Spark 2.0 by Matei Zaharia
  • Introduction to MapReduce and Hadoop by Matei Zaharia
  • Relational Operations Using MapReduce

---

Session-5: Wednesday, April 17, 2024

TOPIC: Introduction to Spark

• REQUIRED:

  • A-Gentle-Introduction-to-Apache-Spark
  • Chapters 1, 2 of Data Algorithms with Spark Book by Mahmoud Parsian
  • PySpark Tutorial
  • Classic Word Count in PySpark

• OPTIONAL:

  • Learning Spark (book)
  • Introduction to Apache Spark

---

Session-6: Monday, April 22, 2024

TOPIC: Introduction to Spark and PySpark (Python API for Spark)

• REQUIRED:

  • Classic Word Count in PySpark
  • A-Gentle-Introduction-to-Apache-Spark
  • Chapters 1, 2, 3, 4 of Data Algorithms with Spark Book by Mahmoud Parsian
  • Learning Spark (book)

• OPTIONAL:

  • Introduction to Spark
  • Introduction to Spark by Shannon Quinn

---

Session-7: Wednesday, April 24, 2024

TOPIC: Spark's Nuts and Bolts

• REQUIRED:

  • PySpark Tutorial
  • Chapters 3, 4, 5 of Data Algorithms with Spark Book by Mahmoud Parsian
  • Learning Spark (book)

• OPTIONAL:

  • Introduction to Spark
  • Parallel-Programming-With-Spark-Matei-Zaharia

---

Session-8: Monday, April 29, 2024

TOPIC: Data Design Patterns

• REQUIRED:
  • PySpark Tutorial
  • MinMax Algorithm
  • Top-10 Algorithm

---

Session-9: Wednesday, May 1, 2024

TOPIC: Data Design Patterns

• REQUIRED:

  • Chapters 3, 4, 5 of Data Algorithms with Spark Book by Mahmoud Parsian
  • Data Design Patterns: InMapper Combiner, mapPartitions
  • Top-10 Algorithm
  • MinMax Algorithm

• OPTIONAL:

  • Chapters 4, 6, 7, 12 of PySpark Algorithms Book by Mahmoud Parsian

---

Session-10: Monday, May 6, 2024

TOPIC: RDD Design Patterns

• REQUIRED:
  • Spark's RDD Partitioning
  • Chapters 3, 4, 5 of Data Algorithms with Spark Book by Mahmoud Parsian
  • Spark's mapPartitions() Transformation
  • mapPartitions() Tutorial
  • Review reducers: groupByKey(), reduceByKey(), and combineByKey()

---

Session-11: Wednesday, May 8, 2024

• Review for Midterm Exam
• Problem solving & Q/A session

---

Session-12: Monday, May 13, 2024

• Midterm Exam
• closed book/notes/friend/internet/software

---

Session-13: Wednesday, May 15, 2024

• Midterm Exam Discussion and Review

---

Session-14: Monday, May 20, 2024

• Spark's DataFrames (1)
• Chapters 4, 6, 7, 12 of PySpark Algorithms Book by Mahmoud Parsian
• Video: Structuring Spark: SQL, DataFrames, Datasets And Streaming - 28 mins

---

Session-15: Wednesday, May 22, 2024

• Spark's DataFrames (2)
• Chapters 4, 6, 7, 12 of PySpark Algorithms Book by Mahmoud Parsian
• Video: Structuring Spark: SQL, DataFrames, Datasets And Streaming - 28 mins

---

Session-16: Monday, May 27, 2024

• Memorial Day; academic and administrative holiday
• No classes, offices closed

---

Session-17: Wednesday, May 29, 2024

• Introduction to Graph data structures
• MapReduce Design Pattern: Graph Algorithms
• Chapter 6 of Data Algorithms with Spark Book by Mahmoud Parsian
• Chapters 11 of PySpark Algorithms Book by Mahmoud Parsian

---

Session-18: Monday, June 3, 2024

• MapReduce Design Pattern: Graph Algorithms
• Chapter 6 of Data Algorithms with Spark Book by Mahmoud Parsian
• Chapters 11 of PySpark Algorithms Book by Mahmoud Parsian

---

Session-19: Wednesday, June 5, 2024

• Introduction to Serverless Analytics
• SQL Access to Big Data
  • SQL Access: Amazon Athena
  • SQL Access: Google BigQuery
• Review for Final Exam
• Q/A session

---

Session-20: Final Exam

• Date: June 10-13, 2024
• closed book/notes/friend/internet/software

---