Intro to the Foundations of Deep Learning

Summary

Hello and welcome to the repo for Breakout Mentors Machine Learning (ML) Academy! My name is Kai Middlebrook. I'm a full-stack data scientist with lots of experience building ML models that solve real-world problems. I'm also the co-creator of the Breakout Mentors ML Academy and the primary author of this repo.

The motivation for creating this repo and the Breakout Mentors ML Academy was to make ML accessible and inclusive to high school students from a wide range of backgrounds (don't worry if you're not in high school, as we've helped plenty of people both young and old learn ML). We subscribe to the idea that you don't need a Ph.D. to benefit from ML, nor do you need to be a mathematical wizard to grasp it. Anyone can learn ML if they have an open mind, a passion for learning, and a desire to solve the problems that matter to them. If that's you, then you've come to the right place!

To help you along in your ML journey, we (Breakout Mentors) adopt a philosophy of teaching the "whole game" and integrating "just-in-time" teaching/learning. Meaning, you as the student get to play with actual code and solve real-world problems using modern data science tools right away instead of being overwhelmed by an endless stream of equations and complex theories. You'll pick up the math and theory as you progress through the material and work on projects.

As you'll soon find, each lesson contains some new concepts, a practical application of those concepts on real-world data, and is followed by a challenge problem or two for you to practice your understanding. We introduce concepts at a high-level and occasionally opt to omit a few fine-grained details that might have otherwise stunted rather than elevated your understanding, as our primary goal is to develop your intuition for ML concepts. After completing all the lessons, you'll be ready to learn more advanced ML concepts and build tools to solve problems you care about.

Course outline: We start our journey by learning linear regression - a fundamental algorithm in ML to solve problems where the goal is to predict a numerical output value. We begin by learning how we can reinterpret the familiar equation for a line y=mx+by=mx+b from an ML perspective. We next introduce logistic and softmax regression, which extends linear regression to classification problems by showing how we can use special functions called activation functions to convert numerical outputs to valid probability scores. We then introduce deep linear neural networks by leveraging our understanding of regression and learning how we can stack many linear layers with activation functions to make learning complex relationships possible. Along the way, you'll discover how to build neural networks from scratch and come to understand how to train and evaluate them.

Some of the data science tools we will work with include: Pandas and NumPy (for manipulating data), Matplotlib's Pyplot (for visualizing data), and Tensorflow and PyTorch (for building and training machine learning models).

Always Open in Colab - many of the lessons contain content that may not appear on Github. For the best learning experience, open lessons in Google Colab by clicking the button 'Open in Colab' when you view a notebook.

Section 0: Introduction to Python Libraries used in Machine Learning

Learning Objective: Refresh memory of Python and learn common Python libraries and processes needed for Machine Learning work.

• ML Packet: Intro to Python and ML Tools
• Object Oriented Programming (OOP) with Python
  • OOP Challenge
• NumPy Basics
  • NumPy Challenge
• Data Manipulation with Pandas
  • Data Manipulation Challenge
• Data Visualization with Seaborn and Matplotlib
  • Data Visualization Challenge

Extra: Deep Learning Concepts and Python tools:

Learning Objective: Learn at the highest level the fundamental terminology and concepts behind deep learning.

• Video: Intro to Google Colab (~5 min)
• Terminology and Concepts Introduction
• Gradio MNIST

Section 1: Linear Regression

Learning Objective: Understand how to build, train, and evaluate single-layer linear neural networks to solve regression problems.

• Part 1: Single-variable Linear Regression using Sklearn (Beginner/Friendly)

• Part 2: Single-variable Linear Regression using PyTorch (Intermediate/More Experienced)

  • Pre-lesson: Video: Crash Course - Linear Regression (~13 min)
  • Other resources: Pandas Basics, Sklearn Basics, Matplotlib Pyplot Basics, NumPy Basics
  • Challenges:
    • Predict your school's comedy show attendance (First challenge from the Single-variable Linear Regression Colab lesson)

• Finding a Dataset

• Cleaning and Preparing Data

  • Challenges:
    • Cleaning and Preparing your own dataset
    • Single-variable Linear Regression with student's dataset

• Single-variable Linear Regression as Neural Networks, a Deep Learning Perspective

  • Challenges:
    • Revisiting Single-variable Linear Regression with Airbnb data
    • Revisiting Single-variable Linear Regression with Student's dataset

• Multiple Linear Regression

  • Challenges:
    • Multiple Linear Regression with Airbnb data
    • Multiple Linear Regression with student's dataset

Section 2: Classification with Logistic and Softmax Regression

Learning Objective: Understand how to build, train, and evaluate single-layer linear neural networks to solve binary and multi-class classification problems.

• Logistic Regression Classifier

  • Pre-lesson: Video: Logistic Regression (~9 min)
  • Other resources: Beyond Accuracy: Precision and Recall
  • Challenges:
    • Logistic Regression with Airbnb data
    • Logistic Regression with student's dataset

• Softmax Regression Classifier

  • Challenges:
    • Softmax Regression with Airbnb data

• How to prepare image data for neural networks

  • Challenges:
    • Softmax regression with student's dataset

Section 3: Image Classification with Deep Neural Networks

Learning Objective: Understand how to build, train, and evaluate deep linear neural networks to solve regression and classification problems.

• Introduction to Deep Linear Neural Networks

  • Pre-lesson: Video: 3blue1Brown - But What is a Neural Network? (~19 min)
  • Other resources: Blog post: Jay Alammar - A visual guide to the basics of Neural Networks (level: intermediate), Blog post: Jay Alammar - A visual guide to the math behind the basics of Neural Networks (level: intermediate)
  • Challenges:
    • Feed Forward Neural Network with Fashion-MNIST

• Using external image datasets in Colab

  • Challenges:
    • Neural Network with your own image dataset

• Introduction to CNNs for Medicine

Section 4: Building ML Apps

Learning Objective: Understand how to deploy web apps that use our models.

• Gradio (High Level)
  • Digit and Letter Classifier
  • Transfer Learning Tutorial
    • Finetuning a Pretrained Network
    • Deploy Image Classification App
  • Transfer Learning Challenge
    • Finetuning a Pretrained Network
    • Deploy Image Classification App

Reinforcement Learning (optional)

⚠️🛑 ML Mentors: Please connect with Kai (via email) to discuss whether jumping into the reinforcement learning content makes sense for your student.

Learning Objective: Learn the foundations of reinforcement learning (RL) and understand when and how to implement RL algorithms in your projects

Learning RL can be difficult as most content is not accessible to younger students. That's why we built our own RL curriculum to enable our students to learn the foundations of RL and leverage RL algorithms in their projects! The curriculum is hands-on, and interactive, just like all the other Breakout Mentors machine learning curriculums! It's designed to be accessible for students who have completed sections 1-4 of our deep learning foundations course. Students interested in game development, robotics, recommendation systems (e.g., Spotify, Netflix, etc.), dynamic decision-making (finance, economics, etc.), and/or machine learning with human feedback may benefit from our RL curriculum. Talk to your ML Mentor to see if learning RL makes sense for your goals and timeline (ML Mentors, see note at the beginning of this section).

About: Reinforcement Learning (RL) is a machine learning paradigm designed to enable agents (e.g., computers) to learn optimal decision-making strategies by interacting with an environment. The primary goal of RL is to train agents to make a sequence of actions that achieve the desired outcome over time. RL is invaluable in situations where explicit instructions or labeled data are scarce or impractical, as it allows agents to autonomously discover optimal behaviors through trial and error. It operates on the principle of feedback: an agent takes actions, observes the environment's response, and uses this feedback to adjust its decision-making process over successive interactions. Through this iterative process of exploration and exploitation, RL algorithms learn to make intelligent decisions, making them particularly suitable for applications in robotics, game playing, recommendation systems, and other domains where dynamic decision-making is essential.

Advanced Deep Learning Resources

Learning Objective: continue learning about deep learning!

• Lecture: MIT Deep Learning Basics
• Course: Fastai
• Book: Dive into Deep Learning
• Github Repo on ML topics with Colab Lessons: ML Notebooks

Other resources

• Video: Intro to Google Colab (~5 min), Notebook: Intro to Google Colab
• "Open in Colab" browser extension - Open any jupyter notebook on GitHub in Google Colab
• Pandas Basics
• Matplotlib Pyplot Basics
• Graphing Basics
• Sklearn Basics
• NumPy Basics
• Beyond Accuracy: Precision and Recall
• Video Series: 3Blue1Brown - Neural Networks (4 videos)
• Blog post: Jay Alammar's visual guide to the basics of Neural Networks (level: intermediate)
• Blog post: Jay Alammar's visual guide to the math behind the basics of Neural Networks (level: intermediate)
• CNN Explainer