t81_558_class_01_1_overview.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "whjsJasuhstV"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_01_1_overview.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "euOZxlIMhstX"
   },
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 1: Python Preliminaries**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d4Yov72PhstY"
   },
   "source": [
    "# Module 1 Material\n",
    "\n",
    "* **Part 1.1: Course Overview** [[Video]](https://www.youtube.com/watch?v=IzZSwS45vt4&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_01_1_overview.ipynb)\n",
    "* Part 1.2: Introduction to Python [[Video]](https://www.youtube.com/watch?v=czq5d53vKvo&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_01_2_intro_python.ipynb)\n",
    "* Part 1.3: Python Lists, Dictionaries, Sets and JSON [[Video]](https://www.youtube.com/watch?v=kcGx2I5akSs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_01_3_python_collections.ipynb)\n",
    "* Part 1.4: File Handling [[Video]](https://www.youtube.com/watch?v=FSuSLCMgCZc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_01_4_python_files.ipynb)\n",
    "* Part 1.5: Functions, Lambdas, and Map/Reduce [[Video]](https://www.youtube.com/watch?v=jQH1ZCSj6Ng&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_01_5_python_functional.ipynb)\n",
    "\n",
    "Watch one (or more) of these depending on how you want to setup your Python TensorFlow environment:\n",
    "* [How to Submit a Module Assignment locally](https://www.youtube.com/watch?v=hmCGjCVhYNc)\n",
    "* [How to Use Google CoLab and Submit Assignment](https://www.youtube.com/watch?v=Pt-Od-oBgOM)\n",
    "* [Installing TensorFlow, Keras, and Python in Windows CPU or GPU](https://www.youtube.com/watch?v=PnK1jO2kXOQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN)\n",
    "* [Installing TensorFlow, Keras, and Python for an Intel Mac](https://www.youtube.com/watch?v=LnzgQr14p7s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN)\n",
    "* [Installing TensorFlow, Keras, and Python for an M1 Mac](https://www.youtube.com/watch?v=5DgWvU0p2bk&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AcAUP0c3hstY"
   },
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xsI496h5hstZ",
    "outputId": "cbc72cd3-dfc0-4ba1-b88d-e654b514bba6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    from google.colab import drive\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pC9A-LaYhsta"
   },
   "source": [
    "# Part 1.1: Overview\n",
    "\n",
    "Deep learning is a group of exciting new technologies for neural networks. [[Cite:lecun2015deep]](https://www.nature.com/articles/nature14539) By using a combination of advanced training techniques neural network architectural components, it is now possible to train neural networks of much greater complexity. This book introduces the reader to deep neural networks, regularization units (ReLU), convolution neural networks, and recurrent neural networks. High-performance computing (HPC) aspects demonstrate how deep learning can be leveraged both on graphical processing units (GPUs), as well as grids. Deep learning allows a model to learn hierarchies of information in a way that is similar to the function of the human brain. The focus is primarily upon the application of deep learning, with some introduction to the mathematical foundations of deep learning. Readers will make use of the Python programming language to architect a deep learning model for several real-world data sets and interpret the results of these networks. [[Cite:goodfellow2016deep]](https://www.deeplearningbook.org/)\n",
    "\n",
    "## Origins of Deep Learning\n",
    "\n",
    "Neural networks are one of the earliest examples of a machine learning model.  Neural networks were initially introduced in the 1940s and have risen and fallen several times in popularity. The current generation of deep learning begain in 2006 with an improved training algorithm by Geoffrey Hinton. [[Cite:hinton2006fast]](https://www.mitpressjournals.org/doi/abs/10.1162/neco.2006.18.7.1527) This technique finally allowed neural networks with many layers (deep neural networks) to be efficiently trained. Four researchers have contributed significantly to the development of neural networks.  They have consistently pushed neural network research, both through the ups and downs.  These four luminaries are shown in Figure 1.LUM.\n",
    "\n",
    "**Figure 1.LUM: Neural Network Luminaries**\n",
    "\n",
    "![Neural Network Luminaries](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_1_luminaries_ann.jpg \"Neural Network Luminaries\")\n",
    "\n",
    "The current luminaries of artificial neural network (ANN) research and ultimately deep learning, in order as appearing in the figure:\n",
    "\n",
    "* [Yann LeCun](http://yann.lecun.com/), Facebook and New York University - Optical character recognition and computer vision using convolutional neural networks (CNN).  The founding father of convolutional nets.\n",
    "* [Geoffrey Hinton](http://www.cs.toronto.edu/~hinton/), Google and University of Toronto. Extensive work on neural networks. Creator of deep learning and early adapter/creator of backpropagation for neural networks.\n",
    "* [Yoshua Bengio](http://www.iro.umontreal.ca/~bengioy/yoshua_en/index.html), University of Montreal and Botler AI. Extensive research into deep learning, neural networks, and machine learning.  \n",
    "* [Andrew Ng](http://www.andrewng.org/), Badiu and Stanford University.  Extensive research into deep learning, neural networks, and application to robotics.\n",
    "\n",
    "Geoffrey Hinton, Yann LeCun, and Yoshua Bengio won the [Turing Award](https://www.acm.org/media-center/2019/march/turing-award-2018) for their contributions to deep learning.\n",
    "\n",
    "## What is Deep Learning\n",
    "\n",
    "The focus of this book is deep learning, which is a prevalent type of machine learning that builds upon the original neural networks popularized in the 1980s. There is very little difference between how a deep neural network is calculated compared with the first neural network.  We've always been able to create and calculate deep neural networks.  A deep neural network is nothing more than a neural network with many layers.  While we've always been able to create/calculate deep neural networks, we've lacked an effective means of training them.  Deep learning provides an efficient means to train deep neural networks.\n",
    "\n",
    "\n",
    "If deep learning is a type of machine learning, this begs the question, \"What is machine learning?\"  Figure 1.ML-DEV  illustrates how machine learning differs from traditional software development.\n",
    "\n",
    "**Figure 1.ML-DEV: ML vs Traditional Software Development**\n",
    "![ML vs Traditional Software Development](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_1_ml_vs_trad.png \"Machine Learning vs Traditional Software Development\")\n",
    "\n",
    "* **Traditional Software Development** - Programmers create programs that specify how to transform input into the desired output.\n",
    "* **Machine Learning** - Programmers create models that can learn to produce the desired output for given input. This learning fills the traditional role of the computer program. \n",
    "\n",
    "Researchers have applied machine learning to many different areas.  This class explores three specific domains for the application of deep neural networks, as illustrated in Figure 1.ML-DOM.\n",
    "\n",
    "**Figure 1.ML-DOM: Application of Machine Learning**\n",
    "![Application of Machine Learning](https://s3.amazonaws.com/data.heatonresearch.com/images/jupyter/dl-domains.jpg \"Application of Machine Learning\")\n",
    "\n",
    "* **Computer Vision** - The use of machine learning to detect patterns in visual data. For example, is an image a picture of a cat or a dog.\n",
    "* **Tabular Data** - Several named input values allow the neural network to predict another named value that becomes the output. For example, we are using four measurements of iris flowers to predict the species. This type of data is often called tabular data.\n",
    "* **Natural Language Processing (NLP)** - Deep learning transformers have revolutionized NLP, allowing text sequences to generate more text, images, or classifications.\n",
    "* **Reinforcement Learning** - Reinforcement learning trains a neural network to choose ongoing actions so that the algorithm rewards the neural network for optimally completing a task.\n",
    "* **Time Series** - The use of machine learning to detect patterns in time. Typical time series applications are financial applications, speech recognition, and even natural language processing (NLP). \n",
    "* **Generative Models** - Neural networks can learn to produce new original synthetic data from input. We will examine StyleGAN, which learns to create new images similar to those it saw during training.\n",
    "\n",
    "## Regression, Classification and Beyond\n",
    "\n",
    "Machine learning research looks at problems in broad terms of supervised and unsupervised learning. Supervised learning occurs when you know the correct outcome for each item in the training set. On the other hand, unsupervised learning utilizes training sets where no correct outcome is known. Deep learning supports both supervised and unsupervised learning; however, it also adds reinforcement and adversarial learning. Reinforcement learning teaches the neural network to carry out actions based on an environment. Adversarial learning pits two neural networks against each other to learn when the data provides no correct outcomes. Researchers continue to add new deep learning training techniques.\n",
    "\n",
    "Machine learning practitioners usually divide supervised learning into classification and regression. Classification networks might accept financial data and classify the investment risk as risk or safe. Similarly, a regression neural network outputs a number and might take the same data and return a risk score. Additionally, neural networks can output multiple regression and classification scores simultaneously. \n",
    "\n",
    "One of the most powerful aspects of neural networks is that the input and output of a neural network can be of many different types, such as:\n",
    "\n",
    "* An image\n",
    "* A series of numbers that could represent text, audio, or another time series\n",
    "* A regression number\n",
    "* A classification class\n",
    "\n",
    "## Why Deep Learning?\n",
    "\n",
    "For tabular data, neural networks often do not perform significantly better that different than other models, such as:\n",
    "\n",
    "* [Support Vector Machines](https://en.wikipedia.org/wiki/Support-vector_machine)\n",
    "* [Random Forests](https://en.wikipedia.org/wiki/Random_forest)\n",
    "* [Gradient Boosted Machines](https://en.wikipedia.org/wiki/Gradient_boosting)\n",
    "\n",
    "Like these other models, neural networks can perform both **classification** and **regression**. When applied to relatively low-dimensional tabular data tasks, deep neural networks do not necessarily add significant accuracy over other model types. However, most state-of-the-art solutions depend on deep neural networks for images, video, text, and audio data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VuXAE2fthstb"
   },
   "source": [
    "# Python for Deep Learning\n",
    "\n",
    "We will utilize the Python 3.x programming language for this book. Python has some of the widest support for deep learning as a programming language. The two most popular frameworks for deep learning in Python are:\n",
    "\n",
    "* [TensorFlow/Keras](https://www.tensorflow.org/) (Google)\n",
    "* [PyTorch](https://pytorch.org/) (Facebook)\n",
    "\n",
    "\n",
    "This book focuses primarily upon Keras, with some applications in PyTorch. For many tasks, we will utilize Keras directly. We will utilize third-party libraries for higher-level tasks, such as reinforcement learning, generative adversarial neural networks, and others. These third-party libraries may internally make use of either PyTorch or Keras. I chose these libraries based on popularity and application, not whether they used PyTorch or Keras.\n",
    "\n",
    "To successfully use this book, you must be able to compile and execute Python code that makes use of TensorFlow for deep learning. There are two options for you to accomplish this:\n",
    "\n",
    "* Install Python, TensorFlow and some IDE (Jupyter, TensorFlow, and others).\n",
    "* Use [Google CoLab](https://colab.research.google.com/) in the cloud, with free GPU access.\n",
    "\n",
    "If you look at this notebook on Github, near the top of the document, there are links to videos that describe how to use Google CoLab. There are also videos explaining how to install Python on your local computer. The following sections take you through the process of installing Python on your local computer. This process is essentially the same on Windows, Linux, or Mac. For specific OS instructions, refer to one of the tutorial YouTube videos earlier in this document.\n",
    "\n",
    "To install Python on your computer, complete the following instructions:\n",
    "\n",
    "* [Installing Python and TensorFlow - Windows/Linux](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/install/tensorflow-install-jul-2020.ipynb)\n",
    "* [Installing Python and TensorFlow - Mac Intel](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/install/tensorflow-install-mac-jan-2021.ipynb)\n",
    "* [Installing Python and TensorFlow - Mac M1](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/install/tensorflow-install-mac-metal-jul-2021.ipynb)\n",
    "\n",
    "## Check your Python Installation\n",
    "\n",
    "Once you've installed Python, you can utilize the following code to check your Python and library versions. If you have a GPU, you can also check to see that Keras recognize it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "okZQe6lzhstb",
    "outputId": "5c9bdf12-f339-48bc-cb49-0b3d8008ec55"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor Flow Version: 2.8.0\n",
      "Keras Version: 2.8.0\n",
      "\n",
      "Python 3.7.13 (default, Mar 16 2022, 17:37:17) \n",
      "[GCC 7.5.0]\n",
      "Pandas 1.3.5\n",
      "Scikit-Learn 1.0.2\n",
      "GPU is available\n"
     ]
    }
   ],
   "source": [
    "# What version of Python do you have?\n",
    "import sys\n",
    "\n",
    "import tensorflow.keras\n",
    "import pandas as pd\n",
    "import sklearn as sk\n",
    "import tensorflow as tf\n",
    "\n",
    "check_gpu = len(tf.config.list_physical_devices('GPU'))>0\n",
    "\n",
    "print(f\"Tensor Flow Version: {tf.__version__}\")\n",
    "print(f\"Keras Version: {tensorflow.keras.__version__}\")\n",
    "print()\n",
    "print(f\"Python {sys.version}\")\n",
    "print(f\"Pandas {pd.__version__}\")\n",
    "print(f\"Scikit-Learn {sk.__version__}\")\n",
    "print(\"GPU is\", \"available\" if check_gpu \\\n",
    "      else \"NOT AVAILABLE\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GR-0hiBlhstb"
   },
   "source": [
    "# Module 1 Assignment\n",
    "\n",
    "You can find the first assignment here: [assignment 1](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class1.ipynb)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "anaconda-cloud": {},
  "colab": {
   "collapsed_sections": [],
   "name": "t81_558_class_01_1_overview.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.9 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}