Week1-Systems_of_Linear_Equations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9969dc90",
   "metadata": {},
   "source": [
    "# Section 1.1 introduces systems of linear equations. In this notebook, we look at a few examples here and solve these systems with Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3803638f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import packages and print software information\n",
    "import platform\n",
    "import numpy as np\n",
    "import scipy as sp\n",
    "from matplotlib import pyplot as plt \n",
    "%matplotlib inline\n",
    "\n",
    "import sys\n",
    "print(\"Python 3 version is\", sys.version)\n",
    "import matplotlib\n",
    "print(\"Matplotlib version is\", matplotlib.__version__)\n",
    "print(\"Numpy version is\", np.__version__)\n",
    "print(\"Scipy version is\", sp.__version__)\n",
    "\n",
    "print(\"System information: \", platform.uname())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fead5dfc",
   "metadata": {},
   "source": [
    "## We investigate the following two systems are equivalent (page 3.). As they have the same number of variables, if they also have the same solution set, then we say the two systems are equivalent. \n",
    "\n",
    "\\begin{align}\n",
    "3x_1+2x_2-x_3=-2\\\\\n",
    "x_2=3\\\\\n",
    "2x_3=4\\end{align} and \n",
    "\n",
    "\\begin{align}\n",
    "3x_1+2x_2-x_3=-2\\\\\n",
    "-3x_1-x_2+x_3=5\\\\\n",
    "3x_1+2x_2+x_3=2\\end{align}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a073f3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "A1 = np.array([[3,2,-1],[0,1,0],[0,0,2]])\n",
    "b1 = np.array([[-2],[3],[4]])\n",
    "\n",
    "A2 = np.array([[3,2,-1],[-3,-1,1],[3,2,1]])\n",
    "b2 = np.array([[-2],[5],[2]])\n",
    "\n",
    "soln1 = np.linalg.solve(A1,b1)\n",
    "soln2 = np.linalg.solve(A2,b2)\n",
    "print(\"soln1=\",soln1,\"soln2=\",soln2,soln1 - soln2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47dd515b",
   "metadata": {},
   "source": [
    "## We look at systems with more than one solution (infinitely many), as well as with no solution.\n",
    "\\begin{align}\n",
    "x_1 + 3x_2 = 3\\\\\n",
    "2x_1 + 6x_2 = 6\n",
    "\\end{align}\n",
    "\n",
    "\\begin{align}\n",
    "x_1 + 3x_2 = 3\\\\\n",
    "2x_1 + 6x_2 = 5\n",
    "\\end{align}\n",
    "\n",
    "In the following, you should expect error message for soln1 and soln2, as the coefficient matrix will be singular."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c94eb767",
   "metadata": {},
   "outputs": [],
   "source": [
    "A1 = np.array([[1,3],[2,6]])\n",
    "b1 = np.array([[3],[6]])\n",
    "soln1 = np.linalg.solve(A1,b1)\n",
    "print(soln1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a608b906",
   "metadata": {},
   "outputs": [],
   "source": [
    "A2 = np.array([[1,3],[2,6]])\n",
    "b2 = np.array([[3],[5]])\n",
    "soln2 = np.linalg.solve(A2,b2)\n",
    "print(soln2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "223906d5",
   "metadata": {},
   "source": [
    "### When there is no solution, you can use the numpy function 'lstsq' to compute the least square solution to the system. It gives the best $x$ to minimize the quantity $||Ax-b||$ (this is the 2-norm)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed298493",
   "metadata": {},
   "outputs": [],
   "source": [
    "A2 = np.array([[1,3],[2,6]])\n",
    "b2 = np.array([[3],[5]])\n",
    "soln3 = np.linalg.lstsq(A2,b2)\n",
    "print(soln3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2138f17e",
   "metadata": {},
   "source": [
    "## Linear algebra is one of the most important subjects now. One example is its role in deep learning. You can see more detail here: https://www.deeplearningbook.org\n",
    "\n",
    "### Here we will discuss an example from the chapter on Deep Feedforward network (https://www.deeplearningbook.org/contents/mlp.html)\n",
    "Consider for example the MSE loss function\n",
    "\n",
    "$J(\\omega,b)=\\sum_{x\\in \\mathbb{X}}(f^*(x)-f(x;\\omega,b))^2$\n",
    "\n",
    "$f(x;\\omega,b)=x^T\\omega +b$ is an approximate linear model, and our goal is to find $\\omega$ and $b$ by minimizing $J$ with respect to $\\omega$ and $b$.\n",
    "\n",
    "Here both $\\omega$ and $b$ are matrices, and the computation is done as matrix multiplication."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e735d005",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}