10.Intro_to_CNNs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Learning about Edges and Corners: Intro to Convolutional Neural Networks\n",
    "\n",
    "In this chapter, we will:\n",
    "\n",
    "- Re-use network weights in different parts of the input (called weight sharing).\n",
    "- Implement the convolutional layer.\n",
    "\n",
    "> [Geoffrey Hinton] \"The Pooling Operation used in Convolutional Neural Networks is a big mistake, and the fact that is works so well is a disaster\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reusing Weights in multiple places\n",
    "\n",
    "The basic premise behind \"shared weights\" is the following idea: \"if we need to detect the same feature in multiple places, we should use the same weights\".\n",
    "\n",
    "As we know, the greatest challenge in NNs is \"Overfitting\". Overfitting is often caused by having more parameters than necessary to learn a specific data set. When NNs have lots of parameters and few training examples, it becomes very easy for the network to overfit. There's a better method than regularization to counter overfitting, we call it **structure**. \n",
    "\n",
    "Structure is when we selectively use the same group of weights in multiple purposes in a NN because we believe the same pattern needs to be detected in multiple places. This assumption is particularly true in the case of \"images\".  \n",
    "\n",
    "The most famous structure in NNs is called a **convolution**, and When used in a layer, it's called a **convolutional layer**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Convolutional Layer\n",
    "\n",
    "In a convolutional layer, Lots of very small linear layers are reused in every position, instead of a single big one. The core idea behind a convolutional layer is that instead of having a big dense layer connected to the whole input, we have small but many layers with a limited number of inputs (no more than 25) & one output, that scraps the input matrices one element at a time. \n",
    "\n",
    "Each mini-layer is called a **convolutional kernel**, containing the weights/activation function:\n",
    "\n",
    "<div style=\"text-align:center\">\n",
    "    <img style=\"width:33%;\" src=\"static/imgs/10/convolutional_layer.png\">\n",
    "</div>\n",
    "\n",
    "Shown in the above figure is an example of a **3x3 convolutional layer**. It predicts the center pixel values, moves one pixel to the right, then predicts again, and moves again, and so on. It will repeat this until it has made a prediction for every possible position in the image.\n",
    "\n",
    "The resulting image is a 6x6 image. In CNNs, we usually use many more filters.\n",
    "\n",
    "<div style=\"text-align:center\">\n",
    "    <img style=\"width:25%;\" src=\"static/imgs/10/step_0.png\">\n",
    "    <img style=\"width:25%;\" src=\"static/imgs/10/step_1.png\">\n",
    "    <img style=\"width:25%;\" src=\"static/imgs/10/step_2.png\">\n",
    "</div>\n",
    "\n",
    "After calculating the activation maps for each kernel, we have the option to either: \n",
    "- Sum the outputs of the kernels (element-wise, sum pooling)\n",
    "- Take the element-wise mean (mean pooling)\n",
    "- Take the element-wise max value (max pooling).\n",
    "\n",
    "The last version turns out to be the most popular. The final matrix is then forward propagated to later layers.\n",
    "\n",
    "There are a few things to notice in the above figure:\n",
    "- The Top right kernel forward propagate a `1` only for horizonal lines.\n",
    "- The Bottom left kernel forward propagate a `1` only for diagonal lines that point upward to the right.\n",
    "\n",
    "It's important to realize that this technique allows each kernel to search for a certain pattern on every location of the image. The idea of setting a small set of weights to detect location-invariant patterns in large-datasets has a considerable impact on fighting overfitting and increases the networks ability to generalize well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Simple Implementation in NumPy\n",
    "\n",
    "We just need to think about mini-linear layers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "from tensorflow.keras import datasets\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((60000, 28, 28), (60000,), (10000, 28, 28), (10000,))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(X_train, y_train), (X_test, y_test) = datasets.mnist.load_data()\n",
    "X_train.shape, y_train.shape, X_test.shape, y_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x149f26670>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAOfklEQVR4nO3df4xc5XXG8efxYgwYHGzAjguuQ2IHQUjrwMYlEJVE0PBDTQ1VSKCIguTGSMUIECRFREqQqlaoIaGEEFoTUEzEz5Qg3Io0EDcporSUBYyxMS6EuGBwMQQKJhCzXp/+sZdqAzvvLHPnl32+H2k1M/fMvfd4vM/emXnnzuuIEICd36ReNwCgOwg7kARhB5Ig7EAShB1IYpdu7mxXT4ndNLWbuwRS+bV+pbdiq8er1Qq77eMlXSlpQNJ3I+Ky0v1301T9no+ps0sABQ/Eyoa1lp/G2x6QdLWkEyQdIuk024e0uj0AnVXnNftCSU9FxNMR8ZakWyQtak9bANqtTtj3l/TsmNsbq2W/wfYS20O2h4a1tcbuANRRJ+zjvQnwrs/eRsSyiBiMiMHJmlJjdwDqqBP2jZLmjLl9gKTn67UDoFPqhP1BSfNtH2h7V0mnSlrRnrYAtFvLQ28Rsc32Ukk/1ujQ2/URsbZtnQFoq1rj7BFxl6S72tQLgA7i47JAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJGpN2Wx7g6QtkkYkbYuIwXY0BaD9aoW98umIeKkN2wHQQTyNB5KoG/aQdLfth2wvGe8OtpfYHrI9NKytNXcHoFV1n8YfFRHP254p6R7bT0TEvWPvEBHLJC2TpGmeETX3B6BFtY7sEfF8dblZ0h2SFrajKQDt13LYbU+1vdfb1yV9RtKadjUGoL3qPI2fJekO229v56aI+Oe2dAV02FeffrhYXzil3ivOwa+fW6y//2/vr7X9VrQc9oh4WtLvtrEXAB3E0BuQBGEHkiDsQBKEHUiCsANJtONEGPSYd2n83/ji4o8X152x9s1ifdJ9q1ppqS9sXnpkw9r8yf9WXHe7di3Wh7YOFOvT1w8X673AkR1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkmCcfSew8aLG3xny0LlXFtc94xfHF+uvL5pRrI/88uVivZN2mTunWL9g6W0Na++bVB5Hb+YrT/1xsT7lRw/W2n4ncGQHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQYZ68MTJtWrI+89lqXOnm3SVOnFuu/fdyGlrf97bkrivWzZp5V3kAPx9lfO/y3ivUv7LWpYW398Ehx3WUvHl2sj1wzq1iXNjSpdx9HdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgnH2yv/8yUeK9f3+7t87tu+BfcrnjL96497F+r98+NaW9/3p73ypWD9gXfenFn6bp0wp1vc5b0PL2/7zi84r1v93Xvl74fe/o3ePS6uaHtltX297s+01Y5bNsH2P7Sery+mdbRNAXRN5Gv89Se/8OpOLJa2MiPmSVla3AfSxpmGPiHslvfMzkYskLa+uL5d0UnvbAtBurb5BNysiNklSdTmz0R1tL7E9ZHtoWFtb3B2Aujr+bnxELIuIwYgYnKzyGy4AOqfVsL9ge7YkVZeb29cSgE5oNewrJJ1ZXT9T0p3taQdApzQdZ7d9s6RPSdrX9kZJX5N0maTbbC+W9IykUzrZZDd0chy92fnoTcfRP1oeR39je+O5wA/7ybnFdQ++dn2xXj7ru7Oe+fLhxfoj88rfiX/KU59tWHvu2Ciue8CPe/kv74ymYY+I0xqUjmlzLwA6iI/LAkkQdiAJwg4kQdiBJAg7kASnuLZBs1Mxz171aLF+wh6vFOuloTVJOnzFBQ1rHz7ngeK6vRxgGj62PLT2g8XfaLKF8q/v3x/4Dw1rf3hT+dTePe7o3FBsr3BkB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkGGefoIHpjb9A9/Vb9i6ue8Ie9xXr//SrfYr1v7ri9GJ9/jU75pjwcVfcW6zPm1zv1/OsJ09tWJt505qGNUnaXmvP/YkjO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kwTh7ZWDv9xXrr9zUeFrlnx16S3HdV7e/Vax/68LG48GStN8/7pjj6JL0yz/7RMPa56Z9vcna5e8JuPqVg4r1gS+82bA2smVLk33vfDiyA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASjLNX1l96cLH++O9c1fK2p7j8N3XJ5beXN3B5ubxlZLeGtStvXVRc94gTHivWj53+eLE+0OTM7yN3b3wu/6yB8jh6M1c/cnSxPu+lR2ptf2fT9Mhu+3rbm22vGbPsUtvP2V5V/ZzY2TYB1DWRp/Hfk3T8OMuviIgF1c9d7W0LQLs1DXtE3Cvp5S70AqCD6rxBt9T26uppfsMvaLO9xPaQ7aFhba2xOwB1tBr2ayR9SNICSZskNZyBLyKWRcRgRAxObnJiA4DOaSnsEfFCRIxExHZJ10pa2N62ALRbS2G3PXvMzZMllb+XF0DPNR1nt32zpE9J2tf2Rklfk/Qp2wskhaQNks7uXIvdsX3atmJ9Uo23N/bwrsX65/fc3PK2m1m8pPXPB0zEZA8U68Oxe8Pa3W9OLa57/n+Wz/Ofdwbj6O9F07BHxGnjLL6uA70A6CA+LgskQdiBJAg7kARhB5Ig7EASnOJamf/d4WL9kz9b2qVO+svLxzf+OmZJevzo8sDM6rdGGtauXlQ+/faDa1cV63hvOLIDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKMs1d8/6PF+t73d6mRLnv19COK9fMXrCzWf7Ht18X66d+/qGFt7toddyrqHRFHdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgnH2ndwuB84t1s//6q3F+pG7P1us/+nZFxTrc3/EWHq/4MgOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kwzr4TmDS18dTHI9eVp6I+ucl00R+94UvF+oGMo+8wmh7Zbc+x/VPb62yvtX1etXyG7XtsP1ldTu98uwBaNZGn8dskXRgRB0s6QtI5tg+RdLGklRExX9LK6jaAPtU07BGxKSIerq5vkbRO0v6SFklaXt1tuaSTOtQjgDZ4T2/Q2f6ApI9JekDSrIjYJI3+QZA0s8E6S2wP2R4a1taa7QJo1YTDbntPSbdLOj8iXpvoehGxLCIGI2Jwsqa00iOANphQ2G1P1mjQb4yIH1aLX7A9u6rPllR+WxdATzUderNtSddJWhcR3xxTWiHpTEmXVZd3dqRDNPXE5R9pXDvo6uK6h/7rF4v1eX9Z/ort7cUq+slExtmPknSGpMdsr6qWXaLRkN9me7GkZySd0pEOAbRF07BHxH2S3KB8THvbAdApfFwWSIKwA0kQdiAJwg4kQdiBJDjFtQ94l/J/w/qrDivW13722w1rP3lzWnHdeZeVp1ze/sYbxTp2HBzZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJxtn7wM//+uPF+hN/9K1i/dXtww1rV536ueK6sXpNsY6dB0d2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCcfY+sOdBr9Ra/xM/uLBhbd7Qf9TaNnYeHNmBJAg7kARhB5Ig7EAShB1IgrADSRB2IImJzM8+R9INkt6v0em4l0XElbYvlfRFSS9Wd70kIu7qVKOZHbe2PBv2/C8PNaxFu5vBDmsiH6rZJunCiHjY9l6SHrJ9T1W7IiIu71x7ANplIvOzb5K0qbq+xfY6Sft3ujEA7fWeXrPb/oCkj0l6oFq01PZq29fbnt5gnSW2h2wPDWtrvW4BtGzCYbe9p6TbJZ0fEa9JukbShyQt0OiR/xvjrRcRyyJiMCIGJ2tK/Y4BtGRCYbc9WaNBvzEifihJEfFCRIxExHZJ10pa2Lk2AdTVNOy2Lek6Sesi4ptjls8ec7eTJfE1pUAfm8i78UdJOkPSY7ZXVcsukXSa7QUaHd3ZIOnsDvSXwsxFT9Ran+E1TMRE3o2/T5LHKTGmDuxA+AQdkARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCUd072xo2y9K+u8xi/aV9FLXGnhv+rW3fu1LordWtbO3uRGx33iFrob9XTu3hyJisGcNFPRrb/3al0RvrepWbzyNB5Ig7EASvQ77sh7vv6Rfe+vXviR6a1VXeuvpa3YA3dPrIzuALiHsQBI9Cbvt422vt/2U7Yt70UMjtjfYfsz2KtuN50LuTi/X295se82YZTNs32P7yepy3Dn2etTbpbafqx67VbZP7FFvc2z/1PY622ttn1ct7+ljV+irK49b11+z2x6Q9F+S/kDSRkkPSjotIh7vaiMN2N4gaTAiev4BDNu/L+l1STdExKHVsr+R9HJEXFb9oZweEX/RJ71dKun1Xk/jXc1WNHvsNOOSTpJ0lnr42BX6+ry68Lj14si+UNJTEfF0RLwl6RZJi3rQR9+LiHslvfyOxYskLa+uL9foL0vXNeitL0TEpoh4uLq+RdLb04z39LEr9NUVvQj7/pKeHXN7o/prvveQdLfth2wv6XUz45gVEZuk0V8eSTN73M87NZ3Gu5veMc143zx2rUx/Xlcvwj7eVFL9NP53VEQcJukESedUT1cxMROaxrtbxplmvC+0Ov15Xb0I+0ZJc8bcPkDS8z3oY1wR8Xx1uVnSHeq/qahfeHsG3epyc4/7+X/9NI33eNOMqw8eu15Of96LsD8oab7tA23vKulUSSt60Me72J5avXEi21MlfUb9NxX1CklnVtfPlHRnD3v5Df0yjXejacbV48eu59OfR0TXfySdqNF35H8u6Su96KFBXx+U9Gj1s7bXvUm6WaNP64Y1+oxosaR9JK2U9GR1OaOPevu+pMckrdZosGb3qLdPavSl4WpJq6qfE3v92BX66srjxsdlgST4BB2QBGEHkiDsQBKEHUiCsANJEHYgCcIOJPF/c8MuT3PJpwMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualizing one example:\n",
    "plt.imshow(X_train[89].squeeze())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get example\n",
    "image_example = X_train[89].squeeze() / 255."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_image_section(layer, row_from, row_to, col_from, col_to):\n",
    "    return layer[row_from:row_to+1, col_from:col_to+1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x149f82880>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAALdklEQVR4nO3df6zddX3H8eert79oGRP8scyWCC6IQ5YFc8cQErNRTeo0sD9mAgvOmS3NfoDVkBlclvDv/jBE/0CSDnEmMoipLGOO+COoMUbTeSksUq5EUlxbKWs3ozhUWuC9P+7RXS6tlPP9nHsOfJ6PhPSecw+f7zttn/2eX/dzUlVIevlbM+0BJK0OY5c6YexSJ4xd6oSxS51Yu5oHW58NtZHNq3lIqSs/40mO1VM50fdWNfaNbOZ3s201Dyl1ZU/de9LveTde6oSxS50wdqkTxi51wtilTgyKPcn2JA8neSTJDa2GktTe2LEnmQNuBt4BXABcneSCVoNJamvImf1i4JGq2l9Vx4A7gSvbjCWptSGxbwEOLrt8aHTdcyTZkWQhycJxnhpwOElDDIn9RG/Je95OGFW1q6rmq2p+HRsGHE7SEENiPwScvezyVuCxYeNImpQhsX8LOC/JuUnWA1cBd7cZS1JrY/8gTFU9neRa4AvAHHBbVe1rNpmkpgb91FtV3QPc02gWSRPkO+ikThi71Aljlzph7FInjF3qhLFLnTB2qRPGLnXC2KVOGLvUCWOXOmHsUieMXeqEsUudMHapE8YudcLYpU4Yu9QJY5c6YexSJ4xd6oSxS50wdqkTxi51wtilThi71Aljlzph7FInjF3qhLFLnTB2qRNjx57k7CRfSbKYZF+SnS0Hk9TW2gH/79PA9VW1N8mvAPcl+VJVPdRoNkkNjX1mr6rDVbV39PWPgUVgS6vBJLU15Mz+C0nOAS4C9pzgezuAHQAb2dTicJLGMPgJuiSnA58FPlBVT6z8flXtqqr5qppfx4ahh5M0pkGxJ1nHUui3V9VdbUaSNAlDno0P8AlgsapuajeSpEkYcma/DHgPcHmSB0b//UGjuSQ1NvYTdFX1dSANZ5E0Qb6DTuqEsUudMHapE8YudcLYpU4Yu9QJY5c6YexSJ4xd6oSxS50wdqkTxi51wtilThi71Aljlzph7FInjF3qhLFLnTB2qRPGLnXC2KVOGLvUCWOXOtHkgx2lrG37V+nIn/9O0/Ve+eBPm6215usPNFtrNXlmlzph7FInjF3qhLFLnTB2qRPGLnVicOxJ5pLcn+RzLQaSNBktzuw7gcUG60iaoEGxJ9kKvBO4tc04kiZl6Jn9o8CHgGdPdoMkO5IsJFk4zlMDDydpXGPHnuRdwJGquu+X3a6qdlXVfFXNr2PDuIeTNNCQM/tlwBVJvgfcCVye5NNNppLU3NixV9WHq2prVZ0DXAV8uaquaTaZpKZ8nV3qRJOfS6yqrwJfbbGWpMnwzC51wtilThi71AljlzrhHnRq4uDfXNx0vX3Xfbzpen/86O83W+uHV5zVbC2AZ/7nB03XOxnP7FInjF3qhLFLnTB2qRPGLnXC2KVOGLvUCWOXOmHsUieMXeqEsUudMHapE8YudcLYpU4Yu9QJY5c6YexSJ4xd6oSxS51Y1T3oMreGudPPaLbeM0880Wyt3qzZvLnper+xfX/T9Vq75XX3NFvr6tf8SbO1AHAPOkktGbvUCWOXOmHsUieMXeqEsUudGBR7klck2Z3kO0kWk7yl1WCS2hr6OvvHgM9X1R8lWQ9sajCTpAkYO/YkZwBvBf4UoKqOAcfajCWptSF3418PHAU+meT+JLcmed7bspLsSLKQZOHYsz8bcDhJQwyJfS3wZuCWqroIeBK4YeWNqmpXVc1X1fz6NRsHHE7SEENiPwQcqqo9o8u7WYpf0gwaO/aqehw4mOT80VXbgIeaTCWpuaHPxl8H3D56Jn4/8L7hI0mahEGxV9UDwHybUSRNku+gkzph7FInjF3qhLFLnVjVPeiOn3Uaj7/7Tc3We/Ut32y21qybe+VZTdf733/61abrfe28f266XmuX3nx9s7W2Ln6j2VqryTO71Aljlzph7FInjF3qhLFLnTB2qRPGLnXC2KVOGLvUCWOXOmHsUieMXeqEsUudMHapE8YudcLYpU4Yu9QJY5c6YexSJ1Z1D7q1R5/sZt+4NZuf94G2gzTfM+632u4Z95Nn235a94X3/kXT9d74Dw83W+uZZiutLs/sUieMXeqEsUudMHapE8YudWJQ7Ek+mGRfkgeT3JFkY6vBJLU1duxJtgDvB+ar6kJgDriq1WCS2hp6N34tcFqStcAm4LHhI0mahLFjr6rvAx8BDgCHgR9V1RdX3i7JjiQLSRaO89T4k0oaZMjd+DOBK4FzgdcCm5Ncs/J2VbWrquaran4dG8afVNIgQ+7Gvw14tKqOVtVx4C7g0jZjSWptSOwHgEuSbEoSYBuw2GYsSa0Necy+B9gN7AW+PVprV6O5JDU26KfequpG4MZGs0iaIN9BJ3XC2KVOGLvUCWOXOrGq21LNumxo96afnf+x0GwtgO2b2r77sPU2Um/612ubrveGv/z3puu9VLeSaskzu9QJY5c6YexSJ4xd6oSxS50wdqkTxi51wtilThi71Aljlzph7FInjF3qhLFLnTB2qRPGLnXC2KVOGLvUCWOXOmHsUide0nvQzZ15ZtP1fvqZM5qt1XrPuH/7ycam6/3dTX/VdL03fPwbTddTe57ZpU4Yu9QJY5c6YexSJ4xd6oSxS514wdiT3JbkSJIHl113VpIvJfnu6Ne2r4FJau5Uzuz/CGxfcd0NwL1VdR5w7+iypBn2grFX1deAH6y4+krgU6OvPwX8YduxJLU27mP2X6uqwwCjX19zshsm2ZFkIcnCcdq+q0zSqZv4E3RVtauq5qtqfh3tPhJZ0oszbuz/leTXAUa/Hmk3kqRJGDf2u4H3jr5+L/AvbcaRNCmn8tLbHcA3gfOTHEryZ8DfA29P8l3g7aPLkmbYC/6Ia1VdfZJvbWs8i6QJ8h10UieMXeqEsUudMHapE6mq1TtYchT4z1O46auA/57wOOOa5dlgtueb5dng5THf66rq1Sf6xqrGfqqSLFTV/LTnOJFZng1me75Zng1e/vN5N17qhLFLnZjV2HdNe4BfYpZng9meb5Zng5f5fDP5mF1Se7N6ZpfUmLFLnZip2JNsT/JwkkeSzNS+dknOTvKVJItJ9iXZOe2ZVkoyl+T+JJ+b9iwrJXlFkt1JvjP6PXzLtGf6uSQfHP2ZPpjkjiRtP1jvxc8zkU1eZyb2JHPAzcA7gAuAq5NcMN2pnuNp4Pqq+k3gEuCvZ2w+gJ3A4rSHOImPAZ+vqjcCv82MzJlkC/B+YL6qLgTmgKumO9VkNnmdmdiBi4FHqmp/VR0D7mRpY8uZUFWHq2rv6Osfs/SXdct0p/p/SbYC7wRunfYsKyU5A3gr8AmAqjpWVT+c6lDPtRY4LclaYBPw2DSHmdQmr7MU+xbg4LLLh5ihmJZLcg5wEbBnyqMs91HgQ8CzU57jRF4PHAU+OXqYcWuSzdMeCqCqvg98BDgAHAZ+VFVfnO5UJ3TKm7yezCzFnhNcN3OvCyY5Hfgs8IGqemLa8wAkeRdwpKrum/YsJ7EWeDNwS1VdBDzJjHzWwOix75XAucBrgc1JrpnuVJMxS7EfAs5ednkrU747tVKSdSyFfntV3TXteZa5DLgiyfdYevhzeZJPT3ek5zgEHKqqn98T2s1S/LPgbcCjVXW0qo4DdwGXTnmmExm8yessxf4t4Lwk5yZZz9KTJHdPeaZfSBKWHnMuVtVN055nuar6cFVtrapzWPp9+3JVzczZqaoeBw4mOX901TbgoSmOtNwB4JIkm0Z/xtuYkScPVxi8yesL7kG3Wqrq6STXAl9g6RnR26pq35THWu4y4D3At5M8MLrub6vqnumN9JJyHXD76B/y/cD7pjwPAFW1J8luYC9Lr7jcz5TfNjva5PX3gFclOQTcyNKmrp8Zbfh6AHj3i17Xt8tKfZilu/GSJsjYpU4Yu9QJY5c6YexSJ4xd6oSxS534P510m37VOA7PAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(get_image_section(image_example, 3, 13, 5, 15))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# book implementation.\n",
    "def get_image_section(layer, row_from, row_to, col_from, col_to):\n",
    "    sub_section = layer[:, row_from:row_to, col_from:col_to]\n",
    "    return sub_section.reshape(-1, 1, row_to-row_from, col_to-col_from)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x149fe4580>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPUAAAD4CAYAAAA0L6C7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAKfElEQVR4nO3d76/edX3H8eer5/QHLWOKczdsieDCdMiyYM4YQmIy6g2cBu5sCSy4zGRptgyshszg7vgPGKM3GEmHekcmNypLnCHq4o8sxqWzFBYplYQURysQuhuIskmLvHfjHJeu7en59ur15XvOe89HQtJzrtNPX2nOk+91rl7nOqkqJPWxaeoBkubLqKVmjFpqxqilZoxaamZxjEO3ZGttY8cYR0sCfsErnKxXc67bRol6Gzv4g+we42hJwIH61qq3efdbasaopWaMWmrGqKVmjFpqxqilZgZFneSWJE8leTrJvWOPkjS7NaNOsgDcB3wAuAa4I8k1Yw+TNJshV+rrgaer6mhVnQQeAm4bd5akWQ2Jeidw7LS3j6+87/9IsifJwSQHT/HqvPZJukBDoj7X80vPermUqtpXVUtVtbSZrRe/TNJMhkR9HLjitLd3Ac+NM0fSxRoS9Q+Aq5NclWQLcDvw1XFnSZrVmt+lVVWvJbkL+AawAHyhqg6PvkzSTAZ962VVPQI8MvIWSXPgM8qkZoxaasaopWaMWmrGqKVmjFpqxqilZoxaasaopWaMWmrGqKVmjFpqxqilZoxaasaopWaMWmrGqKVmjFpqxqilZoxaasaopWaMWmrGqKVmjFpqxqilZoxaasaopWaMWmrGqKVmjFpqxqilZoxaasaopWaMWmrGqKVm1ow6yRVJvpPkSJLDSfa+EcMkzWZxwMe8BtxTVYeS/BrwaJJ/rqonR94maQZrXqmr6vmqOrTy658BR4CdYw+TNJshV+r/leRK4DrgwDlu2wPsAdjG9nlskzSDwQ+UJbkU+Arwsap6+czbq2pfVS1V1dJmts5zo6QLMCjqJJtZDvrBqnp43EmSLsaQR78DfB44UlWfGX+SpIsx5Ep9E/Bh4OYkj6/890cj75I0ozUfKKuq7wF5A7ZImgOfUSY1Y9RSM0YtNWPUUjNGLTVj1FIzRi01Y9RSM0YtNWPUUjNGLTVj1FIzRi01Y9RSM0YtNWPUUjNGLTVj1FIzRi01Y9RSM0YtNWPUUjNGLTVj1FIzRi01Y9RSMxf086mlC5XFcT7FXvyL35/7mW954r/nfibApu89Psq5q/55b+ifJml0Ri01Y9RSM0YtNWPUUjNGLTVj1FIzg6NOspDksSRfG3OQpItzIVfqvcCRsYZImo9BUSfZBXwQeGDcOZIu1tAr9WeBTwCvr/YBSfYkOZjk4Clencc2STNYM+okHwJerKpHz/dxVbWvqpaqamkzW+c2UNKFGXKlvgm4NcmPgYeAm5N8adRVkma2ZtRV9cmq2lVVVwK3A9+uqjtHXyZpJv47tdTMBX2za1V9F/juKEskzYVXaqkZo5aaMWqpGaOWmjFqqRlfTVSjOvY3149y7uG7/27uZ/7pM3849zMBXrr18rmfmZcWVr3NK7XUjFFLzRi11IxRS80YtdSMUUvNGLXUjFFLzRi11IxRS80YtdSMUUvNGLXUjFFLzRi11IxRS80YtdSMUUvNGLXUjFFLzRi11MworyaahU0sXHrZ3M/95csvz/1MLdu0Y8co5/7WLUdHOXcM97/9kVHOveM3/2z+h/589XS9UkvNGLXUjFFLzRi11IxRS80YtdSMUUvNDIo6yZuS7E/yoyRHkrx37GGSZjP0ySefA75eVX+cZAuwfcRNki7CmlEnuQx4H/DnAFV1Ejg57ixJsxpy9/sdwAngi0keS/JAkrOeU5hkT5KDSQ6efP0Xcx8qaZghUS8C7wHur6rrgFeAe8/8oKraV1VLVbW0ZdO2Oc+UNNSQqI8Dx6vqwMrb+1mOXNI6tGbUVfUCcCzJO1fetRt4ctRVkmY29NHvu4EHVx75Pgp8ZLxJki7GoKir6nFgadwpkubBZ5RJzRi11IxRS80YtdSMUUvNjPJqoqcuv4QX/uTdcz/3rff/69zP3GgW3nL5KOf+/B9+fZRz/+Xqfxzl3DHceN89o5y768j3535m1aur3uaVWmrGqKVmjFpqxqilZoxaasaopWaMWmrGqKVmjFpqxqilZoxaasaopWaMWmrGqKVmjFpqxqilZoxaasaopWaMWmrGqKVmRnnhwcUTr/y/f5HATTvO+hHeczHaCwT+7jgvEPhfr58c5dxrv/WXcz/zXX//1NzPBPjlKKeuziu11IxRS80YtdSMUUvNGLXUjFFLzRi11MygqJN8PMnhJE8k+XKSbWMPkzSbNaNOshP4KLBUVdcCC8DtYw+TNJuhd78XgUuSLALbgefGmyTpYqwZdVX9BPg08CzwPPDTqvrmmR+XZE+Sg0kOnmL1n50raVxD7n6/GbgNuAp4G7AjyZ1nflxV7auqpapa2szW+S+VNMiQu9/vB56pqhNVdQp4GLhx3FmSZjUk6meBG5JsTxJgN3Bk3FmSZjXka+oDwH7gEPDDld+zb+RdkmY06Pupq+pTwKdG3iJpDnxGmdSMUUvNGLXUjFFLzRi11Mworya60WTr/J8Bt/ffD879TIBbto/zFNyxXvXz3f901yjn/vZf/dvcz3yjX/VzLF6ppWaMWmrGqKVmjFpqxqilZoxaasaopWaMWmrGqKVmjFpqxqilZoxaasaopWaMWmrGqKVmjFpqxqilZoxaasaopWaMWmrGqKVmUlXzPzQ5AfzHgA/9DeA/5z5gPBtp70baChtr73rY+vaqeuu5bhgl6qGSHKyqpckGXKCNtHcjbYWNtXe9b/Xut9SMUUvNTB31Rvvh9Rtp70baChtr77reOunX1JLmb+ortaQ5M2qpmcmiTnJLkqeSPJ3k3ql2rCXJFUm+k+RIksNJ9k69aYgkC0keS/K1qbecT5I3Jdmf5Ecrf8fvnXrT+ST5+MrnwRNJvpxk29SbzjRJ1EkWgPuADwDXAHckuWaKLQO8BtxTVb8D3AD89Treerq9wJGpRwzwOeDrVfUu4PdYx5uT7AQ+CixV1bXAAnD7tKvONtWV+nrg6ao6WlUngYeA2ybacl5V9XxVHVr59c9Y/qTbOe2q80uyC/gg8MDUW84nyWXA+4DPA1TVyap6adJRa1sELkmyCGwHnpt4z1mminoncOy0t4+zzkMBSHIlcB1wYOIpa/ks8Ang9Yl3rOUdwAngiytfKjyQZMfUo1ZTVT8BPg08CzwP/LSqvjntqrNNFXXO8b51/W9rSS4FvgJ8rKpennrPapJ8CHixqh6dessAi8B7gPur6jrgFWA9P77yZpbvUV4FvA3YkeTOaVedbaqojwNXnPb2Ltbh3ZhfSbKZ5aAfrKqHp96zhpuAW5P8mOUva25O8qVpJ63qOHC8qn51z2c/y5GvV+8HnqmqE1V1CngYuHHiTWeZKuofAFcnuSrJFpYfbPjqRFvOK0lY/prvSFV9Zuo9a6mqT1bVrqq6kuW/129X1bq7mgBU1QvAsSTvXHnXbuDJCSet5VnghiTbVz4vdrMOH9hbnOIPrarXktwFfIPlRxC/UFWHp9gywE3Ah4EfJnl85X1/W1WPTDeplbuBB1f+534U+MjEe1ZVVQeS7AcOsfyvIo+xDp8y6tNEpWZ8RpnUjFFLzRi11IxRS80YtdSMUUvNGLXUzP8AYR42CXuiZt0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(get_image_section(image_example.reshape(1,28,28), 3, 13, 5, 15).squeeze())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to call this method multiple times on every location within the image. Such \"for\" loop can be implemented as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images = X_train\n",
    "images.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7, 28, 28)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_start, batch_end = 0, 7\n",
    "layer_0 = images[batch_start:batch_end]\n",
    "layer_0.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "sections = list()\n",
    "kernel_rows, kernel_cols = 3, 3\n",
    "for row_start in range(layer_0.shape[1]-kernel_rows):\n",
    "    for col_start in range(layer_0.shape[2]-kernel_cols):\n",
    "        section = get_image_section(layer_0, row_start, row_start+kernel_rows, col_start, col_start+kernel_cols)\n",
    "        sections.append(section)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7, 625, 3, 3)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "expanded_input = np.concatenate(sections, axis=1)\n",
    "expanded_input.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4375, 9)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "es = expanded_input.shape\n",
    "flattened_input = expanded_input.reshape(es[0]*es[1], -1)\n",
    "flattened_input.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Following the above implementation, `layer_0` will be a batch of 28x28 images. The `for` loop slides over every subregion in the images, extract and append it to a list called `sections`. `sections` is then concatenated and reshaped in a peculiar way. We Pretend for now that each individual subregion is its own image.\n",
    "\n",
    "Forward-propagating the subregions through a linear layer with one output neuron is the same as predicting that linear layer over every sub-region in every batch. The following shows the entire NumPy implementation: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sys import exit\n",
    "np.random.seed(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import mnist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1000, 784), (1000,))"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "images, labels = (x_train[:1000].reshape(1000, 28*28) / 255, y_train[:1000])\n",
    "images.shape, labels.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "one_hot_labels = np.zeros(shape=(labels.shape[0], 10))\n",
    "for i, v in enumerate(labels):\n",
    "    one_hot_labels[i][v] = 1\n",
    "labels = one_hot_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_images = x_test.reshape(x_test.shape[0], 28*28) / 255\n",
    "test_labels = np.zeros(shape=(y_test.shape[0], 10))\n",
    "for i, v in enumerate(y_test):\n",
    "    test_labels[i][v] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# activation functions\n",
    "def tanh(x):\n",
    "    return np.tanh(x)\n",
    "\n",
    "def grad_tanh(y):\n",
    "    return 1 - y**2\n",
    "\n",
    "def softmax(x):\n",
    "    exps = np.exp(x)\n",
    "    return exps/np.sum(exps, axis=1, keepdims=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr, epochs = (2, 300)\n",
    "pixels_per_image, num_labels = (784, 10)\n",
    "batch_size = 128\n",
    "\n",
    "input_rows, input_cols = (28, 28)\n",
    "kernel_rows, kernel_cols = (3, 3)\n",
    "num_kernels = 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# `Hidden` is of that size because we are training a 1-D Convolutional Layer, not a 2D, and so it's taking more spaces\n",
    "hidden_size = ((input_rows - kernel_rows + 1) * (input_cols - kernel_cols + 1)) * num_kernels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((9, 4), (2704, 10))"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kernels = (0.02 * np.random.random((kernel_rows*kernel_cols, num_kernels))) - 0.01\n",
    "weights_1_2 = (0.2 * np.random.random((hidden_size, num_labels))) - 0.1\n",
    "\n",
    "kernels.shape, weights_1_2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_image_section(layer, row_from, row_to, col_from, col_to):\n",
    "    sub_section = layer[:, row_from:row_to, col_from:col_to]\n",
    "    return sub_section.reshape(-1, 1, row_to-row_from, col_to-col_from)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training Loop\n",
    "for epoch in range(epochs):  # for each epoch\n",
    "    correct_count = 0\n",
    "    for batch in range(int(len(images) / batch_size)):  # for each batch\n",
    "        batch_start, batch_end = (batch*batch_size), ((batch+1)*batch_size)\n",
    "        \n",
    "        # get input\n",
    "        layer_0 = images[batch_start:batch_end]\n",
    "        layer_0 = layer_0.reshape(layer_0.shape[0], 28, 28)\n",
    "        \n",
    "        # extract all sections from all batch images\n",
    "        sections = list()\n",
    "        for row_start in range(layer_0.shape[1] - kernel_rows + 1):  # +1 by akramz.\n",
    "            for col_start in range(layer_0.shape[2]-kernel_cols+1):\n",
    "                section = get_image_section(layer_0, row_start, row_start+kernel_rows, col_start, col_start+kernel_cols)\n",
    "                sections.append(section)\n",
    "        # [0] 128 batch elements, [1] 676 sections/image, 3x3 section size \n",
    "        expanded_input = np.concatenate(sections, axis=1)\n",
    "        \n",
    "        es = expanded_input.shape\n",
    "        # shape: (128x676, 3x3)\n",
    "        flattened_input = expanded_input.reshape(es[0] * es[1], -1)\n",
    "        \n",
    "        # forward propagation\n",
    "        kernel_output = flattened_input.dot(kernels)  # 16 kernels\n",
    "        layer_1 = tanh(kernel_output.reshape(es[0], -1))\n",
    "        dropout_mask = np.random.randint(2, size=layer_1.shape)\n",
    "        layer_1 *= dropout_mask*2\n",
    "        layer_2 = softmax(np.dot(layer_1, weights_1_2))\n",
    "        \n",
    "        # count corrects\n",
    "        for k in range(batch_size):\n",
    "            labelset = labels[batch_start+k:batch_start+k+1]\n",
    "            _inc = int(np.argmax(layer_2[k:k+1]) == np.argmax(labelset))\n",
    "            correct_count += _inc\n",
    "        \n",
    "        # back propagation\n",
    "        layer_2_delta = (labels[batch_start:batch_end]-layer_2)/(batch_size*layer_2.shape[0])\n",
    "        layer_1_delta = layer_2_delta.dot(weights_1_2.T) * grad_tanh(layer_1)\n",
    "        layer_1_delta *= dropout_mask\n",
    "        weights_1_2 += lr * layer_1.T.dot(layer_2_delta)\n",
    "        l1d_reshape = layer_1_delta.reshape(kernel_output.shape)\n",
    "        k_udpate = flattened_input.T.dot(l1d_reshape)\n",
    "        kernels -= lr * k_udpate\n",
    "        \n",
    "        # test\n",
    "        test_correct_count = 0\n",
    "        for i in range(len(test_images)):\n",
    "            layer_0 = test_images[i:i+1]\n",
    "            layer_0 = layer_0.reshape(layer_0.shape[0], 28, 28)\n",
    "            sections = list()\n",
    "            for row_start in range(layer_0.shape[1]-kernel_rows+1):\n",
    "                for col_start in range(layer_0.shape[2]-kernel_cols+1):\n",
    "                    section = get_image_section(layer_0, row_start, row_start+kernel_rows, col_start, col_start+kernel_cols)\n",
    "                    sections.append(section)\n",
    "            expanded_input = np.concatenate(sections, axis=1)\n",
    "            es = expanded_input.shape\n",
    "            flattened_input = expanded_input.reshape(es[0]*es[1],-1)\n",
    "            kernel_output = flattened_input.dot(kernels) \n",
    "            layer_1 = tanh(kernel_output.reshape(es[0],-1)) \n",
    "            layer_2 = np.dot(layer_1,weights_1_2)\n",
    "            test_correct_count += int(np.argmax(layer_2) == np.argmax(test_labels[i:i+1]))\n",
    "        \n",
    "    if(epoch % 1 == 0):\n",
    "        print(f\"I: {epoch} | Test-Acc: {round(test_correct_count / float(len(test_images)), 5)} | Train-Acc: {round(correct_count/float(len(images)), 5)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, swapping the first layer with a convolutional layer gives another few percentage points in error reduction. The output of the convolutional layer is itself also a series of two-dimensional images. \n",
    "\n",
    "Most uses of convolutional layers stack multiple layers on top of each other such that each layer treats the previous output as its input. Stacked Convolutional Layers are one of the main developments that allowed for very deep neural networks and, by extension, the popularization of the phrase \"deep learning\".\n",
    "\n",
    "It can't be overstressed that this invension was a landmark moment for the field; without it, we might still be living the previous AI Winter even as of now.s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "### Reusing weights is one of the most important innovations in deep learning\n",
    "\n",
    "The notion of reusing weights to increase accuracy is hugely important and has an intuitive basis. Let's consider what we would need to detect that a cat is in an image:\n",
    "1. Colors.\n",
    "2. lines and edges.\n",
    "3. Corners & small shapes.\n",
    "4. finally the combination of such low-level features that corresponds to a cat.\n",
    "\n",
    "Notice that such primitive building blocks exist all over the image pixels. The intelligence for detecting lines and edges is learned in the neural network's weights, but if we use different weights to analyze different parts of an image, each section of the weights has to learn what a line is, which is not efficient and can quickly lead to overfitting.\n",
    "\n",
    "When a Neural Network needs to use the same idea in multiple places, try to use the same weights in multiple places. This will make the weights more intelligent by giving them more samples (sections) to learn from, increasing generalization.\n",
    "\n",
    "Many of the biggest developments in deep learning over the past five years are iterations of this idea: CNNs, RNNs, Word Embeddings, & the recently published capsule networks can all be viewed through this lens. \n",
    "\n",
    "In summary, When we have the same features to be learned in multiple places, we should force the network to use the same weights in those places.\n",
    "\n",
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}