Tensorflow_1.1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tensorflow-2.0 in Action "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](img/tensorflow1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> TensorFlow is an open source software library for high performance numerical computation. Its flexible architecture allows easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters of servers to mobile and edge devices.\n",
    "> Originally developed by researchers and engineers from the Google Brain team within Google’s AI organization, it comes with strong support for machine learning and deep learning and the flexible numerical computation core is used across many other scientific domains."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why Tensorflow?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](img/comparision.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](img/comparision2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation of Tensorflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">TensorFlow is tested and supported on the following 64-bit systems:\n",
    "\n",
    ">1.Ubuntu 16.04 or later\n",
    "\n",
    ">2.Windows 7 or later\n",
    "\n",
    ">3.macOS 10.12.6 (Sierra) or later (no GPU support)\n",
    "\n",
    ">4.Raspbian 9.0 or later"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **pip install tensorflow==2.0.0**\n",
    "\n",
    "> To run from Anaconda Prompt\n",
    "\n",
    "> **!pip install tensorflow==2.0.0**\n",
    "\n",
    "> To run from Jupyter Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Both Tensorflow 2.0 and Keras have been released for four years (Keras was released in March 2015, and Tensorflow was released in November of the same year). The rapid development of deep learning in the past days, we also know some problems of Tensorflow1.x and Keras:`\n",
    "\n",
    "* Using Tensorflow means programming static graphs, which is difficult and inconvenient for programs that are familiar with imperative programming\n",
    "\n",
    "* Tensorflow api is powerful and flexible, but it is more complex, confusing and difficult to use.\n",
    "\n",
    "* Keras api is productive and easy to use, but lacks flexibility for research\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Version Check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow version: 2.0.0\n",
      "Keras version: 2.2.4-tf\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print(\"TensorFlow version: {}\".format(tf.__version__))\n",
    "print(\"Keras version: {}\".format(tf.keras.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Tensorflow2.0 is a combination design of Tensorflow1.x and Keras. Considering user feedback and framework development over the past four years, it largely solves the above problems and will become the future machine learning platform.`\n",
    "\n",
    "> Tensorflow 2.0 is built on the following core ideas:\n",
    "\n",
    "\n",
    "* The coding is more pythonic, so that users can get the results immediately like they are programming in numpy\n",
    "* Retaining the characteristics of static graphs (for performance, distributed, and production deployment), this makes TensorFlow fast, scalable, and ready for production.\n",
    "* Using Keras as a high-level API for deep learning, making Tensorflow easy to use and efficient\n",
    "* Make the entire framework both high-level features (easy to use, efficient, and not flexible) and low-level features (powerful and scalable, not easy to use, but very flexible)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Eager execution is by default in TensorFlow 2.0 and, it needs no special setup.\n",
    ">The following below code can be used to find out whether a CPU or GPU is in use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GPU/CPU Check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU\n"
     ]
    }
   ],
   "source": [
    "variable = tf.Variable([3, 3])\n",
    "if tf.test.is_gpu_available():\n",
    "    print('GPU')\n",
    "    print('GPU #0?')\n",
    "    print(var.device.endswith('GPU:0'))\n",
    "else:\n",
    "    print('CPU')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tensor Constant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=8, shape=(), dtype=int32, numpy=42>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ineuron = tf.constant(42)\n",
    "ineuron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ineuron.numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=22, shape=(), dtype=int64, numpy=1>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ineuron1 = tf.constant(1, dtype = tf.int64)\n",
    "ineuron1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[4 2]\n",
      " [9 5]], shape=(2, 2), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "ineuron_x = tf.constant([[4,2],[9,5]])\n",
    "print(ineuron_x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[4, 2],\n",
       "       [9, 5]])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ineuron_x.numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: (2, 2)\n",
      "<dtype: 'int32'>\n"
     ]
    }
   ],
   "source": [
    "print('shape:',ineuron_x.shape)\n",
    "print(ineuron_x.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Commonly used method is to generate constant tf.ones and the tf.zeros like of numpy np.ones & np.zeros"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[1. 1. 1.]\n",
      " [1. 1. 1.]], shape=(2, 3), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "print(tf.ones(shape=(2,3)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[0. 0.]\n",
      " [0. 0.]\n",
      " [0. 0.]], shape=(3, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "print(tf.zeros(shape=(3,2)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[4 6 8]\n",
      " [4 6 8]], shape=(2, 3), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "const2 = tf.constant([[3,4,5], [3,4,5]]);tf\n",
    "const1 = tf.constant([[1,2,3], [1,2,3]]);\n",
    "result = tf.add(const1, const2);\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">We have defined two constants and we add one value to the other. \n",
    ">As a result, we got a Tensor object with the result of the adding. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Random constant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=38, shape=(2, 2), dtype=float32, numpy=\n",
       "array([[ 0.79373837,  1.3887426 ],\n",
       "       [-0.58256006, -1.5399069 ]], dtype=float32)>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.random.normal(shape=(2,2),mean=0,stddev=1.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=42, shape=(2, 2), dtype=int32, numpy=\n",
       "array([[1, 8],\n",
       "       [4, 5]])>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.random.uniform(shape=(2,2),minval=0,maxval=10,dtype=tf.int32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">A variable is a special tensor that is used to store variable values ​​and needs to be initialized with some values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Declaring variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(24,\n",
       " <tf.Variable 'Variable:0' shape=() dtype=int32, numpy=42>,\n",
       " <tf.Variable 'Variable:0' shape=(2, 2, 3) dtype=float32, numpy=\n",
       " array([[[ 0.,  1.,  2.],\n",
       "         [ 3.,  4.,  5.]],\n",
       " \n",
       "        [[ 6.,  7.,  8.],\n",
       "         [ 9., 10., 11.]]], dtype=float32)>)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var0 = 24 # python variable\n",
    "var1 = tf.Variable(42) # rank 0 tensor\n",
    "var2 = tf.Variable([ [ [0., 1., 2.], [3., 4., 5.] ], [ [6., 7., 8.], [9., 10., 11.] ] ]) #rank 3 tensor\n",
    "var0, var1, var2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">TensorFlow will infer the datatype, defaulting to tf.float32 for floats and tf.int32 for integers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The datatype can be explicitly specified"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tf.float64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "float_var64 = tf.Variable(89, dtype = tf.float64)\n",
    "float_var64.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`TensorFlow has a large number of built-in datatypes.`\n",
    "* tf.float16: 16-bit half-precision floating-point.\n",
    "* tf.float32: 32-bit single-precision floating-point.\n",
    "* tf.float64: 64-bit double-precision floating-point.\n",
    "* tf.bfloat16: 16-bit truncated floating-point.\n",
    "* tf.complex64: 64-bit single-precision complex.\n",
    "* tf.complex128: 128-bit double-precision complex.\n",
    "* tf.int8: 8-bit signed integer.\n",
    "* tf.uint8: 8-bit unsigned integer.\n",
    "* tf.uint16: 16-bit unsigned integer.\n",
    "* tf.uint32: 32-bit unsigned integer.\n",
    "* tf.uint64: 64-bit unsigned integer.\n",
    "* tf.int16: 16-bit signed integer.\n",
    "* tf.int32: 32-bit signed integer.\n",
    "* tf.int64: 64-bit signed integer.\n",
    "* tf.bool: Boolean.\n",
    "* tf.string: String.\n",
    "* tf.qint8: Quantized 8-bit signed integer.\n",
    "* tf.quint8: Quantized 8-bit unsigned integer.\n",
    "* tf.qint16: Quantized 16-bit signed integer.\n",
    "* tf.quint16: Quantized 16-bit unsigned integer.\n",
    "* tf.qint32: Quantized 32-bit signed integer.\n",
    "* tf.resource: Handle to a mutable resource.\n",
    "* tf.variant: Values of arbitrary types.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### To reassign a variable, use var.assign()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=89.0>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var_reassign = tf.Variable(89.)\n",
    "var_reassign"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=98.0>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var_reassign.assign(98.)\n",
    "var_reassign"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.Variable 'Variable:0' shape=(2, 2) dtype=float32, numpy=\n",
      "array([[ 1.7069958 , -0.02613977],\n",
      "       [-0.6598021 , -1.0498033 ]], dtype=float32)>\n"
     ]
    }
   ],
   "source": [
    "initial_value = tf.random.normal(shape=(2,2))\n",
    "a = tf.Variable(initial_value)\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">We can assign \"=\" with assign (value), or assign_add (value) with \"+ =\", or assign_sub (value) with \"-=\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_value = tf.random.normal(shape=(2, 2))\n",
    "a.assign(new_value)\n",
    "for i in range(2):\n",
    "    for j in range(2):\n",
    "        assert a[i, j] == new_value[i, j]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "added_value = tf.random.normal(shape=(2,2))\n",
    "a.assign_add(added_value)\n",
    "for i in range(2):\n",
    "    for j in range(2):\n",
    "        assert a[i,j] == new_value[i,j]+added_value[i,j]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Shaping a tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2, 2, 3)\n"
     ]
    }
   ],
   "source": [
    "tensor = tf.Variable([ [ [10., 11., 12.], [13., 14., 15.] ], [ [16., 17., 18.], [19., 20., 21.] ] ]) # tensor variable\n",
    "print(tensor.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tensors can be reshaped and retain the same values which is required for constructing Neural networks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=210, shape=(2, 6), dtype=float32, numpy=\n",
       "array([[10., 11., 12., 13., 14., 15.],\n",
       "       [16., 17., 18., 19., 20., 21.]], dtype=float32)>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor1 = tf.reshape(tensor,[2,6]) # 2 rows 6 cols\n",
    "#tensor2 = tf.reshape(tensor,[1,12]) # 1 rows 12 cols\n",
    "tensor1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=213, shape=(1, 12), dtype=float32, numpy=\n",
       "array([[10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.]],\n",
       "      dtype=float32)>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor2 = tf.reshape(tensor,[1,12]) # 1 row 12 columns\n",
    "tensor2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rank of a tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">The rank of a tensor is defined as the number of dimensions, which is the number of indices that are required to specify any particular element of that tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=215, shape=(), dtype=int32, numpy=3>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.rank(tensor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">(the shape is () because the output here is a scalar value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Specifying an element of a tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=220, shape=(), dtype=float32, numpy=18.0>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor3 = tensor[1, 0, 2] # slice 1, row 0, column 2\n",
    "tensor3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Casting a tensor to a NumPy variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[[10. 11. 12.]\n",
      "  [13. 14. 15.]]\n",
      "\n",
      " [[16. 17. 18.]\n",
      "  [19. 20. 21.]]]\n"
     ]
    }
   ],
   "source": [
    "print(tensor.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "18.0\n"
     ]
    }
   ],
   "source": [
    "print(tensor[1, 0, 2].numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Finding the size or length of a tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor_size = tf.size(input=tensor).numpy()\n",
    "tensor_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tf.float32"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#the datatype of a tensor\n",
    "tensor3.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tensorflow mathematical operations\n",
    ">Can be used as numpy for artificial operations. Tensorflow can not execute these operations on the GPU or TPU.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[-1.2265512  -1.4969218 ]\n",
      " [ 0.01587692  0.5437373 ]], shape=(2, 2), dtype=float32)\n",
      "tf.Tensor(\n",
      "[[-0.90549344  0.50011796]\n",
      " [ 1.4741822  -0.03823759]], shape=(2, 2), dtype=float32)\n",
      "tf.Tensor(\n",
      "[[-2.1320446 -0.9968038]\n",
      " [ 1.4900591  0.5054997]], shape=(2, 2), dtype=float32)\n",
      "tf.Tensor(\n",
      "[[4.545614   0.99361783]\n",
      " [2.220276   0.25552997]], shape=(2, 2), dtype=float32)\n",
      "tf.Tensor(\n",
      "[[0.11859457 0.36905712]\n",
      " [4.4373584  1.6578138 ]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "a = tf.random.normal(shape=(2,2))\n",
    "b = tf.random.normal(shape=(2,2))\n",
    "c = a + b\n",
    "d = tf.square(c)\n",
    "e = tf.exp(c)\n",
    "print(a)\n",
    "print(b)\n",
    "print(c)\n",
    "print(d)\n",
    "print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Performing element-wise primitive tensor operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=246, shape=(2, 2, 3), dtype=float32, numpy=\n",
       "array([[[100., 121., 144.],\n",
       "        [169., 196., 225.]],\n",
       "\n",
       "       [[256., 289., 324.],\n",
       "        [361., 400., 441.]]], dtype=float32)>"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor*tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Broadcasting in Tensorflow\n",
    "\n",
    ">Element-wise tensor operations support broadcasting in the same way that NumPy arrays do.\n",
    "\n",
    ">The simplest example is multiplication of  a tensor by a scalar value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[[40. 44. 48.]\n",
      "  [52. 56. 60.]]\n",
      "\n",
      " [[64. 68. 72.]\n",
      "  [76. 80. 84.]]], shape=(2, 2, 3), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "tensor4 = tensor*4\n",
    "print(tensor4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transpose Matrix multiplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=254, shape=(1, 1), dtype=int32, numpy=array([[64]])>"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matrix_u = tf.constant([[6,7,6]])\n",
    "matrix_v = tf.constant([[3,4,3]])\n",
    "tf.matmul(matrix_u, tf.transpose(a=matrix_v))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Casting a tensor to another datatype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=255, shape=(2, 6), dtype=int32, numpy=\n",
       "array([[10, 11, 12, 13, 14, 15],\n",
       "       [16, 17, 18, 19, 20, 21]])>"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "i = tf.cast(tensor1, dtype=tf.int32)\n",
    "i"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Casting with truncation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: id=257, shape=(), dtype=int32, numpy=4>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "j = tf.cast(tf.constant(4.9), dtype=tf.int32)\n",
    "j"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Ragged tensors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`A ragged tensor is a tensor having one or more ragged dimensions. Ragged dimensions are dimensions that have slices having various lengths.There are a variety of methods for the declaration of ragged arrays, the simplest way is declaring a constant ragged array.`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Below example shows how to declare a constant ragged array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.RaggedTensor [[9, 7, 4, 3], [], [11, 12, 8], [3], [7, 8]]>\n",
      "tf.Tensor([9 7 4 3], shape=(4,), dtype=int32)\n",
      "tf.Tensor([], shape=(0,), dtype=int32)\n",
      "tf.Tensor([11 12  8], shape=(3,), dtype=int32)\n",
      "tf.Tensor([3], shape=(1,), dtype=int32)\n",
      "tf.Tensor([7 8], shape=(2,), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "ragged =tf.ragged.constant([[9, 7, 4, 3], [], [11, 12, 8], [3], [7,8]])\n",
    "print(ragged)\n",
    "print(ragged[0,:])\n",
    "print(ragged[1,:])\n",
    "print(ragged[2,:])\n",
    "print(ragged[3,:])\n",
    "print(ragged[4,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Squared difference of tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(5,), dtype=int32, numpy=array([16,  9,  4, 49, 36])>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "varx = [4,5,6,1,2]\n",
    "vary = 8\n",
    "varz = tf.math.squared_difference(varx,vary)\n",
    "varz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculate the mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Function available\n",
    ">tf.reduce_mean()."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Similar to np.mean, except that it infers the return datatype from the input tensor, whereas np.mean allows you to specify the output type`\n",
    "\n",
    "`tf.reduce_mean(input_tensor, axis=None, keepdims=None, name=None)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining a constant\n",
    "numbers = tf.constant([[8., 9.], [1., 2.]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculate the mean across all axes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=5.0>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.reduce_mean(input_tensor=numbers) #default axis = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculate the mean across columns (reduce rows) with this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=float32, numpy=array([4.5, 5.5], dtype=float32)>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.reduce_mean(input_tensor=numbers, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### When keepdims = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(1, 2), dtype=float32, numpy=array([[4.5, 5.5]], dtype=float32)>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.reduce_mean(input_tensor=numbers, axis=0, keepdims=True) #the reduced axis is retained with a length of 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculate the mean across rows (reduce columns) with this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=float32, numpy=array([8.5, 1.5], dtype=float32)>"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.reduce_mean(input_tensor=numbers, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### When keepdims= True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 1), dtype=float32, numpy=\n",
       "array([[8.5],\n",
       "       [1.5]], dtype=float32)>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.reduce_mean(input_tensor=numbers, axis=1, keepdims=True) #the reduced axis is retained with a length of 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  Random values generation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### tf.random.normal()\n",
    "\n",
    ">tf.random.normal() outputs a tensor of the given shape filled with values of the dtype type from a normal distribution.\n",
    "\n",
    ">The function is as follows:\n",
    "    \n",
    "`tf.random.normal(shape, mean = 0, stddev =2, dtype=tf.float32, seed=None, name=None)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[11.926843 11.951866]\n",
      " [11.554054  8.495991]\n",
      " [11.13671  11.879669]], shape=(3, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "tf.random.normal(shape = (3,2), mean=10, stddev=2, dtype=tf.float32, seed=None, name=None)\n",
    "randon_num = tf.random.normal(shape = (3,2), mean=10.0, stddev=2.0)\n",
    "print(randon_num)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  tf.random.uniform()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">The function is this:\n",
    "    \n",
    ">tf.random.uniform(shape, minval = 0, maxval= None, dtype=tf.float32, seed=None, name=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`This outputs a tensor of the given shape filled with values from a uniform distribution in the range minval to maxval, where the lower bound is inclusive but the upper bound isn't.\n",
    "\n",
    "Example:`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 4), dtype=float32, numpy=\n",
       "array([[0.24019718, 0.6144402 , 0.6651863 , 0.4577496 ],\n",
       "       [0.95334494, 0.8498589 , 0.86562073, 0.8837615 ]], dtype=float32)>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.random.uniform(shape = (2,4), minval=0, maxval=None, dtype=tf.float32, seed=None, name=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Setting the seed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[4 6]\n",
      " [5 2]], shape=(2, 2), dtype=int32)\n",
      "tf.Tensor(\n",
      "[[9 7]\n",
      " [9 4]], shape=(2, 2), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "tf.random.set_seed(11)\n",
    "random_num1 = tf.random.uniform(shape = (2,2), maxval=10, dtype = tf.int32)\n",
    "random_num2 = tf.random.uniform(shape = (2,2), maxval=10, dtype = tf.int32)\n",
    "print(random_num1) #Call 1\n",
    "print(random_num2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[4 6]\n",
      " [5 2]], shape=(2, 2), dtype=int32)\n",
      "tf.Tensor(\n",
      "[[9 7]\n",
      " [9 4]], shape=(2, 2), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "tf.random.set_seed(11) #same seed\n",
    "random_num1 = tf.random.uniform(shape = (2,2), maxval=10, dtype = tf.int32)\n",
    "random_num2 = tf.random.uniform(shape = (2,2), maxval=10, dtype = tf.int32)\n",
    "print(random_num1) #Call 2\n",
    "print(random_num2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Practical example of Random values using Dices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[ 2  2  4]\n",
      " [ 1  6  7]\n",
      " [ 6  1  7]\n",
      " [ 6  6 12]\n",
      " [ 4  4  8]\n",
      " [ 6  2  8]\n",
      " [ 5  6 11]\n",
      " [ 5  5 10]\n",
      " [ 4  6 10]\n",
      " [ 6  6 12]], shape=(10, 3), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "dice11 = tf.Variable(tf.random.uniform([10, 1], minval=1, maxval=7, dtype=tf.int32))\n",
    "dice12 = tf.Variable(tf.random.uniform([10, 1], minval=1, maxval=7, dtype=tf.int32))\n",
    "\n",
    "# lets ADD\n",
    "dice_sum1 = dice11 + dice12\n",
    "# We've got three separate 10x1 matrices. To produce a single 10x3 matrix, we'll concatenate them along dimension 1.\n",
    "finale_matrix = tf.concat(values=[dice11, dice12, dice_sum1], axis=1)\n",
    "print(finale_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Finding the indices of the largest and smallest element"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">The following functions are available:\n",
    "    \n",
    ">`tf.argmax(input, axis=None, name=None, output_type=tf.int64 )`\n",
    "\n",
    ">`tf.argmin(input, axis=None, name=None, output_type=tf.int64 )`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([ 12  11  51  42   6  16  -8 -19  31], shape=(9,), dtype=int32)\n",
      "index of max;  tf.Tensor(2, shape=(), dtype=int64)\n",
      "Max element:  51\n",
      "index of min:  7\n",
      "Min element:  -19\n"
     ]
    }
   ],
   "source": [
    "# 1-D tensor\n",
    "tensor_1d = tf.constant([12, 11, 51, 42, 6, 16, -8, -19, 31])\n",
    "print(tensor_1d)\n",
    "\n",
    "\n",
    "i = tf.argmax(input=tensor_1d)\n",
    "print('index of max; ', i)\n",
    "print('Max element: ',tensor_1d[i].numpy())\n",
    "\n",
    "\n",
    "i = tf.argmin(input=tensor_1d,axis=0).numpy()\n",
    "print('index of min: ', i)\n",
    "print('Min element: ',tensor_1d[i].numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Saving and restoring using a checkpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.Variable 'Variable:0' shape=(2, 4) dtype=int32, numpy=\n",
      "array([[ 5,  6,  9,  3],\n",
      "       [14, 15, 16, 18]])>\n"
     ]
    }
   ],
   "source": [
    "variable1 = tf.Variable([[5,6,9,3],[14,15,16,18]])\n",
    "checkpoint= tf.train.Checkpoint(var=variable1)\n",
    "savepath = checkpoint.save('./vars')\n",
    "variable1.assign([[0,0,0,0],[0,0,0,0]])\n",
    "variable1\n",
    "checkpoint.restore(savepath)\n",
    "print(variable1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using tf.function\n",
    "\n",
    "`tf.function is a function that will take a Python function and return a TensorFlow graph. The advantage of this is that graphs can apply optimizations and exploit parallelism in the Python function (func). tf.function is new to TensorFlow 2.`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Its function is as follows:\n",
    "    \n",
    "`tf.function(\n",
    "func=None,input_signature=None,autograph=True,experimental_autograph_options=None\n",
    ")`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f1(x, y):\n",
    "    return tf.reduce_mean(input_tensor=tf.multiply(x ** 3, 6) + y**3)\n",
    "func = tf.function(f1)\n",
    "x = tf.constant([3., -4.])\n",
    "y = tf.constant([1., 4.])\n",
    "# f1 and f2 return the same value, but f2 executes as a TensorFlow graph\n",
    "assert f1(x,y).numpy() == func(x,y).numpy()\n",
    "#The assert passes, so there is no output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculate the gradient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GradientTape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Another difference from numpy is that it can automatically track the gradient of any variable.\n",
    "\n",
    ">Open one GradientTape and `tape.watch()` track variables through"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[-0.47931308 -0.9943058 ]\n",
      " [ 0.9353223   0.80492437]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "a = tf.random.normal(shape=(2,2))\n",
    "b = tf.random.normal(shape=(2,2))\n",
    "with tf.GradientTape() as tape:\n",
    "    tape.watch(a)\n",
    "    c = tf.sqrt(tf.square(a)+tf.square(b))\n",
    "    dc_da = tape.gradient(c,a)\n",
    "    print(dc_da)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">For all variables, the calculation is tracked by default and used to find the gradient, so do not `usetape.watch()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[-0.47931308 -0.9943058 ]\n",
      " [ 0.9353223   0.80492437]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "a = tf.Variable(a)\n",
    "with tf.GradientTape() as tape:\n",
    "    c = tf.sqrt(tf.square(a)+tf.square(b))\n",
    "    dc_da = tape.gradient(c,a)\n",
    "    print(dc_da)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> You can GradientTapefind higher-order derivatives by opening a few more:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[0.5205455  0.00913197]\n",
      " [0.23476017 0.47745854]], shape=(2, 2), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "with tf.GradientTape() as outer_tape:\n",
    "    with tf.GradientTape() as tape:\n",
    "        c = tf.sqrt(tf.square(a)+tf.square(b))\n",
    "        dc_da = tape.gradient(c,a)\n",
    "    d2c_d2a = outer_tape.gradient(dc_da,a)\n",
    "    print(d2c_d2a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keras, a High-Level API for TensorFlow 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](img/keras.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Keras Sequential model\n",
    "\n",
    "`To build a Keras Sequential model, you add layers to it in the same order that you want the computations to be undertaken by the network.`\n",
    "\n",
    "`After you have built your model, you compile it; this optimizes the computations that are to be undertaken, and is where you allocate the optimizer and the loss function you want your model to use.`\n",
    "\n",
    "`The next stage is to fit the model to the data. This is commonly known as training the model, and is where all the computations take place. It is possible to present the data to the model either in batches, or all at once.`\n",
    "\n",
    "`Next, you evaluate your model to establish its accuracy, loss, and other metrics. Finally, having trained your model, you can use it to make predictions on new data. So, the workflow is: build, compile, fit, evaluate, make predictions.`\n",
    "\n",
    "`There are two ways to create a Sequential model. Let's take a look at each of them.`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](img/kerasmodelling.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Sequential model\n",
    "\n",
    "`Firstly, you can pass a list of layer instances to the constructor, as in the following example.For now, we will just explain enough to allow you to understand what is happening here.`\n",
    "\n",
    "`Acquire the data. MNIST is a dataset of hand-drawn numerals, each on a 28 x 28 pixel grid. Every individual data point is an unsigned 8-bit integer (uint8), as are the labels:`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loading the datset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
      "11493376/11490434 [==============================] - 14s 1us/step\n"
     ]
    }
   ],
   "source": [
    "mnist_data = tf.keras.datasets.mnist\n",
    "(train_x,train_y), (test_x, test_y) = mnist_data.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Definning the variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs=10\n",
    "batch_size = 32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# normalize all the data points and cast the labels to int64\n",
    "train_x, test_x = tf.cast(train_x/255.0, tf.float32), tf.cast(test_x/255.0, tf.float32)\n",
    "train_y, test_y = tf.cast(train_y,tf.int64),tf.cast(test_y,tf.int64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Building the Architecture "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnistmodel1 = tf.keras.models.Sequential([\n",
    "tf.keras.layers.Flatten(),\n",
    "tf.keras.layers.Dense(512,activation=tf.nn.relu),\n",
    "tf.keras.layers.Dropout(0.2),\n",
    "tf.keras.layers.Dense(10,activation=tf.nn.softmax)\n",
    "])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Compiling the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimiser = tf.keras.optimizers.Adam()\n",
    "mnistmodel1.compile (optimizer= optimiser, loss='sparse_categorical_crossentropy', metrics = ['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fitting the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples\n",
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 12s 205us/sample - loss: 0.2185 - accuracy: 0.9348\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - 13s 208us/sample - loss: 0.0986 - accuracy: 0.9701\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 12s 193us/sample - loss: 0.0689 - accuracy: 0.9789\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 12s 193us/sample - loss: 0.0536 - accuracy: 0.9825\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 12s 199us/sample - loss: 0.0439 - accuracy: 0.9858\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1bc798fb548>"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mnistmodel1.fit(train_x, train_y, batch_size=32, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluate the mnistmodel1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 1s 108us/sample - loss: 0.0751 - accuracy: 0.9771\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.07506247069942765, 0.9771]"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mnistmodel1.evaluate(test_x, test_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">This represents a loss of 0.09 and an accuracy of 0.9801 on the test data. \n",
    "\n",
    ">An accuracy of 0.98 means that out of 100 test data points, 98 were, on average, correctly identified by the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "The second way to create a Sequential model\n",
    "The alternative to passing a list of layers to the Sequential model's constructor is to use the add method, as follows, for the same architecture:\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Building the Architecture & Compiling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnistmodel2 = tf.keras.models.Sequential();\n",
    "mnistmodel2.add(tf.keras.layers.Flatten())\n",
    "mnistmodel2.add(tf.keras.layers.Dense(512, activation='relu'))\n",
    "mnistmodel2.add(tf.keras.layers.Dropout(0.2))\n",
    "mnistmodel2.add(tf.keras.layers.Dense(10,activation=tf.nn.softmax))\n",
    "mnistmodel2.compile (optimizer= tf.keras.optimizers.Adam(), loss='sparse_categorical_crossentropy',metrics = ['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fitting the mnistmodel2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples\n",
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 6s 105us/sample - loss: 0.2445 - accuracy: 0.9281\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - ETA: 0s - loss: 0.1054 - accuracy: 0.96 - 6s 96us/sample - loss: 0.1054 - accuracy: 0.9682\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 6s 105us/sample - loss: 0.0741 - accuracy: 0.9773\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 5s 86us/sample - loss: 0.0555 - accuracy: 0.9819\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 6s 92us/sample - loss: 0.0440 - accuracy: 0.9861\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x22960c23888>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mnistmodel2.fit(train_x, train_y, batch_size=64, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluate the mnistmodel2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 1s 90us/sample - loss: 0.0637 - accuracy: 0.9810\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.06365259396769106, 0.981]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mnistmodel2.evaluate(test_x, test_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](img/kerasmodelling2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Keras functional API\n",
    "\n",
    "`The functional API lets you build much more complex architectures than the simple linear stack of Sequential models we have seen previously. It also supports more advanced models. These models include multi-input and multi-output models, models with shared layers, and models with residual connections.`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "mnist = tf.keras.datasets.mnist\n",
    "(train_x,train_y), (test_x, test_y) = mnist.load_data()\n",
    "train_x, test_x = train_x/255.0, test_x/255.0\n",
    "epochs=10\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Building the Architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = tf.keras.Input(shape=(28,28)) # Returns a 'placeholder' tensor\n",
    "x = tf.keras.layers.Flatten()(inputs)\n",
    "x = tf.keras.layers.Dense(512, activation='relu',name='d1')(x)\n",
    "x = tf.keras.layers.Dropout(0.2)(x)\n",
    "predictions = tf.keras.layers.Dense(10,activation=tf.nn.softmax, name='d2')(x)\n",
    "mnistmodel3 = tf.keras.Model(inputs=inputs, outputs=predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Compile & Fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimiser = tf.keras.optimizers.Adam()\n",
    "mnistmodel3.compile (optimizer= optimiser, loss='sparse_categorical_crossentropy', metrics = ['accuracy'])\n",
    "mnistmodel3.fit(train_x, train_y, batch_size=32, epochs=epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluate the mnistmodel3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnistmodel3.evaluate(test_x, test_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Subclassing the Keras Model class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Building the subclass architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MNISTModel(tf.keras.Model):\n",
    "    def __init__(self, num_classes=10):\n",
    "        super(MNISTModel, self).__init__()\n",
    "        # Define your layers here.\n",
    "        inputs = tf.keras.Input(shape=(28,28)) # Returns a placeholder tensor\n",
    "        self.x0 = tf.keras.layers.Flatten()\n",
    "        self.x1 = tf.keras.layers.Dense(512, activation='relu',name='d1')\n",
    "        self.x2 = tf.keras.layers.Dropout(0.2)\n",
    "        self.predictions = tf.keras.layers.Dense(10,activation=tf.nn.softmax, name='d2')\n",
    "\n",
    "\n",
    "    def call(self, inputs):\n",
    "    # This is where to define your forward pass\n",
    "    # using the layers previously defined in `__init__`\n",
    "        x = self.x0(inputs)\n",
    "        x = self.x1(x)\n",
    "        x = self.x2(x)\n",
    "        return self.predictions(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnistmodel4 = MNISTModel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Compile & Fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1875\n",
      "Train on 60000 samples\n",
      "Epoch 1/10\n",
      "60000/60000 [==============================] - 12s 199us/sample - loss: 0.2182 - accuracy: 0.9355\n",
      "Epoch 2/10\n",
      "60000/60000 [==============================] - 12s 201us/sample - loss: 0.0961 - accuracy: 0.9709\n",
      "Epoch 3/10\n",
      "60000/60000 [==============================] - 12s 194us/sample - loss: 0.0697 - accuracy: 0.9783\n",
      "Epoch 4/10\n",
      "60000/60000 [==============================] - 12s 205us/sample - loss: 0.0536 - accuracy: 0.9826\n",
      "Epoch 5/10\n",
      "60000/60000 [==============================] - 12s 193us/sample - loss: 0.0424 - accuracy: 0.9861\n",
      "Epoch 6/10\n",
      "60000/60000 [==============================] - 12s 195us/sample - loss: 0.0363 - accuracy: 0.9878\n",
      "Epoch 7/10\n",
      "60000/60000 [==============================] - 11s 190us/sample - loss: 0.0304 - accuracy: 0.9890\n",
      "Epoch 8/10\n",
      "60000/60000 [==============================] - 12s 199us/sample - loss: 0.0276 - accuracy: 0.9906\n",
      "Epoch 9/10\n",
      "60000/60000 [==============================] - 12s 196us/sample - loss: 0.0244 - accuracy: 0.9913\n",
      "Epoch 10/10\n",
      "60000/60000 [==============================] - 12s 198us/sample - loss: 0.0219 - accuracy: 0.9925\n",
      "10000/10000 [==============================] - 1s 120us/sample - loss: 0.0698 - accuracy: 0.9837\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.06979138339359196, 0.9837]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 32\n",
    "steps_per_epoch = len(train_x.numpy())//batch_size\n",
    "print(steps_per_epoch)\n",
    "mnistmodel4.compile (optimizer= tf.keras.optimizers.Adam(), loss='sparse_categorical_crossentropy',metrics = ['accuracy'])\n",
    "mnistmodel4.fit(train_x, train_y, batch_size=batch_size, epochs=epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluate the mnistmodel4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnistmodel4.evaluate(test_x, test_y)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}