Activations.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "young-concentration",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchvision\n",
    "import numpy as np\n",
    "from torch import nn\n",
    "\n",
    "from helpers import Model, train, visualize_1d, visualize_2d"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6765d6d1",
   "metadata": {},
   "source": [
    "# Activations\n",
    "Today we are learning about various activations used in neural networks. These activations are (typically) non-linear functions allowing the network to learn more complex internal representations."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "960b4a59",
   "metadata": {},
   "source": [
    "## H(y) Step Function\n",
    "In the first journal club of this module we used the step function. Which we learned in the second we are not able to use for training a model because it has 0 derivative almost everywhere"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0dde9f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "class StepFunction(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "\n",
    "    def forward(self, x):\n",
    "        y = torch.zeros_like(x)\n",
    "        y[x > 0] = 1\n",
    "        return y\n",
    "    \n",
    "visualize_1d(StepFunction())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34adb9cb",
   "metadata": {},
   "source": [
    "# Sigmoid\n",
    "How can we fix the derivative problem? Use a fuction that looks similar but is differrentable. For the step function this is the sigmoid function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "875ad655",
   "metadata": {},
   "outputs": [],
   "source": [
    "visualize_1d(nn.Sigmoid(), min_x=-10, max_x=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9673ae2e",
   "metadata": {},
   "source": [
    "## Max Function\n",
    "For classification we would like our output to represent which class is most likely to be right answer one way of doing this is having the largest output set to 1 and all the others to 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4dbf700e",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "class Max(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "\n",
    "    def forward(self, x):\n",
    "        y = torch.zeros_like(x)\n",
    "        max_inds = torch.max(x, dim=-1)[1]\n",
    "        for i in range(y.shape[-1]):\n",
    "            y[max_inds == i, i] = 1\n",
    "        return y\n",
    "\n",
    "visualize_2d(Max())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca566711",
   "metadata": {},
   "source": [
    "## SoftMax\n",
    "Again we made a function we cant use... But we use the same trick and use whats known as  the SoftMax function. This function looks kind of like the above function but smoothly transitions between the 2 maximum values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1cf9eb2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "visualize_2d(nn.Softmax(dim=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90414011",
   "metadata": {},
   "source": [
    "## Training\n",
    "We now have all the pieces we need to do training. We'll be building networks that predict 10 diffrent articles of clothing using the FMNIST dataset. This dataset is similar to the MNIST digit prediction dataset. The model were constructing will use whatever activations you give it. An `internal_activation` for using between hidden layers, and `final_activation` to use at the end before the loss function. The network has two hidden layers with 512 then 128 neurons. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29bf6f77",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Model(nn.Sigmoid(), nn.Softmax(dim=1))\n",
    "model = train(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f69d7f8",
   "metadata": {},
   "source": [
    "# Your Turn\n",
    "Can you beat the score above? Try out whatever activations you'd like and see if you can find anything interesting. And don't worry about rerunning to get better initializations. The random seeds are fixed to stay fair."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74d08c2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "internal_activation = nn.Sigmoid() # Try out diffrent internal activations\n",
    "final_activation = nn.Softmax(dim=1)  # If brave try to find a better function for feeding to cross entropy loss\n",
    "\n",
    "model = Model(internal_activation, final_activation)\n",
    "model = train(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "475edbc3",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}