L10c.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Zadania"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Two-class MNIST \n",
    "\n",
    "from keras.datasets import mnist\n",
    "import numpy as np\n",
    "\n",
    "d1 = 5\n",
    "d2 = 6\n",
    "\n",
    "(mnist_x_train, mnist_y_train), (mnist_x_test, mnist_y_test) = mnist.load_data()\n",
    "\n",
    "X_train = (mnist_x_train.astype('float32') / 255.).reshape((len(mnist_x_train), np.prod(mnist_x_train.shape[1:])))\n",
    "y_train = mnist_y_train\n",
    "X_test = (mnist_x_test.astype('float32') / 255.).reshape((len(mnist_x_test), np.prod(mnist_x_test.shape[1:])))\n",
    "y_test = mnist_y_test\n",
    "\n",
    "X_train = X_train[np.logical_or(y_train == d1, y_train == d2)]\n",
    "y_train = y_train[np.logical_or(y_train == d1, y_train == d2)]\n",
    "y_train[y_train==d1] = 0\n",
    "y_train[y_train==d2] = 1\n",
    "X_test = X_test[np.logical_or(y_test == d1, y_test == d2)]\n",
    "y_test = y_test[np.logical_or(y_test == d1, y_test == d2)]\n",
    "y_test[y_test==d1] = 0\n",
    "y_test[y_test==d2] = 1\n",
    "\n",
    "print X_train.shape\n",
    "print X_test.shape\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ćwiczenie 1 [5 pkt]\n",
    "\n",
    "Uzupełnij metody forward_pass oraz backward_pass w klasach ReLU, Sigmoid i Dense. Metoda forward_pass ma przyjmować batch inputów i zwracać batch outputów. Metoda backward_pass ma przyjmować batch inputów oraz batch pochodnych cząstkowych outputów i zwracać batch pochodnych cząstkowych inputów oraz wektor (**nie batch**) pochodnych cząstkowych wag. Jeśli wagi przechowujemy w macierzy dwuwymiarowej, to możemy najpierw policzyć pochodne cząstkowe w macierzy o takim samym kształcie, a następnie np. użyć .flat. \n",
    "\n",
    "Uwaga: dla warstw bez wag należy zwrócić None.\n",
    "\n",
    "## Ćwiczenie 2 [4 pkt]\n",
    "\n",
    "Uzupełnij metodę _forward_pass klasy Network. Metoda ta ma przyjmować batch inputów (X) i zwracać dwie rzeczy:\n",
    "* inps - lista batchów inputów dla każdej warstwy w sieci (włącznie z X); te wartości będziemy używali w metodzie _backward_pass\n",
    "* output - batch outputów z sieci (czyli $\\mathbf{\\hat y}$); output **nie** powinien być ostatnim elementem inps.\n",
    "\n",
    "## Ćwiczenie 3 [5 pkt]\n",
    "\n",
    "Uzupełnij metodę _backward_pass klasy Network. Zwróć uwagę, że pochodna funkcji kosztu po neuronach ostatniej warstwy jest już liczona w metodzie _fit_on_batch. Metoda ma zwracać listę layer_grads, której elementy to wektory pochodnych cząstkowych funkcji kosztu po kolejnych warstwach (zwrócone przez metodę Layer.backward_pass). Kolejność wektorów w tej liście ma być zgodna z kolejnością warstw w sieci.\n",
    "\n",
    "## Ćwiczenie 4 [3 pkt]\n",
    "Naucz sieć neuronową z jedną warstwą ukrytą i aktywacją Sigmoid na powyższych danych (dwuklasowy MNIST). Użyj MSE jako funkcji kosztu (oznacza to regresję do numeru klasy, co jest złym pomysłem, ale póki nie mamy klasy Crossentropy musi nam to wystarczyć). Użyj GD. Reportuj loss oraz accuracy.\n",
    "\n",
    "## Ćwiczenie 5 [3 pkt]\n",
    "Uzupełnić klasę Crossentropy, wzorując się na klasie MSE.\n",
    "\n",
    "## Ćwiczenie  6 [3 pkt]\n",
    "Uzupełnić klasę Momentum, wzorując się na klasie GD. Wzory można znaleźć tutaj: http://distill.pub/2017/momentum/\n",
    "\n",
    "## Ćwiczenie 7 [3 pkt]\n",
    "Naucz sieć neuronową z jedną warstwą ukrytą. Rozważ dwa przypadki: aktywację ReLU oraz Sigmoid. Czy jest sens używać ReLU jako ostatnią warstwę? Użyj Crossentropy jako funkcji kosztu. Użyj Momentum. Reportuj loss oraz accuracy.\n",
    "\n",
    "## Ćwiczenie 8 [6 pkt]\n",
    "Vanishing gradient.\n",
    "\n",
    "Zadanie polega na zbadaniu zjawiska *vanishing gradient* w głębokich sieciach. Należy zmodyfikować kod warstwy Dense i dodać monitorowanie **normy euklidesowej** wektora delta_weights. Każdą warstwę Dense w trenowanej sieci należy monitorować oddzielnie. Po każdym wywołaniu metody fit_on_batch każdy z monitorów powinien zapamiętać nową normę. Po nauczeniu sieci dla każdej warstwy należy narysować wykres: poziomo - numer wywołania fit_on_batch, pionowo - norma delta_weights. Im niżej znajduje się warstwa Dense, tym silniej będzie zachodziło zjawisko *vanishing gradient*.\n",
    "\n",
    "Naucz dwuwarstwową sieć z aktywacjami Sigmoid, reportując normy delta_weights. Powtórz to dla głębszej sieci (np. 6-10 warstw).\n",
    "\n",
    "## Ćwiczenie 9 [4 pkt]\n",
    "Przetestować kod z ćwiczenia 7. (dwuwarstwowa sieć) stosując inne inicjalizacje wag w warstwach Dense. Napisać własną inicjalizację wag, która sprawi, że sieć niczego się nie nauczy (init='stupid').\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Warstwy\n",
    "\n",
    "class Layer():\n",
    "\n",
    "    def forward_pass(self, input):\n",
    "        # return output\n",
    "        raise NotImplementedError()\n",
    "\n",
    "    def backward_pass(self, input, output_grad):\n",
    "        # return input_grad, weight_grad\n",
    "        raise NotImplementedError()\n",
    "\n",
    "    def update_weights(self, delta_weights):\n",
    "        pass\n",
    "\n",
    "    def debug_grad(self, evaluate_loss):\n",
    "        return None\n",
    "\n",
    "class ReLU(Layer):\n",
    "\n",
    "    def forward_pass(self, input):\n",
    "        # Ćwiczenie 1\n",
    "        pass\n",
    "        \n",
    "    def backward_pass(self, input, output_grad):\n",
    "        # Ćwiczenie 1\n",
    "        pass\n",
    "\n",
    "class Sigmoid(Layer):\n",
    "\n",
    "    def forward_pass(self, input):\n",
    "        # Ćwiczenie 1\n",
    "        pass\n",
    "\n",
    "    def backward_pass(self, input, output_grad):\n",
    "        # Ćwiczenie 1\n",
    "        pass\n",
    "\n",
    "class Dense(Layer):\n",
    "\n",
    "    def __init__(self, input_size, output_size, init = 'gaussian'):\n",
    "        input_size += 1\n",
    "        if init == 'zeros':\n",
    "            self.weights = np.zeros((input_size, output_size))\n",
    "        elif init == 'gaussian':\n",
    "            self.weights = np.random.normal(\n",
    "                0.,\n",
    "                2. / (input_size + output_size),\n",
    "                (input_size, output_size)\n",
    "            )\n",
    "        elif init == 'stupid':\n",
    "            # Ćwiczenie 8\n",
    "            raise NotImplementedError()            \n",
    "        else:\n",
    "            raise NotImplementedError()\n",
    "        self.weights = np.asmatrix(self.weights)\n",
    "\n",
    "    def forward_pass(self, input):\n",
    "        # Ćwiczenie 1\n",
    "        pass\n",
    "    \n",
    "    def backward_pass(self, input, output_grad):\n",
    "        # Ćwiczenie 1\n",
    "        pass\n",
    "        return input_grad, weight_grad.flat\n",
    "\n",
    "    def update_weights(self, delta_weights):\n",
    "        # Ćwiczenie 7 - monitorowanie normy wektora delta_weights\n",
    "        self.weights += delta_weights.reshape(self.weights.shape)\n",
    "\n",
    "    def debug_grad(self, evaluate_loss):\n",
    "        base = evaluate_loss()\n",
    "        grad = []\n",
    "        for (x, y), w in np.ndenumerate(self.weights):\n",
    "            self.weights[x, y] = w + 0.0001\n",
    "            changed = evaluate_loss()\n",
    "            grad.append(10000. * (changed - base))\n",
    "            self.weights[x, y] = w\n",
    "        return grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Optimizers\n",
    "\n",
    "class Optimizer():\n",
    "\n",
    "    def calculate_deltas(self, grad):\n",
    "        raise NotImplementedError()\n",
    "\n",
    "class GD(Optimizer):\n",
    "\n",
    "    def __init__(self, learning_rate):\n",
    "        self.learning_rate = learning_rate\n",
    "\n",
    "    def calculate_deltas(self, grad):\n",
    "        return -self.learning_rate * grad\n",
    "\n",
    "class Momentum(Optimizer):\n",
    "\n",
    "    def __init__(self, alpha, beta):\n",
    "        # Ćwiczenie 6\n",
    "        pass\n",
    "        \n",
    "    def calculate_deltas(self, grad):\n",
    "        # Ćwiczenie 6\n",
    "        pass\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Funkcje kosztu\n",
    "\n",
    "class Loss():\n",
    "\n",
    "    def forward_pass(self, y, t):\n",
    "        # return cost\n",
    "        raise NotImplementedError()\n",
    "\n",
    "    def backward_pass(self, y, t):\n",
    "        # return y_grad\n",
    "        raise NotImplementedError()\n",
    "\n",
    "class MSE(Loss):\n",
    "\n",
    "    def forward_pass(self, y, t):\n",
    "        return np.average(0.5 * np.square(y - t))\n",
    "\n",
    "    def backward_pass(self, y, t):\n",
    "        return (y - t) / y.size\n",
    "\n",
    "class Crossentropy(Loss):\n",
    "\n",
    "    def forward_pass(self, y, t):\n",
    "        # Ćwiczenie 5\n",
    "        pass\n",
    "        \n",
    "    def backward_pass(self, y, t):\n",
    "        # Ćwiczenie 5\n",
    "        pass\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Network():\n",
    "\n",
    "    def __init__(self, loss, optimizer, metrics = []):\n",
    "        self.layers = []\n",
    "        self.loss = loss\n",
    "        self.optimizer = optimizer\n",
    "        self.metrics = metrics\n",
    "\n",
    "    def add(self, layer):\n",
    "        self.layers.append(layer)\n",
    "\n",
    "    def fit(self, X, t, epochs, batch_size=256, print_stats=False):\n",
    "        X = np.array(X)\n",
    "        t = np.array(t)\n",
    "        X = X.reshape(len(X), -1)\n",
    "        t = t.reshape(len(t), -1)\n",
    "        if X.shape[0] != t.shape[0]:\n",
    "            raise ValueError(\"Array sizes don't match\")\n",
    "\n",
    "        for epoch in range(epochs):\n",
    "            if print_stats:\n",
    "                print(\"Epoch %d\" % (epoch+1))\n",
    "                print(\"    -> batch size: %d\" % batch_size)\n",
    "            rng_state = np.random.get_state()\n",
    "            np.random.shuffle(X)\n",
    "            np.random.set_state(rng_state)\n",
    "            np.random.shuffle(t)\n",
    "            pos = 0\n",
    "            while pos < len(X):\n",
    "                batch_X = X[pos:pos+batch_size]\n",
    "                batch_t = t[pos:pos+batch_size]\n",
    "                self._fit_on_batch(batch_X, batch_t)\n",
    "                pos += batch_size\n",
    "            if print_stats:\n",
    "                _, y = self._forward_pass(X)\n",
    "                l = self.loss.forward_pass(y, t)\n",
    "                print(\"    -> loss: %f\" % l)\n",
    "                for m in self.metrics:\n",
    "                    print(\"    -> %s: %f\" % (m.__name__, m(y, t)))\n",
    "\n",
    "    def predict(self, X):\n",
    "        inps, out = self._forward_pass(X)\n",
    "        return out\n",
    "\n",
    "    def _fit_on_batch(self, batch_X, batch_t):\n",
    "        inps, out = self._forward_pass(batch_X)\n",
    "        layer_grads = self._backward_pass(\n",
    "            inps,\n",
    "            self.loss.backward_pass(out, batch_t)\n",
    "        )\n",
    "        grad = self._join(layer_grads)\n",
    "        deltas = self.optimizer.calculate_deltas(grad)\n",
    "        for l, d in zip(self.layers, self._split(deltas, layer_grads)):\n",
    "            if not d is None:\n",
    "                l.update_weights(d)\n",
    "\n",
    "    def _join(self, grads):\n",
    "        return np.concatenate([g for g in grads if not g is None])\n",
    "\n",
    "    def _split(self, grads, layer_grads):\n",
    "        out = []\n",
    "        start = 0\n",
    "        for l in layer_grads:\n",
    "            if l is None:\n",
    "                out.append(None)\n",
    "            else:\n",
    "                out.append(grads[start:start+len(l)])\n",
    "                start += len(l)\n",
    "        return out\n",
    "\n",
    "    def _forward_pass(self, X):\n",
    "        inps = []\n",
    "        output = None\n",
    "\n",
    "        # Ćwiczenie 2\n",
    "        \n",
    "        return inps, output\n",
    "\n",
    "    def _backward_pass(self, inps, grad):\n",
    "        layer_grads = []\n",
    "\n",
    "        # Ćwiczenie 3\n",
    "        \n",
    "        return layer_grads\n",
    "\n",
    "    def _debug_grads(self, X, t):\n",
    "        layer_grads = []\n",
    "        for l in self.layers:\n",
    "            g = l.debug_grad(\n",
    "                lambda: self.loss.forward_pass(self._forward_pass(X)[1], t)\n",
    "            )\n",
    "            if not g is None:\n",
    "                g = np.array(np.array(g).flat)\n",
    "            layer_grads.append(g)\n",
    "        return layer_grads"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Ćwiczenie 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Ćwiczenie 7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Ćwiczenie 8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Ćwiczenie 9"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Asserty"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ćwiczenie 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "np.random.seed(43)\n",
    "d1 = Dense(input_size=4, output_size=3, init=\"gaussian\")\n",
    "d2 = Dense(input_size=4, output_size=3, init=\"zeros\")\n",
    "r = ReLU()\n",
    "s = Sigmoid()\n",
    "inp = np.random.random(4).reshape((1,-1)) - 0.5\n",
    "out_grad_4 = np.random.random(4).reshape((1,-1)) - 0.5\n",
    "out_grad_3 = np.random.random(3).reshape((1,-1)) - 0.5\n",
    "\n",
    "print \"inp:\"\n",
    "print inp.shape\n",
    "print inp\n",
    "print \"out_grad_4:\"\n",
    "print out_grad_4.shape\n",
    "print out_grad_4\n",
    "print \"out_grad_3:\"\n",
    "print out_grad_3.shape\n",
    "print out_grad_3\n",
    "\n",
    "print \"Testing d1...\"\n",
    "print \"d1.forward_pass(inp):\"\n",
    "t = d1.forward_pass(inp)\n",
    "print t.shape\n",
    "print t\n",
    "print \"d1.backward_pass(inp, out_grad_3):\"\n",
    "t = d1.backward_pass(inp, out_grad_3)\n",
    "print t[0].shape\n",
    "print t[0]\n",
    "print t[1].shape\n",
    "print t[1]\n",
    "\n",
    "print \"Testing d2...\"\n",
    "print \"d2.forward_pass(inp):\"\n",
    "t = d2.forward_pass(inp)\n",
    "print t.shape\n",
    "print t\n",
    "print \"d2.backward_pass(inp, out_grad_3):\"\n",
    "t = d2.backward_pass(inp, out_grad_3)\n",
    "print t[0].shape\n",
    "print t[0]\n",
    "print t[1].shape\n",
    "print t[1]\n",
    "\n",
    "print \"Testing r...\"\n",
    "print \"r.forward_pass(inp):\"\n",
    "t = r.forward_pass(inp)\n",
    "print t.shape\n",
    "print t\n",
    "print \"r.backward_pass(inp, out_grad_4):\"\n",
    "t = r.backward_pass(inp, out_grad_4)\n",
    "print t[0].shape\n",
    "print t[0]\n",
    "print t[1]\n",
    "\n",
    "print \"Testing s...\"\n",
    "print \"s.forward_pass(inp):\"\n",
    "t = s.forward_pass(inp)\n",
    "print t.shape\n",
    "print t\n",
    "print \"s.backward_pass(inp, out_grad_4):\"\n",
    "t = s.backward_pass(inp, out_grad_4)\n",
    "print t[0].shape\n",
    "print t[0]\n",
    "print t[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    inp:\n",
    "    (1, 4)\n",
    "    [[-0.05424164  0.16972465 -0.41749995  0.39709858]]\n",
    "    out_grad_4:\n",
    "    (1, 4)\n",
    "    [[-0.2019965  -0.23769518 -0.49487045  0.04320252]]\n",
    "    out_grad_3:\n",
    "    (1, 3)\n",
    "    [[-0.02440363  0.13637368  0.47820413]]\n",
    "    Testing d1...\n",
    "    d1.forward_pass(inp):\n",
    "    (1, 3)\n",
    "    [[-0.02707048 -0.20288487 -0.03403094]]\n",
    "    d1.backward_pass(inp, out_grad_3):\n",
    "    (1, 4)\n",
    "    [[-0.01685764  0.21647157  0.04530229  0.03971692]]\n",
    "    (15,)\n",
    "    [-0.02440363  0.13637368  0.47820413  0.00132369 -0.00739713 -0.02593858\n",
    "     -0.0041419   0.02314597  0.08116303  0.01018852 -0.056936   -0.1996502\n",
    "     -0.00969065  0.05415379  0.18989418]\n",
    "    Testing d2...\n",
    "    d2.forward_pass(inp):\n",
    "    (1, 3)\n",
    "    [[ 0.  0.  0.]]\n",
    "    d2.backward_pass(inp, out_grad_3):\n",
    "    (1, 4)\n",
    "    [[ 0.  0.  0.  0.]]\n",
    "    (15,)\n",
    "    [-0.02440363  0.13637368  0.47820413  0.00132369 -0.00739713 -0.02593858\n",
    "     -0.0041419   0.02314597  0.08116303  0.01018852 -0.056936   -0.1996502\n",
    "     -0.00969065  0.05415379  0.18989418]\n",
    "    Testing r...\n",
    "    r.forward_pass(inp):\n",
    "    (1, 4)\n",
    "    [[ 0.          0.16972465  0.          0.39709858]]\n",
    "    r.backward_pass(inp, out_grad_4):\n",
    "    (1, 4)\n",
    "    [[ 0.         -0.23769518  0.          0.04320252]]\n",
    "    None\n",
    "    Testing s...\n",
    "    s.forward_pass(inp):\n",
    "    (1, 4)\n",
    "    [[ 0.48644291  0.5423296   0.39711514  0.59799036]]\n",
    "    s.backward_pass(inp, out_grad_4):\n",
    "    (1, 4)\n",
    "    [[-0.050462   -0.05899789 -0.11847926  0.01038579]]\n",
    "    None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ćwiczenie 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.seed(43)\n",
    "n = Network(loss=MSE(), optimizer=GD(learning_rate=0.001), metrics=[])\n",
    "n.add(Dense(input_size=4, output_size=3, init=\"gaussian\"))\n",
    "n.add(ReLU())\n",
    "n.add(Dense(input_size=3, output_size=4, init=\"gaussian\"))\n",
    "n.add(ReLU())\n",
    "n.add(Dense(input_size=4, output_size=1, init=\"gaussian\"))\n",
    "n.add(Sigmoid())\n",
    "inp = np.random.random((2,4)) - 0.5\n",
    "inps, out = n._forward_pass(inp)\n",
    "\n",
    "print \"inp:\"\n",
    "print inp\n",
    "for i, inp in enumerate(inps):\n",
    "    print \"inps[\" + str(i) + \"]:\"\n",
    "    print inp\n",
    "print \"out:\"\n",
    "print out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    inp:\n",
    "    [[ 0.06066317 -0.08874451  0.22698799 -0.10080311]\n",
    "     [ 0.17014516  0.20471561  0.10955987  0.04003446]]\n",
    "    inps[0]:\n",
    "    [[ 0.06066317 -0.08874451  0.22698799 -0.10080311]\n",
    "     [ 0.17014516  0.20471561  0.10955987  0.04003446]]\n",
    "    inps[1]:\n",
    "    [[ 0.06231216 -0.25663962 -0.11549112]\n",
    "     [ 0.03836232 -0.10585852 -0.03015728]]\n",
    "    inps[2]:\n",
    "    [[ 0.06231216  0.          0.        ]\n",
    "     [ 0.03836232  0.          0.        ]]\n",
    "    inps[3]:\n",
    "    [[-0.04493685  0.4840028  -0.15711608 -0.22696511]\n",
    "     [-0.0396103   0.48391873 -0.15615311 -0.24031931]]\n",
    "    inps[4]:\n",
    "    [[ 0.          0.4840028   0.          0.        ]\n",
    "     [ 0.          0.48391873  0.          0.        ]]\n",
    "    inps[5]:\n",
    "    [[ 0.29948146]\n",
    "     [ 0.29947621]]\n",
    "    out:\n",
    "    [[ 0.57431575]\n",
    "     [ 0.57431447]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ćwiczenie 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.seed(43)\n",
    "n = Network(loss=MSE(), optimizer=GD(learning_rate=0.001), metrics=[])\n",
    "n.add(Dense(input_size=4, output_size=3, init=\"gaussian\"))\n",
    "n.add(ReLU())\n",
    "n.add(Dense(input_size=3, output_size=4, init=\"gaussian\"))\n",
    "n.add(ReLU())\n",
    "n.add(Dense(input_size=4, output_size=1, init=\"gaussian\"))\n",
    "n.add(Sigmoid())\n",
    "\n",
    "inp = np.random.random((2,4)) - 0.5\n",
    "target = inp[:,0:1]\n",
    "inps, out = n._forward_pass(inp)\n",
    "grad = n.loss.backward_pass(out, target)\n",
    "layer_grads = n._backward_pass(inps, grad)\n",
    "\n",
    "print \"inp:\"\n",
    "print inp\n",
    "print \"target:\"\n",
    "print target\n",
    "for i, inp in enumerate(inps):\n",
    "    print \"inps[\" + str(i) + \"]:\"\n",
    "    print inp\n",
    "print \"out:\"\n",
    "print out\n",
    "print \"grad:\"\n",
    "print grad\n",
    "for i, grad in enumerate(layer_grads):\n",
    "    print \"layer_grads[\" + str(i) + \"]:\"\n",
    "    print grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    inp:\n",
    "    [[ 0.06066317 -0.08874451  0.22698799 -0.10080311]\n",
    "     [ 0.17014516  0.20471561  0.10955987  0.04003446]]\n",
    "    target:\n",
    "    [[ 0.06066317]\n",
    "     [ 0.17014516]]\n",
    "    inps[0]:\n",
    "    [[ 0.06066317 -0.08874451  0.22698799 -0.10080311]\n",
    "     [ 0.17014516  0.20471561  0.10955987  0.04003446]]\n",
    "    inps[1]:\n",
    "    [[ 0.06231216 -0.25663962 -0.11549112]\n",
    "     [ 0.03836232 -0.10585852 -0.03015728]]\n",
    "    inps[2]:\n",
    "    [[ 0.06231216  0.          0.        ]\n",
    "     [ 0.03836232  0.          0.        ]]\n",
    "    inps[3]:\n",
    "    [[-0.04493685  0.4840028  -0.15711608 -0.22696511]\n",
    "     [-0.0396103   0.48391873 -0.15615311 -0.24031931]]\n",
    "    inps[4]:\n",
    "    [[ 0.          0.4840028   0.          0.        ]\n",
    "     [ 0.          0.48391873  0.          0.        ]]\n",
    "    inps[5]:\n",
    "    [[ 0.29948146]\n",
    "     [ 0.29947621]]\n",
    "    out:\n",
    "    [[ 0.57431575]\n",
    "     [ 0.57431447]]\n",
    "    grad:\n",
    "    [[ 0.25682629]\n",
    "     [ 0.20208465]]\n",
    "    layer_grads[0]:\n",
    "    [  2.45711515e-05   0.00000000e+00   0.00000000e+00   2.67516826e-06\n",
    "       0.00000000e+00   0.00000000e+00   9.94708194e-07   0.00000000e+00\n",
    "       0.00000000e+00   4.30677425e-06   0.00000000e+00   0.00000000e+00\n",
    "      -9.52974216e-07   0.00000000e+00   0.00000000e+00]\n",
    "    layer_grads[1]:\n",
    "    None\n",
    "    layer_grads[2]:\n",
    "    [ 0.          0.00700006  0.          0.          0.          0.00036236\n",
    "      0.          0.          0.          0.          0.          0.          0.\n",
    "      0.          0.          0.        ]\n",
    "    layer_grads[3]:\n",
    "    None\n",
    "    layer_grads[4]:\n",
    "    [ 0.11219329  0.          0.05429771  0.          0.        ]\n",
    "    layer_grads[5]:\n",
    "    None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ćwiczenie 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.seed(43)\n",
    "ce = Crossentropy()\n",
    "y = np.random.random(10)\n",
    "t = np.random.random(10)\n",
    "\n",
    "print \"y:\"\n",
    "print y\n",
    "print \"t:\"\n",
    "print t\n",
    "print \"ce.forward_pass(y,t):\"\n",
    "print ce.forward_pass(y,t)\n",
    "print \"ce.backward_pass(y,t):\"\n",
    "print ce.backward_pass(y,t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    y:\n",
    "    [ 0.11505457  0.60906654  0.13339096  0.24058962  0.32713906  0.85913749\n",
    "      0.66609021  0.54116221  0.02901382  0.7337483 ]\n",
    "    t:\n",
    "    [ 0.39495002  0.80204712  0.25442113  0.05688494  0.86664864  0.221029\n",
    "      0.40498945  0.31609647  0.0766627   0.84322469]\n",
    "    ce.forward_pass(y,t):\n",
    "    0.736415962327\n",
    "    ce.backward_pass(y,t):\n",
    "    [-0.27490047 -0.08104869 -0.10469935  0.10054647 -0.24509895  0.5272741\n",
    "      0.11739401  0.0906406  -0.16913545 -0.05603779]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ćwiczenie 6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.seed(43)\n",
    "opt = Momentum(alpha=0.02, beta=0.99)\n",
    "grad1 = np.random.random(5)\n",
    "grad2 = np.random.random(5)\n",
    "grad3 = np.random.random(5)\n",
    "opt.calculate_deltas(grad1)\n",
    "opt.calculate_deltas(grad2)\n",
    "opt.calculate_deltas(grad3)\n",
    "\n",
    "print \"grad1:\"\n",
    "print grad1\n",
    "print \"grad2:\"\n",
    "print grad2\n",
    "print \"grad3:\"\n",
    "print grad3\n",
    "\n",
    "print \"opt.calculate_deltas(grad1):\"\n",
    "print opt.calculate_deltas(grad1)\n",
    "print \"opt.calculate_deltas(grad2):\"\n",
    "print opt.calculate_deltas(grad2)\n",
    "print \"opt.calculate_deltas(grad3):\"\n",
    "print opt.calculate_deltas(grad3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    grad1:\n",
    "    [ 0.11505457  0.60906654  0.13339096  0.24058962  0.32713906]\n",
    "    grad2:\n",
    "    [ 0.85913749  0.66609021  0.54116221  0.02901382  0.7337483 ]\n",
    "    grad3:\n",
    "    [ 0.39495002  0.80204712  0.25442113  0.05688494  0.86664864]\n",
    "    opt.calculate_deltas(grad1):\n",
    "    [-0.02919466 -0.0529381  -0.0209018  -0.01117572 -0.04443381]\n",
    "    opt.calculate_deltas(grad2):\n",
    "    [-0.04608546 -0.06573052 -0.03151603 -0.01164424 -0.05866444]\n",
    "    opt.calculate_deltas(grad3):\n",
    "    [-0.05352361 -0.08111416 -0.03628929 -0.0126655  -0.07541077]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python2",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  },
  "toc": {
   "toc_cell": false,
   "toc_number_sections": true,
   "toc_threshold": 6,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}