week01_intro/homework.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Numpy-имплементация многослойной нейронной сети \n",
    "\n",
    "**Разработчик: Артём Бабенко**\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mryab/dl-hse-ami/blob/master/week01_intro/homework.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "В этом задании необходимо будет самостоятельно реализовать нейронную сеть и обучить ее с помощью метода обратного распространения ошибки."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Load dependencies\n",
    "!wget --quiet --show-progress \"https://raw.githubusercontent.com/mryab/dl-hse-ami/master/week01_intro/util.py\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Сначала определим базовый класс Layer с основными методами .forward() и .backward(). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class Layer:\n",
    "    \"\"\"\n",
    "    A building block. Each layer is capable of performing two things:\n",
    "    \n",
    "    - Process input to get output:           output = layer.forward(input)\n",
    "    \n",
    "    - Propagate gradients through itself:    grad_input = layer.backward(input, grad_output)\n",
    "    \n",
    "    Some layers also have learnable parameters which they update during layer.backward.\n",
    "    \"\"\"\n",
    "    def __init__(self):\n",
    "        \"\"\"Here you can initialize layer parameters (if any) and auxiliary stuff.\"\"\"\n",
    "        # An identity layer does nothing\n",
    "        pass\n",
    "    \n",
    "    def forward(self, input):\n",
    "        \"\"\"\n",
    "        Takes input data of shape [batch, input_units], returns output data [batch, output_units]\n",
    "        \"\"\"\n",
    "        # An identity layer just returns whatever it gets as input.\n",
    "        return input\n",
    "\n",
    "    def backward(self, input, grad_output):\n",
    "        \"\"\"\n",
    "        Performs a backpropagation step through the layer, with respect to the given input.\n",
    "        \n",
    "        To compute loss gradients w.r.t input, you need to apply chain rule (backprop):\n",
    "        \n",
    "        d loss / d x  = (d loss / d layer) * (d layer / d x)\n",
    "        \n",
    "        Luckily, you already receive d loss / d layer as input, so you only need to multiply it by d layer / d x.\n",
    "        \n",
    "        If your layer has parameters (e.g. dense layer), you also need to update them here using d loss / d layer\n",
    "        \"\"\"\n",
    "        # The gradient of an identity layer is precisely grad_output\n",
    "        input_dim = input.shape[1]\n",
    "        \n",
    "        d_layer_d_input = np.eye(input_dim)\n",
    "        \n",
    "        return np.dot(grad_output, d_layer_d_input) # chain rule"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Дальнейший план\n",
    "\n",
    "В этом задании необходимо построить нейросеть для классификации рукописных цифр. Для этого понадобится реализовать следующее:\n",
    "- ReLU слой\n",
    "- Полносвязный слой, $f(X)=W \\cdot X + b$\n",
    "- Функцию потерь, для задачи классификации - cross-entropy\n",
    "- Алгоритм обратного распространения ошибки\n",
    "\n",
    "Далее вам необходимо будет реализовать каждый из пунктов выше, пользуяюсь предоставленными сниппетами кода.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### ReLU слой\n",
    "В этом слое применяется поэлементная нелинейность $f(X)=max(X,0)$. В этом слое необходимо реализовать только .forward() метод."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class ReLU(Layer):\n",
    "    def __init__(self):\n",
    "        \"\"\"ReLU layer simply applies elementwise rectified linear unit to all inputs\"\"\"\n",
    "        pass\n",
    "    \n",
    "    def forward(self, input):\n",
    "        \"\"\"Apply elementwise ReLU to [batch, input_units] matrix\"\"\"\n",
    "        # <your code>\n",
    "        return output\n",
    "    \n",
    "    def backward(self, input, grad_output):\n",
    "        \"\"\"Compute gradient of loss w.r.t. ReLU input\"\"\"\n",
    "        relu_grad_mask = input > 0\n",
    "        return grad_output * relu_grad_mask        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Для отладки градиентов, вычисленных аналитически, рекомендуется использовать сравнение с градиентами, вычисленными численно, как рассказывалось на лекции. Если вы корректно реализовали методы .forward() и .backward(), то значения градиентов окажутся достаточно близкими. Функция численного подсчета градиентов уже реализована в файле util.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# some tests\n",
    "from util import eval_numerical_gradient\n",
    "points = np.linspace(-1,1,10*32).reshape([10,32])\n",
    "l = ReLU()\n",
    "grads = l.backward(points, np.ones([10,32])/(32*10))\n",
    "numeric_grads = eval_numerical_gradient(lambda x: l.forward(x).mean(), x=points)\n",
    "assert np.allclose(grads, numeric_grads, rtol=1e-3, atol=0),\\\n",
    "    \"gradient returned by your layer does not match the numerically computed gradient\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Полносвязный слой\n",
    "\n",
    "У полносвязного слоя есть обучаемые параметры: матрица линейного преобразования и столбец свободных членов.\n",
    "\n",
    "$$f(X)= W \\cdot X + b $$\n",
    "\n",
    "* X - матрица входных данных размера [batch_size, num_features],\n",
    "* W - матрица преобразования размера [num_features, num_outputs] \n",
    "* b - столбец свободных членов размера [num_outputs]\n",
    "\n",
    "W и b необходимо инициализировать во время создания слоя и обновлять при каждом вызове метода .backward().\n",
    "Для этого слоя вам необходимо реализовать как прямой, так и обратный проход. Формулы для аналитического подсчета градиентов можно взять из лекции, но рекомендуется проверить их самостоятельно на бумаге."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class Dense(Layer):\n",
    "    def __init__(self, input_units, output_units, learning_rate=0.1):\n",
    "        \"\"\"\n",
    "        A dense layer is a layer which performs a learned affine transformation:\n",
    "        f(x) = <W*x> + b\n",
    "        \"\"\"\n",
    "        self.learning_rate = learning_rate\n",
    "        \n",
    "        # initialize weights with small random numbers from normal distribution\n",
    "        self.weights = np.random.randn(input_units, output_units)*0.01\n",
    "        self.biases = np.zeros(output_units)\n",
    "        \n",
    "    def forward(self,input):\n",
    "        \"\"\"\n",
    "        Perform an affine transformation:\n",
    "        f(x) = <W*x> + b\n",
    "        \n",
    "        input shape: [batch, input_units]\n",
    "        output shape: [batch, output units]\n",
    "        \"\"\"\n",
    "        #<your code here>\n",
    "    \n",
    "    def backward(self,input,grad_output):\n",
    "        \n",
    "        # compute d f / d x = d f / d dense * d dense / d x\n",
    "        # where d dense/ d x = weights transposed\n",
    "        #grad_input = <your code here>\n",
    "        \n",
    "        # compute gradient w.r.t. weights and biases\n",
    "        #grad_weights = <your code here>\n",
    "        #grad_biases = <your code here>\n",
    "        \n",
    "        assert grad_weights.shape == self.weights.shape and grad_biases.shape == self.biases.shape\n",
    "        # Here we perform a stochastic gradient descent step. \n",
    "        self.weights = self.weights - self.learning_rate * grad_weights\n",
    "        self.biases = self.biases - self.learning_rate * grad_biases\n",
    "        \n",
    "        return grad_input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Тестирование полносвязного слоя\n",
    "\n",
    "Три следующих блока тестируют вашу реализацию, если все корректно, вы трижды получите \"Well done!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "l = Dense(128, 150)\n",
    "\n",
    "assert -0.05 < l.weights.mean() < 0.05 and 1e-3 < l.weights.std() < 1e-1,\\\n",
    "    \"The initial weights must have zero mean and small variance. \"\\\n",
    "    \"If you know what you're doing, remove this assertion.\"\n",
    "assert -0.05 < l.biases.mean() < 0.05, \"Biases must be zero mean. Ignore if you have a reason to do otherwise.\"\n",
    "\n",
    "# To test the outputs, we explicitly set weights with fixed values. DO NOT DO THAT IN ACTUAL NETWORK!\n",
    "l = Dense(3,4)\n",
    "\n",
    "x = np.linspace(-1,1,2*3).reshape([2,3])\n",
    "l.weights = np.linspace(-1,1,3*4).reshape([3,4])\n",
    "l.biases = np.linspace(-1,1,4)\n",
    "\n",
    "assert np.allclose(l.forward(x),np.array([[ 0.07272727,  0.41212121,  0.75151515,  1.09090909],\n",
    "                                          [-0.90909091,  0.08484848,  1.07878788,  2.07272727]]))\n",
    "print(\"Well done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# To test the grads, we use gradients obtained via finite differences\n",
    "\n",
    "from util import eval_numerical_gradient\n",
    "\n",
    "x = np.linspace(-1,1,10*32).reshape([10,32])\n",
    "l = Dense(32,64,learning_rate=0)\n",
    "\n",
    "numeric_grads = eval_numerical_gradient(lambda x: l.forward(x).sum(),x)\n",
    "grads = l.backward(x,np.ones([10,64]))\n",
    "\n",
    "assert np.allclose(grads,numeric_grads,rtol=1e-3,atol=0), \"input gradient does not match numeric grad\"\n",
    "print(\"Well done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "#test gradients w.r.t. params\n",
    "def compute_out_given_wb(w,b):\n",
    "    l = Dense(32,64,learning_rate=1)\n",
    "    l.weights = np.array(w)\n",
    "    l.biases = np.array(b)\n",
    "    x = np.linspace(-1,1,10*32).reshape([10,32])\n",
    "    return l.forward(x)\n",
    "    \n",
    "def compute_grad_by_params(w,b):\n",
    "    l = Dense(32,64,learning_rate=1)\n",
    "    l.weights = np.array(w)\n",
    "    l.biases = np.array(b)\n",
    "    x = np.linspace(-1,1,10*32).reshape([10,32])\n",
    "    l.backward(x,np.ones([10,64]) / 10.)\n",
    "    return w - l.weights, b - l.biases\n",
    "    \n",
    "w,b = np.random.randn(32,64), np.linspace(-1,1,64)\n",
    "\n",
    "numeric_dw = eval_numerical_gradient(lambda w: compute_out_given_wb(w,b).mean(0).sum(),w )\n",
    "numeric_db = eval_numerical_gradient(lambda b: compute_out_given_wb(w,b).mean(0).sum(),b )\n",
    "grad_w,grad_b = compute_grad_by_params(w,b)\n",
    "\n",
    "assert np.allclose(numeric_dw,grad_w,rtol=1e-3,atol=0), \"weight gradient does not match numeric weight gradient\"\n",
    "assert np.allclose(numeric_db,grad_b,rtol=1e-3,atol=0), \"weight gradient does not match numeric weight gradient\"\n",
    "print(\"Well done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Функция потерь\n",
    "\n",
    "Вычисление функции потерь для задачи классификации, а также ее градиента уже сделано для вас."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def softmax_crossentropy_with_logits(logits,reference_answers):\n",
    "    \"\"\"Compute crossentropy from logits[batch,n_classes] and ids of correct answers\"\"\"\n",
    "    logits_for_answers = logits[np.arange(len(logits)),reference_answers]\n",
    "    \n",
    "    xentropy = - logits_for_answers + np.log(np.sum(np.exp(logits),axis=-1))\n",
    "    \n",
    "    return xentropy\n",
    "\n",
    "def grad_softmax_crossentropy_with_logits(logits,reference_answers):\n",
    "    \"\"\"Compute crossentropy gradient from logits[batch,n_classes] and ids of correct answers\"\"\"\n",
    "    ones_for_answers = np.zeros_like(logits)\n",
    "    ones_for_answers[np.arange(len(logits)),reference_answers] = 1\n",
    "    \n",
    "    softmax = np.exp(logits) / np.exp(logits).sum(axis=-1,keepdims=True)\n",
    "    \n",
    "    return (- ones_for_answers + softmax) / logits.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "logits = np.linspace(-1,1,500).reshape([50,10])\n",
    "answers = np.arange(50)%10\n",
    "\n",
    "softmax_crossentropy_with_logits(logits,answers)\n",
    "grads = grad_softmax_crossentropy_with_logits(logits,answers)\n",
    "numeric_grads = eval_numerical_gradient(lambda l: softmax_crossentropy_with_logits(l,answers).mean(),logits)\n",
    "\n",
    "assert np.allclose(numeric_grads,grads,rtol=1e-3,atol=0), \"The reference implementation has just failed. Someone has just changed the rules of math.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Итоговая нейросеть\n",
    "\n",
    "Все готово для запуска нейросети. Нейросеть будем тестировать на классическом датасете MNIST. Код ниже визуализирует несколько примеров из этого датасета."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "from util import load_mnist\n",
    "X_train, y_train, X_val, y_val, X_test, y_test = load_mnist(flatten=True)\n",
    "\n",
    "plt.figure(figsize=[6,6])\n",
    "for i in range(4):\n",
    "    plt.subplot(2,2,i+1)\n",
    "    plt.title(\"Label: %i\"%y_train[i])\n",
    "    plt.imshow(X_train[i].reshape([28,28]),cmap='gray');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "В нашей реализации сеть - просто список (Python-list) слоев. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "network = []\n",
    "hidden_layers_size = 40\n",
    "network.append(Dense(X_train.shape[1], hidden_layers_size))\n",
    "network.append(ReLU())\n",
    "network.append(Dense(hidden_layers_size, hidden_layers_size))\n",
    "network.append(ReLU())\n",
    "network.append(Dense(hidden_layers_size, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Реализуйте прямой проход по целой сети, последовательно вызывая .forward() для каждого слоя."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def forward(network, X):\n",
    "    \"\"\"\n",
    "    Compute activations of all network layers by applying them sequentially.\n",
    "    Return a list of activations for each layer. \n",
    "    Make sure last activation corresponds to network logits.\n",
    "    \"\"\"\n",
    "    activations = []\n",
    "    input = X\n",
    "    \n",
    "    # <your code here>\n",
    "    \n",
    "    assert len(activations) == len(network)\n",
    "    return activations\n",
    "\n",
    "def predict(network, X):\n",
    "    \"\"\"\n",
    "    Use network to predict the most likely class for each sample.\n",
    "    \"\"\"\n",
    "    logits = forward(network, X)[-1]\n",
    "    return logits.argmax(axis=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def train(network,X,y):\n",
    "    \"\"\"\n",
    "    Train your network on a given batch of X and y.\n",
    "    You first need to run forward to get all layer activations.\n",
    "    Then you can run layer.backward going from last to first layer.\n",
    "    \n",
    "    After you called backward for all layers, all Dense layers have already made one gradient step.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Get the layer activations\n",
    "    layer_activations = forward(network,X)\n",
    "    layer_inputs = [X] + layer_activations  #layer_input[i] is an input for network[i]\n",
    "    logits = layer_activations[-1]\n",
    "    \n",
    "    # Compute the loss and the initial gradient\n",
    "    loss = softmax_crossentropy_with_logits(logits,y)\n",
    "    loss_grad = grad_softmax_crossentropy_with_logits(logits,y)\n",
    "        \n",
    "    # propagate gradients through network layers using .backward\n",
    "    # hint: start from last layer and move to earlier layers\n",
    "    #<YOUR CODE>\n",
    "        \n",
    "    return np.mean(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Все готово для запуска обучения. Если все реализовано корректно, то точность классификации на валидационном множестве должна превысить 97%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from tqdm.auto import tqdm\n",
    "def iterate_minibatches(inputs, targets, batchsize, shuffle=False):\n",
    "    assert len(inputs) == len(targets)\n",
    "    if shuffle:\n",
    "        indices = np.random.permutation(len(inputs))\n",
    "    for start_idx in tqdm(range(0, len(inputs) - batchsize + 1, batchsize)):\n",
    "        if shuffle:\n",
    "            excerpt = indices[start_idx:start_idx + batchsize]\n",
    "        else:\n",
    "            excerpt = slice(start_idx, start_idx + batchsize)\n",
    "        yield inputs[excerpt], targets[excerpt]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from IPython.display import clear_output\n",
    "train_log = []\n",
    "val_log = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "for epoch in range(15):\n",
    "\n",
    "    for x_batch,y_batch in iterate_minibatches(X_train, y_train, batchsize=32, shuffle=True):\n",
    "        train(network, x_batch, y_batch)\n",
    "    \n",
    "    train_log.append(np.mean(predict(network, X_train) == y_train))\n",
    "    val_log.append(np.mean(predict(network, X_val) == y_val))\n",
    "    \n",
    "    clear_output()\n",
    "    print(\"Epoch\",epoch)\n",
    "    print(\"Train accuracy:\",train_log[-1])\n",
    "    print(\"Val accuracy:\",val_log[-1])\n",
    "    plt.plot(train_log,label='train accuracy')\n",
    "    plt.plot(val_log,label='val accuracy')\n",
    "    plt.legend(loc='best')\n",
    "    plt.grid()\n",
    "    plt.show()\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Нейросеть для извлечения представлений\n",
    "Помимо собственно классификации, нейросети часто используют для получения векторных описаний (embeddings) объектов различной природы. Часто в качестве представлений можно взять просто векторы активаций нейронов с одного из последних слоев. Вычислим такие представления:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "train_activations = forward(network, X_train)\n",
    "train_hidden_representations = train_activations[2]\n",
    "test_activations = forward(network, X_test)\n",
    "test_hidden_representations = test_activations[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Полученные представления можно использовать в любых целях, в комбинации с другими инструментами машинного обучения.\n",
    "Сравним обученные представления для изображений из MNIST c исходными 784-мерными представлениями. Постройте t-SNE визуализации 1000 случайных изображений из X_train, полученные на основе исходных 784-мерных представлений и обученных представлений. Сделайте вывод о качестве представлений."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from util import plot_embedding\n",
    "from sklearn.manifold import TSNE\n",
    "subset_ids = np.random.permutation(X_train.shape[0])[:1000]\n",
    "#x_original_tsne = <your code>\n",
    "#x_learnt_tsne = <your code>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "plot_embedding(x_original_tsne, y_train[subset_ids])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "plot_embedding(x_learnt_tsne, y_train[subset_ids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Наконец, исследуем универсальность полученных представлений. Используем их для решения другой задачи классификации: предсказание четности изображенной цифры. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "new_y_train = y_train % 2\n",
    "new_y_test = y_test % 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Обучите простейшую логистическую регресиию, предсказывающую четность изображенной цифры, используя в качестве вектора признаков:\n",
    "\n",
    "1) исходные 784-мерные представления\n",
    "\n",
    "2) обученные представления\n",
    "\n",
    "Сравните качество и сделайте вывод об универсальности обученных представлений."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "lr = LogisticRegression(solver=\"lbfgs\")\n",
    "#<your code>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hse_dl_year": "2021-fall"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}