lab_day2/day2_code04_feature_selection_[PILL].ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Can Neural Network be used for feature selection?\n",
        "\n",
        "We'll use the breast cancer dataset, already solved in another notebook."
      ],
      "metadata": {
        "id": "m-bk4DL_WVRx"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Setup(s)"
      ],
      "metadata": {
        "id": "bKE0Xq6M79co"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Libraries setup"
      ],
      "metadata": {
        "id": "J18SWcB0WnLg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#very common libraries, that we are for sure using\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import tensorflow as tf"
      ],
      "metadata": {
        "id": "qrRIdp56WpET"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Seed setup"
      ],
      "metadata": {
        "id": "Rc1kzLhAhC73"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from numpy.random import seed\n",
        "myseed = 0\n",
        "seed(myseed)\n",
        "tf.random.set_seed(myseed)"
      ],
      "metadata": {
        "id": "krBiyWMihEnp"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Data setup"
      ],
      "metadata": {
        "id": "YxGmXCFNWhu9"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lQJoQjGZWHY0"
      },
      "outputs": [],
      "source": [
        "#libraries for this block\n",
        "import sklearn.datasets\n",
        "from sklearn.model_selection import StratifiedShuffleSplit\n",
        "\n",
        "# loading data\n",
        "from sklearn.datasets import load_breast_cancer\n",
        "bcancer = load_breast_cancer()\n",
        "y = bcancer.target\n",
        "X = pd.DataFrame(bcancer.data, columns=bcancer.feature_names)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Adding a dud feature\n",
        "\n",
        "We add a fake, meaningless (\"dud\") feature called \"RANDOM_VALUES\" that contains random values between zero and one. This feature has no meaning and if the feature selection process works well it should be filtered out (or at least highlighted as not interesting)."
      ],
      "metadata": {
        "id": "vFyh_liF59ev"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from numpy.random import uniform\n",
        "X['RANDOM_VALUES'] = np.random.uniform(size=X.shape[0])"
      ],
      "metadata": {
        "id": "aaM_AolJ6Bnx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Normalizing"
      ],
      "metadata": {
        "id": "rt0ilcdt6IBM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# normalizing\n",
        "X = (X - X.mean())/X.std()"
      ],
      "metadata": {
        "id": "1zknPXiX6Jd5"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 1-Node Logistic Network"
      ],
      "metadata": {
        "id": "why07Wbm8COl"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Setup"
      ],
      "metadata": {
        "id": "1GpC08GkYSRk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#libraries for this block\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "\n",
        "# Configuration\n",
        "input_shape = (X.shape[1],) ## tuple that specifies the number of features\n",
        "output_activation = 'sigmoid'\n",
        "loss_function = 'binary_crossentropy'\n",
        "extra_metrics = ['binary_accuracy']\n",
        "optimizer_used = 'sgd' ##stochastic gradient descent\n",
        "num_epochs = 100\n",
        "\n",
        "# 1-node logistic neural network\n",
        "model_1LN = Sequential()\n",
        "model_1LN.add(Dense(1, activation=output_activation))\n",
        "\n",
        "#compiling, training\n",
        "model_1LN.compile(optimizer=optimizer_used, loss=loss_function, metrics=extra_metrics)\n",
        "history = model_1LN.fit(X, y, epochs=num_epochs, validation_split=0.2, verbose=0)"
      ],
      "metadata": {
        "id": "UfEJHmoTW993"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Verify training"
      ],
      "metadata": {
        "id": "z68dni5maoqG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def plot_history(h, title, target='loss'):\n",
        "    plt.rcParams[\"figure.figsize\"]=5,5\n",
        "    plt.plot(h.history[target], label = \"Train \" + target)\n",
        "    plt.plot(h.history['val_'+target], label = \"Validation \" + target)\n",
        "    plt.xlabel('Epochs')\n",
        "    plt.title(title)\n",
        "    plt.legend()\n",
        "    plt.show()\n",
        "\n",
        "plot_history(history, '1-Node logistic network', target = 'loss')\n",
        "plot_history(history, '1-Node logistic network', target = 'binary_accuracy')"
      ],
      "metadata": {
        "id": "1RLBirG7aIm-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Taking a look at the weights"
      ],
      "metadata": {
        "id": "GlVsqsm-cT4Z"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#extracting weights and biases from the first (and only) layer\n",
        "weights = model_1LN.layers[0].get_weights()[0]\n",
        "biases  = model_1LN.layers[0].get_weights()[1]\n",
        "\n",
        "if True:\n",
        "  #can you guess what this would print?\n",
        "  print(weights.shape)\n",
        "  print(biases.shape)"
      ],
      "metadata": {
        "id": "5JnHK0_xaP-d"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "An image is worth a thousand words. Let's plot the weights."
      ],
      "metadata": {
        "id": "ytKj1Il-f06z"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#praparing data for the plot, we must have one column per feature, not per node, and properly named\n",
        "mydata = pd.DataFrame(np.transpose(weights), columns = X.columns)\n",
        "\n",
        "#a pretty plot, for more click here ;) https://www.physalia-courses.org/courses-workshops/course38/\n",
        "import seaborn as sb\n",
        "plt.rcParams[\"figure.figsize\"]=20,10\n",
        "ax = sb.barplot(data=mydata)\n",
        "ax.set(title='Weights for 1-Node Logistic Network, seed=' + str(myseed))\n",
        "ax.set_xticklabels(labels = mydata.columns, rotation = -90)\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "1J_B6S2Md2v0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Same thing, 10 times (for statistics)"
      ],
      "metadata": {
        "id": "ZWQZ2jH7enB0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for i in range(10):\n",
        "  print('Doing iteration ' + str(i))\n",
        "\n",
        "  # declaring a new model instance\n",
        "  model_1LN = Sequential()\n",
        "  model_1LN.add(Dense(1, activation=output_activation))\n",
        "\n",
        "  #compiling, training\n",
        "  model_1LN.compile(optimizer=optimizer_used, loss=loss_function, metrics=extra_metrics)\n",
        "  history = model_1LN.fit(X, y, epochs=num_epochs, validation_split=0.2, verbose=0)\n",
        "\n",
        "  #extracting weights from the current model\n",
        "  weights_current = model_1LN.layers[0].get_weights()[0]\n",
        "\n",
        "  #adding weights what we stored so far\n",
        "  weights = np.concatenate((weights, weights_current), axis = 1)"
      ],
      "metadata": {
        "id": "ybdWtd8F0HPN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "if True:\n",
        "  #can you guess what this would print?\n",
        "  print(weights.shape)\n",
        "  print(biases.shape)"
      ],
      "metadata": {
        "id": "ciLsdepg1bF1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Taking a look at the weights distribution"
      ],
      "metadata": {
        "id": "U0E4UBya3JEg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#praparing data for the plot, we must have one column per feature, not per node, and properly named\n",
        "mydata = pd.DataFrame(np.transpose(weights), columns = X.columns)\n",
        "\n",
        "plt.rcParams[\"figure.figsize\"]=20,10\n",
        "ax = sb.boxplot(data=mydata)\n",
        "ax.set(title='Distribution of weights for 1-Node Logistic Network, seed=' + str(myseed))\n",
        "ax.set_xticklabels(labels = mydata.columns, rotation = -90)\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "bTpqGj4511D1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Sorting by descending absolute values"
      ],
      "metadata": {
        "id": "FX1wcT1M3kGI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#absolute values\n",
        "mydata = mydata.abs()\n",
        "\n",
        "#sorting by median\n",
        "mydata = mydata.reindex(mydata.median().sort_values(ascending=False).index, axis=1)\n",
        "\n",
        "#distribution plot\n",
        "plt.rcParams[\"figure.figsize\"]=20,10\n",
        "ax = sb.boxplot(data=mydata)\n",
        "ax.set(title='Distributions of absolute weights for 1-Node Logistic Network, sorted by descending median, seed=' + str(myseed))\n",
        "ax.set_xticklabels(labels = mydata.columns, rotation = -90)\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "yMty_U_Q3WXG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Shallow (2 layers) neural network"
      ],
      "metadata": {
        "id": "66dFfZKU8Nnb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Training"
      ],
      "metadata": {
        "id": "PBCOvmhu8zyj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# newtwork configuration\n",
        "input_shape = (X.shape[1],) ## tuple that specifies the number of features\n",
        "hidden_nodes = 16\n",
        "hidden_activation = 'relu'\n",
        "output_activation = 'sigmoid'\n",
        "loss_function = 'binary_crossentropy'\n",
        "extra_metrics = ['binary_accuracy']\n",
        "optimizer_used = 'sgd' ##stochastic gradient descent\n",
        "num_epochs = 100\n",
        "\n",
        "# training\n",
        "weights_1stLayer = None\n",
        "weights_2ndLayer = None\n",
        "for i in range(10):\n",
        "  print('Doing iteration ' + str(i))\n",
        "\n",
        "  # declaring a new model instance\n",
        "  model_shallow = Sequential()\n",
        "  model_shallow.add(Dense(units=hidden_nodes, input_shape=input_shape, activation=hidden_activation))\n",
        "  model_shallow.add(Dense(1, activation=output_activation))\n",
        "\n",
        "  # compiling, training\n",
        "  model_shallow.compile(optimizer=optimizer_used, loss=loss_function, metrics=extra_metrics)\n",
        "  history = model_shallow.fit(X, y, epochs=num_epochs, validation_split=0.2, verbose=0)\n",
        "\n",
        "  # extracting weights from the current model\n",
        "  weights_1stLayer_current = model_shallow.layers[0].get_weights()[0]\n",
        "  weights_2ndLayer_current = model_shallow.layers[1].get_weights()[0]\n",
        "\n",
        "  # adding weights what we stored so far\n",
        "  if weights_1stLayer is None:\n",
        "     weights_1stLayer = weights_1stLayer_current\n",
        "     weights_2ndLayer = weights_2ndLayer_current\n",
        "  else:\n",
        "    weights_1stLayer = np.concatenate((weights_1stLayer, weights_1stLayer_current), axis = 1)\n",
        "    weights_2ndLayer = np.concatenate((weights_2ndLayer, weights_2ndLayer_current), axis = 1)"
      ],
      "metadata": {
        "id": "uckGPYYj31bd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "if False:\n",
        "  #can you guess what this would print?\n",
        "  print(weights_1stLayer.shape)\n",
        "  print(weights_2ndLayer.shape)"
      ],
      "metadata": {
        "id": "zTB0kMQLAb0q"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Plot"
      ],
      "metadata": {
        "id": "o-Lo-Mc_83TX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#absolute values\n",
        "mydata_1st = pd.DataFrame(np.transpose(weights_1stLayer), columns = X.columns).abs()\n",
        "mydata_2nd = pd.DataFrame(np.transpose(weights_2ndLayer)).abs() #no column names here, can you guess why?\n",
        "\n",
        "#sorting by median\n",
        "mydata_1st = mydata_1st.reindex(mydata_1st.median().sort_values(ascending=False).index, axis=1)\n",
        "mydata_2nd = mydata_2nd.reindex(mydata_2nd.median().sort_values(ascending=False).index, axis=1)\n",
        "\n",
        "#distribution plot, 1st layer\n",
        "plt.rcParams[\"figure.figsize\"]=20,10\n",
        "ax = sb.boxplot(data=mydata_1st)\n",
        "ax.set(title='Weights for 2-layers Shallow Network, first layer, sorted by descending median, seed=' + str(myseed))\n",
        "ax.set_xticklabels(labels = mydata_1st.columns, rotation = -90)\n",
        "plt.show()\n",
        "\n",
        "#distribution plot, 2nd layer\n",
        "ax = sb.boxplot(data=mydata_2nd)\n",
        "ax.set(title='Weights for 2-layers Shallow Network, second layer, sorted by descending median, seed=' + str(myseed))\n",
        "ax.set_xticklabels(labels = mydata_2nd.columns)\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "uMH4Dcrh8eXe"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Shedding some light in the blackbox?\n",
        "\n",
        "There are options to try and make some sense out of the weights of neural networks. This is however a rather advanced topic, still the subject of cutting-edge current scientific research.\n",
        "\n",
        "We list here a few websites where you can start this journey, in case you're interested:\n",
        "\n",
        "- [Variable importance in neural networks](https://www.r-bloggers.com/2013/08/variable-importance-in-neural-networks/) (an R blog)\n",
        "- [Feature Importance with Neural Network](https://towardsdatascience.com/feature-importance-with-neural-network-346eb6205743) (a Python blog)\n",
        "- [Nonparametric variable importanceusing an augmented neural network with multi-task learning](https://proceedings.mlr.press/v80/feng18a/feng18a.pdf) (scientific article, *International conference on machine learning*, 2018)\n",
        "- [VtNet: A neural network with variable importance assessment](https://https://onlinelibrary.wiley.com/doi/abs/10.1002/sta4.325) (scientific article, *Stat*, 2020)\n",
        "- [An accurate comparison of methods for quantifying variable importance in artificial neural networks using simulated data](https://https://www.sciencedirect.com/science/article/pii/S0304380004001565?via%3Dihub) (scientific article, *Ecological Modelling*, 2004)"
      ],
      "metadata": {
        "id": "_CzOULgME889"
      }
    }
  ]
}