train_and_test_set_creation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training and test sets creation\n",
    "\n",
    "This notebook is used to create our training and test set from the HAM10000 dataset, and ensures that there are no duplicate images of lesions. These are saved as two pickles of dataframes which will be used in other parts of our code. \n",
    "\n",
    "The HAM10000 dataset can be found here https://www.kaggle.com/kmader/skin-cancer-mnist-ham10000\n",
    "\n",
    "## Reference\n",
    "Tschandl, P., Rosendahl, C., Kittler, H.: The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data 5, 180161 (2018)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EjBGSEQbOI9s"
   },
   "source": [
    "### Import statements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CY1evZeOOI9w",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os\n",
    "from glob import glob\n",
    "from PIL import Image\n",
    "np.random.seed(123)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Nh59Cy6fOI98"
   },
   "source": [
    "### Get all paths\n",
    "\n",
    "You should have the unzipped dataset, the images are saved in the two subfolders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 164
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1188,
     "status": "error",
     "timestamp": 1548055008486,
     "user": {
      "displayName": "Kyle Young",
      "photoUrl": "https://lh6.googleusercontent.com/-GytXuqoBZcY/AAAAAAAAAAI/AAAAAAAAACs/u0zFywWbmkI/s64/photo.jpg",
      "userId": "16479215267355514043"
     },
     "user_tz": -600
    },
    "id": "DGLA0FE6OI9_",
    "outputId": "8d044a82-aa60-4492-b6e5-f498e5dfbfa5"
   },
   "outputs": [],
   "source": [
    "path_list = glob('HAM10000_images_part_1/*.jpg') + \\\n",
    "        glob('HAM10000_images_part_2/*.jpg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10015"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(path_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lMPwjFe7OI-H"
   },
   "outputs": [],
   "source": [
    "imageid_path_dict = {}\n",
    "for x in path_list:\n",
    "    imageid_path_dict[os.path.splitext(os.path.basename(x))[0]] = x;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YDOwdtLMQ_pj"
   },
   "outputs": [],
   "source": [
    "lesion_type_dict = {\n",
    "    'nv': 'Melanocytic nevi',\n",
    "    'mel': 'Melanoma',\n",
    "    'bkl': 'Benign keratosis-like lesions ',\n",
    "    'bcc': 'Basal cell carcinoma',\n",
    "    'akiec': 'Actinic keratoses',\n",
    "    'vasc': 'Vascular lesions',\n",
    "    'df': 'Dermatofibroma'\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xSOOiGBYOI-Q"
   },
   "source": [
    "Load metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gTFwQXvtOI-S"
   },
   "outputs": [],
   "source": [
    "skin_df = pd.read_csv('HAM10000_metadata.csv')\n",
    "skin_df['path'] = skin_df['image_id'].map(imageid_path_dict.get)\n",
    "skin_df['cell_type'] = skin_df['dx'].map(lesion_type_dict.get) \n",
    "skin_df['cell_type_idx'] = pd.Categorical(skin_df['cell_type']).codes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 414
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 109489,
     "status": "ok",
     "timestamp": 1543979772000,
     "user": {
      "displayName": "Kyle Young",
      "photoUrl": "https://lh6.googleusercontent.com/-GytXuqoBZcY/AAAAAAAAAAI/AAAAAAAAACs/u0zFywWbmkI/s64/photo.jpg",
      "userId": "16479215267355514043"
     },
     "user_tz": -600
    },
    "id": "FANELBq0OI-W",
    "outputId": "5740a995-e576-450b-ad68-c47421d0accc",
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lesion_id</th>\n",
       "      <th>image_id</th>\n",
       "      <th>dx</th>\n",
       "      <th>dx_type</th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>localization</th>\n",
       "      <th>path</th>\n",
       "      <th>cell_type</th>\n",
       "      <th>cell_type_idx</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>HAM_0000118</td>\n",
       "      <td>ISIC_0027419</td>\n",
       "      <td>bkl</td>\n",
       "      <td>histo</td>\n",
       "      <td>80.0</td>\n",
       "      <td>male</td>\n",
       "      <td>scalp</td>\n",
       "      <td>./../../Resources/HAM10000\\HAM10000_images_par...</td>\n",
       "      <td>Benign keratosis-like lesions</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>HAM_0000118</td>\n",
       "      <td>ISIC_0025030</td>\n",
       "      <td>bkl</td>\n",
       "      <td>histo</td>\n",
       "      <td>80.0</td>\n",
       "      <td>male</td>\n",
       "      <td>scalp</td>\n",
       "      <td>./../../Resources/HAM10000\\HAM10000_images_par...</td>\n",
       "      <td>Benign keratosis-like lesions</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>HAM_0002730</td>\n",
       "      <td>ISIC_0026769</td>\n",
       "      <td>bkl</td>\n",
       "      <td>histo</td>\n",
       "      <td>80.0</td>\n",
       "      <td>male</td>\n",
       "      <td>scalp</td>\n",
       "      <td>./../../Resources/HAM10000\\HAM10000_images_par...</td>\n",
       "      <td>Benign keratosis-like lesions</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>HAM_0002730</td>\n",
       "      <td>ISIC_0025661</td>\n",
       "      <td>bkl</td>\n",
       "      <td>histo</td>\n",
       "      <td>80.0</td>\n",
       "      <td>male</td>\n",
       "      <td>scalp</td>\n",
       "      <td>./../../Resources/HAM10000\\HAM10000_images_par...</td>\n",
       "      <td>Benign keratosis-like lesions</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>HAM_0001466</td>\n",
       "      <td>ISIC_0031633</td>\n",
       "      <td>bkl</td>\n",
       "      <td>histo</td>\n",
       "      <td>75.0</td>\n",
       "      <td>male</td>\n",
       "      <td>ear</td>\n",
       "      <td>./../../Resources/HAM10000\\HAM10000_images_par...</td>\n",
       "      <td>Benign keratosis-like lesions</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     lesion_id      image_id   dx dx_type   age   sex localization  \\\n",
       "0  HAM_0000118  ISIC_0027419  bkl   histo  80.0  male        scalp   \n",
       "1  HAM_0000118  ISIC_0025030  bkl   histo  80.0  male        scalp   \n",
       "2  HAM_0002730  ISIC_0026769  bkl   histo  80.0  male        scalp   \n",
       "3  HAM_0002730  ISIC_0025661  bkl   histo  80.0  male        scalp   \n",
       "4  HAM_0001466  ISIC_0031633  bkl   histo  75.0  male          ear   \n",
       "\n",
       "                                                path  \\\n",
       "0  ./../../Resources/HAM10000\\HAM10000_images_par...   \n",
       "1  ./../../Resources/HAM10000\\HAM10000_images_par...   \n",
       "2  ./../../Resources/HAM10000\\HAM10000_images_par...   \n",
       "3  ./../../Resources/HAM10000\\HAM10000_images_par...   \n",
       "4  ./../../Resources/HAM10000\\HAM10000_images_par...   \n",
       "\n",
       "                        cell_type  cell_type_idx  \n",
       "0  Benign keratosis-like lesions               2  \n",
       "1  Benign keratosis-like lesions               2  \n",
       "2  Benign keratosis-like lesions               2  \n",
       "3  Benign keratosis-like lesions               2  \n",
       "4  Benign keratosis-like lesions               2  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skin_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7470,)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skin_df['lesion_id'].unique().shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the lesion id metadata to find duplicates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "skin_df = skin_df.drop_duplicates(subset='lesion_id')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZrRg7gFIOI-Z"
   },
   "source": [
    "Get rid of null values for age and set them to the mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 109485,
     "status": "ok",
     "timestamp": 1543979772001,
     "user": {
      "displayName": "Kyle Young",
      "photoUrl": "https://lh6.googleusercontent.com/-GytXuqoBZcY/AAAAAAAAAAI/AAAAAAAAACs/u0zFywWbmkI/s64/photo.jpg",
      "userId": "16479215267355514043"
     },
     "user_tz": -600
    },
    "id": "wy-Aj9dnOI-b",
    "outputId": "4dd31c38-5317-4106-c158-8316c975c979",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "lesion_id         0\n",
       "image_id          0\n",
       "dx                0\n",
       "dx_type           0\n",
       "age              52\n",
       "sex               0\n",
       "localization      0\n",
       "path              0\n",
       "cell_type         0\n",
       "cell_type_idx     0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skin_df.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-SKmPzf_OI-f"
   },
   "outputs": [],
   "source": [
    "skin_df['age'].fillna((skin_df['age'].mean()), inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vBTaDKHP0hJZ"
   },
   "source": [
    "Checks that all our null values have been filled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 109475,
     "status": "ok",
     "timestamp": 1543979772002,
     "user": {
      "displayName": "Kyle Young",
      "photoUrl": "https://lh6.googleusercontent.com/-GytXuqoBZcY/AAAAAAAAAAI/AAAAAAAAACs/u0zFywWbmkI/s64/photo.jpg",
      "userId": "16479215267355514043"
     },
     "user_tz": -600
    },
    "id": "lN45CPMIOI-h",
    "outputId": "8f64df46-6d46-4c5e-cf63-e76420dd339d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "lesion_id        0\n",
       "image_id         0\n",
       "dx               0\n",
       "dx_type          0\n",
       "age              0\n",
       "sex              0\n",
       "localization     0\n",
       "path             0\n",
       "cell_type        0\n",
       "cell_type_idx    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skin_df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uTiSMq0BOI-2"
   },
   "source": [
    "Open all images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "A8iV0vyqOI-2",
    "outputId": "b961d047-0b7a-42db-d2e1-5629ced1c5f4"
   },
   "outputs": [],
   "source": [
    "skin_df['image'] = skin_df['path'].map(lambda x: np.asarray(Image.open(x)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Zawm5BV0OI-7",
    "outputId": "22ad7d3a-0450-47b8-a045-41650c0ec298",
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(450, 600, 3)    7470\n",
       "Name: image, dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Checking the image size distribution\n",
    "skin_df['image'].map(lambda x: x.shape).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lesion_id</th>\n",
       "      <th>image_id</th>\n",
       "      <th>dx</th>\n",
       "      <th>dx_type</th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>localization</th>\n",
       "      <th>path</th>\n",
       "      <th>cell_type</th>\n",
       "      <th>cell_type_idx</th>\n",
       "      <th>image</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>HAM_0000118</td>\n",
       "      <td>ISIC_0027419</td>\n",
       "      <td>bkl</td>\n",
       "      <td>histo</td>\n",
       "      <td>80.0</td>\n",
       "      <td>male</td>\n",
       "      <td>scalp</td>\n",
       "      <td>./../../Resources/HAM10000\\HAM10000_images_par...</td>\n",
       "      <td>Benign keratosis-like lesions</td>\n",
       "      <td>2</td>\n",
       "      <td>[[[188, 147, 191], [186, 148, 189], [187, 150,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>HAM_0002730</td>\n",
       "      <td>ISIC_0026769</td>\n",
       "      <td>bkl</td>\n",
       "      <td>histo</td>\n",
       "      <td>80.0</td>\n",
       "      <td>male</td>\n",
       "      <td>scalp</td>\n",
       "      <td>./../../Resources/HAM10000\\HAM10000_images_par...</td>\n",
       "      <td>Benign keratosis-like lesions</td>\n",
       "      <td>2</td>\n",
       "      <td>[[[186, 128, 140], [188, 128, 136], [183, 126,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>HAM_0001466</td>\n",
       "      <td>ISIC_0031633</td>\n",
       "      <td>bkl</td>\n",
       "      <td>histo</td>\n",
       "      <td>75.0</td>\n",
       "      <td>male</td>\n",
       "      <td>ear</td>\n",
       "      <td>./../../Resources/HAM10000\\HAM10000_images_par...</td>\n",
       "      <td>Benign keratosis-like lesions</td>\n",
       "      <td>2</td>\n",
       "      <td>[[[122, 80, 102], [124, 82, 104], [127, 83, 10...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>HAM_0002761</td>\n",
       "      <td>ISIC_0029176</td>\n",
       "      <td>bkl</td>\n",
       "      <td>histo</td>\n",
       "      <td>60.0</td>\n",
       "      <td>male</td>\n",
       "      <td>face</td>\n",
       "      <td>./../../Resources/HAM10000\\HAM10000_images_par...</td>\n",
       "      <td>Benign keratosis-like lesions</td>\n",
       "      <td>2</td>\n",
       "      <td>[[[188, 142, 118], [189, 141, 119], [189, 142,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>HAM_0005132</td>\n",
       "      <td>ISIC_0025837</td>\n",
       "      <td>bkl</td>\n",
       "      <td>histo</td>\n",
       "      <td>70.0</td>\n",
       "      <td>female</td>\n",
       "      <td>back</td>\n",
       "      <td>./../../Resources/HAM10000\\HAM10000_images_par...</td>\n",
       "      <td>Benign keratosis-like lesions</td>\n",
       "      <td>2</td>\n",
       "      <td>[[[94, 58, 58], [91, 61, 59], [91, 61, 59], [9...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     lesion_id      image_id   dx dx_type   age     sex localization  \\\n",
       "0  HAM_0000118  ISIC_0027419  bkl   histo  80.0    male        scalp   \n",
       "2  HAM_0002730  ISIC_0026769  bkl   histo  80.0    male        scalp   \n",
       "4  HAM_0001466  ISIC_0031633  bkl   histo  75.0    male          ear   \n",
       "6  HAM_0002761  ISIC_0029176  bkl   histo  60.0    male         face   \n",
       "8  HAM_0005132  ISIC_0025837  bkl   histo  70.0  female         back   \n",
       "\n",
       "                                                path  \\\n",
       "0  ./../../Resources/HAM10000\\HAM10000_images_par...   \n",
       "2  ./../../Resources/HAM10000\\HAM10000_images_par...   \n",
       "4  ./../../Resources/HAM10000\\HAM10000_images_par...   \n",
       "6  ./../../Resources/HAM10000\\HAM10000_images_par...   \n",
       "8  ./../../Resources/HAM10000\\HAM10000_images_par...   \n",
       "\n",
       "                        cell_type  cell_type_idx  \\\n",
       "0  Benign keratosis-like lesions               2   \n",
       "2  Benign keratosis-like lesions               2   \n",
       "4  Benign keratosis-like lesions               2   \n",
       "6  Benign keratosis-like lesions               2   \n",
       "8  Benign keratosis-like lesions               2   \n",
       "\n",
       "                                               image  \n",
       "0  [[[188, 147, 191], [186, 148, 189], [187, 150,...  \n",
       "2  [[[186, 128, 140], [188, 128, 136], [183, 126,...  \n",
       "4  [[[122, 80, 102], [124, 82, 104], [127, 83, 10...  \n",
       "6  [[[188, 142, 118], [189, 141, 119], [189, 142,...  \n",
       "8  [[[94, 58, 58], [91, 61, 59], [91, 61, 59], [9...  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skin_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7470, 11)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skin_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "skin_df = skin_df.drop('lesion_id', axis=1).drop('image_id', axis=1).drop('dx_type', axis=1).\\\n",
    "        drop('age', axis=1).drop('sex', axis=1).drop('localization', axis=1).drop('path', axis=1).\\\n",
    "        drop('cell_type', axis=1).drop('cell_type_idx', axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Drop all images that are not nv or mel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "skin_df = skin_df[(skin_df.dx == 'mel') | (skin_df.dx == 'nv')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "skin_df['id'] = skin_df.pop('dx')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "skin_df = skin_df.reset_index()\n",
    "skin_df = skin_df.drop('index', axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image</th>\n",
       "      <th>id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[[[163, 137, 166], [163, 137, 162], [164, 138,...</td>\n",
       "      <td>nv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[[[229, 146, 164], [229, 146, 162], [233, 149,...</td>\n",
       "      <td>nv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[[[120, 99, 104], [125, 102, 108], [128, 103, ...</td>\n",
       "      <td>mel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[[[128, 106, 129], [129, 107, 128], [130, 106,...</td>\n",
       "      <td>mel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[[[24, 12, 22], [21, 9, 19], [18, 6, 16], [19,...</td>\n",
       "      <td>mel</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               image   id\n",
       "0  [[[163, 137, 166], [163, 137, 162], [164, 138,...   nv\n",
       "1  [[[229, 146, 164], [229, 146, 162], [233, 149,...   nv\n",
       "2  [[[120, 99, 104], [125, 102, 108], [128, 103, ...  mel\n",
       "3  [[[128, 106, 129], [129, 107, 128], [130, 106,...  mel\n",
       "4  [[[24, 12, 22], [21, 9, 19], [18, 6, 16], [19,...  mel"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skin_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save a seperate test set \n",
    "\n",
    "Our test set contains 200 mel and 200 nv."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6017, 2)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skin_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(200, 2)\n"
     ]
    }
   ],
   "source": [
    "mel = skin_df[skin_df.id == 'mel'].sample(n=200,random_state=200)\n",
    "mel.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "skin_df = skin_df.drop(skin_df.index[mel.index])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5817, 2)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skin_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "skin_df = skin_df.reset_index()\n",
    "skin_df = skin_df.drop('index', axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(200, 2)\n"
     ]
    }
   ],
   "source": [
    "nv = skin_df[skin_df.id == 'nv'].sample(n=200,random_state=200)\n",
    "nv.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Drop the test images from the training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "skin_df = skin_df.drop(skin_df.index[nv.index])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5617, 2)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skin_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "skin_df = skin_df.reset_index()\n",
    "skin_df = skin_df.drop('index', axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image</th>\n",
       "      <th>id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[[[163, 137, 166], [163, 137, 162], [164, 138,...</td>\n",
       "      <td>nv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[[[229, 146, 164], [229, 146, 162], [233, 149,...</td>\n",
       "      <td>nv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[[[120, 99, 104], [125, 102, 108], [128, 103, ...</td>\n",
       "      <td>mel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[[[24, 12, 22], [21, 9, 19], [18, 6, 16], [19,...</td>\n",
       "      <td>mel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[[[180, 146, 144], [182, 147, 145], [182, 147,...</td>\n",
       "      <td>mel</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               image   id\n",
       "0  [[[163, 137, 166], [163, 137, 162], [164, 138,...   nv\n",
       "1  [[[229, 146, 164], [229, 146, 162], [233, 149,...   nv\n",
       "2  [[[120, 99, 104], [125, 102, 108], [128, 103, ...  mel\n",
       "3  [[[24, 12, 22], [21, 9, 19], [18, 6, 16], [19,...  mel\n",
       "4  [[[180, 146, 144], [182, 147, 145], [182, 147,...  mel"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skin_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df = pd.concat((mel, nv))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(400, 2)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image</th>\n",
       "      <th>id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>379</th>\n",
       "      <td>[[[176, 157, 185], [176, 157, 185], [178, 159,...</td>\n",
       "      <td>mel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105</th>\n",
       "      <td>[[[157, 133, 159], [157, 133, 155], [156, 131,...</td>\n",
       "      <td>mel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>163</th>\n",
       "      <td>[[[48, 28, 37], [51, 30, 39], [52, 32, 41], [6...</td>\n",
       "      <td>mel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>455</th>\n",
       "      <td>[[[156, 111, 106], [153, 112, 106], [157, 117,...</td>\n",
       "      <td>mel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>386</th>\n",
       "      <td>[[[255, 194, 189], [255, 194, 191], [255, 191,...</td>\n",
       "      <td>mel</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 image   id\n",
       "379  [[[176, 157, 185], [176, 157, 185], [178, 159,...  mel\n",
       "105  [[[157, 133, 159], [157, 133, 155], [156, 131,...  mel\n",
       "163  [[[48, 28, 37], [51, 30, 39], [52, 32, 41], [6...  mel\n",
       "455  [[[156, 111, 106], [153, 112, 106], [157, 117,...  mel\n",
       "386  [[[255, 194, 189], [255, 194, 191], [255, 191,...  mel"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df = test_df.reset_index()\n",
    "test_df = test_df.drop('index', axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image</th>\n",
       "      <th>id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[[[163, 137, 166], [163, 137, 162], [164, 138,...</td>\n",
       "      <td>nv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[[[229, 146, 164], [229, 146, 162], [233, 149,...</td>\n",
       "      <td>nv</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[[[120, 99, 104], [125, 102, 108], [128, 103, ...</td>\n",
       "      <td>mel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[[[24, 12, 22], [21, 9, 19], [18, 6, 16], [19,...</td>\n",
       "      <td>mel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[[[180, 146, 144], [182, 147, 145], [182, 147,...</td>\n",
       "      <td>mel</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               image   id\n",
       "0  [[[163, 137, 166], [163, 137, 162], [164, 138,...   nv\n",
       "1  [[[229, 146, 164], [229, 146, 162], [233, 149,...   nv\n",
       "2  [[[120, 99, 104], [125, 102, 108], [128, 103, ...  mel\n",
       "3  [[[24, 12, 22], [21, 9, 19], [18, 6, 16], [19,...  mel\n",
       "4  [[[180, 146, 144], [182, 147, 145], [182, 147,...  mel"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skin_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5617, 2)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skin_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image</th>\n",
       "      <th>id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[[[176, 157, 185], [176, 157, 185], [178, 159,...</td>\n",
       "      <td>mel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[[[157, 133, 159], [157, 133, 155], [156, 131,...</td>\n",
       "      <td>mel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[[[48, 28, 37], [51, 30, 39], [52, 32, 41], [6...</td>\n",
       "      <td>mel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[[[156, 111, 106], [153, 112, 106], [157, 117,...</td>\n",
       "      <td>mel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[[[255, 194, 189], [255, 194, 191], [255, 191,...</td>\n",
       "      <td>mel</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               image   id\n",
       "0  [[[176, 157, 185], [176, 157, 185], [178, 159,...  mel\n",
       "1  [[[157, 133, 159], [157, 133, 155], [156, 131,...  mel\n",
       "2  [[[48, 28, 37], [51, 30, 39], [52, 32, 41], [6...  mel\n",
       "3  [[[156, 111, 106], [153, 112, 106], [157, 117,...  mel\n",
       "4  [[[255, 194, 189], [255, 194, 191], [255, 191,...  mel"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(400, 2)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hhQ6NxlQ0hJk"
   },
   "source": [
    "Save the dataframe as a pickle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "skin_df.to_pickle('NvAndMelNoDuplicatesFullSize.zip')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df.to_pickle('NvAndMelNoDuplicatesFullSizeTestSet.zip')"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "DataLoading.ipynb",
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}