first_glimpse.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cover Type Prediction: First glimpse\n",
    "\n",
    "Sébastien Meyer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "font = {\"size\": 18}\n",
    "mpl.rc(\"font\", **font)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic information about our dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read files\n",
    "test_df = pd.read_csv(\"data/covtype.csv\", index_col=[\"Id\"])\n",
    "\n",
    "training_ids = []\n",
    "\n",
    "with open(\"data/training_ids.txt\", \"r\", encoding=\"utf-8\") as f:\n",
    "\n",
    "    training_ids = f.read().split(\",\")\n",
    "    training_ids = [int(x) for x in training_ids]\n",
    "\n",
    "train_df = test_df.iloc[training_ids, :].copy()\n",
    "\n",
    "print(train_df.shape)\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We observe that the dataset is made of 15120 training samples, where there are 54 variables and 1 target (the cover type)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(test_df.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 581012 test samples. Clearly, the test set is much larger than the training set. Therefore, we must be paying attention to overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is our task ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to know if we are facing a regression or a classification problem, and to know how many classes there are if we are facing a classification situation. Also, we need to know if the training set is imbalanced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imbalanced or balanced ?\n",
    "print(train_df[\"Cover_Type\"].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The task is thus a classification task, with 7 possible labels. The training dataset is imbalanced, we might need to weight a class differently from another."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Repartition of cover types\n",
    "plt.figure(figsize=(10, 8))\n",
    "\n",
    "color = iter(plt.cm.tab10(np.linspace(0, 1, 7)))\n",
    "\n",
    "plt.hist(train_df.loc[train_df[\"Cover_Type\"] == 1, \"Cover_Type\"], label=\"Cover_Type = 1\", bins=np.arange(1, 9)-0.5, alpha=0.75, color=next(color))\n",
    "plt.hist(train_df.loc[train_df[\"Cover_Type\"] == 2, \"Cover_Type\"], label=\"Cover_Type = 2\", bins=np.arange(1, 9)-0.5, alpha=0.75, color=next(color))\n",
    "plt.hist(train_df.loc[train_df[\"Cover_Type\"] == 3, \"Cover_Type\"], label=\"Cover_Type = 3\", bins=np.arange(1, 9)-0.5, alpha=0.75, color=next(color))\n",
    "plt.hist(train_df.loc[train_df[\"Cover_Type\"] == 4, \"Cover_Type\"], label=\"Cover_Type = 4\", bins=np.arange(1, 9)-0.5, alpha=0.75, color=next(color))\n",
    "plt.hist(train_df.loc[train_df[\"Cover_Type\"] == 5, \"Cover_Type\"], label=\"Cover_Type = 5\", bins=np.arange(1, 9)-0.5, alpha=0.75, color=next(color))\n",
    "plt.hist(train_df.loc[train_df[\"Cover_Type\"] == 6, \"Cover_Type\"], label=\"Cover_Type = 6\", bins=np.arange(1, 9)-0.5, alpha=0.75, color=next(color))\n",
    "plt.hist(train_df.loc[train_df[\"Cover_Type\"] == 7, \"Cover_Type\"], label=\"Cover_Type = 7\", bins=np.arange(1, 9)-0.5, alpha=0.75, color=next(color))\n",
    "\n",
    "plt.tight_layout()\n",
    "# plt.savefig(\"report/figures/label_repart.png\", facecolor=\"white\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to check is there are any missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Are there missing values ?\n",
    "train_df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset is clearly clean. Next, we will dive into the data types."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Looking at the available features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Numerical / Categorical features\n",
    "train_df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we can see that all the features are numerical. We also want to know if there are discrete or continuous. Continuous features take unique values for every data points, while discrete features have a limited amount of values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Discrete / Continuous features\n",
    "train_df.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems that all the features are discrete to some extent. Furthermore, Wilderness_AreaX and Soil_TypeX features are binary features with values 0 and 1. The question we want to answer is whether a unique value of each of these features is assigned to each data point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Binary variables and target\n",
    "wild_var = [f\"Wilderness_Area{i}\" for i in range(1, 5)]\n",
    "soil_var = [f\"Soil_Type{i}\" for i in range(1, 41)]\n",
    "label_var = [\"Cover_Type\"]\n",
    "\n",
    "# Print the number of positive class for both types of binary features and the missing spots\n",
    "print(\"Total number of 1 for Wilderness_AreaX features: \", train_df[wild_var].sum().sum())\n",
    "print(\"Total number of 1 for Soil_TypeX features: \", train_df[soil_var].sum().sum())\n",
    "\n",
    "print(\"\\n\")\n",
    "\n",
    "print(\"Number of data point(s) with no Wilderness_AreaX: \", (train_df[wild_var].sum(axis=1) == 0).sum())\n",
    "print(\"Number of data point(s) with no Soil_TypeX: \", (train_df[soil_var].sum(axis=1) == 0).sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This exploration ensures that each data point has one and only one corresponding Wilderness_AreaX value set to 1 and one and only one corresponding Soil_TypeX value set to 1.\n",
    "\n",
    "Now, let's verify the values of numerical features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Separate discrete and continuous features\n",
    "all_var = list(train_df.columns)\n",
    "disc_var = wild_var + soil_var + label_var\n",
    "cont_var = [x for x in all_var if x not in disc_var]\n",
    "\n",
    "# Look at the coherence of the data\n",
    "train_df[cont_var].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option(\"display.max_columns\", None)\n",
    "\n",
    "def describe(df, stats):\n",
    "    d = df.describe()\n",
    "    return d.append(df.reindex(d.columns, axis=1).agg(stats))\n",
    "\n",
    "print(describe(train_df, [\"sum\"]))\n",
    "\n",
    "pd.set_option(\"display.max_columns\", 15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The details given in the data description are as follows:\n",
    "\n",
    "- Elevation, quantitative (meters): Elevation in meters\n",
    "- Aspect, quantitative (azimuth): Aspect in degrees azimuth\n",
    "- Slope, quantitative (degrees): Slope in degrees\n",
    "- *Horizontal_Distance_To_Hydrology* , quantitative (meters): Horz Dist to nearest surface water features\n",
    "- *Vertical_Distance_To_Hydrology* , quantitative (meters): Vert Dist to nearest surface water features\n",
    "- *Horizontal_Distance_To_Roadways* , quantitative (meters ): Horz Dist to nearest roadway\n",
    "- *Hillshade_9am* , quantitative (0 to 255 index): Hillshade index at 9am, summer solstice\n",
    "- *Hillshade_Noon*, quantitative (0 to 255 index): Hillshade index at noon, summer soltice\n",
    "- *Hillshade_3pm*, quantitative (0 to 255 index): Hillshade index at 3pm, summer solstice\n",
    "- *Horizontal_Distance_To_Fire_Points*, quantitative (meters): Horz Dist to nearest wildfire ignition points\n",
    "\n",
    "The description of the features correspond to the given details. Moreover, the values seem reasonable. \n",
    "\n",
    "Then, we will look at the distributions and correlation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find correlations with the target and sort\n",
    "corr = train_df.corr()[\"Cover_Type\"].sort_values()\n",
    "print(corr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It appears that some of the Soil_TypeX and Wilderness_AreaX variables have the largest correlation with the target variable. Let's look at them.\n",
    "\n",
    "Also, a NaN value is set for Soil_Type15, which shows that the value is always 0 for all data points. This variable cannot help with our task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at the repartition of the most correlated binary features\n",
    "print(\"Number of positive Soil_Type38 values among training set: \", (train_df[\"Soil_Type38\"] == 1).sum())\n",
    "print(\"Number of positive Soil_Type39 values among training set: \", (train_df[\"Soil_Type39\"] == 1).sum())\n",
    "print(\"Number of positive Wilderness_Area1 values among training set: \", (train_df[\"Wilderness_Area1\"] == 1).sum())\n",
    "print(\"Number of positive Soil_Type29 values among training set: \", (train_df[\"Soil_Type29\"] == 1).sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Repartition of cover types when Soil_Type38 is equal to 1 and 0\n",
    "plt.figure(1, figsize=(10, 8))\n",
    "sns.kdeplot(train_df.loc[train_df[\"Soil_Type38\"] == 0, \"Cover_Type\"], label=\"Soil_Type38 = 0\", color=\"blue\")\n",
    "sns.kdeplot(train_df.loc[train_df[\"Soil_Type38\"] == 1, \"Cover_Type\"], label=\"Soil_Type38 = 1\", color=\"red\")\n",
    "\n",
    "plt.legend(loc=\"best\")\n",
    "plt.title(\"Repartition of Soil_Type38 among Cover_Type\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the case of Soil_Type38, we indeed observe that data points with value 1 are divided into cover types 1 and 7, while data points with value 0 can be of all cover types, except 7 with less probability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Repartition of cover types when Soil_Type39 is equal to 1 and 0\n",
    "plt.figure(1, figsize=(10, 8))\n",
    "sns.kdeplot(train_df.loc[train_df[\"Soil_Type39\"] == 0, \"Cover_Type\"], label=\"Soil_Type39 = 0\", color=\"blue\")\n",
    "sns.kdeplot(train_df.loc[train_df[\"Soil_Type39\"] == 1, \"Cover_Type\"], label=\"Soil_Type39 = 1\", color=\"red\")\n",
    "\n",
    "plt.legend(loc=\"best\")\n",
    "plt.title(\"Repartition of Soil_Type39 among Cover_Type\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The repartition among Soil_Type39 is almost the same as for Soil_Type38, which give an insight that some soil types are largely related to specific cover types. These features will be of great importance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Repartition of cover types when Soil_Type29 is equal to 1 and 0\n",
    "plt.figure(1, figsize=(10, 8))\n",
    "sns.kdeplot(train_df.loc[train_df[\"Soil_Type29\"] == 0, \"Cover_Type\"], label=\"Soil_Type29 = 0\", color=\"blue\")\n",
    "sns.kdeplot(train_df.loc[train_df[\"Soil_Type29\"] == 1, \"Cover_Type\"], label=\"Soil_Type29 = 1\", color=\"red\")\n",
    "\n",
    "plt.legend(loc=\"best\")\n",
    "plt.title(\"Repartition of Soil_Type29 among Cover_Type\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For Soil_Type29 feature, the repartition is less precise in case of positive values. However, we can see that some of the cover types are immediately eliminated when the class is positive. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Repartition of cover types when Wilderness_Area1 is equal to 1 and 0\n",
    "plt.figure(1, figsize=(10, 8))\n",
    "sns.kdeplot(train_df.loc[train_df[\"Wilderness_Area1\"] == 0, \"Cover_Type\"], label=\"Wilderness_Area1 = 0\", color=\"blue\")\n",
    "sns.kdeplot(train_df.loc[train_df[\"Wilderness_Area1\"] == 1, \"Cover_Type\"], label=\"Wilderness_Area1 = 1\", color=\"red\")\n",
    "\n",
    "plt.legend(loc=\"best\")\n",
    "plt.title(\"Repartition of Wilderness_Area1 among Cover_Type\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interestingly, we observe that the repartition of cover types regarding Wilderness_Area1 is similar to the one for Soil_Type29. Later, we will see if these features are correlated.\n",
    "\n",
    "Finally, take a look at the repartition for the most correlated continuous features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Repartition of cover types wrt Horizontal_Distance_To_Roadways\n",
    "plt.figure(figsize=(10, 8))\n",
    "\n",
    "color = iter(plt.cm.brg(np.linspace(0, 1, 7)))\n",
    "\n",
    "sns.kdeplot(train_df.loc[train_df[\"Cover_Type\"] == 1, \"Horizontal_Distance_To_Roadways\"], label=\"Cover_Type = 1\", color=next(color))\n",
    "sns.kdeplot(train_df.loc[train_df[\"Cover_Type\"] == 2, \"Horizontal_Distance_To_Roadways\"], label=\"Cover_Type = 2\", color=next(color))\n",
    "sns.kdeplot(train_df.loc[train_df[\"Cover_Type\"] == 3, \"Horizontal_Distance_To_Roadways\"], label=\"Cover_Type = 3\", color=next(color))\n",
    "sns.kdeplot(train_df.loc[train_df[\"Cover_Type\"] == 4, \"Horizontal_Distance_To_Roadways\"], label=\"Cover_Type = 4\", color=next(color))\n",
    "sns.kdeplot(train_df.loc[train_df[\"Cover_Type\"] == 5, \"Horizontal_Distance_To_Roadways\"], label=\"Cover_Type = 5\", color=next(color))\n",
    "sns.kdeplot(train_df.loc[train_df[\"Cover_Type\"] == 6, \"Horizontal_Distance_To_Roadways\"], label=\"Cover_Type = 6\", color=next(color))\n",
    "sns.kdeplot(train_df.loc[train_df[\"Cover_Type\"] == 7, \"Horizontal_Distance_To_Roadways\"], label=\"Cover_Type = 7\", color=next(color))\n",
    "\n",
    "plt.legend(loc=\"best\")\n",
    "plt.title(\"Repartition of Horizontal_Distance_To_Roadways among Cover_Type\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This graph show that medium values of cover types (around 3-6) correspond to smaller values of horz dist to roadways, while cover types 1, 2 and 7 can be assigned to larger values. However, wee see that in both groups, the value of horz dist to roadways does not help much at differentiating the cover types.\n",
    "\n",
    "In terms of correlation between the features, we might not be able to get a lot of information. Indeed, the most correlated features with the target are unique features such as Soil_TypeX and Wilderness_AreaX. Let's plot a correlation heatmap between Soil_TypeX and Wilderness_AreaX for the highest correlated features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "most_corr_disc = train_df[[\"Soil_Type38\", \"Soil_Type39\", \"Soil_Type29\", \"Wilderness_Area1\", \"Cover_Type\"]]\n",
    "most_corr_disc_corr = most_corr_disc.corr()\n",
    "\n",
    "plt.figure(figsize = (8, 6))\n",
    "\n",
    "# Heatmap of correlations\n",
    "sns.heatmap(most_corr_disc_corr, cmap=plt.cm.RdYlBu_r, vmin=-0.23, annot=True, vmax=0.55)\n",
    "\n",
    "plt.xticks(rotation=60)\n",
    "\n",
    "plt.tight_layout()\n",
    "# plt.savefig(\"report/figures/basecorr.png\", facecolor=\"white\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These results confirm what we observed, that is, Soil_Type29 and Wilderness_Area1 give similar information about the cover types, that are more likely to be small if the values of these features are larger. Also, we see that two groups of cover types appear to be less easy to discriminate: 1-2-7 and 3-4-5-6."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}