Exercise-14.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Data-Riddle on the Riddler  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prelude: Riddler is a weekly column at the [https://fivethirtyeight.com/](https://fivethirtyeight.com/), where they offer puzzles on maths, logic and probability. These puzzles are often solved using data analysis approaches. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Are you The Best Warlord?\n",
    "[Article Page](https://fivethirtyeight.com/features/are-you-the-best-warlord/)\n",
    "\n",
    "\n",
    "\n",
    "In a distant, war-torn land, there are ten castles. There are two warlords: you and your archenemy. Each castle has its own strategic value for a would-be conqueror. Specifically, the castles are worth 1, 2, 3, …, 9, and 10 victory points (corresponding to the castle number). You and your enemy each have 100 soldiers to distribute, any way you like, to fight at any of the ten castles. Whoever sends more soldiers to a given castle conquers that castle and wins its victory points. If you each send the same number of troops, you split the points. You don’t know what distribution of forces your enemy has chosen until the battles begin. Whoever wins the most points wins the war.\n",
    "\n",
    "The puzzle received submissions from around 1500 users, and the submissions are available at [https://github.com/fivethirtyeight/data/blob/master/riddler-castles/castle-solutions-3.csv](https://github.com/fivethirtyeight/data/blob/master/riddler-castles/castle-solutions-3.csv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Read the file and remove the invalid entries from the file.\n",
    "- Constraints: \n",
    "    - The sum of entries in each row should add up to 100 (A soldier needs to be sent to some castle)\n",
    "    - The number of soldiers sent to a castle should be a whole number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the data you obtain after the above cleaning, perform the following:\n",
    "\n",
    "1. For each castle, try to fit (using `stats.norm.fit`) the distribution as normal distribution and obtain its mean, `mu` and standard deviation, `sigma.` \n",
    "2. For the castle with the highest mean value, plot the histogram and the corresponding normal distribution obtained using `stats.norm.fit`.\n",
    "3. For the castle with the lowest standard deviation value,  plot the histogram and the corresponding normal distribution obtained using `stats.norm.fit`.\n",
    "4. For the castle with the highest standard deviation value,  plot the histogram and the corresponding normal distribution obtained using `stats.norm.fit`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the castles in questions 2, 3 and 4 in the previous question box, perform the following:\n",
    "- Plot the Q-Q plots and verify, and if the above data distributions are good fits as normal distribution "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Find the mode(s) and median for each of the castle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- For each submission (a row), compare it with every other row and calculate (roughly $1500 \\times 1500$ comparisons) the score for each player in each of the comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " - Calculate the number of wins, losses and ties for each of the submissions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Plot the histogram based on the number of wins per submission\n",
    "- Plot the histogram based on the number of losses per submission\n",
    "- Plot the histogram based on the number of losses per submission\n",
    "- Check if any of the above distribution follows a normal disstribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " - Find top 5 submissions based on their number of wins."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Consider the discrete data distrubtions (column vector for each castle) and the corresponding normal distribution obtained using `stats.norm.fit`\n",
    "- Consider the submission with the highest number of wins\n",
    "- FOr each entry in the submission, calcualte its probability from the corresponding discrete data distribution\n",
    "- For each entry in the submission, calculatte its  probability from the corresponding normal distribution, obtained using `stats.norm.fit`\n",
    "- For each entry, calculate the differences of the corresponding probabilities (data distribution and the normal distribution obtained) for each castle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "- You are required to generatet 2000 different submissions, one each for a minion, where each submission is a vector of 10 integers, randomly sampled from a normal distribution.\n",
    "\n",
    "    - In reality, it is not possible to generate vectors as decribe in the question above. Could you reason why?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detour from warlords: Being both rounded and normal.\n",
    "\n",
    "- Sample 100000 entries from a normal distribution.\n",
    "- Round off each draw to the nearest integer.\n",
    "- Check if the resulting disttribution is also normal\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 14"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " - Plot boxplots for each of the castle as vertical boxplots and visually idnetify box longest height. What does the height of the box imply?\n",
    " - For the castle with the largest box, plot separate boxplots for the data by grouping number of soldiers above the mdeian as a single group and the remaining as a single group.\n",
    " - For the castle with the smallest box, plot separate boxplots for the data by grouping number of soldiers above the mdeian as a single group and the remaining as a single group.\n",
    " - Identify the castle with the largest number of outliers (boxplot returns a value `fliers`, use it as outliers)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Consider all the pairwise combinations of castles.\n",
    "- Perform regression using, `scipy.stats.linregress` for each of such pairs\n",
    "- Find the top 5 pairs, based on the correlation coefficient `r`\n",
    "- Take castle pair with the highest value of `r`, say `(castle i, castle j)`. Then, what will be th value of `r` for `(castle j, castle i)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Its time for some `%magic`\n",
    "\n",
    "- You might have wondered, what does the peculiar statement `%matplotlib inline` do. \n",
    "- These are known in Jupyter as magic commands, and are prefixed by the `% `character. These magic commands are designed to succinctly solve various common problems in standard data analysis. Magic commands come in two flavors: line magics, which are denoted by a single `%` prefix and operate on a single line of input, and cell magics, which are denoted by a double `%%` prefix and operate on multiple lines of input. We'll demonstrate and discuss a few brief examples here, and come back to more focused discussion of several useful magic commands later in the chapter.\n",
    "- Another example of a useful magic function is `%timeit`, which will automatically determine the execution time of the single-line Python statement that follows it. For example, we may want to check the performance of a list comprehension:\n",
    "\n",
    "```py\n",
    "In [8]:  %timeit L = [n ** 2 for n in range(1000)]`\n",
    "```\n",
    "`1000 loops, best of 3: 325 µs per loop`\n",
    "\n",
    "The benefit of `%timeit` is that for short commands it will automatically perform multiple runs in order to attain more robust results. For multi line statements, adding a second `%` sign will turn this into a cell magic that can handle multiple lines of input. For example, here's the equivalent construction with a for-loop:\n",
    "\n",
    "```py \n",
    "In [9]: %%timeit\n",
    "   ...: L = []\n",
    "   ...: for n in range(1000):\n",
    "   ...:     L.append(n ** 2)\n",
    "   ...:\n",
    "``` \n",
    "\n",
    "`1000 loops, best of 3: 373 µs per loop`\n",
    "\n",
    "We can immediately see that list comprehensions are about 10% faster than the equivalent for-loop construction in this case. \n",
    "\n",
    "- For more such magic commands, refer to [https://github.com/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/01.03-Magic-Commands.ipynb](https://github.com/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/01.03-Magic-Commands.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `%%timeit` magic can be used to see roughly how much time does your code take to run (on your machine). \n",
    "- For the above code for winding, correlation coefficient for all the pairwise combinations of castles, use the magic command and see the time taken for executing your code\n",
    "- Can it be improved?\n",
    "    - For instance, if you are considering the pairwise combination of castlees using two for loops. This might result in 100 comparisons (or 90, if you added a comparison check to ignore correlation of pairs with the same castle numbers).\n",
    "    - But given `r` is symmetric, could wee reduce it to 45 comparisons? If so, modify the code to perform the comparison only for 45 pairs. \n",
    "    - use `timeit` to check the running time and see the difference in execution time for yourself"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the pair of castles with the highest `r`, perform the following:\n",
    "    - Plot a scatter plot, with each castle in the x and y axes\n",
    "    - take data from each castle and find their median. Categorise the values (number of soldiers) into group A and group B, based on whether the value is above the median or below.\n",
    "    - This should crerate four different categories, based on the category in which the castle's value is present \n",
    "    - Plot the scatter plot with a legend and color palette to distinguish between the points belonging to different categories\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the pair of castles with the highest `r`, say (castle i, castle j), perform the following: \n",
    "    - Using the lineregress function, obtain the slope and intercept for the castle pair\n",
    "    - Would the slope and interecept be same for (castle j, castle i). Verify for yourself"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the pair of castles with the highest `r`, say (castle i, castle j), perform the following: \n",
    "    - Take the castle pair and perform regression, but do not use data from the last 10 lines.\n",
    "    - Using the slope and intercept, predict the value for castle j based on the value for castle i in the last 10 lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}