Python for Data Analytics - Module 2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<font size=\"3\" color=\"brown\">Author : UTSAV AWASTHI</font><br>\n",
    "<font size=\"1\" color=\"brown\">About Me: http://iamawesome.com/ </font> <br>\n",
    "Subscribe to my YouTube channel <a href=\"https://www.youtube.com/channel/UCJpp55YcP5swWj8TmIOd0OA/\">here</a><br> \n",
    "For serious conversations related to career and jobs, follow me on <a href=\"https://www.linkedin.com/in/utsavawasthi/\">LinkedIn</a><br>\n",
    "For low-effort dank memes and one-liners, follow me on <a href=\"https://twitter.com/utsavsaidwhat\">Twitter</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<div align=\"center\">\n",
    "    <font size = 3 color=\"orange\" style=\"font-family: 'Comic Sans MS'\">// ज्ञानं परमं बलम् //</font><br>\n",
    "</div>\n",
    "<img src = \"https://www.python.org/static/community_logos/python-logo-master-v3-TM.png\" width=\"50%\" height=\"50%\">\n",
    "<div align=\"center\" id=\"contents\"> <font size=\"5\" color=\"orange\"><b>PYTHON PROGRAMMING LANGUAGE</b></font> </div>\n",
    "\n",
    "<div align=\"left\"> <font size=\"4\" color=\"red\"><b>TABLE OF CONTENTS</b></font> </div>\n",
    "<ol type=\"square\">\n",
    "    <li><a href=\"#regex\">Regular Expressions</a></li>\n",
    "    <li><a href=\"#numpy\">NumPy</a></li>\n",
    "    <li><a href=\"#pandas\">Pandas</a></li>\n",
    "    <li><a href=\"#mat\">matplotlib.pyplot</a></li>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"left\" id=\"regex\"><a href=\"#contents\"><font size=\"4\" color=\"red\"><b>REGULAR EXPRESSIONS</b></font></a></div>\n",
    "<ol type=\"square\">\n",
    "    <li><a href=\"#regex_simple\">Simple Matching</a></li>\n",
    "    <li><a href=\"#regex_re\">re module</a></li>\n",
    "    <li><a href=\"#regex_wild\">Wild Card Characters: Special Characters</a></li>\n",
    "    <li><a href=\"#regex_rep\">Repetitions</a></li>\n",
    "    <li><a href=\"#regex_groups\">Groups and Grouping</a></li>\n",
    "    <li><a href=\"#regex_greedy\">Greedy vs Non-Greedy Matching</a></li>\n",
    "</ol>\n",
    "\n",
    "Regular Expressions (or regex in short) are a powerful tool to find patterns in a text data. For example if we have a really big word document, given enough time (and motivation) a human can read the whole document and make a list of all emails and phone numbers mentioned anywhere in the document.<br>\n",
    "With computers, you can do the same thing just by telling it what pattern it should look for. Like in Phone number the pattern could be any one of these XXXXXXXXXX or XXX XXX XXXX or XXXXX XXXXX etc. Or in case of emails it is SOME&#95;TEXT&#64;SOME&#95;TEXT.SOME&#95;TEXT "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"left\" id =\"regex_simple\"> <a href=\"#regex\"><font size=\"3\" color=\"black\"><b>Simple Matching</b></font></a></div>\n",
    "\n",
    "There are multiple string methods available by default, which you use to perform basic pattern matching<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#checking presence of a substring\n",
    "my_string = \"Hi! How are you?\"\n",
    "if \"are\" in my_string:\n",
    "    print (\"Present\")\n",
    "else:\n",
    "    print (\"Absent\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#matching is case-sensitive\n",
    "if \"Are\" in my_string:\n",
    "    print (\"Present\")\n",
    "else:\n",
    "    print (\"Absent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we have seen in Module 1, the \"in\" operator also works with lists, tubles, sets and dictionaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#mystr.find(’am’, <start to search from index>, <end search at index>) - outputs \"-1\" if not found\n",
    "print (my_string.find(\"are\"))\n",
    "print (my_string.find(\"Hi\",1))\n",
    "print (my_string.find(\"Hi\",0,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#rfind() finds from back\n",
    "print (\"rfind example : \", my_string.rfind(\"are\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#index is similar to find, but raises ValueError if not present.\n",
    "print (\"index example : \", my_string.index(\"are\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#You can also check if a string starts or ends with a particular substring\n",
    "print (my_string.startswith(\"H\"))\n",
    "print (my_string.endswith(\"!\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following methods return True if the string only contains characters of the appropiate type, False otherwise :<br>\n",
    "<b>isalpha()</b> - alphabetic<br>\n",
    "<b>isdigit()</b> - digits<br>\n",
    "<b>isdecimal()</b> - float numbers, etc<br>\n",
    "<b>isnumeric()</b> - similar, covers special chars like ½<br>\n",
    "<b>isalnum()</b> - all of above<br>\n",
    "<b>islower()</b> - contains only lowercase<br>\n",
    "<b>isupper()</b> - contains only uppercase<br>\n",
    "<b>isspace()</b> - contains only whitespace<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (\"Hello world!\".isalpha()) \n",
    "print (\"Helloworld\".isalpha()) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#replace doesn't modify the string\n",
    "my_string = \"My Grade is D!\"\n",
    "print (my_string.replace(\"D!\",\"A+!\"))\n",
    "print (my_string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#You can use lstrip(), rstrip() or strip(). These are used to remove leading or trailing white spaces\n",
    "my_string = \"\\tHow are you?   \"\n",
    "print (\">\"+my_string.strip()+\"<\")\n",
    "#Again the string is not modified\n",
    "print (\">\"+my_string+\"<\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#split() can be used to split the line into list of words\n",
    "my_string = \"\\tHow are\\tyou?\\t\\t\"\n",
    "print (my_string.split())\n",
    "print (my_string.split(\" \")) #split by spaces\n",
    "print (my_string.split(\"\\t\"))#split by tabs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"left\" id =\"regex_re\"> <a href=\"#regex\"><font size=\"3\" color=\"black\"><b>re module</b></font></a></div>\n",
    "\n",
    "Regular expressions are used to identify whether a pattern exists in a given sequence of characters (string) or not. They help in manipulating text data, which is often a pre-requisite for data science projects that involve text mining<br><br>\n",
    "In Python, the \"re\" module provides the support for regular expressions. We will discuss few methods inside this module<br><br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "pattern = r\"Cookie\" # r at the begining is called a raw string literal\n",
    "sequence = \"Cookie\"\n",
    "if re.match(pattern, sequence):\n",
    "     print (\"Match!\")\n",
    "else: print(\"Not a match!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "r at the begining is called a raw string literal. It changes how the string literal is interpreted. Such literals are stored as they appear. <br><br>\n",
    "search ⇒ find something anywhere in the string and return a match object.<br>\n",
    "match ⇒ find something at the beginning of the string and return a match object.\n",
    "\n",
    "Both of them return a match object if something is found, otherwise \"None\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# example code:\n",
    "string_with_newlines = \"\"\"something\n",
    "someotherthing\"\"\"\n",
    "\n",
    "import re\n",
    "\n",
    "print (re.match('some', string_with_newlines)) # matches\n",
    "print (re.match('someother', string_with_newlines)) # won't match\n",
    "print (re.search('someother', string_with_newlines)) # finds something"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>group, start, end, and span</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r1 = re.search(\"a*b\",\"fooaaabcde\") #search occurance of \"a\", followed by anything, followed by \"b\"\n",
    "print (r1)\n",
    "print (r1.group())  # group returns string matched\n",
    "print (r1.start())  # index of the match start\n",
    "print (r1.end())    # index of the match end\n",
    "print (r1.span())   # tuple of (start, end)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"left\" id =\"regex_wild\"> <a href=\"#regex\"><font size=\"3\" color=\"black\"><b>Wild Card Characters: Special Characters</b></font></a></div>\n",
    "\n",
    "Special characters are characters which do not match themselves as seen but actually have a special meaning when used in a regular expression.\n",
    "\n",
    "The most widely used special characters are as listed below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#. - A period. Matches any single character except newline character\n",
    "print (re.search(r'Co.k.e', 'Cookie').group())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\\w - Lowercase w. Matches any single letter, digit or underscore\n",
    "print (re.search(r'Co\\wk\\we', 'Cookie').group())\n",
    "#\\W - Uppercase w. Matches any character not part of \\w (lowercase w).\n",
    "print (re.search(r'C\\Wke', 'C@ke').group())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\\s - Lowercase s. Matches a single whitespace character like: space, newline, tab, return. \n",
    "print (re.search(r'Eat\\scake', 'Eat cake').group())\n",
    "#\\S - Uppercase s. Matches any character not part of \\s (lowercase s).\n",
    "print (re.search(r'Cook\\Se', 'Cookie').group())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\\t - Lowercase t. Matches tab.\n",
    "print (re.search(r'Eat\\tcake', 'Eat\tcake').group())\n",
    "#\\n - Lowercase n. Matches newline.\n",
    "#\\r - Lowercase r. Matches return."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\\d - Lowercase d. Matches decimal digit 0-9.\n",
    "print (re.search(r'c\\d\\dkie', 'c08kie').group())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#^ - Caret. Matches a pattern at the start of the string.\n",
    "print (re.search(r'^Eat', 'Eat cake').group())\n",
    "\n",
    "#$ - Matches a pattern at the end of string.\n",
    "print (re.search(r'cake$', 'Eat cake').group())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#[abc] - Matches a or b or c. The first occurance only\n",
    "print (re.search(r'[abc]', 'abccdbc').group())\n",
    "#[a-zA-Z0-9] - Matches any letter from (a to z) or (A to Z) or (0 to 9). \n",
    "#Characters that are not within a range can be matched by complementing the set. \n",
    "#If the first character of the set is ^, all the characters that are not in the set will be matched."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (re.search(r'Number: [0-6]', 'Number: 54').group())\n",
    "\n",
    "# Matches any character except 5\n",
    "print (re.search(r'Number: [^5]', 'Number: 03').group())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#re.findall() finds all matches\n",
    "print(re.findall(\"\\d+\",\"12 dogs,11 cats, 1 egg\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#re.sub substitutes one string for a pattern\n",
    "re.sub('(blue|white|red)', 'black', 'blue socks and red shoes')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"left\" id =\"regex_rep\"> <a href=\"#regex\"><font size=\"3\" color=\"black\"><b>Repetitions</b></font></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It becomes quite tedious if you are looking to find long patterns in a sequence. Fortunately, the re module handles repetitions using the following special characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#+ - Checks for one or more characters to its left.\n",
    "re.search(r'Co+kie', 'Cooookie').group()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#* - Checks for zero or more characters to its left.\n",
    "re.search(r'Ca*o*kie', 'Caokie').group()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#? - Checks for exactly zero or one character to its left.\n",
    "re.search(r'Colou?r', 'Color').group()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check for exact number of repetitions too"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#{x} - Repeat exactly x number of times.\n",
    "#{x,} - Repeat at least x times or more.\n",
    "#{x, y} - Repeat at least x times but no more than y times.\n",
    "re.search(r'\\d{9,10}', '09876543219').group()\n",
    "#goes till only 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"left\" id =\"regex_groups\"> <a href=\"#regex\"><font size=\"3\" color=\"black\"><b>Groups and Grouping</b></font></a></div>\n",
    "\n",
    "Now suppose when you're validating email addresses and want to check the user name and host separately. This is when the 'group' feature of regular expression comes in handy. It allows you to pick up parts of the matching text. The groups are seperated using parentheses i.e. ()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "email_address = 'Please contact us at: support@citi.com'\n",
    "match = re.search(r'([\\w\\.-]+)@([\\w\\.-]+)', email_address) \n",
    "if match: \n",
    "    print (match.group()) # The whole matched text \n",
    "    print (match.group(1)) # The username (group 1) \n",
    "    print (match.group(2)) # The host (group 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#We can also group 'groups'\n",
    "\n",
    "pat2 = \"(\\w+)@((\\w+\\.)+(com|org|net|edu))\"\n",
    "r2 = re.match(pat2,\"finin@cs.umbc.edu\")\n",
    "print (r2.group(1))\n",
    "print (r2.group(2))\n",
    "print (r2.groups())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#We can ‘label’ the groups as well and reference the matching parts by the labels\n",
    "pat3 =\"(?P<name>\\w+)@(?P<host>(\\w+\\.)+(com|org|net|edu))\"\n",
    "r3 = re.match(pat3,\"finin@cs.umbc.edu\")\n",
    "print (r3.group('name'))\n",
    "print (r3.group('host'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"left\" id =\"regex_greedy\"> <a href=\"#regex\"><font size=\"3\" color=\"black\"><b>Greedy vs Non-Greedy Matching</b></font></a></div>\n",
    "\n",
    "\n",
    "When a special character matches as much of the search sequence (string) as possible, it is said to be a \"Greedy Match\". It is the normal behavior of a regular expression but sometimes this behavior is not desired:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "heading  = r'<h1>TITLE</h1>'\n",
    "re.match(r'<.*>', heading).group()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"left\" id=\"numpy\"><a href=\"#contents\"><font size=\"4\" color=\"red\"><b>NUMPY</b></font></a></div>\n",
    "<ol type=\"square\">\n",
    "    <li><a href=\"#numpy_arrays\">Arrays</a></li>\n",
    "    <li><a href=\"#numpy_matrix\">Matrix Operations</a></li>\n",
    "    <li><a href=\"#numpy_index\">Indexing and Slicing</a></li>\n",
    "    <li><a href=\"#numpy_tricks\">Some Tricks</a></li>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NumPy is a library for the Python programming language, adding support for large, multi-dimensional arrays and matrices. It has a large collection of high-level mathematical functions to operate on these arrays. Numpy is not a default pacakge in Python and you will need to install it first. Luckily it comes pre-installed in Anaconda<br>\n",
    "You might ask why a seperate package/module for matrics. The answer is that all the algorithms in machine learning or Deep Learning or AI internally work their magic using matrices. So it is very important to have a module which has highly optimized mathematical operations on matrices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"left\" id =\"numpy_arrays\"> <a href=\"#numpy\"><font size=\"3\" color=\"black\"><b>Arrays</b></font></a></div>\n",
    "<br>\n",
    "NumPy Array is a datatype defined inside numpy library, just like list is a datatype available by default. Computation with NumPy arrays is much faster than the native lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Creating array with list of lists\n",
    "#1D array\n",
    "print (\"1d : \\n\", np.array([1,2,3]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#2D array\n",
    "print (\"2d : \\n\", np.array([[1,2,3],[4,5,6]])) #notice the list of lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#With arange() funciton\n",
    "print (np.arange(10))  # like Python's range, but returns an array\n",
    "print (np.arange( 10, 30, 5 )) #start, end, step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# it accepts float arguments\n",
    "print (np.arange( 0, 2, 0.3 )) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#With rearranging\n",
    "print (np.arange(6).reshape((3, 2))) #reshaping arange to a 3x2 array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#reshape is also a function\n",
    "print (np.reshape([1,2,3,4,5,6], (3,2)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#an array of zeroes\n",
    "print (np.zeros( (3,4) ))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#a 3d array of ones\n",
    "print (np.ones( (2,3,4), dtype=np.int16 ))  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#an identity matrix\n",
    "print (np.identity(3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Certain attributes\n",
    "a = np.arange(30).reshape(3, 5, 2) #3 arrays of dimension 5x2\n",
    "print (\"a :\\n\", a)\n",
    "print ('Attribuites of a:\\n')\n",
    "print (\"Shape:\", a.shape)\n",
    "print (\"Dimensions:\", a.ndim)\n",
    "print (\"itemsize:\", a.itemsize) #Length of one array element in bytes\n",
    "print (\"Data type:\", a.dtype)\n",
    "print (\"Size:\", a.size) #number of elements\n",
    "print (\"Type:\", type(a))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#You can specify the data type while creating arrays\n",
    "a = np.array( [ [1,2], [3,4] ], dtype=complex )\n",
    "print (\"a: \\n\", a)\n",
    "print (\"a.dtype:\", a.dtype)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Using linspace you can create evenly spaced numbers over a specified interval\n",
    "print (np.linspace( 0, 2, 9 ))                 # 9 numbers from 0 to 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "a = np.linspace( 0, 2*np.pi, 100 ) #numpy has pi defined inside of it\n",
    "print (a)\n",
    "print (np.sin(a))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#You can create random distributions too\n",
    "np.random.normal(0.0, 1.0, (2, 2, 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "help(np.random.normal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you haven't figured out already, numpy arrays are MUTABLE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Simple assignments make no copy of array objects or of their data because numpy arrays are mutable too\n",
    "a = np.arange(12)\n",
    "b = a            # no new object is created\n",
    "print (\"b is a?\", b is a)           # a and b are two names for the same ndarray object\n",
    "print(id(a), id(b))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (\"Shape before:\", a.shape)\n",
    "b.shape = 3,4    # changes the shape of a\n",
    "print (\"Shape before:\", a.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"left\" id =\"numpy_matrix\"> <a href=\"#numpy\"><font size=\"3\" color=\"black\"><b>Matrix Operations</b></font></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NumPy supports all major matrix operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.arange(6).reshape(2, 3)\n",
    "b = np.arange(2,8).reshape(3, 2)\n",
    "c = np.arange(6).reshape(3, 2)\n",
    "print (\"a:\\n\", a)\n",
    "print (\"b:\\n\", b)\n",
    "print (\"c:\\n\", c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "b+c #try b-c also"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Dot Product\n",
    "np.dot(a,b) #try b,a too \n",
    "#or a.dot(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Transpose\n",
    "np.transpose(a)\n",
    "#or simple a.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from numpy.linalg import inv\n",
    "inv(a[0:2,0:2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.dot(inv(a[0:2,0:2]),a[0:2,0:2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Functions get applied to all the elements\n",
    "a*2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a**2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a<3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.mean(a) #min, max etc are there too"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#You can take column sums(or min/max etc) OR row sums too\n",
    "b.sum(axis=1)   #1 means along rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "b.mean(axis=0)   #0 means along column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# returns the array, flattened\n",
    "b.ravel()  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#You can stack the arrays\n",
    "a = np.zeros((2,2))\n",
    "b = np.ones((2,2))\n",
    "print (np.vstack((a,b)))\n",
    "print (np.hstack((a,b)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"left\" id =\"numpy_index\"> <a href=\"#numpy\"><font size=\"3\" color=\"black\"><b>Indexing and Slicing</b></font></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember numbering/indexing starts from 0 in Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.arange(15).reshape(3, 5)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#first row\n",
    "print (\"First row of a: \", a[0, :])# : means everything"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#second columns\n",
    "print (\"Second column of a: \", a[:, 2] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#slicing works too\n",
    "print (a[1:3, 2:4]) #From row index 1 to row index 3-1, column index 2 to column index 4-1 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#You can use indexing/slicing to assign values\n",
    "a[1:3, 2:4] = -1\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Using steps\n",
    "a = np.arange(10)**3\n",
    "print (a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a[:6:2] = -1000  #change every second element till index 5 (i.e. 6-1)\n",
    "print (a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#You can use slicing to create new arrays\n",
    "a = np.arange(12).reshape(2,3,2) #two arrays of 3x2\n",
    "b = a[:,:,:]    #3 dimensions three columns \n",
    "print (id(a), id(b))\n",
    "print (a)\n",
    "print (b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Or you can use copy() method\n",
    "b = a.copy()    #3 dimensions three columns \n",
    "print (id(a), id(b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"left\" id =\"numpy_tricks\"> <a href=\"#numpy\"><font size=\"3\" color=\"black\"><b>Some Tricks</b></font></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Replacing elements based on a condition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.arange(12).reshape((3,4))\n",
    "print (\"a:\\n\", a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "b = a>6 #Here a>6 creates a boolean array and then that is use as index below\n",
    "print (\"b:\\n\", b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a[b] = -1\n",
    "print (\"a finally :\\n\", a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here a>6 creates a boolean array and then that is use to index "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#pearson correlation\n",
    "data = np.random.normal(size=[100, 5]) #100 rows, 5 columns\n",
    "print(np.corrcoef(data.T))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#solving equations ax = b\n",
    "a = np.array([[1,2],[3,-4]])\n",
    "b = np.array([2,16])\n",
    "print (np.linalg.solve(a,b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"left\" id=\"pandas\"><a href=\"#contents\"><font size=\"4\" color=\"red\"><b>PANDAS</b></font></a></div>\n",
    "<ol type=\"square\">\n",
    "    <li><a href=\"#pandas_df\">Introduction to Dataframes</a></li>\n",
    "    <li><a href=\"#pandas_manu\">Manipulating Dataframes</a></li>\n",
    "    <li><a href=\"#pandas_stats\">Statistics with Dataframes</a></li>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas is a high-level data manipulation tool. It is built on the Numpy package and its key data structure is called the DataFrame (along with one more known as series)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"left\" id =\"pandas_df\"> <a href=\"#pandas\"><font size=\"3\" color=\"black\"><b>Introduction to Dataframes</b></font></a></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#You can create dataframes with list of lists, where each list is one row in the data\n",
    "data = [['ABC', -3.5, 0.01], ['ABC', -2.3, 0.12], \n",
    "        ['DEF', None, 0.03], ['DEF', 3.7, 0.01], \n",
    "        ['GHI', 0.04, 0.43], ['GHI', -0.1, 0.67]]\n",
    "df = pd.DataFrame(data, columns=['gene', 'log2FC', 'pval'])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#or you can create using dictionaries, where keys are column names and values are the values in the column\n",
    "pd.DataFrame({'name':['Anil', 'John', 'Ali'], 'age':[5,6,7]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But mostly we will be reading data from files. Here is how you read a csv file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Let's first set the working directory\n",
    "import os\n",
    "os.chdir(r\"C:\\Users\\utsav.awasthi\\Documents\\Personal\\Python for Data Analytics\")\n",
    "\n",
    "#Now load the dataset\n",
    "iris = pd.read_csv(\"iris.csv\")\n",
    "print (type(iris))\n",
    "iris.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "read_csv() allows mutiple options like skipping first n rows/columns, define delimiter. As Thor says, let's do 'get help' :P"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(pd.read_csv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pd.read_excel and pd.read_sas are also supported"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are some key things you should immediately do after importing a data set\n",
    "<ol>\n",
    "    <li>Check <b>shape</b> of the data i.e. #rows and #columns using df.shape</li>\n",
    "    <li>Check <b>data type</b> of each column using df.info()</li>\n",
    "    <li>Check <b>uniqueness</b> of the data i.e. ensure the data is unique at the primary keys' level. e.g for students data, check if the data is unique at roll_no and class_std or not, or customer data being unique at cust_id level using df.shape, df[['pk1','pk2']].drop_duplicates().shape</li>\n",
    "    <li>Check <b>frequency</b> of key columns - like months, dates, etc. This will ensure you have complete data and not missing out some key information using df.col.value_counts(dropna=False)</li>\n",
    "    <li>Check number of <b>null</b> values in each column using df.isna().sum()</li>\n",
    "    <li>Some <b>basic stats</b> of each column using df.describe()</li>\n",
    "    <li>Finally analyze some <b>random sample</b> of the data using df.sample(100), if you want to be extra sure. You can export sample to using df.sample(100).to_csv('sample.csv')</li>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Shape of the data\n",
    "iris.shape "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Data type - 'object' means its a non-number(i.e. not int/float) type column e.g. string \n",
    "iris.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#ensuring data is unique at primary key(s) level\n",
    "iris.shape, iris[['Id']].drop_duplicates().shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#frequency of some key columns\n",
    "iris.Species.value_counts(dropna=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you do not use \"dropna = False\" above, the frequency for null values in the column will not be shown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#count of nulls in each column\n",
    "iris.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Some Basic stats\n",
    "iris.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above code gave descriptive statistics on numerical columns only. If you want the same for text columns, use the code below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.describe(include=['O'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Viewing data\n",
    "iris.sample(10) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use iris.tail() or iris head() if you want to see top of bottom rows only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.tail(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Slicing Dataframes using \"iloc\"</b><br>\n",
    "Indexing and slicing works pretty much the same way as python lists - remember lst[a:b:c] ?. Just that here you need to provide a,b,c to slice index for both rows and columns. Let's see some examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#gives everything\n",
    "iris.iloc[:,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#taking out the value at row index = 3, and column index = 3\n",
    "iris.iloc[3,3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#taking out the values at row index between 0 and 2, and column index between 0 and 2\n",
    "iris.iloc[0:3,0:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#taking out the values at row index between 0 and 2, and column index between 0 and 2, \n",
    "#while skipping 1 row and column\n",
    "iris.iloc[0:3:2,0:3:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>A better way of slicing Dataframes using \"loc\"</b><br>\n",
    "For \"iloc\" you need to provide the location or index of columns or rows. That becomes troublesome if have a huge dataset with many rows/columns and you do not know the index of a particular row or column that you want. For example in case of banking data you might have thousands of rows or columns, finding the index of lets say 'sales' column is hard. <br><br>\n",
    "So what can you do? You can use 'loc' and just ask pandas to display rows which satisfy particular condition(s) and also specify the columns by their names(i.e. not with index) within a list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.loc[iris.Species == 'Iris-setosa', ['SepalLengthCm','SepalWidthCm','Species']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can specify multiple conditions on row selection using ampersand \"&\" and pipe \"|\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.loc[(iris.Species == 'Iris-setosa') & \n",
    "         (iris.SepalWidthCm >= 3.1), ['SepalLengthCm','SepalWidthCm','Species']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice pandas labels the rows by default (see the very first column without any column name). That serves as the row name. It is NOT the index of rows, while it does look like it. Generally you will be using row names with 'loc' and will only use conditions on rows to filter data. Still here is one example of how it works"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#will print rows with names 1 to 4, NOT index 1 to 3 which happens in case of iloc\n",
    "iris.loc[1:4,[\"Species\",\"SepalWidthCm\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a code which shows you the difference between 'loc' and 'iloc' in using row names. Notice how 'loc' above printed rows 1 to 4 because it prints all row names mentioned, but in case of 'iloc' below , it does not print the row at index 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.iloc[1:4,0:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just do yourself a favor and avoid iloc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Getting column names\n",
    "print (iris.columns)\n",
    "print (type(iris.columns)) #Pandas data series is similar to lists in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Ways to get specific columns</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#prints column as series. Using head() to show only few values\n",
    "iris.Species.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#another similar way. This is a preferred way\n",
    "iris['Species'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#following prints the column as a dataframe. Notice the two square brackets\n",
    "iris[['Species']].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#The above method can be used to print multiple columns \n",
    "iris[['Species', 'SepalWidthCm']].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#same example as above using loc\n",
    "iris.loc[:, ['Species', 'SepalWidthCm']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"left\" id =\"pandas_manu\"> <a href=\"#pandas\"><font size=\"3\" color=\"black\"><b>Manipulating Dataframes</b></font></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While working with any dataset, a good practice is to convert all the column names to either upper or lower case to avoid any error due to Case-sensitive nature of Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.columns = iris.columns.str.upper()\n",
    "iris.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#You can rename columns by using 'rename' method and specifiying old and new names in a dictionary \n",
    "#and also what is it that you are renaming - columns or rows\n",
    "iris.rename({'PETALLENGTHCM':'PETALLENGTHCM_NEW',\n",
    "             'PETALWIDTHCM':'PETALWIDTHCM_NEW'},axis = 'columns').head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Let's see if iris dataset is changed or not\n",
    "iris.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait what?! Why didn't it rename but during print it showed new names? Answer: Any method in pandas generally does not make changes to the original dataset and only prints the output. You will need to store the output in a separate variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = iris.rename({'PETALLENGTHCM':'PETALLENGTHCM_NEW','PETALWIDTHCM':'PETALWIDTHCM_NEW'},axis = 'columns')\n",
    "iris.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or (a less preferred way) you can use an option called \"inplace = True\" which is available in most pandas methods to enforce changes in the dataset during the use of a method. For example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "iris.rename({'SEPALLENGTHCM':'SEPALLENGTHCM_NEW','SEPALWIDTHCM':'SEPALWIDTHCM_NEW'},\n",
    "            axis = 'columns',inplace = True)\n",
    "iris.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's revert back to original names. You can rename columns like this also, but it is not preferred if you have many columns in the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.columns = ['ID','SEPALLENGTHCM','SEPALWIDTHCM','PETALLENGTHCM','PETALWIDTHCM','SPECIES']\n",
    "iris.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#if you want to sort the values for which you are getting frequency\n",
    "iris.PETALLENGTHCM.value_counts(dropna=False).sort_index(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#if you want to sort by frequency\n",
    "iris.PETALLENGTHCM.value_counts(dropna=False).sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = iris.sort_values(by=['ID'], ascending = [False])\n",
    "# or you could have used below code\n",
    "#iris.sort_values(by=['ID'], ascending = [True], inplace=True)\n",
    "iris.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#You can also sort by multiple columns. \n",
    "#Specify the columns in 'by' option \n",
    "#Speficy the sort order in 'ascending' option\n",
    "iris = iris.sort_values(by=['PETALLENGTHCM', 'PETALWIDTHCM'], ascending = [False, True])\n",
    "iris.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Adding a new column with a static value\n",
    "iris['dummy'] = 1\n",
    "\n",
    "#Addng a new column using some calculation on other columns\n",
    "iris['SEPALLENGTHCM_SQ'] = iris['SEPALLENGTHCM'] * iris['SEPALLENGTHCM']\n",
    "iris['LENGTH_AVG'] = (iris['SEPALLENGTHCM'] + iris['PETALLENGTHCM'])/2\n",
    "iris['WIDTH_AVG'] = (iris['SEPALWIDTHCM'] + iris['PETALWIDTHCM'])/2\n",
    "\n",
    "iris.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Adding columns basis some condition - basically pandas's if-else version\n",
    "iris['new'] = 3\n",
    "iris.loc[iris.SPECIES=='Iris-setosa', 'new'] = 1\n",
    "iris.loc[iris.SPECIES=='Iris-virginica', 'new'] = 2\n",
    "iris.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#You can create cross-tabs between two columns too\n",
    "pd.crosstab(iris.SPECIES, iris.new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "reset_index() converts outputs like above to dataframe. Notice the difference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.crosstab(iris.SPECIES, iris.new).reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dropping Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = iris.drop(['dummy', 'new'], axis = 'columns')\n",
    "iris.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stacking two separate dataframes can be done using append() or concat()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df1 = pd.DataFrame([[1, 2], [3, 4]], columns=list('AB'))\n",
    "df2 = pd.DataFrame([[5, 6], [7, 8]], columns=list('AB'))\n",
    "print(df1)\n",
    "print()\n",
    "print(df2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df1.append(df2, ignore_index = True)\n",
    "#If you do not ignore index, pandas will save the original indexes from the two datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Stacking up side by side\n",
    "pd.concat([df1,df2.rename(columns={'A':'C', 'B':'D'})],axis='columns') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>SQL joins in pandas</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new = pd.DataFrame([['Iris-virginica', 'IV'],['Iris-setosa', 'IS']], columns = ['Flowers', 'ShortForm'])\n",
    "new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.SPECIES.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#inner join \n",
    "temp1 = iris.merge(right = new, #what you want to merge\n",
    "          how = 'inner', #how you want to merge i.e. type of join \n",
    "          left_on = ['SPECIES'], #join columns from left table\n",
    "          right_on = ['Flowers'] #join columns from right table\n",
    "          )\n",
    "temp1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(temp1.shape)\n",
    "temp1.ShortForm.value_counts(dropna=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#left join \n",
    "temp2 = iris.merge(right = new, #what you want to merge\n",
    "          how = 'left', #how you want to merge i.e. type of join \n",
    "          left_on = ['SPECIES'], #join columns from left table\n",
    "          right_on = ['Flowers'] #join columns from right table\n",
    "          )\n",
    "temp2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(temp2.shape)\n",
    "temp2.ShortForm.value_counts(dropna=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"left\" id =\"pandas_stats\"> <a href=\"#pandas\"><font size=\"3\" color=\"black\"><b>Statistics with Dataframes</b></font></a></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can apply certain functions on whole dataset by columns - like min(), max(), std(), mean(), nunique() etc. For Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can get specific percentiles using 'quantile()'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.quantile([0.0, 0.01, 0.50, 0.99, 1.0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can calculate correlations between numerical columns using corr()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "iris.corr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Missing value Treament"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Missing Value Treatment\n",
    "df = pd.DataFrame([[np.nan, 2, np.nan, 0],\n",
    "                    [3, 4, np.nan, 1],\n",
    "                    [np.nan, np.nan, np.nan, 5],\n",
    "                    [np.nan, 3, np.nan, 4]],\n",
    "                    columns=list('ABCD'))\n",
    "#Missing values in pandas are represented by NaN\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Fill nulls with some value\n",
    "df.fillna(0) #this doesn't replace df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#or just drop all rows with missing\n",
    "df.dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b> groupby() </b><br>\n",
    "One of the most important utility in pandas is it's groupby() function, which allows you to aggregate information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Aggregation for all columns\n",
    "iris.groupby(['SPECIES']).sum().reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Aggregation for specific columns\n",
    "iris.groupby(['SPECIES'])['LENGTH_AVG','WIDTH_AVG'].sum().reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#more than one aggregation \n",
    "iris.groupby(['SPECIES'])['LENGTH_AVG','WIDTH_AVG'].agg(['min','max', 'mean']).reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#different aggregation for different columns\n",
    "iris.groupby(['SPECIES']).agg({'LENGTH_AVG':'min', 'WIDTH_AVG':'max'}).reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Get unique values in a column\n",
    "iris.SPECIES.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Get number of unique values in a column\n",
    "iris.SPECIES.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"left\" id=\"mat\"><a href=\"#contents\"><font size=\"4\" color=\"red\"><b>MATPLOTLIB.PYPLOT</b></font></a></div>\n",
    "<ol type=\"square\">\n",
    "    <li><a href=\"#mat_basic\">Basic Plotting</a></li>\n",
    "    <li><a href=\"#mat_mult\">Multiple plots</a></li>\n",
    "    <li><a href=\"#mat_pix\">Plotting Pixels</a></li>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Matplotlib is a plotting library for the Python programming language and its numerical mathematics extension NumPy. matplotlib.pyplot is a collection of command style functions that make matplotlib work like MATLAB. Each pyplot function makes some change to a figure: e.g., creates a figure, creates a plotting area in a figure, plots some lines in a plotting area, decorates the plot with labels, etc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"left\" id =\"mat_basic\"> <a href=\"#mat\"><font size=\"3\" color=\"black\"><b>Basic Plotting</b></font></a></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot([1, 2, 3, 4], [1, 4, 9, 16])\n",
    "plt.ylabel('some numbers')\n",
    "plt.show() #You always need to use show() at the end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot([1, 2, 3, 4], [1, 4, 9, 16], 'ro') #ro => r = red, 0 = points\n",
    "plt.ylabel('some numbers')\n",
    "plt.show() #You always need to use show() at the end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# evenly sampled time at 200ms intervals\n",
    "t = np.arange(0., 5., 0.2)\n",
    "\n",
    "# red dashes, blue squares and green triangles\n",
    "plt.plot(t, t, 'r--', t, t**2, 'bs', t, t**3, 'g^') #red dashes, blue squares, green triangles\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Histogram\n",
    "mu, sigma = 100, 15\n",
    "x = mu + sigma*np.random.randn(10000)\n",
    "plt.hist(x, bins=100, facecolor='orange')  #try changing value of bins parameter\n",
    "plt.xlabel('Smarts')\n",
    "plt.ylabel('Probability')\n",
    "plt.title('Histogram of IQ')\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"left\" id =\"mat_mult\"> <a href=\"#mat\"><font size=\"3\" color=\"black\"><b>Multiple Plots</b></font></a></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.linspace(0.0, 2*np.pi, 100)\n",
    "_sin = np.sin(x)\n",
    "_cos = np.cos(x)\n",
    "\n",
    "plt.subplot(2, 1, 1)\n",
    "plt.plot(x, _sin, 'y*', linewidth = 1)\n",
    "plt.title('A tale of 2 subplots')\n",
    "plt.ylabel('sin curve')\n",
    "\n",
    "plt.subplot(2, 1, 2)\n",
    "plt.plot(x, _cos, 'r--')\n",
    "plt.xlabel('x')\n",
    "plt.ylabel('cos curve')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"left\" id =\"mat_pix\"> <a href=\"#mat\"><font size=\"3\" color=\"black\"><b>Plotting Pixels</b></font></a></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = np.random.random(size=(3, 3))\n",
    "print(data)\n",
    "plt.imshow(data, interpolation='nearest')\n",
    "plt.xticks(np.arange(0.0, 2.5, 1), np.arange(0.5, 2, 0.5))\n",
    "plt.yticks(np.arange(2, -0.5, -1), np.arange(0.5, 2, 0.5))\n",
    "plt.title('The value of pixels refer to the color')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We encourage you to explore plot() function <a href=\"https://matplotlib.org/api/_as_gen/matplotlib.pyplot.plot.html\">here</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size = 10 color=\"green\">\"</font><br>\n",
    "<div align = center> \n",
    "    <font size = 5 color=\"green\" style=\"font-family: 'Comic Sans MS'\">Pass on what you have learned</font><br>\n",
    "</div>\n",
    "<div align = right> \n",
    "    <font size = 10 color=\"green\"t>\"</font><br>\n",
    "</div>\n",
    "<img align = \"right\" src = \"https://orig00.deviantart.net/ff5e/f/2013/102/4/4/yoda_by_valval-d61g9rh.png\" width=\"20%\" height=\"20%\">"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}