2_Data_Science_Basic_Concepts.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python for Data Science"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Joe McCarthy](http://interrelativity.com/joe), \n",
    "*Data Scientist*, [Indeed](http://www.indeed.com/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from IPython.display import display, Image, HTML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Navigation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notebooks in this primer:\n",
    "\n",
    "1. [Introduction](1_Introduction.ipynb)\n",
    "2. **Data Science: Basic Concepts** (*you are here*)\n",
    "3. [Python: Basic Concepts](3_Python_Basic_Concepts.ipynb)\n",
    "4. [Using Python to Build and Use a Simple Decision Tree Classifier](4_Python_Simple_Decision_Tree.ipynb)\n",
    "5. [Next Steps](5_Next_Steps.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Data Science: Basic Concepts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Science and Data Mining"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"http://data-science-for-biz.com/\"><img src=\"http://akamaicovers.oreilly.com/images/0636920028918/cat.gif\" style=\"margin: 0px 0px 5px 20px; width: 125px; float: right;\" title=\"Data Science for Business, by Provost and Fawcett\" alt=\"DataScienceForBusiness_cover.jpg\" /></a>\n",
    "Foster Provost and [Tom Fawcett](http://home.comcast.net/~tom.fawcett/public_html/index.html) offer succinct descriptions of data science and data mining in [Data Science for Business](http://data-science-for-biz.com/):\n",
    "\n",
    "> **Data science** involves principles, processes and techniques for understanding phenomena via the (automated) analysis of data.\n",
    "> \n",
    "> **Data mining** is the extraction of knowledge from data, via technologies that incorporate these principles."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Knowledge Discovery, Data Mining and Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Provost & Fawcett also offer some history and insights into the relationship between *data mining* and *machine learning*, terms which are often used somewhat interchangeably:\n",
    "\n",
    "> The field of Data Mining (or KDD: Knowledge Discovery and Data Mining) started as an offshoot of Machine Learning, and they remain closely linked. Both fields are concerned with the analysis of data to find useful or informative patterns. Techniques and algorithms are shared between the two; indeed, the areas are so closely related that researchers commonly participate in both communities and transition between them seamlessly. Nevertheless, it is worth pointing out some of the differences to give perspective.\n",
    "> \n",
    ">Speaking generally, because Machine Learning is concerned with many types of performance improvement, it includes subfields such as robotics and computer vision that are not part of KDD. It also is concerned with issues of agency and cognition — how will an intelligent agent use learned knowledge to reason and act in its environment — which are not concerns of Data Mining.\n",
    "> \n",
    ">Historically, KDD spun off from Machine Learning as a research field focused on concerns raised by examining real-world applications, and a decade and a half later the KDD community remains more concerned with applications than Machine Learning is. As such, research focused on commercial applications and business issues of data analysis tends to gravitate toward the KDD community rather than to Machine Learning. KDD also tends to be more concerned with the entire process of data analytics: data preparation, model learning, evaluation, and so on.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross Industry Standard Process for Data Mining (CRISP-DM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [Cross Industry Standard Process for Data Mining](https://en.wikipedia.org/wiki/Cross_Industry_Standard_Process_for_Data_Mining) introduced a process model for data mining in 2000 that has become widely adopted.\n",
    "\n",
    "<a href=\"https://en.wikipedia.org/wiki/Cross_Industry_Standard_Process_for_Data_Mining\"><img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/b/b9/CRISP-DM_Process_Diagram.png/479px-CRISP-DM_Process_Diagram.png\" title=\"Cross Industry Standard Process for Data Mining\" alt=\"CRISP-DM_Process_Diagram\" /></a>\n",
    "\n",
    "The model emphasizes the ***iterative*** nature of the data mining process, distinguishing several different stages that are regularly revisited in the course of developing and deploying data-driven solutions to business problems:\n",
    "\n",
    "* Business understanding\n",
    "* Data understanding\n",
    "* Data preparation\n",
    "* Modeling \n",
    "* Deployment\n",
    "\n",
    "We will be focusing primarily on using Python for **data preparation** and **modeling**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Science Workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Philip Guo](http://www.pgbovine.net/) presents a [Data Science Workflow](http://cacm.acm.org/blogs/blog-cacm/169199-data-science-workflow-overview-and-challenges/fulltext) offering a slightly different process model emhasizing the importance of **reflection** and some of the meta-data, data management and bookkeeping challenges that typically arise in the data science process. His 2012 PhD thesis, [Software Tools to Facilitate Research Programming](http://pgbovine.net/projects/pubs/guo_phd_dissertation.pdf), offers an insightful and more comprehensive description of many of these challenges.\n",
    "\n",
    "<a href=\"http://cacm.acm.org/blogs/blog-cacm/169199-data-science-workflow-overview-and-challenges/fulltext\"><img src=\"http://cacm.acm.org/system/assets/0001/3678/rp-overview.jpg\" title=\"Data Science Workflow, by Philip Guo\" alt=\"pguo-data-science-overview.jpg\" style=\"width: 500px\" /></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Provost & Fawcett list a number of different tasks in which data science techniques are employed:\n",
    "\n",
    "* Classification and class probability estimation \n",
    "* Regression (aka value estimation) \n",
    "* Similarity matching \n",
    "* Clustering \n",
    "* Co-occurrence grouping (aka frequent itemset mining, association rule discovery, market-basket analysis) \n",
    "* Profiling (aka behavior description, fraud / anomaly detection) \n",
    "* Link prediction \n",
    "* Data reduction \n",
    "* Causal modeling \n",
    "\n",
    "We will be focusing primarily on **classification** and **class probability estimation** tasks, which are defined by Provost & Fawcett as follows:\n",
    "\n",
    "> *Classification* and *class probability estimation* attempt to predict, for each individual in a population, which of a (small) set of classes this individual belongs to. Usually the classes are mutually exclusive. An example classification question would be: “Among all the customers of MegaTelCo, which are likely to respond to a given offer?” In this example the two classes could be called will respond and will not respond.\n",
    "\n",
    "To further simplify this primer, we will focus exclusively on **supervised** methods, in which the data is explicitly labeled with classes. There are also *unsupervised* methods that involve working with data in which there are no pre-specified class labels."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Supervised Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [Natural Language Toolkit (NLTK) book](http://www.nltk.org/book) provides a diagram and succinct description (below, with italics and bold added for emphasis) of supervised classification:\n",
    "\n",
    "<a href=\"http://www.nltk.org/book/ch06.html\"><img src=\"http://www.nltk.org/images/supervised-classification.png\" title=\"Supervised Classification, from NLTK book, Chapter 6\" alt=\"nltk_ch06_supervised-classification.png\" style=\"width: 500px\" /></a>\n",
    "\n",
    "> *Supervised Classification*. (a) During *training*, a **feature extractor** is used to convert each **input value** to a **feature set**. These feature sets, which capture the basic information about each input that should be used to classify it, are discussed in the next section. Pairs of feature sets and **labels** are fed into the **machine learning algorithm** to generate a **model**. (b) During *prediction*, the same feature extractor is used to convert **unseen inputs** to feature sets. These feature sets are then fed into the model, which generates **predicted labels**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Mining Terminology"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Structured** data has simple, well-defined patterns (e.g., a table or graph)\n",
    "* **Unstructured** data has less well-defined patterns (e.g., text, images)\n",
    "* **Model**: a pattern that captures / generalizes regularities in data (e.g., an equation, set of rules, decision tree)\n",
    "* **Attribute** (aka *variable*, *feature*, *signal*, *column*): an element used in a model\n",
    "* **Instance** (aka *example*, *feature vector*, *row*): a representation of a single entity being modeled\n",
    "* **Target attribute** (aka *dependent variable*, *class label*): the class / type / category of an entity being modeled"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Mining Example: UCI Mushroom dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [Center for Machine Learning and Intelligent Systems](http://cml.ics.uci.edu/) at the University of California, Irvine (UCI), hosts a  [Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets.html) containing over 200 publicly available data sets.\n",
    "\n",
    "<a href=\"https://archive.ics.uci.edu/ml/datasets/Mushroom\"><img src=\"https://archive.ics.uci.edu/ml/assets/MLimages/Large73.jpg\"  style=\"margin: 0px 0px 5px 20px; width: 125px; float: right;\" title=\"Mushrooms from Agaricus and Lepiota Family\" alt=\"mushroom\"/></a>\n",
    "We will use the [mushroom](https://archive.ics.uci.edu/ml/datasets/Mushroom) data set, which forms the basis of several examples in Chapter 3 of the Provost & Fawcett data science book.\n",
    "\n",
    "The following description of the dataset is provided at the UCI repository:\n",
    "\n",
    ">This data set includes descriptions of hypothetical samples corresponding to 23 species of gilled mushrooms in the Agaricus and Lepiota Family (pp. 500-525 [The Audubon Society Field Guide to North American Mushrooms, 1981]). Each species is identified as definitely edible, definitely poisonous, or of unknown edibility and not recommended. This latter class was combined with the poisonous one. The Guide clearly states that there is no simple rule for determining the edibility of a mushroom; no rule like leaflets three, let it be'' for Poisonous Oak and Ivy.\n",
    "> \n",
    "> **Number of Instances**: 8124\n",
    "> \n",
    "> **Number of Attributes**: 22 (all nominally valued)\n",
    "> \n",
    "> **Attribute Information**: (*classes*: edible=e, poisonous=p)\n",
    "> \n",
    "> 1. *cap-shape*: bell=b, conical=c, convex=x, flat=f, knobbed=k, sunken=s\n",
    "> 2. *cap-surface*: fibrous=f, grooves=g, scaly=y, smooth=s\n",
    "> 3. *cap-color*: brown=n ,buff=b, cinnamon=c, gray=g, green=r, pink=p, purple=u, red=e, white=w, yellow=y\n",
    "> 4. *bruises?*: bruises=t, no=f\n",
    "> 5. *odor*: almond=a, anise=l, creosote=c, fishy=y, foul=f, musty=m, none=n, pungent=p, spicy=s\n",
    "> 6. *gill-attachment*: attached=a, descending=d, free=f, notched=n\n",
    "> 7. *gill-spacing*: close=c, crowded=w, distant=d\n",
    "> 8. *gill-size*: broad=b, narrow=n\n",
    "> 9. *gill-color*: black=k, brown=n, buff=b, chocolate=h, gray=g, green=r, orange=o, pink=p, purple=u, red=e, white=w, yellow=y\n",
    "> 10. *stalk-shape*: enlarging=e, tapering=t\n",
    "> 11. *stalk-root*: bulbous=b, club=c, cup=u, equal=e, rhizomorphs=z, rooted=r, missing=?\n",
    "> 12. *stalk-surface-above-ring*: fibrous=f, scaly=y, silky=k, smooth=s\n",
    "> 13. *stalk-surface-below-ring*: fibrous=f, scaly=y, silky=k, smooth=s\n",
    "> 14. *stalk-color-above-ring*: brown=n, buff=b, cinnamon=c, gray=g, orange=o, pink=p, red=e, white=w, yellow=y\n",
    "> 15. *stalk-color-below-ring*: brown=n, buff=b, cinnamon=c, gray=g, orange=o, pink=p, red=e, white=w, yellow=y\n",
    "> 16. *veil-type*: partial=p, universal=u\n",
    "> 17. *veil-color*: brown=n, orange=o, white=w, yellow=y\n",
    "> 18. *ring-number*: none=n, one=o, two=t\n",
    "> 19. *ring-type*: cobwebby=c, evanescent=e, flaring=f, large=l, none=n, pendant=p, sheathing=s, zone=z\n",
    "> 20. *spore-print-color*: black=k, brown=n, buff=b, chocolate=h, green=r, orange=o, purple=u, white=w, yellow=y\n",
    "> 21. *population*: abundant=a, clustered=c, numerous=n, scattered=s, several=v, solitary=y\n",
    "> 22. *habitat*: grasses=g, leaves=l, meadows=m, paths=p, urban=u, waste=w, woods=d\n",
    "> \n",
    "> **Missing Attribute Values**: 2480 of them (denoted by \"?\"), all for attribute #11.\n",
    "> \n",
    "> **Class Distribution**: -- edible: 4208 (51.8%) -- poisonous: 3916 (48.2%) -- total: 8124 instances\n",
    "\n",
    "The [data file](https://archive.ics.uci.edu/ml/machine-learning-databases/mushroom/agaricus-lepiota.data) associated with this dataset has one instance of a hypothetical mushroom per line, with abbreviations for the values of the class and each of the other 22 attributes separated by commas.\n",
    "\n",
    "Here is a sample line from the data file:\n",
    "\n",
    "p,k,f,n,f,n,f,c,n,w,e,?,k,y,w,n,p,w,o,e,w,v,d\n",
    "\n",
    "This instance represents a mushroom with the following attribute values (highlighted in **bold**):\n",
    "\n",
    "*class*: edible=e, **poisonous=p**\n",
    "\n",
    "1. *cap-shape*: bell=b, conical=c, convex=x, flat=f, **knobbed=k**, sunken=s\n",
    "2. *cap-surface*: **fibrous=f**, grooves=g, scaly=y, smooth=s\n",
    "3. *cap-color*: **brown=n** ,buff=b, cinnamon=c, gray=g, green=r, pink=p, purple=u, red=e, white=w, yellow=y\n",
    "4. *bruises?*: bruises=t, **no=f**\n",
    "5. *odor*: almond=a, anise=l, creosote=c, fishy=y, foul=f, musty=m, **none=n**, pungent=p, spicy=s\n",
    "6. *gill-attachment*: attached=a, descending=d, **free=f**, notched=n\n",
    "7. *gill-spacing*: **close=c**, crowded=w, distant=d\n",
    "8. *gill-size*: broad=b, **narrow=n**\n",
    "9. *gill-color*: black=k, brown=n, buff=b, chocolate=h, gray=g, green=r, orange=o, pink=p, purple=u, red=e, **white=w**, yellow=y\n",
    "10. *stalk-shape*: **enlarging=e**, tapering=t\n",
    "11. *stalk-root*: bulbous=b, club=c, cup=u, equal=e, rhizomorphs=z, rooted=r, **missing=?**\n",
    "12. *stalk-surface-above-ring*: fibrous=f, scaly=y, **silky=k**, smooth=s\n",
    "13. *stalk-surface-below-ring*: fibrous=f, **scaly=y**, silky=k, smooth=s\n",
    "14. *stalk-color-above-ring*: brown=n, buff=b, cinnamon=c, gray=g, orange=o, pink=p, red=e, **white=w**, yellow=y\n",
    "15. *stalk-color-below-ring*: **brown=n**, buff=b, cinnamon=c, gray=g, orange=o, pink=p, red=e, white=w, yellow=y\n",
    "16. *veil-type*: **partial=p**, universal=u\n",
    "17. *veil-color*: brown=n, orange=o, **white=w**, yellow=y\n",
    "18. *ring-number*: none=n, **one=o**, two=t\n",
    "19. *ring-type*: cobwebby=c, **evanescent=e**, flaring=f, large=l, none=n, pendant=p, sheathing=s, zone=z\n",
    "20. *spore-print-color*: black=k, brown=n, buff=b, chocolate=h, green=r, orange=o, purple=u, **white=w**, yellow=y\n",
    "21. *population*: abundant=a, clustered=c, numerous=n, scattered=s, **several=v**, solitary=y\n",
    "22. *habitat*: grasses=g, leaves=l, meadows=m, paths=p, urban=u, waste=w, **woods=d**\n",
    "\n",
    "Building a model with this data set will serve as a motivating example throughout much of this primer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Navigation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notebooks in this primer:\n",
    "\n",
    "1. [Introduction](1_Introduction.ipynb)\n",
    "2. **Data Science: Basic Concepts** (*you are here*)\n",
    "3. [Python: Basic Concepts](3_Python_Basic_Concepts.ipynb)\n",
    "4. [Using Python to Build and Use a Simple Decision Tree Classifier](4_Python_Simple_Decision_Tree.ipynb)\n",
    "5. [Next Steps](5_Next_Steps.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}