Lecture 5 - Overview of Data Science.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border:5px outset; width:75%; margin:auto\">\n",
    "<h1 style=\"text-align:center\">Contents Overview</h1>\n",
    "<ul>\n",
    "    <li><h5><a href=\"#c0\">0: Preface</a></h5></li>\n",
    "    <li><h5><a href=\"#c1\">1: What is Data Science?</a></h5></li>\n",
    "    <li><h5><a href=\"#c2\">2: Workflow</a></h5></li>\n",
    "    <li><h5><a href=\"#c3\">3: Goals & Applications</a></h5></li>\n",
    "    <li><h5><a href=\"#c4\">4: Landscape</a></h5></li>\n",
    "    <li><h5><a href=\"#c5\">5: Skillset</a></h5></li>\n",
    "</ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border:2px solid black\" />\n",
    "<h1><a id=\"c0\">Preface</a></h1>\n",
    "<div style=\"padding: 5% 5% 5% 5%\">\n",
    "    <p>This lecture aims to take a high-level view at data science to help put the field into perspective for those seeking to move into the field of data. The goal is to give students a better sense of what may interest them and which areas of the field they wish to pursue.</p>\n",
    "    <p>The details in this lecture are not required to proceed with Lantern's other courses, however, I recommend all student review Chapter 2: Workflow as it will aid students in completing data analysis projects in the succeeding courses. Understanding the process and workflow of data analysis and data science will ensure students have a mental map to keep them on track as they progress through the work.</p> \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border:2px solid black\" />\n",
    "<h1><a id=\"c1\">Chapter 1: What is Data Science?</a></h1>\n",
    "<div style=\"padding: 5% 5% 5% 5%\">\n",
    "    <p>\"Data Science\" is an incredibly broad term which is often used in a variety of work involving datasets. In the modern business space it has become a buzzword along side terms like \"big data\" and \"AI\", however in the most basic terms, data science is a field of study similar to mathematics or computer science. Every field has its area of interest and expertise while also being applicable to any number of jobs and tasks. For example, statistics is its own field of study but it's also used in every other field that works with data (e.g. the sciences & engineering); it's not possible to perform rigourous research without the proper use of statistics.</p>\n",
    "    <img src=\"images/5.1_data_science_disciplines.png\" width=550 />\n",
    "    <p><b>Data science</b> is an inter-disciplinary field of study that combines domain/business expertise, scientific methods, programming skills, and knowledge of statistics to extract meaningful insights from structured and unstructured data, and apply that knowledge across a range of applications.</p>\n",
    "    <img src=\"images/5.1_venn_diagram.png\" width=550 />\n",
    "    <p>The four main pillars of a data scientist are business/domain, computer science, mathematics, and communication. The perfect data scientist should be well versed in all four pillars, in reality people are often great at only one or two and sub-par at the rest. The necessities of a data scientist will also vary quite a bit based on the company they're working for, the goal of their project, and their area of expertise.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border:2px solid black\" />\n",
    "<h1><a id=\"c2\">Chapter 2: Workflow</a></h1>\n",
    "<div style=\"padding: 5% 5% 5% 5%\">\n",
    "    <p>Whenever entering a new technical field it can be useful to map out the process or workflow of the field/job to get a sense of what the work entails and which skills will be needed for different parts. When it comes to something as broad as data science you will come across many varying workflows that different people have mapped out but the core workflow will be the same across all maps (some just focus more on the science and other on the business).</p>\n",
    "    <p>The following a diagram of the GABDO process model that it typically followed, especially when creating models using machine learning and related techniques.</p>\n",
    "    <img src=\"images/5.2_workflow.jpg\" width=650 />\n",
    "    <p>The model consists of five iterative phases: Goals, Acquire, Build, Deliver, and Optimize.</p>\n",
    "    <p>Goals</p>\n",
    "    <ul>\n",
    "        <li>The first step is to identify the goals of the project. These goals are typically born out of business or research needs and interest. Broad goals are typically set by higher-level executives which are then broken down into more granular goals that can be pursued.</li>\n",
    "        <li>The next step is to identitfy potential opportunities to use data to achive the specific goals. This process involves asking lots of questions of what is available, what is needed, and how goals can be achieved.</li>\n",
    "        <li>To round off our project definition we create a hypothesis that we can test as part of our project. Setting the hypothesis will help validate that the opportunities are viable and feasible.</li>\n",
    "    </ul>\n",
    "    <p>Acquire</p>\n",
    "    <ul>\n",
    "        <li>Once our project is defined we need to acquire the necessary data to build our solution. This starts with identifying available and useful data sources. This can include internal or external sources and the tools needed to access the data (e.g. API, SQL, etc.). If the data isn't readily available this step may also include identifying ways to generate and collect new data.</li>\n",
    "        <li>Next we need to actually acquire the data. Depending on the complexity of the project this process may involve integrating data from various sources into a uniform data store (e.g. local computer, cloud database, data warehouse) that is better suited for efficient access and analysis.</li>\n",
    "        <li>Lastly the data needs to be prepared for use. This step can include a number of steps including data cleaning, wrangling, feature selection, and feature engineering.</li>\n",
    "    </ul>\n",
    "    <p>Build</p>\n",
    "    <ul>\n",
    "        <li>With the data prepared and ready to go we need to start by understanding the data being used for our hypothesis test. This is done through exploratory data analysis (EDA) using tools like descriptive statistics, summary statistics, and data visualizations. The objective of our exploration is to see the extent of our data (what is here and what is missing) as well as any potential relationships, correlations, and issues with the data.</li>\n",
    "        <li>The next step is to make the appropriate selections for each hypothesis we wish to test. This includes model/algorithm selection, performance metric selection, and initial model parameter selection. Often multiple models are chosen, tested, and compared to each other to achieve the best performance.</li>\n",
    "        <li>Training, validating, and testing our model is the next part of the process. The data we are working with to test the hypothesis is often split into train-test or train-validate-test sets using a 70/30 or 60/20/20 split. The train subset is then used by the model to learn, i.e. update the parameters to fit the data and application. The validate subset is then used to confirm the model on new data (that wasn't used in training) and validate its performance impact of hyperparameter tuning. Once all of the training and validating is complete the model's performance is measured using the test subset.</li>\n",
    "        <li>Finally, the performance of promising model can be improved further through fine-tuning of hyperparameters, dataset refinement, feature selection, feature engineering, and other techniques.</li>\n",
    "    </ul>\n",
    "    <p>Deliver</p>\n",
    "    <ul>\n",
    "        <li>The next phase of the process is to deliver the results (this may vary by project). Often when data is analysis for stakeholders the insights found need to be presented through a variety of methods such as presentation, reports, dashboards, or mobile apps.</li>\n",
    "        <li>Action can then be taken based on relevant actionable insights that were generated through the project. This can include making data-informed and data-driven decision or deploying standalone applications or integrated products.</li>\n",
    "    </ul>\n",
    "    <p>Optimize</p>\n",
    "    <ul>\n",
    "        <li>In the case of deployed AI solutions, the last part of the process is to optimize the performance over time as the application and its use grows. This starts by monitoring the solution and its performance over time, analyzing the results it produces, and continuously improving and optimizing its function for the changing environment.</li>\n",
    "    </ul>\n",
    "    <p>Note: visual maps are a great tool in the learning process however they should always be taken with a grain of salt. No static map is capable of showing all information and as a result they are typically oversimplified and subjective to the opinions of their creators. I like to use maps to get a picture of the branches and processes that are available (and not focus on the exact relations that the map may depict). Learning roadmaps can also be useful for beginners (I encourage you to look up data science roadmaps and compare some of them) as they list, in varying degrees of detail, the subject areas and topics of a field. This lets students get a bigger picture and keep track of what they need to study, however the order used in roadmaps is completely arbitrary and I encourage students to study topics in the order that interests and works best for them.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border:2px solid black\" />\n",
    "<h1><a id=\"c3\">Chapter 3: Goals & Applications</a></h1>\n",
    "<div style=\"padding: 5% 5% 5% 5%\">\n",
    "    <img src=\"images/5.3_prediction.png\" width=450 />\n",
    "    <p>To better understand the disciplines and processes that happen in the field of data science, it can beneficial to familiarize yourself with the typical goals and deliverables associated with data science projects. Here's a list of common goals:</p>\n",
    "    <ul>\n",
    "        <li>Prediction (predict values based on inputs</li>\n",
    "        <li>Classification (e.g. spam or not spam)</li>\n",
    "        <li>Recommendations (e.g. products to buy or next movie to watch)</li>\n",
    "        <li>Pattern detection and grouping</li>\n",
    "        <li>Anomaly detection (e.g. fraud detection)</li>\n",
    "        <li>Recognition (e.g. image, text, audio, video, facial)</li>\n",
    "        <li>Actionable insights (via dashboards, reports, visuals)</li>\n",
    "        <li>Automated processes and decision-making</li>\n",
    "        <li>Scoring and ranking</li>\n",
    "        <li>Segmentation (e.g. demographic-based marketing)</li>\n",
    "        <li>Optimization (e.g. risk management)</li>\n",
    "        <li>Forecasts (e.g. sales and revenue)</li>\n",
    "    </ul>\n",
    "    <img src=\"images/5.3_recommendations.png\" width=550 />\n",
    "    <p>There are no doubt hundreds of great examples, you can look up online, for each of the goals, that have already been implemented in the past.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border:2px solid black\" />\n",
    "<h1><a id=\"c4\">Chapter 4: Landscape</a></h1>\n",
    "<div style=\"padding: 5% 5% 5% 5%\">\n",
    "    <p>Client, Product Owner, Project Manager, Team Lead:</p>\n",
    "    <ul>\n",
    "        <li>A subject matter expert in their field.</li>\n",
    "        <li>Provides high-level information about the problem.</li>\n",
    "        <li>Defines acceptance criteria for the end solution/product.</li>\n",
    "    </ul>\n",
    "    <p>Data Engineers</p>\n",
    "    <ul>\n",
    "        <li>They Extract, Transform, and Load (ETL) data into data warehouses.</li>\n",
    "        <li>Ingest or prepare different data sources for Data Scientists.</li>\n",
    "        <li>They create data pipelines.</li>\n",
    "        <li>Use Big Data tools like Sqoop, Kafka, etc.</li>\n",
    "    </ul>\n",
    "    <p>DevOps</p>\n",
    "    <ul>\n",
    "        <li>They help Data Scientists to put their models into production.</li>\n",
    "        <li>Spin up clusters.</li>\n",
    "        <li>Run tests.</li>\n",
    "        <li>Monitor builds and pipelines.</li>\n",
    "    </ul>\n",
    "    <p>Data Analysts</p>\n",
    "    <ul>\n",
    "        <li>Create dashboards and reports.</li>\n",
    "        <li>Create charts and summary tables.</li>\n",
    "        <li>Find meadningful patterns and insights in the data.</li>\n",
    "        <li>Run tests and analyze and compare results.</li>\n",
    "        <li>Use SQL and access data.</li>\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"border:2px solid black\" />\n",
    "<h1><a id=\"c5\">Chapter 5: Skillset</a></h1>\n",
    "<div style=\"padding: 5% 5% 5% 5%\">\n",
    "    <img src=\"images/5.5_tools.webp\" width=550 />\n",
    "    <ul>\n",
    "        <li>A bare minimum knowledge of the subject, good enough to understand the problem.</li>\n",
    "        <ul>\n",
    "            <li>Banking</li>\n",
    "            <li>Pharmaceutical</li>\n",
    "            <li>Fraud</li>\n",
    "            <li>Stocks</li>\n",
    "            <li>Marketing</li>\n",
    "        </ul>\n",
    "        <li>Being able to run queries and access data (most of the time big data) from databases.</li>\n",
    "        <ul>\n",
    "            <li>SQL</li>\n",
    "            <li>Spark</li>\n",
    "            <li>Hive</li>\n",
    "        </ul>\n",
    "        <li>Knowledge of Python programming as one of the most popular Data Science languages.</li>\n",
    "        <ul>\n",
    "            <li>Numpy</li>\n",
    "            <li>Pandas</li>\n",
    "            <li>Matplotlib</li>\n",
    "            <li>Scikit-learn</li>\n",
    "        </ul>\n",
    "        <li>Ability to visualize and explore data to find patterns, outliers, etc.</li>\n",
    "        <li>Know how to clean and preprocess your data.</li>\n",
    "        <li>Knowledge of Feature Engineering, tranform data to ML features.</li>\n",
    "        <ul>\n",
    "            <li>Facial recognition: colors, pixel location, edges, etc.</li>\n",
    "            <li>NLP: text, words, word frequency, etc.</li>\n",
    "            <li>Marketing: time of click, number of clicks, average clicks, etc.</li>\n",
    "        </ul>\n",
    "        <li>Knowledge of Machine Learning (ML) algorithms.</li>\n",
    "        <ul>\n",
    "            <li>Clustering</li>\n",
    "            <li>Classification</li>\n",
    "            <li>Regression</li>\n",
    "            <li>Neural Networks</li>\n",
    "            <li>Decision Trees</li>\n",
    "        </ul>\n",
    "        <li>Basic knowledge of statistics, probability, etc.</li>\n",
    "        <li>Know how to run experiements and analyse outcomes.</li>\n",
    "        <li>Familiarity with cloud platforms.</li>\n",
    "        <ul>\n",
    "            <li>Amazon Web Services</li>\n",
    "            <li>Microsoft Azure</li>\n",
    "            <li>Google Cloud Platform</li>\n",
    "        </ul>\n",
    "    </ul>\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}