AB_Testing_with_Python.ipynb

{
  "cells": [
    {
      "metadata": {
        "_cell_guid": "19088ede-8fec-494e-bc1c-46a78211478c",
        "_uuid": "e0d4dfc3eb59412c9046298044b673bcb1f32746"
      },
      "cell_type": "markdown",
      "source": "# AB Testing With Python - Walkthrough Udacity's Course Final Project\n *By Tammy Rotem*\n1. [About Udacity's A/B Testing Course](#about) <br>\n2. [Experiment Overview](#overview)<br>\n2.1 [Current Conditions Before Change](#current)<br>\n2.2 [Description of Experimented Change](#description)<br>\n2.3 [Experiment Hypothesis](#hypothesis)<br>\n2.4 [Experiment Details](#details)<br>\n3. [Metric Choice](#metricchoice)<br>\n3.1 [Invariate Metrics - Sanity Checks](#invariate)<br>\n3.2 [Evaluation Metrics - Performance Indicators](#evaluation)<br>\n4. [Estimating the baseline values of metrics](#baseline)<br>\n4.1 [Collecting estimators data](#collect)<br>\n4.2 [Estimating Stadard Deviation](#sd)<br>\n4.2.1 [Scaling Collected Data](#scale)<br>\n4.2.2 [Estimating Analytically](#estimate)<br>\n5. [Experiment Sizing](#sizing)<br>\n5.1 [Get Z-score critical value and Standard Deviations](#z)<br>\n5.2 [Calculate Sample Size per Metric](#calc) <br>\n6. [Analyzing Collected Data](#analysis)<br>\n6.1 [Loading collected data](#collect_results)<br>\n6.2 [Sanity Checks](#sanity)<br>\n6.2.1 [Sanity Checks for difference between Counts](#check_counts)<br>\n6.2.2 [Sanity Checks for difference between probabilities](#check_probs)<br>\n6.3 [Measuring effect size](#effect)<br>\n6.4 [Double check with Sign Tests](#sign_tests)<br>\n6.4.1 [Data Preparation](#prep)<br>\n6.4.2 [Building a Sign Test](#sign)<br>\n7. [Conclusions & Recommendations](#conclusions)<br>"
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "9a04b6ea-111a-4627-849f-53adf7efd40b",
        "_uuid": "f4949401e2dab760c957525f17e90addf45fc9e8",
        "trusted": false
      },
      "cell_type": "code",
      "source": "import math as mt\nimport numpy as np\nimport pandas as pd\nfrom scipy.stats import norm",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "27cb4b93-34ec-4469-9963-cc61ab80b193",
        "_uuid": "f5f7c36658e5469932f8c5cd84a1d1a4b0eff924"
      },
      "cell_type": "markdown",
      "source": "## 1 Udacity's A/B Testing Course <a class=\"anchor\" id=\"about\"></a>\n[Udacity](https://www.udacity.com/) published a great free course for [A/B Testing](https://en.wikipedia.org/wiki/A/B_testing), also known as split tests, which are online experiments used to test potential improvements to a website or mobile app.\nThis Python notebook is a walkthrough solution of the final project.\n\nUdacity's [AB Testing course](https://www.udacity.com/course/ab-testing--ud257) is presented by Google and focuses on design and analysis of A/B tests. The course covers how to choose and characterize metrics to evaluate your experiments, how to design an experiment with enough statistical power and how to analyze the results and draw valid conclusions."
    },
    {
      "metadata": {
        "_cell_guid": "82414b3c-6e0c-43e8-9cf8-51303729f493",
        "_uuid": "5f4067cba7cfdf087ee2d5cd7bd3333e2d63f6b1"
      },
      "cell_type": "markdown",
      "source": "## 2 Experiment Overview <a class=\"anchor\" id=\"overview\"></a>\n\n**Experiment Name:** \"Free Trial\" Screener. <br>\nIt is conducted by Udacity, a website dedicated to teaching online - with the overall business goal of maximizing course completion by students.\n### 2.1 Current Conditions Before Change <a class=\"anchor\" id=\"current\"></a>\n* At the time of this experiment, Udacity courses currently have two options on the course overview page: \"start free trial\", and \"access course materials\". <br>\n* If the student clicks \"start free trial\", they will be asked to enter their credit card information, and then they will be enrolled in a free trial for the paid version of the course. After 14 days, they will automatically be charged unless they cancel first.\n* If the student clicks \"access course materials\", they will be able to view the videos and take the quizzes for free, but they will not receive coaching support or a verified certificate, and they will not submit their final project for feedback.\n\n\n### 2.2 Description of Experimented Change <a class=\"anchor\" id=\"description\"></a>\n\n* In the experiment, Udacity tested a change where if the student clicked \"start free trial\", they were asked how much time they had available to devote to the course.\n* If the student indicated 5 or more hours per week, they would be taken through the checkout process as usual. If they indicated fewer than 5 hours per week, a message would appear indicating that Udacity courses usually require a greater time commitment for successful completion, and suggesting that the student might like to access the course materials for free.\n* At this point, the student would have the option to continue enrolling in the free trial, or access the course materials for free instead. [This screenshot](https://drive.google.com/file/d/0ByAfiG8HpNUMakVrS0s4cGN2TjQ/view) shows what the experiment looks like.\n\n### 2.3 Experiment Hypothesis <a class=\"anchor\" id=\"hypothesis\"></a>\nThe hypothesis was that this might set clearer expectations for students upfront, thus reducing the number of frustrated students who left the free trial because they didn't have enough time—without significantly reducing the number of students to continue past the free trial and eventually complete the course. If this hypothesis held true, Udacity could improve the overall student experience and improve coaches' capacity to support students who are likely to complete the course.\n\n### 2.4 Experiment Details <a class=\"anchor\" id=\"details\"></a>\nThe unit of diversion is a cookie, although if the student enrolls in the free trial, they are tracked by user-id from that point forward. The same user-id cannot enroll in the free trial twice. For users that do not enroll, their user-id is not tracked in the experiment, even if they were signed in when they visited the course overview page."
    },
    {
      "metadata": {
        "_cell_guid": "1055062b-40d7-400c-8721-653f71250bbc",
        "_uuid": "d4200f0848220af981f13946919ebf0c5cee0f00"
      },
      "cell_type": "markdown",
      "source": "## 3 Metric Choice <a class=\"anchor\" id=\"metric\"></a>\nWe need two types of metrics for a successful experiment (or at least, a safe one); Invariate and evaluation metrics.\nInvariate metrics are used for \"sanity checks\", that is, to make sure our experiment (the way we presented a change to a part of the population, as well as the way we collected the data) is not inherently wrong. Basically, this means we pick metrics which we consider not to change (not to be affected) because of our experiment and later make sure these metrics don't change drastically between our control and experiment groups.<br>\nEvaluation metrics on the other hand, are the metrics in which we expect to see a change, and are relevant to the business goals we aim to achieve. For each metric we state a $Dmin$ - which marks the minimum change which is practically significant to the business. For instance, stating that any increase in retention that is under 2%, even if statistically significant, is not practical to the business.\n### 3.1 Invariate Metrics - Sanity Checks <a class=\"anchor\" id=\"invariate\"></a>\n\n| Metric Name  | Metric Formula  | $Dmin$  | Notation |\n|:-:|:-:|:-:|:-:|\n| Number of Cookies in Course Overview Page  | # unique daily cookies on page | 3000 cookies  | $C_k$ |\n| Number of Clicks on Free Trial Button  | # unique daily cookies who clicked  | 240 clicks | $C_l$ |\n| Free Trial button Click-Through-Probability  | $\\frac{C_l}{C_k}$ | 0.01  | $CTP$ | \n\n### 3.2 Evaluation Metrics - Performance Indicators <a class=\"anchor\" id=\"evaluation\"></a>\n| Metric Name  | Metric Formula  | $Dmin$  | Notation |\n|:-:|:-:|:-:|:-:|\n| Gross Conversion   |  $\\frac{enrolled}{C_l}$  | 0.01  | $Conversion_{Gross}$ |\n| Retention   | $\\frac{paid}{enrolled}$  | 0.01  | $Retention$ |\n| Net Conversion  |  $\\frac{paid}{C_l}$  | 0.0075 | $Conversion_{Net}$ |"
    },
    {
      "metadata": {
        "_cell_guid": "8532b7b5-e365-4468-aa5c-839114fbe07b",
        "_uuid": "dc01c453a9b39e69a4b25699444828f42814f490"
      },
      "cell_type": "markdown",
      "source": "## 4 Estimating the baseline values of metrics <a class=\"anchor\" id=\"baseline\"></a>\nBefore we start our experiment we should know how these metrics behave before the change - that is, what are their baseline values.\n### 4.1 Collecting estimators data <a class=\"anchor\" id=\"collect\"></a>\nUdacity gives the following rough estimates for these metrics (presumably collected from aggregates on daily traffic) <br>\n\n| Item | Description  | Estimator  |\n|:-:|:-:|:-:|\n| Number of cookies | Daily unique cookies to view course overview page  | 40,000  |\n| Number of clicks | Daily unique cookies to click Free Trial button  | 3,200 |\n| Number of enrollments | Free Trial enrollments per day  | 660  |\n| CTP | CTP on Free Trial button  | 0.08  |\n| Gross Conversion | Probability of enrolling, given a click  | 0.20625  |\n| Retention | Probability of payment, given enrollment  | 0.53  |\n| Net Conversion | Probability of payment, given click  | 0.109313 |"
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "fc93fbb9-b9f0-4b51-8dad-c4a8ee7b4320",
        "_uuid": "a3b1eab0da4bf7262ff7d98a41bed840d835762a",
        "trusted": false
      },
      "cell_type": "code",
      "source": "#Let's place this estimators into a dictionary for ease of use later\nbaseline = {\"Cookies\":40000,\"Clicks\":3200,\"Enrollments\":660,\"CTP\":0.08,\"GConversion\":0.20625,\n           \"Retention\":0.53,\"NConversion\":0.109313}",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "a459663f-a85a-4843-b190-39cf3ae1ae8d",
        "_uuid": "9883558511af822cd1a26ae9ce838428d05309df"
      },
      "cell_type": "markdown",
      "source": "### 4.2 Estimating Standard Deviation <a class=\"anchor\" id=\"sd\"></a>\nOnce we collected these estimates, we should estimate the standard deviation of a metric, this is computed for sample size calculations and confidence intervals for our results. The more variant a metric is, the harder it is to reach a significant result. Assuming a sample size of 5,000 cookies visiting the course overview page per day (as given in project's instructions) - we want to estimate a standard deviation, for the evaluation metrics only. The sample size we are considering should be smaller than the \"population\" we collected and small enough to have two groups with that size.\n\n#### 4.2.1 Scaling Collected Data <a class=\"anchor\" id=\"scale\"></a>\nFor all the calculations to follow we need to scale our collected counts estimates of metrics with the sample size we specified for variance estimation. In this case, from 40000 unique cookies to visit the course overview page per day, to 5000."
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "d99f7ee0-4a65-402b-a709-a949cd549bc3",
        "_uuid": "019536837b5aa146997b693563cfb22af1fb85ee",
        "trusted": false
      },
      "cell_type": "code",
      "source": "#Scale The counts estimates\nbaseline[\"Cookies\"] = 5000\nbaseline[\"Clicks\"]=baseline[\"Clicks\"]*(5000/40000)\nbaseline[\"Enrollments\"]=baseline[\"Enrollments\"]*(5000/40000)\nbaseline",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "b66cdcd4-1ad0-4536-b75e-f3fec164ffc4",
        "_uuid": "c8dc256a506700380e3b5595d2697ab3a5b1b809"
      },
      "cell_type": "markdown",
      "source": "#### 4.2.2 Estimating Analytically <a class=\"anchor\" id=\"estimate\"></a>\nIn order to estimate variance analytically, we can assume metrics which are probabilities ($\\hat{p}$) are binomially distributed, so we can use this formula for the standard deviation: <br>\n<center><font size=\"4\">$SD=\\sqrt{\\frac{\\hat{p}*(1-\\hat{p})}{n}}$</font></center><br>\nThis assumption is only valid when the **unit of diversion** of the experiment is equal to the **unit of analysis** (the denominator of the metric formula). In the cases when this is not valid, the actual variance might be different and it is recommended to estimate it empirically.\n\nFor each metric, we need to plug two variables into the formula: <br>\n$\\hat{p}$ - baseline probability of the event to occur <br>\n$ n $ - sample size <br>\n\n\n* **Gross Conversion** - The baseline probability for Gross Conversion can be calculated by the number of users to enroll in a free trial divided by the number of cookies clicking the free trial. In other words, the probability of enrollment given a click. In this case, the unit of diversion (Cookies), that is the element by which we differentiate samples and assign them to control and experiment groups, is equall to the unit of analysis (cookies who click), that is the denominator of the formula to calculate Gross Conversion (GC). When this is the case, this analytic estimate of variance is sufficient.\n"
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "4c47c350-99c8-4990-8456-cbbe014a972f",
        "_uuid": "2c6930584c39024455900fce7e0e92e5812814b3",
        "trusted": false
      },
      "cell_type": "code",
      "source": "# Let's get the p and n we need for Gross Conversion (GC)\n# and compute the Stansard Deviation(sd) rounded to 4 decimal digits.\nGC={}\nGC[\"d_min\"]=0.01\nGC[\"p\"]=baseline[\"GConversion\"]\n#p is given in this case - or we could calculate it from enrollments/clicks\nGC[\"n\"]=baseline[\"Clicks\"]\nGC[\"sd\"]=round(mt.sqrt((GC[\"p\"]*(1-GC[\"p\"]))/GC[\"n\"]),4)\nGC[\"sd\"]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "8b7b3d37-69e5-49e5-9e60-4c6a1dae00f9",
        "_uuid": "362a79099101a5d72ee5d78689459d8b2f9c2104"
      },
      "cell_type": "markdown",
      "source": "* **Retention** - The baseline probability for retention is the number of paying users (enrolled after 14 free days) divided by the number of total enrolled users. In other words, the probability of payment, given enrollment. The sample size is the number of enrolled users. In this case, unit of diversion is not equal to unit of analysis (users who enrolled) so an analytical estimation is not enough - if we had the data for these estimates, we would want to estimate this variance empirically as well."
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "654d08d4-6638-4f9f-a327-6f2f5a6a9365",
        "_uuid": "334e65cddd5a800a396fa630b676ca12bec42114",
        "trusted": false
      },
      "cell_type": "code",
      "source": "# Let's get the p and n we need for Retention(R)\n# and compute the Stansard Deviation(sd) rounded to 4 decimal digits.\nR={}\nR[\"d_min\"]=0.01\nR[\"p\"]=baseline[\"Retention\"]\nR[\"n\"]=baseline[\"Enrollments\"]\nR[\"sd\"]=round(mt.sqrt((R[\"p\"]*(1-R[\"p\"]))/R[\"n\"]),4)\nR[\"sd\"]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "6625822a-c58b-4d1d-b3a8-ed8541796213",
        "_uuid": "37d04b56f1b9e5d8fa5d08c0242c317ec39c959c"
      },
      "cell_type": "markdown",
      "source": "* **Net Conversion** - The baseline probability for the net conversion is the number of paying users divided by the number of cookies that clicked the free trial button. In other words, the probability of payment, given a click. The sample size is the number of cookies that clicked. In this case, the unit of analysis and diversion are equal so we expect a good enough estimation analytically."
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "162f3d90-467a-41e6-9489-f1ea01f08afa",
        "_uuid": "be4b9c808cf098efde59ea1d17e1a35799873e5f",
        "trusted": false
      },
      "cell_type": "code",
      "source": "# Let's get the p and n we need for Net Conversion (NC)\n# and compute the Standard Deviation (sd) rounded to 4 decimal digits.\nNC={}\nNC[\"d_min\"]=0.0075\nNC[\"p\"]=baseline[\"NConversion\"]\nNC[\"n\"]=baseline[\"Clicks\"]\nNC[\"sd\"]=round(mt.sqrt((NC[\"p\"]*(1-NC[\"p\"]))/NC[\"n\"]),4)\nNC[\"sd\"]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "cc418f7c-0573-45c5-9ebc-f70b0916a090",
        "_uuid": "a62be65b6b882a33a76f7402f52f7c83f4ccff75"
      },
      "cell_type": "markdown",
      "source": "## 5 Experiment Sizing <a class=\"anchor\" id=\"sizing\"></a>\nAt this point, once we have estimated our metrics in the baseline (most importantly, their estimated variance) we cancalculate the munumal number of samples we need so that our experiment will have enough statistical power, as well as siginificance.\n\nGiven $\\alpha=0.05$ (significance level ) and $\\beta=0.2$ (power), we want to estimate how many total pageviews (cookies who viewed the course overview page) we need in the experiment. This amount will be divided into tthe two groups: control and experiment. This calculation can be done using an [online calculator](http://www.evanmiller.org/ab-testing/sample-size.html) or by calculating directly using the required formula.\n\nThe minimum sample size for control and experiment groups, which provides probability of Type I Error $\\alpha$, Power $1-\\beta$, detectable effect $d$ and baseline conversion rate $p$ (simple hypothesis $H_0 : P_{cont} - P_{exp} = 0$ against simple alternative $H_A : P_{cont} - P_{exp} = d$  is:\n\n<center> <font size=\"5\"> $n = \\frac{(Z_{1-\\frac{\\alpha}{2}}sd_1 + Z_{1-\\beta}sd_2)^2}{d^2}$</font>, with: <br><br>\n$sd_1 = \\sqrt{p(1-p)+p(1-p)}$<br><br>\n$sd_2 = \\sqrt{p(1-p)+(p+d)(1-(1-(p+d))}$ </center><br>\n\nNow, let's break down what inputs we need and which calculations still need to be made. Regarding inputs, we have all the data we need:\nType 1 error ($\\alpha$), power ($1-\\beta$), detectable change ($d = D_{min}$) and baseline conversion rate (our $\\hat{p}$ ).\nWhat we need to calculate: \n* Get Z score for $1-\\frac{\\alpha}{2}$ and for $1-\\beta$\n* Get standard deviations 1 & 2, that is for both the baseline and for expected changed rate\nAll these components will finally yield the number we need.\n\n### 5.1 Get z-score critical value and Standard Deviations <a class=\"anchor\" id=\"side_methods\"></a>\nWe are used to looking up this value in a table, but gladly we can use python's `scipy.stats.norm` package to get all the required methods for normal distribution. The `ppf` method gives us access to the [Percent Point Function (ppf)](https://en.wikipedia.org/wiki/Quantile_function) or Quantile Function, and besides it being the inverse of the [Cummulative Distribution Function (cdf)](https://en.wikipedia.org/wiki/Cumulative_distribution_function), this is the functions that will give back our required critical z-score."
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "1cedf3ae-f318-4036-b966-a299350fb048",
        "_uuid": "8956f99d8ea1b9bb6d8b294a6e333c86a51d88d7",
        "trusted": false
      },
      "cell_type": "code",
      "source": "def get_sds(p,d):\n    sd1=mt.sqrt(2*p*(1-p))\n    sd2=mt.sqrt(p*(1-p)+(p+d)*(1-(p+d)))\n    x=[sd1,sd2]\n    return x",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "230d398f-5692-4f04-a8a9-316c9c661801",
        "_uuid": "c1c7a715d55432c5d19fef049094527cf1f72343",
        "trusted": false
      },
      "cell_type": "code",
      "source": "#Inputs: required alpha value (alpha should already fit the required test)\n#Returns: z-score for given alpha\ndef get_z_score(alpha):\n    return norm.ppf(alpha)\n\n# Inputs p-baseline conversion rate which is our estimated p and d-minimum detectable change\n# Returns\ndef get_sds(p,d):\n    sd1=mt.sqrt(2*p*(1-p))\n    sd2=mt.sqrt(p*(1-p)+(p+d)*(1-(p+d)))\n    sds=[sd1,sd2]\n    return sds\n\n# Inputs:sd1-sd for the baseline,sd2-sd for the expected change,alpha,beta,d-d_min,p-baseline estimate p\n# Returns: the minimum sample size required per group according to metric denominator\ndef get_sampSize(sds,alpha,beta,d):\n    n=pow((get_z_score(1-alpha/2)*sds[0]+get_z_score(1-beta)*sds[1]),2)/pow(d,2)\n    return n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "ee719e64-ee76-45a7-8267-ba41c27feb11",
        "_uuid": "3982b947794bbcd660991ca9087227d5294d2075"
      },
      "cell_type": "markdown",
      "source": "### 5.2 Calculate Sample Size per Metric <a class=\"anchor\" id=\"calc\"></a>\nOK! Looks like we set up all the tools required for this part. We are now going to calculate the number of samples required for the experiment per metric, and we are subected to the fact that the highest sample size will be the effective size. This size should be considered in terms of efficacy of duration and exposure: how long will it take to get this many samples for the experiment.\n\nSo, to work more easily, let's add the d parameter to each of the metrics characteristics of each metric:"
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "e9b0d74a-35ba-461e-9b04-7527cc0147f9",
        "_uuid": "f44e06b60b027dd83ec3734c52ed1347d2a80c0c",
        "trusted": false
      },
      "cell_type": "code",
      "source": "GC[\"d\"]=0.01\nR[\"d\"]=0.01\nNC[\"d\"]=0.0075",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "eb361734-4ab5-4ba6-800c-0271152b61fd",
        "_uuid": "54336bb19b3564f95a4698903806d4e7edaeb8c4"
      },
      "cell_type": "markdown",
      "source": "Now, for the calculations\n* **Gross Conversion**\n"
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "28cf1802-c0ce-4189-b438-86504e53a721",
        "_uuid": "b443cb437954e04ef889ef4b365d8875da2833d1",
        "trusted": false
      },
      "cell_type": "code",
      "source": "# Let's get an integer value for simplicity\nGC[\"SampSize\"]=round(get_sampSize(get_sds(GC[\"p\"],GC[\"d\"]),0.05,0.2,GC[\"d\"]))\nGC[\"SampSize\"]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "50f65b60-e745-44fc-8803-84f358dc8c26",
        "_uuid": "4382f761b5282a009eb8265ddfcf6a12276382ec"
      },
      "cell_type": "markdown",
      "source": "This means we need at least 25,835 cookies who click the Free Trial button - per group! That means that if we got 400 clicks out of 5000 pageviews (`400/5000 = 0.08`) -> So, we are going to need `GC[\"SampSize\"]/0.08 = 322,938` pageviews, again ; per group! Finally, the total amount of samples per the Gross Conversion metric is:"
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "ba7702c9-b092-404b-a212-8713a517d9bc",
        "_uuid": "530c3e906980568677ce0a7fc5c92f5f9a26302b",
        "trusted": false
      },
      "cell_type": "code",
      "source": "GC[\"SampSize\"]=round(GC[\"SampSize\"]/0.08*2)\nGC[\"SampSize\"]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "8a145fe1-7493-4eef-9b3e-99138068d2ae",
        "_uuid": "7e0c0bfc939849712c49611f9392abe154e04179"
      },
      "cell_type": "markdown",
      "source": "* Retention"
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "c2c65956-f449-4d28-94b5-c5300aeadb4b",
        "_uuid": "1aa61cc37f8839a44f0773623d41c3684be1a9c7",
        "trusted": false
      },
      "cell_type": "code",
      "source": "# Getting a nice integer value\nR[\"SampSize\"]=round(get_sampSize(get_sds(R[\"p\"],R[\"d\"]),0.05,0.2,R[\"d\"]))\nR[\"SampSize\"]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "5679534f-a863-47af-ba94-7b61d99383de",
        "_uuid": "05ba302c01db5f280d9bd0cc007f289c4174d338"
      },
      "cell_type": "markdown",
      "source": "This means that we need 39,087 users who enrolled per group! We have to first convert this to cookies who clicked, and then to cookies who viewed the page, then finally to multipky by two for both groups."
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "0281b321-7506-49ba-a33e-29eed30eefbe",
        "_uuid": "16bf69e2a0f6214b91485b736676a1fe7aadfdca",
        "trusted": false
      },
      "cell_type": "code",
      "source": "R[\"SampSize\"]=R[\"SampSize\"]/0.08/0.20625*2\nR[\"SampSize\"]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "42631863-3d60-4870-b6fe-0a6086940b22",
        "_uuid": "7f208b2044a3bb8ac8cc1160d7ca8c18e8403b71"
      },
      "cell_type": "markdown",
      "source": "This takes us as high as over 4 million page views total, this is practically impossible because we know we get about 40,000 a day, this would take well over 100 days. This means we have to drop this metric and not continue to work with it because results from our experiment (which is much smaller) will be biased.\n* **Net Conversion**"
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "56be284a-1c49-4c65-91d2-e007ae5b327d",
        "_uuid": "42607629817a83c2be769165cff7a30d74067320",
        "trusted": false
      },
      "cell_type": "code",
      "source": "# Getting a nice integer value\nNC[\"SampSize\"]=round(get_sampSize(get_sds(NC[\"p\"],NC[\"d\"]),0.05,0.2,NC[\"d\"]))\nNC[\"SampSize\"]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "9ad6d399-8b5f-41d6-be72-ce428e13340d",
        "_uuid": "1ffe24cdbd36a2327cf73dd307fb0900a0f3364c"
      },
      "cell_type": "markdown",
      "source": "So, needing 27,413 cookies who click per group takes us all the way up to:"
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "e4202dd9-7cb6-455c-9630-dd9277f84da6",
        "_uuid": "38b580f63f9e25fa7e19958d7b12a14d6e2b8838",
        "trusted": false
      },
      "cell_type": "code",
      "source": "NC[\"SampSize\"]=NC[\"SampSize\"]/0.08*2\nNC[\"SampSize\"]",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "24687192-ce64-4c1c-9f30-8aa80058bc88",
        "_uuid": "37be6b0fc9775b2bd645ea017bb551ee227c0022"
      },
      "cell_type": "markdown",
      "source": "We are all the way up to 685,325 cookies who view the page. This is more than what was needed for Gross Conversion, so this will be our number. Assuming we take 80% of each days pageviews, the data collection period for this experiment (the period in which the experiment is revealed) will be about 3 weeks.\n"
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "a117e9b7-bfb5-4444-b1ed-e64903008bf9",
        "_uuid": "6e8be49265f041c4e92e6b59f4c9d249176583c6"
      },
      "cell_type": "markdown",
      "source": "## 6 Analyzing Collected Data <a class=\"anchor\" id=\"analysis\"></a>\nFinally, the moment we've all been waiting for, after so much preparation we finally get to see what this experiment will prove!\nThe data is presented as two spreadsheets. I will load each spreadshot into a pandas dataframe.\n\n### 6.1 Loading collected data <a class=\"anchor\" id=\"collect_results\"></a>"
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "58b7a728-3b45-4867-969b-48959085498c",
        "_uuid": "4c29c3bce964e643ef30633cab8b95f268f76abc",
        "trusted": false
      },
      "cell_type": "code",
      "source": "# we use pandas to load datasets\ncontrol=pd.read_csv(\"../input/control-data/control_data.csv\")\nexperiment=pd.read_csv(\"../input/experiment-data/experiment_data.csv\")\ncontrol.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "9f84eaa2-cf6a-49dc-8725-da01a50a8098",
        "_uuid": "eeb7dfe54121cf61212f66bba3d886a54778e5c7"
      },
      "cell_type": "markdown",
      "source": "### 6.2 Sanity Checks <a class=\"anchor\" id=\"sanity\"></a>\nFirst thing we have to do before even beginning to analyze this experiment's results is sanity checks. These checks help verify that the experiment was conducted as expected and that other factors did not influence the data which we collected. This also makes sure that data collection was correct.\n\nWe have 3 Invariant metrics:: \n* Number of Cookies in Course Overview Page\n* Number of Clicks on Free Trial Button\n* Free Trial button Click-Through-Probability\n\nTwo of these metrics are simple counts like number of cookies or number of clicks and the third is a probability (CTP). We will use two different ways of checking whether these obsereved values are like we expect (if in fact the experiment was not damaged.\n\n#### 6.2.1 Sanity Checks for differences between counts <a class=\"anchor\" id=\"check_counts\"></a>\n* **Number of cookies who viewed the course overview page** - Starting from this simpel invariant metric, we want to count the total amount of cookie pageviews we diverted to each group and see if there is a significant difference int he amount of cookies. A significant difference will imply a biased experiment that we should not rely on it's results."
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "37ab672f-be1c-46bc-ae53-edf722bef4bc",
        "_uuid": "15652351bc793b528ca853666ddb4b7defe4f4ff",
        "trusted": false
      },
      "cell_type": "code",
      "source": "pageviews_cont=control['Pageviews'].sum()\npageviews_exp=experiment['Pageviews'].sum()\npageviews_total=pageviews_cont+pageviews_exp\nprint (\"number of pageviews in control:\", pageviews_cont)\nprint (\"number of Pageviewsin experiment:\" ,pageviews_exp)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "fc7fdfbe-d9d7-4239-a7be-994a8b112ce4",
        "_uuid": "fb716ffd3eb94fe822c7bcb226d05aceba25a732"
      },
      "cell_type": "markdown",
      "source": "Ok so these look like pretty close numbers. Now, let's make sure this difference in amounts is not significant and is random and even like we expected. We can model this diversion in the following way: <br>\nWe expect the amount of pageviews in the control group to be about a half (50%) of the total pageviews in both groups, so we can define a random variable with an easy to use distribution. <br>\nA binomial random variable will be the number of successes we can expect to get out of N experiments, given the probability of a single success. So, if we consider being assigned to a group (control, for example) a success with probability 0.5 (random!), the number of samples which get assigned to the group is the value of our random binomial variable! <br>\n\nThis get's easier thanks to the central limit theorem which let's us approximate the binomial distribution to a normal distribution (when n is large enough) with a mean of $p$ and a standard deviation $\\sqrt{\\frac{p(1-p)}{N}}$\n<center> <font size=\"4\"> $ X$~$N( p,\\sqrt{\\frac{p(1-p)}{N}})$ </font></center>\nWhat we want to test is whether our observed $\\hat{p}$ (number of samples in control divided by total number of damples in both groups) is not significantly different than $p=0.5$. In order to do that we can calculate the margin of error acceptable at a 95% confidence level:\n<center> <font size=\"4\"> $ ME=Z_{1-\\frac{\\alpha}{2}}SD$ </font></center>\nFinally, a [confidence interval](https://en.wikipedia.org/wiki/Confidence_interval) can be derived to tell us in which range an observed $p$ can exist and be acceptable as \"the same\" as the expected value.\n<center> <font size=\"4\"> $ CI=[\\hat{p}-ME,\\hat{p}+ME]$ </font></center>\nWhen our obsereved $\\hat{p}$ is within this range, all is well and the test was passed."
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "e939e7d9-59e6-41a3-b6cf-49e9adeb17a1",
        "_uuid": "e9826a63b91cf63f0f0ab5781cf05b39105a21f8",
        "trusted": false
      },
      "cell_type": "code",
      "source": "p=0.5\nalpha=0.05\np_hat=round(pageviews_cont/(pageviews_total),4)\nsd=mt.sqrt(p*(1-p)/(pageviews_total))\nME=round(get_z_score(1-(alpha/2))*sd,4)\nprint (\"The confidence interval is between\",p-ME,\"and\",p+ME,\"; Is\",p_hat,\"inside this range?\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "d1e68346-0006-4344-b9a8-b158bcda1709",
        "_uuid": "76eb5d80bcd328f7beba1ac639dda8992e412b43"
      },
      "cell_type": "markdown",
      "source": "Our observed $\\hat{p}$ is inside this range which means the difference in number of samples between groups is expected. So far so good, since this invariant metric sanity test passes!\n* **Number of cookies who clicked the Free Trial Button**\nWe are going to address this count with the same strategy as before."
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "4b99eec8-db1f-47b6-824b-d9dfa6ddb2de",
        "_uuid": "c60467b13e7411252af75b15c1cec19a7ab175c0",
        "trusted": false
      },
      "cell_type": "code",
      "source": "clicks_cont=control['Clicks'].sum()\nclicks_exp=experiment['Clicks'].sum()\nclicks_total=clicks_cont+clicks_exp\n\np_hat=round(clicks_cont/clicks_total,4)\nsd=mt.sqrt(p*(1-p)/clicks_total)\nME=round(get_z_score(1-(alpha/2))*sd,4)\nprint (\"The confidence interval is between\",p-ME,\"and\",p+ME,\"; Is\",p_hat,\"inside this range?\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "112496ac-2c13-47da-b075-f41e06ae91d1",
        "_uuid": "31749d1a6e8dda73238267ed00d2f0fa3021e760"
      },
      "cell_type": "markdown",
      "source": "We have another pass! Great, so far it still seems all is well with our experiment results. Now, for the final metric which is a probability.\n\n#### 6.2.2 Sanity Checks for differences between probabilities <a class=\"anchor\" id=\"check_prob\"></a>\n* ** Click-through-probability of the Free Trial Button**\nIn this case, we want to make sure the proportion of clicks given a pageview (our observed CTP) is about the same in both groups (since this was not expected to change due to the experiment). In order to check this out we will calculate the CTP in each group and calculate a confidence interval for the expected difference between them. \n\nIn other words, we expect to see no difference ($CTP_{exp}-CTP_{cont}=0$), with an acceptable margin of error, dictated by our calculated confidence interval. The changes we should notice are for the calculation of the standard error - which in this case is a pooled standard error.\n\n<center><font size=\"4\">$SD_{pool}=\\sqrt{\\hat{p_{pool}}(1-\\hat{p_{pool}}(\\frac{1}{N_{cont}}+\\frac{1}{N_{exp}})}$</font></center>\nwith <br> <center><font size=\"5\"> $\\hat{p_{pool}}=\\frac{x_{cont}+x_{exp}}{N_{cont}+N_{exp}}$ </font></center>\nWe should understand that CTP is a proportion in a population (amount of events x in a population n) like the amount of clicks out of the amount of pageviews.."
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "8f1b7453-11b0-47e7-a53d-c28d18eceb0a",
        "_uuid": "1dd583d7d3698e0a1a8281c7f1c6c46ce2964f48",
        "trusted": false
      },
      "cell_type": "code",
      "source": "ctp_cont=clicks_cont/pageviews_cont\nctp_exp=clicks_exp/pageviews_exp\nd_hat=round(ctp_exp-ctp_cont,4)\np_pooled=clicks_total/pageviews_total\nsd_pooled=mt.sqrt(p_pooled*(1-p_pooled)*(1/pageviews_cont+1/pageviews_exp))\nME=round(get_z_score(1-(alpha/2))*sd_pooled,4)\nprint (\"The confidence interval is between\",0-ME,\"and\",0+ME,\"; Is\",d_hat,\"within this range?\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "626356a4-5f7c-4e5b-a236-f355d8462ef8",
        "_uuid": "cfb988eee490c8ac43589c61718433a6d6f0f88f"
      },
      "cell_type": "markdown",
      "source": "Wonderful. It seems this test has passed with flying colors as well.\n\n### 6.3 Examining effect size <a class=\"anchor\" id=\"effect\"></a>\nThe next step is looking at the changes between the control and experiment groups with regard to our evaluation metrics to make sure the difference is there, that it is statistically significant and most importantly practically significant (the difference is \"big\" enough to make the experimented change beneficial to the company).\n\nNow, all that is left is to measure for each evaluation metric, the difference between the values from both groups. Then, we compute the confidence interval for that difference and test whether or not this confidence interval is both statistically and practically significant.\n\n* **Gross Conversion**\nA metric is statistically significant if the confidence interval does not include 0 (that is, you can be confident there was a change), and it is practically significant if the confidence interval does not include the practical significance boundary (that is, you can be confident there is a change that matters to the business.)\n\n> **Important:** The given spreadsheet lists pageviews and clicks for 39 days, while it only lists enrollments and payments for 23 days. So, when working with enrollments and payments we should notice using only the corresponding pageviews and clicks, and not all of them."
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "8e13eba4-0daa-4220-a772-e1f371854d60",
        "_uuid": "6d4d94df9e5afdfdf97703292b0ef9ac6c3e4251",
        "trusted": false
      },
      "cell_type": "code",
      "source": "# Count the total clicks from complete records only\nclicks_cont=control[\"Clicks\"].loc[control[\"Enrollments\"].notnull()].sum()\nclicks_exp=experiment[\"Clicks\"].loc[experiment[\"Enrollments\"].notnull()].sum()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "5615673e-529c-452a-ad48-9186cb88f74b",
        "_uuid": "3f17a8086726ee8cc01f150808f88fe8693a95a2",
        "trusted": false
      },
      "cell_type": "code",
      "source": "#Gross Conversion - number of enrollments divided by number of clicks\nenrollments_cont=control[\"Enrollments\"].sum()\nenrollments_exp=experiment[\"Enrollments\"].sum()\n\nGC_cont=enrollments_cont/clicks_cont\nGC_exp=enrollments_exp/clicks_exp\nGC_pooled=(enrollments_cont+enrollments_exp)/(clicks_cont+clicks_exp)\nGC_sd_pooled=mt.sqrt(GC_pooled*(1-GC_pooled)*(1/clicks_cont+1/clicks_exp))\nGC_ME=round(get_z_score(1-alpha/2)*GC_sd_pooled,4)\nGC_diff=round(GC_exp-GC_cont,4)\nprint(\"The change due to the experiment is\",GC_diff*100,\"%\")\nprint(\"Confidence Interval: [\",GC_diff-GC_ME,\",\",GC_diff+GC_ME,\"]\")\nprint (\"The change is statistically significant if the CI doesn't include 0. In that case, it is practically significant if\",-GC[\"d_min\"],\"is not in the CI as well.\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "cbc93acb-0d0a-448c-8826-d69d515bfc3d",
        "_uuid": "6f1263f81625002f07c5c015ccaa8a692a7ef35f"
      },
      "cell_type": "markdown",
      "source": "According to this result there was a change due to the experiment, that change was both statistically and practically significant. \nWe have a negative change of 2.06%, when we were willing to accept any change greater than 1%. This means the Gross Conversion rate of the experiment group (the one exposed to the change, i.e. asked how many hours they can devote to studying) has decreased as expected by 2% and this change was significant. This means  less people enrolled in the Free Trial after due to the pop-up. \n* **Net Conversion** \nThe hypothesis is the same as before just with net conversion instead of gross. At this point we expect the fraction of payers (out of the clicks) to decrease as well."
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "8cb352ad-fc92-4f21-b939-3874314ba8f4",
        "_uuid": "83e99af5dea50f22629ccbd34faa196d4065172f",
        "trusted": false
      },
      "cell_type": "code",
      "source": "#Net Conversion - number of payments divided by number of clicks\npayments_cont=control[\"Payments\"].sum()\npayments_exp=experiment[\"Payments\"].sum()\n\nNC_cont=payments_cont/clicks_cont\nNC_exp=payments_exp/clicks_exp\nNC_pooled=(payments_cont+payments_exp)/(clicks_cont+clicks_exp)\nNC_sd_pooled=mt.sqrt(NC_pooled*(1-NC_pooled)*(1/clicks_cont+1/clicks_exp))\nNC_ME=round(get_z_score(1-alpha/2)*NC_sd_pooled,4)\nNC_diff=round(NC_exp-NC_cont,4)\nprint(\"The change due to the experiment is\",NC_diff*100,\"%\")\nprint(\"Confidence Interval: [\",NC_diff-NC_ME,\",\",NC_diff+NC_ME,\"]\")\nprint (\"The change is statistically significant if the CI doesn't include 0. In that case, it is practically significant if\",NC[\"d_min\"],\"is not in the CI as well.\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "d36717f0-2607-4f67-b16f-593f3e620b81",
        "_uuid": "fb574d1d1a255e7e9c7c12f72a1e15b34396b871"
      },
      "cell_type": "markdown",
      "source": "In this case we got a change size of less than a 0.5%, a very small decrease which is not statistically significant, and as such not practically significant.\n\n## 6.4 Double check with Sign Tests <a class=\"anchor\" id=\"sign_tests\"></a>\nIn a sign test we get another angle at analyzing the results we got - we check if the trend of change we observed (increase or decrease) was evident in the daily data. We are goint to compute the metric's value per day and then count on how many days the metric was lower in the experiment group and this will be the number of succssesses for our binomial variable. Once this is defined we can look at the proportion of days of success out of all the available days.\n\n### 6.4.1 Data Preparation <a class=\"anchor\" id=\"prep\"></a>"
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "d59fb882-ec0a-4cf9-9fd6-5e0271302ea6",
        "_uuid": "aae6e4d96ac0d06b3bdfece143be74d58c0fade3",
        "trusted": false
      },
      "cell_type": "code",
      "source": "#let's first create the dataset we need for this:\n# start by merging the two datasets\nfull=control.join(other=experiment,how=\"inner\",lsuffix=\"_cont\",rsuffix=\"_exp\")\n#Let's look at what we got\nfull.count()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "97817e5e-9c6b-4347-96f2-ad56a11bc60f",
        "_uuid": "e2302f51793b1fcdb6c671783f0f7ab3ada7f62e",
        "trusted": false
      },
      "cell_type": "code",
      "source": "#now we only need the complete data records\nfull=full.loc[full[\"Enrollments_cont\"].notnull()]\nfull.count()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "09f17781-7a21-4c4b-a239-e8ed2495f147",
        "_uuid": "a5b14c1c27a3e2c78a1313c08824e05da12b2057",
        "trusted": false
      },
      "cell_type": "code",
      "source": "# Perfect! Now, derive a new column for each metric, so we have it's daily values\n# We need a 1 if the experiment value is greater than the control value=\nx=full['Enrollments_cont']/full['Clicks_cont']\ny=full['Enrollments_exp']/full['Clicks_exp']\nfull['GC'] = np.where(x<y,1,0)\n# The same now for net conversion\nz=full['Payments_cont']/full['Clicks_cont']\nw=full['Payments_exp']/full['Clicks_exp']\nfull['NC'] = np.where(z<w,1,0)\nfull.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "1b2a2c0e-bf4e-4a39-8e46-3d2c8ad17352",
        "_uuid": "904d0fb8aee7d7c40a29174197b493716adee9fa",
        "trusted": false
      },
      "cell_type": "code",
      "source": "GC_x=full.GC[full[\"GC\"]==1].count()\nNC_x=full.NC[full[\"NC\"]==1].count()\nn=full.NC.count()\nprint(\"No. of cases for GC:\",GC_x,'\\n',\n      \"No. of cases for NC:\",NC_x,'\\n',\n      \"No. of total cases\",n)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "e57851c4-5cfc-4a22-b85a-db313de228e6",
        "_uuid": "b6a395fc010e3843670a3f4a396340ee1dae5740"
      },
      "cell_type": "markdown",
      "source": "### 6.4.2 Building a Sign Test <a class=\"anchor\" id=\"sign\"></a>\nWe can forget all about this part and just use an [online sign test calculator](https://www.graphpad.com/quickcalcs/binomial2/), but for me that is just no fun - so I will implement the calculations behind it. <br>\nWhat we want to do after we count the amount of days in which the experiment group had a higher metric value than that of the control group, is to see if that number is likely to be seen again in a new experiment (significance). We assume the chance of a day like this is random (50% chance to happen) and then use the binomial distribution with $p=0.5$ and the number of experiments (days) to tell us the probability of this happening according to a random chance.<br>\nSo, according to the binomial distribution with $p=0.5$ and $n=$total number of days; we want to now the probability of $x$ days being a success (higher metric value in experiment). Because we are doing a two-tailed test we want to double this probability and once we have we can call it the $p-value$ and compare it to our $\\alpha$. If the $p-value$ is greater than the $\\alpha$ the result is not significant and vice-versa.<br>\n<center><font size=\"4\"> $p(successes )=\\frac{n!}{x!(n-x)!}p^x(1-p)^{n-x}$ </font></center>\nRecall that a $p-value$ is the probability of observing a test statistic as or more extreme than that observed. If we observed 2 days like that, the $p-value$ for the test is: $p-value = P(x <= 2)$. We only need to remember the following:<br>\n<center>$P(x<=2)=P(0)+P(1)+P(2)$.</center><br>For more detailed information, visit [this excellent page](http://sphweb.bumc.bu.edu/otlt/mph-modules/bs/bs704_nonparametric/BS704_Nonparametric5.html).\n"
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "0cc3b899-b54b-4504-bca4-4e63fa594adb",
        "_uuid": "e4db8dc9ead5aa4dd2389d03d8c037b2a3a30f28",
        "trusted": false
      },
      "cell_type": "code",
      "source": "#first a function for calculating probability of x=number of successes\ndef get_prob(x,n):\n    p=round(mt.factorial(n)/(mt.factorial(x)*mt.factorial(n-x))*0.5**x*0.5**(n-x),4)\n    return p\n#next a function to compute the pvalue from probabilities of maximum x\ndef get_2side_pvalue(x,n):\n    p=0\n    for i in range(0,x+1):\n        p=p+get_prob(i,n)\n    return 2*p",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "47e53586-6a19-4bc6-a625-cbdf23b29673",
        "_uuid": "1742c1f207eba696a02adf533c03215d7fec00c8"
      },
      "cell_type": "markdown",
      "source": "Finally, to conduct the sign test itself: we will calculate the p-value for each metric, using the counts `GC_x`,`NC_x` and `n` and the function `get_2side_pvalue`."
    },
    {
      "metadata": {
        "collapsed": true,
        "_cell_guid": "04d7c5b8-e859-4040-88e0-61180d05b167",
        "_uuid": "f1f7624b436d510f36ca22df49eb1455b196485b",
        "trusted": false
      },
      "cell_type": "code",
      "source": "print (\"GC Change is significant if\",get_2side_pvalue(GC_x,n),\"is smaller than 0.05\")\nprint (\"NC Change is significant if\",get_2side_pvalue(NC_x,n),\"is smaller than 0.05\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "2add23f9-3e0c-4b73-8f02-758dd9bb4f8a",
        "_uuid": "e3ce6967a1ebfc3f84b268645bc71f16df53f36a"
      },
      "cell_type": "markdown",
      "source": "We get the same conclusions as we got from our effect size calculation: the change in Gross conversion was indeed significant, while the change in Net conversion was not.\n\n## 7 Conclusions & Recommendations <a class=\"anchor\" id=\"conclusions\"></a>\nAt this point, once we have seen that the actual underlying goal we had was not reached (increase fraction of paying users by asking them in advance if they have the time to invest in the course), we can only recommend to not continue with change. It may have caused a change in Gross conversion, but it didn't for net conversion.\n\nA wonderful summary of this experiment, analysis and results, by the amazing Andrew Bauman, can be found [here](https://github.com/baumanab/udacity_ABTesting#summary)."
    },
    {
      "metadata": {
        "_cell_guid": "3f84c57a-4c16-425d-9230-d32c9222161b",
        "_uuid": "fac12ff041fc6e22f790ff4836a09c487f5bae14"
      },
      "cell_type": "markdown",
      "source": ""
    }
  ],
  "metadata": {
    "language_info": {
      "version": "3.6.3",
      "name": "python",
      "codemirror_mode": {
        "version": 3,
        "name": "ipython"
      },
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "mimetype": "text/x-python",
      "file_extension": ".py"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 1
}