chap9_ex.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Chapter 9: Model Comparison"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "(Latex macros)\n",
    "$\n",
    "    \\newcommand{\\tage}{t_{age}}\n",
    "    \\newcommand{\\dataerr}{\\overrightarrow{\\sigma_d}}\n",
    "    \\newcommand{\\nclobs}{n_{c,obs}}\n",
    "    \\newcommand{\\nclpred}{n_{c,pred}}\n",
    "    \\newcommand{\\birthrate}{\\dot n_{birth}}\n",
    "    \\newcommand{\\Mi}{M_{i}}\n",
    "    \\newcommand{\\ts}{\\tilde{t}}\n",
    "    \\newcommand{\\PI}{\\overrightarrow{\\pi}}\n",
    "    \\newcommand{\\dif}{\\,\\text{d}} \n",
    "    \\newcommand{\\dt}{\\Delta t}\n",
    "    \\newcommand{\\Ge}{\\gamma_e}\n",
    "    \\newcommand{\\Gd}{\\gamma_d} \n",
    "    \\newcommand{\\given}{\\,\\mid\\,}\n",
    "    \\newcommand{\\prob}{\\mathcal{P}}\n",
    "    \\newcommand{\\data}{\\overrightarrow{d}}\n",
    "    \\newcommand{\\T}{\\overrightarrow{\\theta}}\n",
    "    \\newcommand{\\err}{\\overrightarrow{\\sigma}}\n",
    "    \\newcommand{\\like}{\\mathcal{L}}\n",
    "    \\newcommand{\\best}{\\mathrm{best}}\n",
    "    \\newcommand{\\normal}{\\mathcal{N}}\n",
    "    \\newcommand{\\Pbad}{\\prob_{\\mathrm{b}}}\n",
    "    \\newcommand{\\union}{\\cup}\n",
    "    \\newcommand{\\intersect}{\\cap}\n",
    "    \\newcommand{\\argmax}{\\operatornamewithlimits{arg\\,max}}\n",
    "    \\newcommand{\\argmin}{\\operatornamewithlimits{arg\\,min}}\n",
    "    \\newcommand{\\card}{\\operatorname{card}}\t\n",
    "    \\newcommand{\\sgn}{\\operatorname{sgn}}\n",
    "    \\newcommand{\\rank}{\\operatorname{rank}}\t\t\t\t\n",
    "    \\newcommand{\\EE}{\\operatornamewithlimits{E}}\n",
    "    \\newcommand{\\id}{\\operatorname{id}}\t\t\t\n",
    "   \\newcommand{\\abs}[1]{\\left| #1 \\right|}\n",
    "   \\newcommand{\\norm}[1]{\\left\\| #1 \\right\\|}\n",
    "   \\newcommand{\\pa}[1]{\\left(#1\\right)}\n",
    "   \\newcommand{\\bra}[1]{\\left[#1\\right]}\n",
    "   \\newcommand{\\cbra}[1]{\\left\\{#1\\right\\}} \t\n",
    "   \\newcommand{\\Vec}[1]{\\overrightarrow}\n",
    "   \\newcommand{\\mmatrix}[1]{\\boldsymbol{#1}}\n",
    "   \\newcommand{\\inverse}[1]{{#1}^{-1}}\n",
    "   \\newcommand{\\transpose}[1]{{#1}^{\\scriptscriptstyle \\top}}\n",
    "   \\newcommand{\\mean}[1]{\\left<{#1}\\right>}\n",
    "   \\newcommand{\\Proba}[1]{\\prob\\left(\\, #1 \\,\\right)}\n",
    "   \\newcommand{\\proba}[1]{\\prob(\\, #1 \\,)}\n",
    "   \\newcommand{\\set}[1]{\\left\\{\\,#1\\,\\right\\}}\n",
    "   \\newcommand{\\set}[1]{\\left\\{\\,#1\\,\\right\\}}\n",
    "$\n",
    "$\n",
    "  \\newcommand{\\Unit}[1]{{\\mathrm{~#1}}} % define unit\n",
    "  \\newcommand{\\um}{\\mu\\mathrm{m}}\n",
    "  \\newcommand{\\erg}{\\Unit{erg}}\n",
    "  \\newcommand{\\ergs}{\\Unit{erg~s}}\n",
    "  \\newcommand{\\yr}{\\Unit{yr}}\n",
    "  \\newcommand{\\Myr}{\\Unit{Myr}}\n",
    "  \\newcommand{\\Gyr}{\\Unit{Gyr}}\n",
    "  \\newcommand{\\pc}{\\Unit{pc}}\n",
    "  \\newcommand{\\kpc}{\\Unit{kpc}}\n",
    "  \\newcommand{\\Mpc}{\\Unit{Mpc}}\n",
    "  \\newcommand{\\Lsun}{\\Unit{L}_{\\odot}}\n",
    "  \\newcommand{\\Zsun}{\\Unit{Z}_{\\odot}}\n",
    "  \\newcommand{\\msun}{\\Unit{M}_{\\odot}}\n",
    "  \\newcommand{\\kms}{\\Unit{km\\,s^{-1}}}\n",
    "  \\newcommand{\\ang}{\\AA}               %Angstrom unit\n",
    "  \\newcommand{\\degpoint}{\\mbox{$^\\circ\\mskip-7.0mu.\\,$}}\n",
    "  \\newcommand{\\halpha}{\\mbox{H$\\alpha$}}\n",
    "  \\newcommand{\\hbeta}{\\mbox{H$\\beta$}}\n",
    "  \\newcommand{\\hgamma}{\\mbox{H$\\gamma$}}\n",
    "  \\newcommand{\\lya}{\\mbox{Ly$\\alpha$}}\n",
    "  \\newcommand{\\lyb}{\\mbox{Ly$\\beta$}}\n",
    "  \\newcommand{\\minpoint}{\\mbox{$'\\mskip-4.7mu.\\mskip0.8mu$}}\n",
    "  \\newcommand{\\mv}{\\mbox{$m_{_V}$}}\n",
    "  \\newcommand{\\Mv}{\\mbox{$M_{_V}$}}\n",
    "  \\newcommand{\\peryr}{\\mbox{$\\>\\rm yr^{-1}$}}\n",
    "  \\newcommand{\\secpoint}{\\mbox{$''\\mskip-7.6mu.\\,$}}\n",
    "  \\newcommand{\\sqdeg}{\\mbox{${\\rm deg}^2$}}\n",
    "  \\newcommand{\\squig}{\\sim\\!\\!}\n",
    "  \\newcommand{\\subsun}{\\mbox{$_{\\normalsize\\odot}$}}\n",
    "  \\newcommand{\\sq}{\\mbox{\\rlap{$\\sqcap$}$\\sqcup$}}% \n",
    "  \\newcommand{\\arcdeg}{\\mbox{$^\\circ$}}% \n",
    "  \\newcommand{\\arcmin}{\\mbox{$^\\prime$}}% \n",
    "  \\newcommand{\\arcsec}{\\mbox{$^{\\prime\\prime}$}}% \n",
    "  \\newcommand{\\fd}{\\mbox{$.\\!\\!^{\\mathrm d}$}}% \n",
    "  \\newcommand{\\fh}{\\mbox{$.\\!\\!^{\\mathrm h}$}}% \n",
    "  \\newcommand{\\fm}{\\mbox{$.\\!\\!^{\\mathrm m}$}}% \n",
    "  \\newcommand{\\fs}{\\mbox{$.\\!\\!^{\\mathrm s}$}}% \n",
    "  \\newcommand{\\fdg}{\\mbox{$.\\!\\!^\\circ$}}% \n",
    "  \\newcommand{\\slantfrac}{\\case}% \n",
    "  \\newcommand{\\onehalf}{\\slantfrac{1}{2}}% \n",
    "  \\newcommand{\\onethird}{\\slantfrac{1}{3}}% \n",
    "  \\newcommand{\\twothirds}{\\slantfrac{2}{3}}% \n",
    "  \\newcommand{\\onequarter}{\\slantfrac{1}{4}}% \n",
    "  \\newcommand{\\threequarters}{\\slantfrac{3}{4}}% \n",
    "  \\newcommand{\\ubvr}{\\mbox{$U\\!BV\\!R$}}%% UBVR system \n",
    "  \\newcommand{\\ub}{\\mbox{$U\\!-\\!B$}}%   % U-B \n",
    "  \\newcommand{\\bv}{\\mbox{$B\\!-\\!V$}}%   % B-V \n",
    "  \\newcommand{\\vr}{\\mbox{$V\\!-\\!R$}}%   % V-R \n",
    "  \\newcommand{\\ur}{\\mbox{$U\\!-\\!R$}}%   % U-R \n",
    "  \\newcommand{\\ion}[2]{#1$\\;${\\small\\rmfamily\\@Roman{#2}}\\relax}% \n",
    "  \\newcommand{\\nodata}{ ~$\\cdots$~ }% \n",
    "  \\newcommand{\\diameter}{\\ooalign{\\hfil/\\hfil\\crcr\\mathhexbox20D}}% \n",
    "  \\newcommand{\\degr}{\\arcdeg}% \n",
    "  \\newcommand{\\sun}{\\odot}% \n",
    "  \\newcommand{\\Sun}{\\sun}% \n",
    "  \\newcommand{\\Sol}{\\sun}% \n",
    "  \\newcommand{\\Av}{\\ensuremath{{\\mathrm{A}}_{\\mathrm{V}}}}\n",
    "$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "# Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Exercise 1 (30 points) \n",
    "\n",
    "A survey of $50$ galaxies classifies $17$ of them as having \"active nuclei\". \n",
    "On the basis of one theory, model $M_1$ says that the fraction, $p$, of active\n",
    "nuclei in such a survey is $1/3$. Another model, $M_2$, says that this fraction is $1/2$ with a standard\n",
    "deviation, $\\sigma$, of $0.1$ (i.e, this model says that there is a natural variation in $p$). \n",
    "\n",
    "Interpreting this latter statement as a beta distribution, what is the Bayes factor, $\\proba{D \\given M_1} / \\proba{D \\given M_2}$, for these two models? \n",
    "\n",
    "There is some dispute amongst supporters of $M_2$ as to the value of $\\sigma$. Plot the Bayes\n",
    "factor as a function of $\\sigma$ as you vary it from $0.02$ to $0.4$ in small steps. \n",
    "\n",
    "For what values of $\\sigma$, if any, do the evidence favour $M_2$ over $M_1$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Exercise 2 (40 points)\n",
    "\n",
    "Use Bayesian model comparison to decide whether a quadratic model or a linear model is a better\n",
    "fit of $y$ vs. $x$ to the data in file 2Dline `modelcomparison.dat`. That is, calculate the evidence for \n",
    "each model and then their Bayes factor. The quadratic model is defined as \n",
    "$$y = a_0 + a_1\\,x + a_2\\,x^2 + \\mathcal{N}(0, \\sigma)$$\n",
    "The linear model is the same with $a_2 = 0$. \n",
    "\n",
    "Use the following priors for the quadratic model:\n",
    "\\begin{eqnarray}\n",
    "    \\proba{a_0} = \\mathcal{N}(0, 4)\\\\\n",
    "    \\proba{\\arctan a_1} = \\mathcal{U}(-\\pi/2, \\pi/2)\\\\\n",
    "    \\proba{a_2} = \\mathcal{N}(0, 1)\\\\\n",
    "    \\proba{\\log_{10} \\sigma} = \\mathcal{U}(-\\log_{10} 2, \\log_{10} 2)\n",
    "\\end{eqnarray}\n",
    "and the same for the linear model, but of course without $a_2$. $\\mathcal{N}(\\alpha, \\beta)$ indicates a Gaussian with\n",
    "mean $\\alpha$ and standard deviation $\\beta$. $\\mathcal{U}(\\alpha, \\beta)$ indicates a distribution which is uniform between $\\alpha$ and $\\beta$, and zero outside. Use at least $10^5$ samples of the prior. How conclusive is your analysis?\n",
    "Investigate how sensitive your results are to the priors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Exercise 3 (30 points)\n",
    "\n",
    "\n",
    "When Maximum a posteriori differs from Bayesian approach.\n",
    "\n",
    "Carol wants to play a strange game with Bob and Alice. In front on them is a rectangular table. Carol ask Bob and Alice to take positions along one of the short sides of the table. Virtually Carol cuts the table at a random position parallel to the long edge defining a line that Alice and Bob don't know. This stays fixed for the whole game. Carol explains the game to the players: \"You will throw pucks onto the table where ever you want. Any puck stopping on the left side of the line I already defined will give 1 point to Alice, and any on the right side will give one to Bob. The first to get 6 points wins.\"\n",
    "\n",
    "After each of the player has thrown 4 pucks (8 in total), Carols announces that Alice has 5 points and Bob has 3.\n",
    "\n",
    "Alice is one point away from winning and the limit placement seems to favor her. Thus, it seems likely that the next round will go to her as well. And she has three opportunities to get a favorable roll before Bob can win, whereas Bob must win the next three rounds to win the game.  Quantitatively, what is the probability that Bob will nonetheless win the game?\n",
    "\n",
    "\n",
    "    ++==============================================================+ p=0\n",
    "    ||                                                     x        |\n",
    "    ||                                  x     x                     |\n",
    "    ++----- virtual cut --------------------------------------------+ p\n",
    "    ||                                       x            x         |\n",
    "    ||                                                              |\n",
    "    ||                                   x                          |\n",
    "    ||                                                    x         |\n",
    "    ||                                         x                    |\n",
    "    ++==============================================================+ p=1\n",
    "\n",
    "\n",
    "To answer this problem, we can adopt at least 2 approaches: Maximum a posteriori (MAP) or Bayesian. The following questions will help you to answer using both methods.\n",
    "\n",
    "1) **MAP approach**\n",
    "\n",
    "1.1) Given the current score, estimate where the dividing line sits. Use $p$ as the probability that any toss will land in Alice's favor.\n",
    "    \n",
    "1.2) Write down an analytic expression for the probability that Bob wins the game and evaluate it using the given data.\n",
    "\n",
    "\n",
    "2) **Bayesian approach**\n",
    "To help you out, we consider the following random variables:\n",
    "\n",
    "* $B$ = Bob Wins the game,\n",
    "* $D$ = observed data, i.e., $D = (n_A,n_B)=(5, 3)$ the respective number of points for Alice and Bob,\n",
    "* $p$ = unknown probability that a ball lands on Alice's side in any throw.\n",
    "\n",
    "The problem translates into finding $\\proba{B \\given D}$, the posterior probability that Bob wins given our observation that Alice currently has five points and Bob has only three.\n",
    "$\\proba{B \\given D}$ does not depend on $p$. Therefore, we consider the marginalization over the unknown $p$ parameter (nuisance parameter). In this case, that means we express the posterior probability as\n",
    "$$\\proba{B \\given D} = \\int_{0}^{1} \\proba{B, p \\given D}\\, dp$$\n",
    "\n",
    "2.1) Write down an expression for $\\proba{B, p \\given D}$ in terms of known quantities using the low of joint probabilities and Bayes' theorem. And then, show that\n",
    "$$\\proba{B \\given D} = \\frac{\\int_0^1 \\pa{1 - p}^6 p^5 dp}{\\int_0^1 \\pa{1 - p}^3 p^5 dp}$$\n",
    "\n",
    "2.2) give the numerical value of $\\proba{B \\given D}$ in this approach. \n",
    "\n",
    "**hint**: Realize that you known $\\proba{B, p \\given D}$\n",
    "\n",
    "3) Compare the values from both approaches.\n",
    "\n",
    "4) We will make a Monte Carlo simulation of the entire game to see which approach gives a result closer to the simulation result. \n",
    "\n",
    "4.1) How many pucks at most need to be thrown to get to the end of any one game?\n",
    "\n",
    "4.2) Draw $10^6$ random values of $p$ and for each draw as many puck positions (from 0 to 1 corresponding to the left to right postion) as necessary to finish the game and count the points. Select only the games that lead to the current conditions after 8 pucks are thrown, and conclude from this simulation on how many times Bob could win.\n",
    "\n",
    "5) In the light of this test, which approach should you adopt for this particular problem?"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}