McCullochPitts.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## McCulloch Pitts Neuron\n",
    "\n",
    "The 'McCulloch Pitts', or 'MP' model is one of the earliest models of a neuron. \n",
    "\n",
    "The MP is a mathematical model of a neuron, that explores if and how neuron models can implement logical operations.\n",
    "\n",
    "It was first described in 1943, by Warren S. McCulloch and Walter Pitts, in the paper \"A logical calculus of the ideas immanent in nervous activity\". "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MP Neurons\n",
    "\n",
    "The McCulloch Pitts Neuron, or 'MP' unit is now sometimes referred to as a 'linear threshold gate model'.\n",
    "\n",
    "MP units have some number of inputs {x_1, ..., x_n}, which can have values of {0, 1}.\n",
    "\n",
    "These inputs are linearly summed, to measure the activation of the unit. \n",
    "\n",
    "There is then a threshold. If the level of activation is higher than the threshold, the unit ouputs a 1, otherwise, it outputs 0. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MP neuron can be visualized like this:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/McCullochPitts.png\" width=\"300px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MP Neuron in Code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MPNeuron():\n",
    "    \"\"\"A McCulloch Pitts neuron model.\"\"\"\n",
    " \n",
    "    def __init__(self, bias=0, threshold_value=0):\n",
    "        \n",
    "        self.bias = bias\n",
    "        self.threshold_value = threshold_value\n",
    " \n",
    "    def fire(self, inputs, threshold_value=0):\n",
    "        \n",
    "        activation = sum(inputs) + self.bias\n",
    "        output = self.threshold(activation)\n",
    "\n",
    "        return output\n",
    "    \n",
    "    def threshold(self, activation):\n",
    "        \n",
    "        if activation > self.threshold_value:\n",
    "            return 1\n",
    "        else:\n",
    "            return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# An example neuron\n",
    "mp = MPNeuron()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the ouput for a given input\n",
    "inputs = [1, 1]\n",
    "mp.fire(inputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the output for a different input\n",
    "inputs = [1, 0, 1]\n",
    "mp.fire(inputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Try a new neuron with a new threshold value\n",
    "mp = MPNeuron(threshold_value=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Try some inputs on our new neuron\n",
    "inputs = [1, 0, 0]\n",
    "mp.fire(inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculating 'AND'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up the inputs and expected outputs for logical 'AND'\n",
    "input_pairs = [(0,0),(0,1),(1,0),(1,1)]\n",
    "expected_outputs = [0,0,0,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input Pair\t Target Output\n",
      "(0, 0)\t\t 0\n",
      "(0, 1)\t\t 0\n",
      "(1, 0)\t\t 0\n",
      "(1, 1)\t\t 1\n"
     ]
    }
   ],
   "source": [
    "# Check out inputs and expected outputs\n",
    "print(\"Input Pair\\t Target Output\")\n",
    "for input_pair, expected_output in zip(input_pairs, expected_outputs):\n",
    "    print(str(input_pair) + \"\\t\\t \" + str(expected_output))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize an MP neuron that can calculate 'AND'\n",
    "mp_and = MPNeuron(0, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input Pair\t Calculated Output\t Target Output\n",
      "(0, 0)\t\t 0\t\t\t 0\n",
      "(0, 1)\t\t 0\t\t\t 0\n",
      "(1, 0)\t\t 0\t\t\t 0\n",
      "(1, 1)\t\t 1\t\t\t 1\n"
     ]
    }
   ],
   "source": [
    "# See if our MP neuron can calculate logical 'AND'\n",
    "print(\"Input Pair\\t Calculated Output\\t Target Output\")\n",
    "for input_pair, expected_output in zip(input_pairs, expected_outputs):\n",
    "    out = mp_and.fire(input_pair)\n",
    "    print(str(input_pair) + \"\\t\\t \" + str(out) + \"\\t\\t\\t \"+ str(expected_output))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculating 'OR'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up the inputs and expected outputs for logical 'AND'\n",
    "input_pairs = [(0,0),(0,1),(1,0),(1,1)]\n",
    "expected_outputs = [0,1,1,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input Pair\t Target Output\n",
      "(0, 0)\t\t 0\n",
      "(0, 1)\t\t 1\n",
      "(1, 0)\t\t 1\n",
      "(1, 1)\t\t 1\n"
     ]
    }
   ],
   "source": [
    "# Check out inputs and expected outputs\n",
    "print(\"Input Pair\\t Target Output\")\n",
    "for input_pair, expected_output in zip(input_pairs, expected_outputs):\n",
    "    print(str(input_pair) + \"\\t\\t \" + str(expected_output))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize an MP neuron that can calculate 'OR'\n",
    "mp_or = MPNeuron(1, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input Pair\t Calculated Output\t Target Output\n",
      "(0, 0)\t\t 0\t\t\t 0\n",
      "(0, 1)\t\t 1\t\t\t 1\n",
      "(1, 0)\t\t 1\t\t\t 1\n",
      "(1, 1)\t\t 1\t\t\t 1\n"
     ]
    }
   ],
   "source": [
    "# See if our MP neuron can calculate logical 'AND'\n",
    "print(\"Input Pair\\t Calculated Output\\t Target Output\")\n",
    "for input_pair, expected_output in zip(input_pairs, expected_outputs):\n",
    "    out = mp_or.fire(input_pair)\n",
    "    print(str(input_pair) + \"\\t\\t \" + str(out) + \"\\t\\t\\t \"+ str(expected_output))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extending the MP Neuron"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The description above is the most simple neuron model we can have. \n",
    "\n",
    "From here, we imagine adding some more complexity to this neuron. \n",
    "\n",
    "In particular, the original MP neuron also has a conceptualization of 'excitatory' and 'inhibitory' neurons. \n",
    "\n",
    "We can implement this with the the concept of 'weights'. \n",
    "\n",
    "In the MP neuron case, remember that our inputs can be {0, 1}. To implement 'excitatory' and 'inhibitory' inputs, we can use weights of {-1, 0, 1}. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MPNeuron():\n",
    "    \"\"\"A McCulloch Pitts neuron model, with weights.\"\"\"\n",
    " \n",
    "    def __init__(self, weights, bias=0, threshold_value=0):\n",
    "        \n",
    "        self.weights = weights\n",
    "        self.bias = bias\n",
    "        self.threshold_value = threshold_value\n",
    " \n",
    "    def fire(self, inputs, threshold_value=0):\n",
    "        \n",
    "        activation = sum([i*w for (i, w) in zip(inputs, self.weights)])\n",
    "        output = self.threshold(activation + self.bias)\n",
    "\n",
    "        return output\n",
    "    \n",
    "    def threshold(self, activation):\n",
    "        if activation > self.threshold_value:\n",
    "            return 1\n",
    "        else:\n",
    "            return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make a neuron, with weights\n",
    "mp = MPNeuron([1, -1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Try some inputs on our MP neuron, that has weights\n",
    "inputs = [1, 1]\n",
    "mp.fire(inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Limitations of the MP Unit\n",
    "\n",
    "- Boolean values: all inputs and ouputs are 0 or 1, which is limiting\n",
    "- We can set, or try to learn, the threshhold value `b`, but it's range is limited\n",
    "- There is no clear training procedure to find the particular model we want to achieve a particular goal"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}