01 TensorFlow Fundamentals.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# TensorFlow Fundamentals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Python > 3, TensorFlow > 1.0\n",
    "# Import core TensorFlow libraries\n",
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# My First TensorFlow Graph\n",
    "\n",
    "![Basic basic graph](images/dataflow_basic.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Define the graph\n",
    "\n",
    "* Nodes represent some sort of computation\n",
    "* Edges represent data transfer from one computation to the next\n",
    "\n",
    "Remember that no computation actually takes place until we run it in a `tf.Session`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# `tf.placeholder` creates an \"input\" node- we will give it value when we run our model\n",
    "a = tf.placeholder(tf.int32, name=\"input_a\")\n",
    "b = tf.placeholder(tf.int32, name=\"input_b\")\n",
    "\n",
    "# `tf.add` creates an addition node\n",
    "c = tf.add(a, b, name=\"add\")\n",
    "\n",
    "# `tf.mul` creates a multiplication node\n",
    "d = tf.multiply(a, b, name=\"multiply\")\n",
    "\n",
    "# Add up the results of the previous two nodes\n",
    "out = tf.add(c, d, name=\"output\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Run the graph\n",
    "\n",
    "* Start a `tf.Session` to launch the graph\n",
    "* Setup any necessary input values\n",
    "* Use `Session.run()` to compute values from the graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "23\n"
     ]
    }
   ],
   "source": [
    "# Start a session\n",
    "sess = tf.Session()\n",
    "\n",
    "# Create a \"feed_dict\" dictionary to define input values\n",
    "# Keys to dictionary are handles to our placeholders\n",
    "# Values to dictionary are values we'd like to feed in\n",
    "feed_dict = { a: 5, b: 3 }\n",
    "\n",
    "output = sess.run(out, feed_dict=feed_dict)\n",
    "\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Once we're done with our compuations, we can close down our `Session`\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# TensorFlow Core API\n",
    "\n",
    "---\n",
    "\n",
    "# `Tensor` Objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is a Tensor?\n",
    "\n",
    "Tensors, simply put, are _n_-dimensional matrices. A 0-dimensional tensor is a single number (or scalar), a 1-dimensional tensor is a vector, and a 2-dimensional tensor is a standard matrix. Higher dimensional tensors are simply referred to as an \"_n_-D tensor\"\n",
    "\n",
    "Every value that is passed through a TensorFlow model is a `Tensor` object- the TensorFlow representation of a tensor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining tensors by hand\n",
    "\n",
    "You can define `Tensor` object values in two main ways:\n",
    "\n",
    "1. Native Python types\n",
    "2. NumPy arrays (recommended)\n",
    "\n",
    "Both of these are able to be automatically converted into TensorFlow `Tensor` objects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Native Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 0-D tensor (scalar)\n",
    "t_0d_py = 4\n",
    "\n",
    "# 1-D tensor (vector)\n",
    "t_1d_py = [1, 2, 3]\n",
    "\n",
    "# 2-D tensor (matrix)\n",
    "t_2d_py = [[1, 2], \n",
    "           [3, 4], \n",
    "           [5, 6]]\n",
    "\n",
    "# 3-D tensor\n",
    "t_3d_py = [[[0, 0], [0, 1], [0, 2]],\n",
    "           [[1, 0], [1, 1], [1, 2]],\n",
    "           [[2, 0], [2, 1], [2, 2]]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NumPy Arrays\n",
    "\n",
    "Pretty much the same as native Python, but with the `numpy.array` function wrapping it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 0-D tensor (scalar)\n",
    "t_0d_np = np.array(4, dtype=np.int32)\n",
    "\n",
    "# 1-D tensor (vector)\n",
    "t_1d_np = np.array([1, 2, 3], dtype=np.int64)\n",
    "\n",
    "# 2-D tensor (matrix)\n",
    "t_2d_np = np.array([[1, 2], \n",
    "                    [3, 4], \n",
    "                    [5, 6]],\n",
    "                   dtype=np.float32)\n",
    "\n",
    "# 3-D tensor\n",
    "t_3d_np = np.array([[[0, 0], [0, 1], [0, 2]],\n",
    "                    [[1, 0], [1, 1], [1, 2]],\n",
    "                    [[2, 0], [2, 1], [2, 2]]],\n",
    "                   dtype=np.int32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data types\n",
    "\n",
    "In general, using `np.array` (or `np.asarray`) is the recommended way of defining values for tensors by hand in TensorFlow. The primary reason for this is that you can specify the exact data type (\"dtype\") you'd like the values to be represented with. For example, there's no way to specify a 32-bit integer vs a 64-bit integer with native Python. TensorFlow is tightly integrated with NumPy, and most TensorFlow data types have a corresponding NumPy `dtype`:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorFlow type | Equivalent NumPy type | Description\n",
    "--- | --- | ---\n",
    "`tf.float32` | `np.float32` | 32 bit floating point.\n",
    "`tf.float64` | `np.float64` | 64 bit floating point.\n",
    "`tf.int8` | `np.int8` | 8 bit signed integer.\n",
    "`tf.int16` | `np.int16` | 16 bit signed integer.\n",
    "`tf.int32` | `np.int32` | 32 bit signed integer.\n",
    "`tf.int64` | `np.int64` | 64 bit signed integer.\n",
    "`tf.uint8` | `np.uint8` | 8 bit unsigned integer.\n",
    "`tf.string` | N/A | String type, as byte array\n",
    "`tf.bool` | `np.bool` | Boolean.\n",
    "`tf.complex64` | `np.complex64` | Complex number made of two 32 bit floating point numbers: real and imaginary parts.\n",
    "`tf.qint8` | N/A | 8 bit signed integer used in quantized Ops.\n",
    "`tf.qint32` | N/A | 32 bit signed integer used in quantized Ops.\n",
    "`tf.quint8` | N/A | 8 bit unsigned integer used in quantized Ops.\n",
    "\n",
    "Slightly modified version of [this table](https://www.tensorflow.org/versions/master/resources/dims_types.html#data-types)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Just to show that they are equivalent\n",
    "(tf.float32 == np.float32 and\n",
    " tf.float64 == np.float64 and\n",
    " tf.int8 == np.int8 and\n",
    " tf.int16 == np.int16 and\n",
    " tf.int32 == np.int32 and\n",
    " tf.int64 == np.int64 and\n",
    " tf.uint8 == np.uint8 and\n",
    " tf.bool == np.bool and\n",
    " tf.complex64 == np.complex64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The primary exception to when you should _not_ use `np.array()` is when defining a `Tensor` of strings. When using strings, just use standard Python lists. It's best practice to include the `b` prefix in front of strings to explicitly define the strings as byte-arrays:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf_string_tensor = [b\"first\", b\"second\", b\"third\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tensor Shapes\n",
    "\n",
    "A common term in TensorFlow is a `Tensor` object's \"shape\". A shape value is a list or tuple containing an ordered set of integers. The _i_-th  element in the list describes the length of the _i_-th dimension in the tensor, while the number of elements in the list defines the dimensionality of the tensor. Here are some examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Shapes corresponding to scalars\n",
    "# Note that either lists or tuples can be used\n",
    "s_0d_list = []\n",
    "s_0d_tuple = ()\n",
    "\n",
    "# Shape corresponding to a vector of length 3\n",
    "s_1d = [3]\n",
    "\n",
    "# Shape corresponding to a 2-by-3 matrix\n",
    "s_2d = (2, 3)\n",
    "\n",
    "# Shape corresponding to a 4-by-4-by-4 cube tensor\n",
    "s_3d = [4, 4, 4]\n",
    "\n",
    "s_var = [None, 4, 4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use the `tf.shape` Operation to get the shape value of `Tensor` objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of tensor: [4 2 3]\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    get_shape = tf.shape([[[1, 2, 3], [1, 2, 3]],\n",
    "                          [[2, 4, 6], [2, 4, 6]],\n",
    "                          [[3, 6, 9], [3, 6, 9]],\n",
    "                          [[4, 8, 12], [4, 8, 12]]])\n",
    "    shape = sess.run(get_shape)\n",
    "    print(\"Shape of tensor: \" + str(shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Constants\n",
    "\n",
    "You can create `Tensor` constants in your TensorFlow graph easily. Just use the `tf.constant` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "my_const = tf.constant(np.array([1, 2, 3], dtype=np.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a set of values is going to be reused all throughout your graph, using constants is an easy way to place that value directly into the graph (instead of reading from a NumPy array or Python list directly)\n",
    "\n",
    "Note: all `Tensor` objects are immutable. The constant type is simply a convenient way to add basic `Tensor` values to a graph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Note on `SparseTensor`\n",
    "\n",
    "TensorFlow has implementations of sparse tensor representations, or tensors whose entries primarily consist of zeros. In some instances, `SparseTensor` and `Tensor` objects can be intermixed, but more often than not they require more care. Because the `SparseTensor` API isn't as robust as the `Tensor` API and for the sake of keeping things digestible, we won't cover `SparseTensor` objects today."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorFlow `Operation` objects (also referred to as \"Ops\" in the TensorFlow documentation- we will avoid that usage today to avoid mixing DevOps and TensorFlow Ops) are nodes that perform compuation on or with Tensor objects. They take as input zero or more `Tensor` objects (or objects that can be converted into tensors- see the previous section), and output zero or more tensors. These outputs can then either be returned to the client or passed on to further Operations. Operations are the fundamental building blocks of any TensorFlow graph- their calculations represent nodes, and data flowing from one to the next represents edges.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've already seen several Operations earlier: `tf.add` and `tf.mul` are classic examples: they both take in two tensors and output one. When given non-scalar values, they do addition/multiplication element-wise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Initialize some tensors\n",
    "a = np.array([1, 2], dtype=np.int32)\n",
    "b = np.array([3, 4], dtype=np.int32)\n",
    "\n",
    "# `tf.add()` creates an \"add\" Operation and places it in the graph\n",
    "# The variable `c` will be a handle to the output of the operation\n",
    "# This output can be passed on to other Operations!\n",
    "c = tf.add(a, b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The important thing to remember is that Operations do not execute when created- that's the reason `tf.add([1, 2],[3, 4])` doesn't return the value `[4, 6]` immediately. It must be passed into a `Session.run()` method, which we'll cover in more detail below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4 6]\n"
     ]
    }
   ],
   "source": [
    "sess = tf.Session()\n",
    "print(sess.run(c))\n",
    "\n",
    "c_result = sess.run(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The majority of the TensorFlow API is Operations. `tf.scalar_summary` and `tf.placeholder` were both Operations we used in the first example- remember that we had to run the `out_summary` variable in `Session.run()`\n",
    "\n",
    "In addition to Operation-specific inputs, each Operation can take in a `name` parameter, which can help identify Operations in TensorBoard and other tools."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "c = tf.add(a, b, name=\"my_add_operation\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting into the habit of adding names to your Operations now will save you headaches later on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow `Graph` Objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When TensorFlow is imported into Python, it automatically creates a `Graph` object and makes it the default graph. You can create more graphs as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create a new graph - constructor takes no parameters\n",
    "new_graph = tf.Graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, operations (such as `tf.add` and `tf.mul`) are added to the default graph when created. To add operations to your new graph, use a `with` statement along with the graph's `as_default()` method. This makes that graph the default while inside of the `with` block:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# DEFAULT GRAPH\n",
    "\n",
    "co = tf.constant(4)\n",
    "\n",
    "with new_graph.as_default():\n",
    "    a = tf.add(3, 4)\n",
    "    b = tf.multiply(a, 2)\n",
    "    other_co = tf.constant(6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sess = tf.Session(graph=tf.get_default_graph())\n",
    "sess.run(co)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default graph, other than being set to the default, is no different than any other `Graph`. If you need to get a handle to the default graph, use the `tf.get_default_graph` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "default_graph = tf.get_default_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: `get_default_graph()` will return whatever graph is set to the default, so if you are inside of a `with g.as_default()` block, `get_default_graph()` will return `g`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "with new_graph.as_default():\n",
    "    print(new_graph is tf.get_default_graph())\n",
    "\n",
    "print(new_graph is tf.get_default_graph())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most TensorFlow models will not require more than one graph per script. However, you may find this useful when defining two independent models side-by-side. Additionally, there are mechanisms to export and import external models and load them in as `Graph` objects, which can allow you to feed the output of existing models into your new model (or vice versa). We won't be able to demonstrate these now, but see [`Graph.as_graph_def()`](https://www.tensorflow.org/versions/master/api_docs/python/framework.html#Graph.as_graph_def) and [`tf.import_graph_def`](https://www.tensorflow.org/versions/master/api_docs/python/framework.html#import_graph_def) in the TensorFlow API for more information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sessions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Sessions\n",
    "\n",
    "As we saw earlier, `Session` objects are used to launch and execute graphs. Earlier, we created a session using its default constructor, but it has three optional parameters:\n",
    "\n",
    "* `target` specifies the execution engine to use. By default it is the empty string, which causes the Session to use the standard local execution context. Typically, this parameter is only used when using TensorFlow in a distributed setting\n",
    "* `graph` specifies which `Graph` object the session should run. The default value is `None`, which causes the `Session` to load in the default graph. Sessions only manage one graph at a time, so executing more than one graph will require more than one session\n",
    "* `config` allows users to specify advanced options to configure the session. We won't cover this today, but some things that are available are: limiting the number of CPUs/GPUs used, logging options, and changing optimization of the graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# A session with the default graph launched\n",
    "# Equivalent to `tf.Session(graph=tf.get_default_graph())`\n",
    "sess_default = tf.Session()\n",
    "\n",
    "# A session with new_graph launched\n",
    "sess_new = tf.Session(graph=new_graph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running Sessions\n",
    "\n",
    "The most important method of a `Session` is its `run()` function. Earlier in this notebook, we saw basic usage of the two primary parameters to `run()`: `fetches` and `feed_dict`.\n",
    "\n",
    "### `fetches`\n",
    "\n",
    "`fetches` expects a list of `Tensor` and/or `Operation` handles (or just a single `Tensor`/`Operation`). The list specifies what computations we would like TensorFlow to run, as well as what we'd like `run()` to output: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sess_default.run(tf.add(3,2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorFlow will only perform calculations necessary to compute the values specified in `fetches`, so it won't waste time if you only need to run a small part of a large, complicated graph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `feed_dict`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`feed_dict` is an optional parameter to `run`, but becomes required when placeholder nodes are included. We saw it used to feed input data to placeholders, but `feed_dict` can actually send values to any node. The keys to the dictionary should be handles to `Tensor` objects (usually outputs of Operations), and the values should be replacement data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create Operations, Tensors, etc (using the default graph)\n",
    "a = tf.add(3, 4)\n",
    "b = tf.multiply(a, 5)\n",
    "\n",
    "# Define a dictionary that says to replace the value of `a` with 15\n",
    "replace_dict = {a: 15}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "35\n"
     ]
    }
   ],
   "source": [
    "# Run the session without feed_dict\n",
    "# Prints (3 + 4) * 5 = 35\n",
    "print(sess_default.run(b))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "75\n"
     ]
    }
   ],
   "source": [
    "# Run the session, passing in `replace_dict` as the value to `feed_dict`\n",
    "# Prints 15 * 5 = 75 instead of 7 * 5 = 35\n",
    "print(sess_default.run(b, feed_dict=replace_dict))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When using placeholders,TensorFlow insists that any calls to `Session.run()` include `feed_dict` values for all placeholders:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a = tf.placeholder(tf.int32, name=\"my_placeholder\")\n",
    "b = tf.add(a, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You must feed a value for placeholder tensor 'my_placeholder_1' with dtype int32\n",
      "\t [[Node: my_placeholder_1 = Placeholder[dtype=DT_INT32, shape=[], _device=\"/job:localhost/replica:0/task:0/cpu:0\"]()]]\n"
     ]
    }
   ],
   "source": [
    "# This raises an error:\n",
    "try:\n",
    "    sess_default.run(b)\n",
    "except tf.errors.InvalidArgumentError as e:\n",
    "    print(e.message)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11\n"
     ]
    }
   ],
   "source": [
    "# Create feed dictionary\n",
    "feed_dict = {a: 8}\n",
    "\n",
    "# Now it works!\n",
    "print(sess_default.run(b, feed_dict=feed_dict))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Closing out the Sessions we opened up\n",
    "sess_default.close()\n",
    "sess_new.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# TensorFlow Variables\n",
    "\n",
    "The last fundamental TensorFlow class is the `Variable`. A TensorFlow `Variable` has persistent state across multiple calls to `Session.run()`, which means that learned parameters in machine\" learning models are Variables. We can create a Variable with a starting value of 0 like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "my_var = tf.Variable(0, name=\"my_var\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, even though the object has been created, the value of the `Variable` has to be initialized separately with either of the `tf.initialize_variables()` or, more commonly, `tf.initialize_all_variables()` Operations. Remember that Operations must be passed into `Session.run()` to be executed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sess = tf.Session()\n",
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having value initialization separated from object creation allows us to have multiple Sessions training variations on the same graph. We can also easily reinitialize the variable later if we'd like.\n",
    "\n",
    "How that the Variable is initialized, we can tweak it's value! Let's do some basic incrementing with the `Variable.assign()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n",
      "4\n",
      "5\n",
      "6\n",
      "7\n",
      "8\n",
      "9\n",
      "10\n"
     ]
    }
   ],
   "source": [
    "increment = my_var.assign(my_var + 1)\n",
    "\n",
    "for i in range(10):\n",
    "    print(sess.run(increment))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may notice that if you run the previous code multiple times in the notebook, the value persists and continues to climb. The Variable's state is maintained by the `Session` object, and the state will persist unless either the session is close, the Variable is re-initialized, or a new value is assigned to the Variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n",
      "4\n",
      "5\n",
      "6\n",
      "7\n",
      "8\n",
      "9\n",
      "10\n"
     ]
    }
   ],
   "source": [
    "# Re-initialize variables\n",
    "sess.run(tf.global_variables_initializer())\n",
    "\n",
    "# Start incrementing, beginning from 0 again\n",
    "for i in range(10):\n",
    "    print(sess.run(increment))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Trainable\n",
    "\n",
    "There are several optional parameters in the `Variable` constructor, but one to pay close attention to is `trainable`. It takes in a boolean value, which defaults to `True`, and specifies to TensorFlow whether the built-in optimization functions (which we will cover in a separate notebook) should affect this `Variable`. **If a `Variable` in your model should _not_ be adjusted during gradient descent, make sure to set its `trainable` parameter to `False`**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `tf.get_variable()`\n",
    "\n",
    "Though the basic `Variable()` constructor is intuitive and good for beginners, eventually we would encourage you to move on to using the `tf.get_variable()` method for creating and accessing `Variable` objects. It allows users to more easily share Variables across complicated models, where handles to exact Variables can be lost or hard to manage. We will show some examples with `tf.get_variable()` in another notebook, but do check out the [official how-to](https://www.tensorflow.org/versions/r0.8/how_tos/variable_scope/index.html), as `tf.get_variable()` is best practice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sess.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}