Vanilla GAN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning Vanilla GAN by code\n",
    "\n",
    "### The theory\n",
    "GANs are deep neural networks scheme comprised of two deep nets, one is called the generator and the other is called the discriminator. First introduced in 2014 by Ian J. Goodfellow[1], the potential for GANs are huge. It can be used for image-to-image translations and general classifications. \n",
    "\n",
    "If you haven't read the paper, I really recommend you to skim it first.  \n",
    "The basic idea of GANs is to make the aforementioned neural networks (generator and discriminator) compete against each other. The process flow of GANs is shown below\n",
    "<img src=\"notebook_images/gan_diagram.png\" width=\"300\" height=\"200\"></png>\n",
    "As an illustration, a single GAN can be seen as the combination of a counterfeiter (generator) and a cop (discriminator). The counterfeiter is trying to make false notes (money) to fool the cop. Meanwhile, the cop is also learning to detect them. Since both are training to beat each other, GANs have potential to do a lot of things better than conventional neural networks.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The maths\n",
    "\n",
    "<img src=\"notebook_images/gan_formula.png\" width=\"500\" height=\"200\"></img>\n",
    "\n",
    "Basically gan is trying to learn the data distribution using the generative model with help of discriminative model.  \n",
    "\n",
    "\n",
    "The formula shown above simply mean that `we are trying to minimize the expected value of of` $D(x)$ `if we sample x from probablity distribution of data1` $P_{data}$ `, and maximize the value of` $G(z)$ `if we sample of z from distribution of noise` $P_{z}$.  \n",
    "\n",
    "We are trying to optimum the value of this formula. \n",
    "\n",
    "<img src=\"notebook_images/gan_graph.png\" width=\"500\" height=\"200\"> </img>\n",
    "\n",
    "- the blue-dashed line is a discriminative distribution.  \n",
    "- black-dathed line is data distribution. $P_{x}$\n",
    "- green-solid line is from generative distribution $P_{g}$\n",
    "\n",
    "This graph shows that first there are sampling from distribution z (vertical line from domain z to domain x). \n",
    "As time goes (picture to right), model G will try to fit the distribution of data. At the end\n",
    "the generator fit the data and the the discriminator can't differentiate the distribution of data and generator. \n",
    "At the end, they will reach a point at which both cannot improve because pg = pdata. \n",
    "The discriminator is unable to differentiate between the two distributions, i.e. D(x) = 1 .\n",
    "\n",
    "### The algorithm\n",
    "Assume we have data distribution $P_{x}$.  \n",
    "We want to learn $P_{g}$ over data $x$. We define prior input noise $P_{z}$. First we take sample $z$ then we feed it into model $g(z,w)$ where w is the parameter to fit model *g* into the distribution $p_{x}$. \n",
    "where $d(q,w)$ is a discriminator model that output single probability (logistic) if data comes from $P_{z}$ or $P_{x}$ (fake vs real)\n",
    "\n",
    "The GAN algorithm can be seen\n",
    "<img src=\"notebook_images/gan_algo.png\" height=\"600\" width=\"550\"></img>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "____\n",
    "### Code implementation\n",
    "The code below show the implementation of GAN on tensorflow  \n",
    "source https://github.com/emsansone/GAN/blob/master/gan.py\n",
    "\n",
    "GAMES is number of iteration all.   \n",
    "DISCR_UPDATE is when step to update discriminator  \n",
    "and GEN_UPDATE is when step to update generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/syahdeini/tools/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "import __future__ \n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.gridspec as gridspec\n",
    "import matplotlib.image as mpimg\n",
    "import matplotlib.animation as animation\n",
    "import seaborn\n",
    "from scipy.stats import norm\n",
    "import os\n",
    "\n",
    "GAMES = 500\n",
    "DISCR_UPDATE = 100\n",
    "GEN_UPDATE = 1\n",
    "\n",
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Frist we define $P_{X}$ which distribution of our real data x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training data\n",
    "class RealDistribution:\n",
    "    def __init__(self):\n",
    "        self.mu = 5\n",
    "        self.sigma = 1\n",
    "\n",
    "    def sample(self, N):\n",
    "        samples = np.random.normal(self.mu, self.sigma, N)\n",
    "        return samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then define $P_{z}$ which is noise distribution, we will use sampling from this distribution and feed it in our generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Noise data\n",
    "class NoiseDistribution:\n",
    "    def __init__(self):\n",
    "        self.low = 0\n",
    "        self.high = 1\n",
    "\n",
    "    def sample(self, N):\n",
    "        samples = np.random.uniform(self.low, self.high, N)\n",
    "        return samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "  \n",
    "In this tutorial we define  \n",
    "$g(x)$ as a linear function ($z*w$) \n",
    "\n",
    "** Tensorflow info: each variable works in different scope"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GAN:\n",
    "    # each scope DISC and GEN have different w and b parameter\n",
    "    def linear(self, input, scope=None):\n",
    "        init_w = tf.random_normal_initializer(stddev=0.1)\n",
    "        init_b = tf.constant_initializer(0.0)\n",
    "        # initializer only works for first time\n",
    "        with tf.variable_scope(scope or 'linear'):              # USING SCOPE FOR FUTURE VERSION WITH MULTIPLE LAYERS\n",
    "            w = tf.get_variable('w', [1,1], initializer=init_w)\n",
    "            b = tf.get_variable('b', [1,1], initializer=init_b)\n",
    "            return tf.add(tf.matmul(w, input), b)\n",
    "\n",
    "    def generator(self, input):\n",
    "        logits = self.linear(input, 'gen')\n",
    "        return logits\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Discrimnator works by differentiate if data come from real or fake.\n",
    "so we use sigmoid as last function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GAN(GAN):\n",
    "    def discriminator(self, input):\n",
    "            logits = self.linear(input, 'discr')\n",
    "            pred = tf.sigmoid(logits)\n",
    "            return pred\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "____\n",
    "**Creating Model**\n",
    "\n",
    "About hessian, if you see the tutorial. they used heissian to know that the descriminator is in the saddle point. \n",
    "I am really not sure and understand about the plot. but if hessian eigen values is used to  see the multivariable domain\n",
    "- If the Hessian at a given point has all positive eigenvalues then the multivariable domain is “concave up”\n",
    "- if it's negative then it's convave down. \n",
    "- if it's one negative and one positive then it's the saddle point, the graph is concave up in one direction and concave down in other direction\n",
    "If all of the eigenvalues are negative, it is said to be a negative-definite matrix. This is like “concave down”\n",
    "### Generator and Discriminator Model\n",
    "1. Generating the variable for generative model (G). \n",
    "2. Generating variable for Discrimnative model (D).\n",
    "3. Generating loss function \n",
    "    * $G = (1-D(G(z))$\n",
    "    * $D = D + (1-D(G(z))$\n",
    "    \n",
    "4. Getting a list of parameter, we use that to feed the optimizer\n",
    "5. Set the optimizer for each variable using gradient descent by attaching the loss function\n",
    "   and which variable to update (minimizing the loss)\n",
    "6. function in tf to generate the gradient (it not important in our graph, this is used to see \n",
    "   the gradient. after\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GAN(GAN):\n",
    "    def __init__(self):\n",
    "        self.games = GAMES\n",
    "        self.discriminator_steps = DISCR_UPDATE\n",
    "        self.generator_steps = GEN_UPDATE\n",
    "        self.learning_rate = 0.1\n",
    "        self.num_samples = 10\n",
    "        self.skip_log = 20\n",
    "        \n",
    "        self.noise = NoiseDistribution()\n",
    "        self.data = RealDistribution()\n",
    "\n",
    "        self.create_model()\n",
    "        \n",
    "    def create_model(self):\n",
    "        # 1. Generator (G)\n",
    "        with tf.variable_scope('GEN'):\n",
    "            self.z = tf.placeholder(tf.float32, shape=(1, self.num_samples))\n",
    "            # z to G. G(z)\n",
    "            self.gen = self.generator(self.z)\n",
    "            \n",
    "        # 2. Discriminator (D)\n",
    "        with tf.variable_scope('DISC') as scope:\n",
    "            self.x = tf.placeholder(tf.float32, shape=(1, self.num_samples))\n",
    "            self.discr_x = self.discriminator(self.x)\n",
    "            scope.reuse_variables()\n",
    "            self.discr_g_x = self.discriminator(self.gen)\n",
    "\n",
    "        # 3. Losses\n",
    "        self.loss_gen = tf.reduce_mean(tf.log(1-self.discr_g_x))\n",
    "        self.loss_discr = tf.reduce_mean(-tf.log(self.discr_x) -tf.log(1-self.discr_g_x))\n",
    "\n",
    "        # 4. Parameters\n",
    "        self.gen_params = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope='GEN')\n",
    "        self.discr_params = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope='DISC')\n",
    "        self.all_params = tf.trainable_variables()\n",
    "        \n",
    "        # 5. Optimizers (this optimizer who's actually updating the parameter)\n",
    "        self.opt_gen = tf.train.GradientDescentOptimizer(self.learning_rate).minimize(\n",
    "            self.loss_gen,\n",
    "            var_list=self.gen_params\n",
    "        )\n",
    "        self.opt_discr = tf.train.GradientDescentOptimizer(self.learning_rate).minimize(\n",
    "            self.loss_discr,\n",
    "            var_list=self.discr_params\n",
    "        )\n",
    "        \n",
    "        \n",
    "        # 6. gradients (This function is just used to calculate the gradient after updating it \n",
    "        # using gradient descent)\n",
    "        self.grad_discr = tf.gradients(self.loss_discr, self.discr_params)[0]\n",
    "        self.grad_gen = tf.gradients(self.loss_gen, self.gen_params)[0]\n",
    "        \n",
    "        \n",
    "        # Hessian computation\n",
    "#         hessian = []\n",
    "#         for v1 in self.all_params:\n",
    "#             temp = []\n",
    "#             for v2 in self.all_params:\n",
    "#                 # computing derivative twice, first w.r.t v2 and then w.r.t v1\n",
    "#                 temp.append(tf.gradients(tf.gradients(-self.loss_discr, v2)[0], v1)[0])\n",
    "#             temp = [tf.constant(0, dtype=tf.float32) if t == None else t for t in temp] # tensorflow returns None when there is no gradient, so we replace None with 0\n",
    "#             temp = tf.stack(temp)\n",
    "#             hessian.append(temp)\n",
    "#         self.hessian = tf.squeeze(tf.stack(hessian))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The loss function for generator and discriminator as seen in the algorithm above and define in the code below.  \n",
    "loss $gen = 1 - D(x)$  \n",
    "loss $discr = D(x) + (1-D(x))$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GAN(GAN):\n",
    "    def train(self):\n",
    "        init = tf.global_variables_initializer()\n",
    "\n",
    "        with tf.Session() as sess:\n",
    "            sess.run(init)\n",
    "            \n",
    "            x = self.data.sample(self.num_samples)\n",
    "            objective_function = []\n",
    "            grad_magn_discr = []\n",
    "            grad_magn_gen = []\n",
    "            eigs = []\n",
    "            frames = 0\n",
    "        \n",
    "            for games in range(self.games):\n",
    "                # Update discrimintator\n",
    "                z = self.noise.sample(self.num_samples)\n",
    "                for discr_steps in range(self.discriminator_steps):\n",
    "                        loss_discr, _ = sess.run([self.loss_discr, self.opt_discr],{\n",
    "                            self.x : np.reshape(x, (1, self.num_samples)),\n",
    "                            self.z : np.reshape(z, (1,self.num_samples))\n",
    "                        })\n",
    "                grad_discr_val = sess.run(self.grad_discr, feed_dict={\n",
    "                    self.x: np.reshape(x, (1, self.num_samples)),\n",
    "                    self.z: np.reshape(z, (1, self.num_samples))\n",
    "                })\n",
    "                grad_magn_discr.append(np.linalg.norm(grad_discr_val))\n",
    "\n",
    "                # intermediate visualization\n",
    "                if games % self.skip_log == 0:\n",
    "                    print('game %d: Loss: %.3f\\tTarget loss: %.3f' %(games, -loss_discr, -2*np.log(2)))\n",
    "                    self.intuition(sess, x)\n",
    "                    frame = plt.gca()\n",
    "                    frame.axes.get_yaxis().set_visible(False)\n",
    "                    plt.draw()\n",
    "                    plt.pause(0.01)\n",
    "                    plt.clf()\n",
    "\n",
    "                # update generator\n",
    "                for gen_steps in range(self.generator_steps):\n",
    "                    z = self.noise.sample(self.num_samples)\n",
    "                    loss_gen, _ = sess.run([self.loss_gen, self.opt_gen], {\n",
    "                        self.z: np.reshape(z, (1, self.num_samples))\n",
    "                    })\n",
    "                grad_gen_val = sess.run(self.grad_gen, feed_dict={\n",
    "                    self.z: np.reshape(z,(1, self.num_samples))\n",
    "                })\n",
    "                grad_magn_gen.append(np.linalg.norm(grad_gen_val))\n",
    "\n",
    "\n",
    "                                # Intermediate visualization\n",
    "                if games % self.skip_log == 0:\n",
    "                    print('game %d: Loss: %.3f\\tTarget loss: %.3f' % (games, -loss_discr, -2*np.log(2)))\n",
    "                    self.intuition(sess, x)\n",
    "                    frame = plt.gca()\n",
    "                    frame.axes.get_yaxis().set_visible(False)\n",
    "                    plt.savefig('img/img-'+str(games)+'.png')\n",
    "                    plt.draw()\n",
    "                    plt.pause(0.01)\n",
    "                    plt.clf()\n",
    "                    frames += 1\n",
    "\n",
    "                objective_function.append(-loss_discr)\n",
    "\n",
    "            ### below this code it's all for visualization    \n",
    "            # Visualization\n",
    "            plt.close()\n",
    "            print('\\nSaving summary...\\n')\n",
    "            gs = gridspec.GridSpec(2, 2)\n",
    "\n",
    "            # Graphical interpretation\n",
    "            plt.subplot(gs[0,0])\n",
    "            self.intuition(sess, x)\n",
    "            frame = plt.gca()\n",
    "            frame.axes.get_yaxis().set_visible(False)\n",
    "\n",
    "            # Objective function\n",
    "            plt.subplot(gs[0,1])\n",
    "            self.objective(objective_function, games)\n",
    "\n",
    "            # Gradient discriminator\n",
    "            plt.subplot(gs[1,0])\n",
    "            plt.plot(range(self.games),grad_magn_discr)\n",
    "            plt.title('Gradient magnitude - Discriminator')\n",
    "\n",
    "            # Gradient generator\n",
    "            plt.subplot(gs[1,1])\n",
    "            plt.plot(range(self.games),grad_magn_gen)\n",
    "            plt.title('Gradient magnitude - Generator')\n",
    "            plt.savefig('img/summary_'+str(self.games)+'_'+str(self.discriminator_steps)+\\\n",
    "                      '_'+str(self.generator_steps)+'.eps')\n",
    "            plt.savefig('img/summary_'+str(self.games)+'_'+str(self.discriminator_steps)+\\\n",
    "                      '_'+str(self.generator_steps)+'.png')\n",
    "\n",
    "            # Animation\n",
    "            print('\\nCreating GIF animation...')\n",
    "            fig = plt.figure()\n",
    "            plt.axis('off')\n",
    "            anim = animation.FuncAnimation(fig, self.animate, frames=frames)\n",
    "            anim.save('img/img_'+str(self.games)+'_'+str(self.discriminator_steps)+\\\n",
    "                      '_'+str(self.generator_steps)+'.gif', writer='imagemagick', fps=int(120/self.skip_log))\n",
    "            self.delete()\n",
    "\n",
    "        \n",
    "        def animate(self, i):\n",
    "            print('Frame {}'.format(i))\n",
    "            img = mpimg.imread('img/img-'+str(i*self.skip_log)+'.png')\n",
    "            ax = plt.imshow(img)\n",
    "            return ax\n",
    "\n",
    "        def delete(self):\n",
    "            i = 0\n",
    "            while True:\n",
    "                try:\n",
    "                    os.remove('img/img-'+str(i*self.skip_log)+'.png')\n",
    "                    i += 1\n",
    "                except:\n",
    "                    return\n",
    "\n",
    "        def intuition(self, sess, x):\n",
    "            min_range = self.noise.low\n",
    "            max_range = self.data.mu+2*self.data.sigma\n",
    "            plt.xlim([min_range,max_range])\n",
    "            plt.ylim([-0.6,1])\n",
    "\n",
    "            # Lines\n",
    "            plt.plot([min_range, max_range], [-0.5,-0.5], 'k-', lw=1)\n",
    "            plt.plot([min_range, max_range], [0,0], 'k-', lw=1)\n",
    "\n",
    "            # Samples\n",
    "            num = 10\n",
    "            z = self.noise.sample(num)\n",
    "            plt.plot(z, -0.5*np.ones(num),'bo')\n",
    "            out = sess.run(self.gen, {self.z: np.reshape(z, (1,self.num_samples))})\n",
    "            plt.plot(np.transpose(out), \\\n",
    "                    np.zeros(num),'bo')\n",
    "\n",
    "            # Arrows\n",
    "            for i in range(num):\n",
    "                plt.plot([z[i],out[0][i]],[-0.49,-0.01],'-k')\n",
    "\n",
    "            # Real distribution\n",
    "            x_range = np.linspace(min_range, max_range, 50)\n",
    "            fit = norm.pdf(x_range, self.data.mu, self.data.sigma)\n",
    "            plt.plot(x_range, fit, '-g')           \n",
    "\n",
    "            # Real data\n",
    "            plt.plot(x, np.zeros(self.num_samples),'go')\n",
    "\n",
    "            # Discriminator\n",
    "            num = 40*self.num_samples\n",
    "            x_range = np.linspace(min_range, max_range, num)\n",
    "            out = []\n",
    "            for i in range(int(num/self.num_samples)):\n",
    "                tmp = x_range[i*self.num_samples:(i+1)*self.num_samples]\n",
    "                tmp = sess.run(self.discr1, {self.x: np.reshape(tmp, (1,self.num_samples))})[0]\n",
    "                for j in range(self.num_samples):\n",
    "                    out.append(tmp[j])\n",
    "            plt.plot(x_range, \\\n",
    "                    np.array(out),'-b')\n",
    "\n",
    "            plt.title('Graphical interpretation')\n",
    "\n",
    "        def objective(self, objective, games):\n",
    "            plt.plot(range(self.games),objective)\n",
    "            plt.plot([1, games], [-2*np.log(2), -2*np.log(2)], 'r-', lw=1)\n",
    "            plt.title('Objective vs. Iterations')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = GAN()\n",
    "model.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}