tf-neural-ode-v1.0.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tensorflow Experiments on Neural Ordinary Differential Equations\n",
    "\n",
    "> You can contact me on twitter as [@mandubian](http://twitter.com/mandubian)\n",
    "\n",
    "This notebook is a sandbox to test concepts exposed in this amazing paper:\n",
    "\n",
    "> Neural Ordinary Differential Equations\n",
    "> http://arxiv.org/abs/1806.07366\n",
    ">\n",
    "> Authors: \n",
    ">    Chen, Ricky T. Q.\n",
    ">    Rubanova, Yulia\n",
    ">    Bettencourt, Jesse\n",
    ">    Duvenaud, David\n",
    "\n",
    "My idea is to reproduce the concepts exposed in the paper fully in Tensorflow using Eager-Mode and GradientTape. \n",
    "\n",
    "I didn't want to depend on Autograd and to be able to use classic Keras models.\n",
    "\n",
    "For ODE Solver, I wanted to compare implementations in TF with Scipy very robust ones (the only one I found is Runge-Kutta Dopri5).\n",
    "\n",
    "## A few preliminary ideas (to let you scratch your head with me)\n",
    "\n",
    "> Skip it if you just want to see it running\n",
    "\n",
    "### Modeling the derivative of hidden state\n",
    "\n",
    "Deep neural networks as we know in general are aimed at simulating a function $y = f(x)$ from a set of known inputs/outputs (a supervised learning based on a groundtruth). The classic approach is to represent the function with inputs, outputs and a _hidden state_ we don't know but want to learn. This hidden state is represented by a sequence of discrete deep neural layers whose internal parameters are learnt by step by step back-propagation of gradients on a chosen derivable loss function and an optimization mechanism to reach the best minimum (ideally global). So the hidden state to be learnt is parameterized by the trainable parameters of those discrete layers.\n",
    "\n",
    ">In _Neural ODE_ paper (http://arxiv.org/abs/1806.07366), the authors propose a new approach by modeling not the hidden state of the function but the derivative of the hidden state.\n",
    "\n",
    "\n",
    "### Residual layer as Ordinary Differential Equation\n",
    "\n",
    "The authors make this assumption starting from the case of well-known residual neural networks _ResNet_ constituted of multiple residual layers. A residual layer is meant to _keep in memory_ what you knew in previous layer and to slighly update it without forgetting it. A residual layer can be represented by the following equation:\n",
    "\n",
    "$$\n",
    "z_{t+1} = z_t + f(z_t, {\\theta}_t) \\\\\n",
    "$$\n",
    "\n",
    "where:\n",
    "- $z_t$ is the input of the layer\n",
    "- $z_{t+1}$ is the output of the layer\n",
    "- $f$ is the function parameterized by $θ_t$ that we want to learn\n",
    "- $t$ can be seen as the depth of this layer in a full ResNet with $0, 1, ..., t, t+1, ..., t+n$ successive residual layers.\n",
    "\n",
    "Previous equation can be rewritten as:\n",
    "\n",
    "$$\n",
    "z_{t+1} - z_t = f(z_t, {\\theta}_t) * ({\\delta}_t = 1) \\\\\n",
    "or \\\\\n",
    "\\frac {z_{t+1} - z_t} {({\\delta}_t = 1)} = f(z_t, {\\theta}_t)\n",
    "$$\n",
    "\n",
    "${\\delta}_t = 1$ can be seen as discrete depth increment going from one residual layer to the next in the ResNet.\n",
    "\n",
    "\n",
    "Let's now consider this depth $t$ is no more discrete but continuous. We can easily infer the following:\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "{\\frac {dz(t)}{dt}} = f(z(t), t, {\\theta})\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "The previous equation is called an _ordinary differential equation (ODE)_. There is a whole field of study in mathematics about solving ordinary differential equations, with so-called _ODE Solvers_, i.e solving equations of kind:\n",
    "\n",
    "$$\n",
    "\\text{Find } z(t) \\text{ knowing }\n",
    "\\begin{aligned}\n",
    "{\\frac {dz(t)}{dt}} & = f(z(t), t) \\\\\n",
    "\\end{aligned}\n",
    "\\text{ given an initial value }\n",
    "\\begin{aligned}\n",
    "z(t_{0}) & = z_{0} \\\\\n",
    "\\end{aligned} \\text{ and a discretized interval }\n",
    "\\begin{aligned}\n",
    "{t_0, t_1, ... t_n}\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "ODE Solvers are numerical methods to compute fast but robust approximated integrals (analytical methods exist but can't solve many cases of ODE like $\\dot{y} = sin(x^2 + y^2)$). During your math study, you might have seen this Euler method https://en.wikipedia.org/wiki/Euler_method of approximating a function by using a naive construction with the tangent, an incremental step and an initial point on the curve. I'm not a mathematician so I won't teach you anything about ODE Solvers, you just need to know that this field has been studied and used intensively in multiple real-life apps (physics, geology, meteo, finance, chemistry etc...) and provide very efficient, optimized algorithms implemented in multiple libs and for example Scipy provides nice implementations of different ODE solvers.\n",
    "\n",
    "\n",
    "\n",
    "### Computing hidden state using ODE Solver\n",
    "\n",
    "Let's imagine we have a ResNet with multiple residual layers of depths $0, 1, ..., t, t+1, ..., t+n$. In a classic ResNet, all layers have their own function $f_t$ but if we mutualize this function for all layers, we can generalize to the whole network:\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "{\\frac {dz(t)}{dt}} & = f(z(t), t)\\hspace{0.5cm}\\forall{t} \\in [t_0, t_1, ...t_n]\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "So the $f$ function of this ResNet can be seen as the derivative of the hidden state ${\\frac {dz(t)}{dt}}$ we are trying to learn.\n",
    "\n",
    "Using ODE Solver, if we know $f$ and an initial value $z(t_{0})$ and this interval discretization ${t_0, t_1, ..., t_n}$, we can compute a robust approximation $z(t)$ in the following way:\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "z(t_n) & = ODESolve(z(t_0), f, [t_0, t_1, ...t_n], θ) \n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "where:\n",
    "- $t_0$ is the initial layer\n",
    "- $[t_0, t_1, ...t_n]$ is the discretized interval\n",
    "- $z(t_0)$ is the initial value of the function\n",
    "- $z(t_n)$ is the output value of our network\n",
    "- $t_n$ is the output layer\n",
    "- ${\\theta}$ are the parameters of the function $f$ we want to learn\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### Depth as an input parameter of neural model\n",
    "\n",
    "$[t_0, t_1, ...t_n]$ represents the depth distribution of the layers of our _mono-function_ ResNet. In a classic network, the depth in terms of layers is a static part of the architecture of the network which can't be changed without retraining completely the network.\n",
    "\n",
    "In our case, depth distribution becomes a continuous space that we discretized and provide as a dynamic parameter to the ODE Solver at training or inference. We can imagine having different depth distribution at training or inference to speed computations for example. We could even imagine using more complicated depth distribution like non linear distribution or even learning an optimized shape for a given problem. Next studies...\n",
    "\n",
    "There is also an immediate analogy between this depth distribution and time distribution. It also quickly reminds of time-series issues and recurrent neural networks... They speak about it in the paper but I still haven't implemented that part... Next steps...\n",
    "\n",
    "\n",
    "### Back-propagation by backward ODE Solving\n",
    "\n",
    "So that's cool but how do you represent and learn those ${\\theta}$???\n",
    "\n",
    "I won't go in deep demonstration as it would be too long. Don't worry if you don't catch this part :D\n",
    "\n",
    "It starts with a classic neural network representing our derivative $f$ with internal parameters $\\theta$.\n",
    "\n",
    "Let simplify to a 1-step time interval $[t_0, t_1]$, an initial input value $z(t_0)$ and our known output $groundtruth(t_1)$.\n",
    "\n",
    "First, it computes current estimated value $z({t_1})$ by a classic forward computation with a call to our ODE Solver using current $\\theta$ and input value $z(t_0)$.\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "z(t_1) = ODESolve(z(t_0), f, [t_0, t_1], {\\theta})\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "Then, it computes the classic Loss function:\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "L(z(t_1), groundtruth(t_1)) & = L(ODESolve(z(t_0), f, [t_0, t_1], {\\theta}), groundtruth(t_1))\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "The authors then consider the so-called _adjoint state_ (adjoint state method is the math _trick_ at the origin of backward auto-differentiation _aka_ gradient back-propagation that often simplifies gradient computation as neural networks are often many-to-scalars functions with lower gradient complexity in backward mode... This is another topic)\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "a(t) = − {\\frac {\\partial L(z(t))}{\\partial z(t)}}\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "\n",
    "The paper appendix proves that:\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "{\\frac {da(t)}{dt}} = - a(t){\\frac {\\partial{f(z(t), t, \\theta)}}{\\partial z}}\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "and the same respectively for ${\\frac {da(t)}{d\\theta}}$ and ${\\frac {da(t)}{dt}}$\n",
    "\n",
    "${\\frac {\\partial{f(z(t), t, \\theta)}}{\\partial z}}$ is the classic gradient we already compute in normal residual layer using back-propagation.\n",
    "\n",
    "Knowing all the derivatives of $a(t)$ with respect to $z(t), t, \\theta$, we can then compute $a(t)$ with a call to ODE Solver going backward in time from $t_1$ to $t_0$. The initial value is the gradient of loss ${\\frac {\\partial L(z(t_1))}{\\partial z}}$ at $t_1$ between our current output $z(t_1)$ and our groundtruth...\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "s0 & = [z(t1), {\\frac {\\partial L}{\\partial z(t_1)}}, 0, −{\\frac {\\partial L}{t_1}}] \\\\\n",
    "[z(t_0), {\\frac {\\partial L}{\\partial z(t_0)}}, {\\frac {\\partial L}{\\partial θ}}, {\\frac {\\partial L}{\\partial t}}] & = ODESolve(z(t_0), augmented_f, [t_1, t_0], {\\theta})\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "By using a bit more machinery (an _augmented state_ grouping $z(t)$, ${\\frac {da(t)}{dz}}$, ${\\frac {da(t)}{d\\theta}}$, ${\\frac {da(t)}{dt}}$ together), the paper demonstrates how all required backward gradients can be computed in one single call to ODESolver and then you can train your parameters using a classic optimizer like SGD or Adam...\n",
    "\n",
    "Not so clear? It's normal :D\n",
    "\n",
    "\n",
    "### Conclusion\n",
    "\n",
    "By representing the residual-like hidden-state through its derivative and a _continuous_ depth distribution, it's possible to train model parameters with ODE Solvers. The hidden-state of the derivative is still represented by a static human-engineered discrete sequence of neural layers but the depth distribution becomes an input parameter of the network that can be changed dynamically at training or inference.\n",
    "\n",
    "Is it going to change your life as a datascientist drastically? Not yet... But it's bringing new cool ideas on how we can represent neural networks in another way, how it learns, how residual layers work. It opens new fields of study, drawing bridges between Deep Learning and existing older branches of mathematics. It also bring diversity in DL domain that IMHO tends to re-use the same tricks without always catching how they work.\n",
    "\n",
    "#### Other sources of inspiration\n",
    "\n",
    "There are more precise details in this nice article https://rkevingibson.github.io/blog/neural-networks-as-ordinary-differential-equations/ and then read the paper first pages, it's quite simple after all. If I can understand it, you can too...\n",
    "\n",
    "This notebook is also very inspired by this cool article on Neural ODE & appealing Julia:\n",
    "\n",
    "https://julialang.org/blog/2019/01/fluxdiffeq\n",
    "\n",
    "\n",
    "If you want to know more about ODE Solvers, there is a whole litterature about it with lots of different methods like `Runge-Kutta` or `LSODA` used in this notebook and you can find more details in scipy\n",
    "https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.odeint.html#scipy.integrate.odeint\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Code under MIT License\n",
    "\n",
    "```\n",
    "Copyright 2019 Pascal Voitot\n",
    "\n",
    "Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n",
    "\n",
    "The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n",
    "\n",
    "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensorflow version:  1.12.0\n"
     ]
    }
   ],
   "source": [
    "from __future__ import absolute_import\n",
    "from __future__ import print_function\n",
    "\n",
    "import time\n",
    "import os\n",
    "import numpy as np\n",
    "import scipy.integrate\n",
    "from plotly import __version__\n",
    "from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot\n",
    "#import plotly.plotly as py\n",
    "import plotly.graph_objs as go\n",
    "import plotly.io as pio\n",
    "\n",
    "import tensorflow as tf\n",
    "print(\"tensorflow version: \", tf.__version__)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script><script type=\"text/javascript\">if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script><script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window._Plotly) {require(['plotly'],function(plotly) {window._Plotly=plotly;});}</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script><script type=\"text/javascript\">if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script><script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window._Plotly) {require(['plotly'],function(plotly) {window._Plotly=plotly;});}</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tf.enable_eager_execution()\n",
    "init_notebook_mode(connected=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = tf.ConfigProto()\n",
    "config.gpu_options.allow_growth=True\n",
    "\n",
    "os.environ['CUDA_VISIBLE_DEVICES'] = '0'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test ODE Solvers (TF and Scipy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let a derivative function:\n",
    "\n",
    "$\n",
    "\\hspace{1cm}f'(t) = 2t\n",
    "$\n",
    "\n",
    "Let a set of values $t \\in [0, 4.0]$\n",
    "\n",
    "Let the initial value be $f(t_0) = 0.0$\n",
    "\n",
    "A good ODE Solver should integrate to a parabolic:\n",
    "\n",
    "$\n",
    "\\hspace{1cm}f(t) = t^2\n",
    "$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test Full-TF ODE Solver (Runge-Kutta - Adaptative - Explicit - Dopri5)\n",
    "\n",
    "> `Dopri5` is an explicit runge-kutta method of order (4)5 due to Dormand & Prince (with stepsize control and dense output).\n",
    ">\n",
    "> Authors:\n",
    "> E. Hairer and G. Wanner Universite de Geneve, Dept. de Mathematiques CH-1211 Geneve 24\n",
    ">\n",
    "> **Please note that this implementation works well but the method tends to be less stable computationally speaking than the scipy LSODA implicit method used below**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "linkText": "Export to plot.ly",
        "plotlyServerURL": "https://plot.ly",
        "showLink": false
       },
       "data": [
        {
         "type": "scatter",
         "uid": "34df5352-b29d-4271-b44f-67bebdc4df75",
         "x": [
          0,
          0.4444444444444444,
          0.8888888888888888,
          1.3333333333333333,
          1.7777777777777777,
          2.2222222222222223,
          2.6666666666666665,
          3.1111111111111107,
          3.5555555555555554,
          4
         ],
         "y": [
          0,
          0.19753089547157288,
          0.790123462677002,
          1.7777777910232544,
          3.160493850708008,
          4.938271522521973,
          7.111111640930176,
          9.679012298583984,
          12.641977310180664,
          16
         ]
        }
       ],
       "layout": {}
      },
      "text/html": [
       "<div id=\"72fe210b-f159-440c-91e2-8629c4aa2450\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"72fe210b-f159-440c-91e2-8629c4aa2450\", [{\"x\": [0.0, 0.4444444444444444, 0.8888888888888888, 1.3333333333333333, 1.7777777777777777, 2.2222222222222223, 2.6666666666666665, 3.1111111111111107, 3.5555555555555554, 4.0], \"y\": [0.0, 0.19753089547157288, 0.790123462677002, 1.7777777910232544, 3.160493850708008, 4.938271522521973, 7.111111640930176, 9.679012298583984, 12.641977310180664, 16.0], \"type\": \"scatter\", \"uid\": \"a08c8069-492f-4099-9658-9e40a96f1e35\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"72fe210b-f159-440c-91e2-8629c4aa2450\"));});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"72fe210b-f159-440c-91e2-8629c4aa2450\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"72fe210b-f159-440c-91e2-8629c4aa2450\", [{\"x\": [0.0, 0.4444444444444444, 0.8888888888888888, 1.3333333333333333, 1.7777777777777777, 2.2222222222222223, 2.6666666666666665, 3.1111111111111107, 3.5555555555555554, 4.0], \"y\": [0.0, 0.19753089547157288, 0.790123462677002, 1.7777777910232544, 3.160493850708008, 4.938271522521973, 7.111111640930176, 9.679012298583984, 12.641977310180664, 16.0], \"type\": \"scatter\", \"uid\": \"a08c8069-492f-4099-9658-9e40a96f1e35\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"72fe210b-f159-440c-91e2-8629c4aa2450\"));});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ts = np.linspace(0., 4.0, 10)\n",
    "\n",
    "ys, info = tf.contrib.integrate.odeint(lambda y, t: tf.to_float(2*t), 0.0, ts, rtol = 0.01, full_output=True)\n",
    "\n",
    "iplot([go.Scatter(x=ts, y=ys.numpy())])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test Scipy ODE Solver (LSODE)\n",
    "\n",
    "> By default Scipy is using on LSODA Solver method (there are other methods, even DOPRI5)\n",
    ">\n",
    "> Real-valued Variable-coefficient Ordinary Differential Equation solver, with fixed-leading-coefficient implementation. It provides automatic method switching between implicit Adams method (for non-stiff problems) and a method based on backward differentiation formulas (BDF) (for stiff problems).\n",
    ">\n",
    "> Source: http://www.netlib.org/odepack\n",
    ">\n",
    "> **Please note that this implementation is more stable computionally speaking than full-TF implementation above**\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "linkText": "Export to plot.ly",
        "plotlyServerURL": "https://plot.ly",
        "showLink": false
       },
       "data": [
        {
         "type": "scatter",
         "uid": "abe12add-b09b-4ee7-b9ae-bc136e25bb49",
         "x": [
          0,
          0.4444444444444444,
          0.8888888888888888,
          1.3333333333333333,
          1.7777777777777777,
          2.2222222222222223,
          2.6666666666666665,
          3.1111111111111107,
          3.5555555555555554,
          4
         ],
         "y": [
          0,
          0.19753087008442466,
          0.790123462677017,
          1.7777777836646713,
          3.160493833047383,
          4.938271610825161,
          7.1111111169980035,
          9.679012351565902,
          12.641975314528871,
          16.00000000588689
         ]
        }
       ],
       "layout": {}
      },
      "text/html": [
       "<div id=\"5cfcf2d4-0bad-4a12-8982-c93c2f11ccd1\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"5cfcf2d4-0bad-4a12-8982-c93c2f11ccd1\", [{\"x\": [0.0, 0.4444444444444444, 0.8888888888888888, 1.3333333333333333, 1.7777777777777777, 2.2222222222222223, 2.6666666666666665, 3.1111111111111107, 3.5555555555555554, 4.0], \"y\": [0.0, 0.19753087008442466, 0.790123462677017, 1.7777777836646713, 3.160493833047383, 4.938271610825161, 7.1111111169980035, 9.679012351565902, 12.641975314528871, 16.00000000588689], \"type\": \"scatter\", \"uid\": \"73ffc33e-69cd-4509-a749-20fa0f3361b0\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"5cfcf2d4-0bad-4a12-8982-c93c2f11ccd1\"));});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"5cfcf2d4-0bad-4a12-8982-c93c2f11ccd1\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"5cfcf2d4-0bad-4a12-8982-c93c2f11ccd1\", [{\"x\": [0.0, 0.4444444444444444, 0.8888888888888888, 1.3333333333333333, 1.7777777777777777, 2.2222222222222223, 2.6666666666666665, 3.1111111111111107, 3.5555555555555554, 4.0], \"y\": [0.0, 0.19753087008442466, 0.790123462677017, 1.7777777836646713, 3.160493833047383, 4.938271610825161, 7.1111111169980035, 9.679012351565902, 12.641975314528871, 16.00000000588689], \"type\": \"scatter\", \"uid\": \"73ffc33e-69cd-4509-a749-20fa0f3361b0\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"5cfcf2d4-0bad-4a12-8982-c93c2f11ccd1\"));});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ts = np.linspace(0., 4.0, 10)\n",
    "\n",
    "ys = scipy.integrate.odeint(lambda y, t: 2*t, [0.0], ts)\n",
    "\n",
    "iplot([go.Scatter(x=ts, y=np.squeeze(ys))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Illustrating ODE Solvers with oscillators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Two-dimensional damped oscillator\n",
    "\n",
    "http://www.dam.brown.edu/people/mraissi/research/7_multistep_neural_networks/\n",
    "\n",
    "Let us consider the two-dimensional damped harmonic oscillator with cubic dynamics:\n",
    "\n",
    "$\n",
    "\\begin{align}\n",
    "{\\frac {dx} {dt}} & = −0.1x^3 + 2.0y^3 \\\\\n",
    "{\\frac {dy} {dt}} & = −2.0x^3 − 0.1y^3 \\\\\n",
    "z & = (x, y)\n",
    "\\end{align}\n",
    "$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# z = [x, y]\n",
    "def damped_oscillator_2d(z, t, A):\n",
    "    return np.dot(z**3, A)\n",
    "\n",
    "A = np.array([\n",
    "    # −0.1x^3 + 2.0y^3\n",
    "    [-0.1, 2.0],\n",
    "    # −2.0x^3 − 0.1y^3\n",
    "    [-2.0, -0.1]])\n",
    "\n",
    "# Initial u0 = [x0, y0]\n",
    "z0 = np.array([2., 0.]).T\n",
    "\n",
    "# Time sampling\n",
    "N = 100  # Dataset size\n",
    "max_T = 6.0 # Time sampling\n",
    "ts = np.linspace(0., max_T, N)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TF-RungeKutta ODE Solver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "linkText": "Export to plot.ly",
        "plotlyServerURL": "https://plot.ly",
        "showLink": false
       },
       "data": [
        {
         "name": "x",
         "type": "scatter",
         "uid": "f59408bc-7823-4f70-a5b7-5f07ddfc90c4",
         "x": [
          0,
          0.06060606060606061,
          0.12121212121212122,
          0.18181818181818182,
          0.24242424242424243,
          0.30303030303030304,
          0.36363636363636365,
          0.42424242424242425,
          0.48484848484848486,
          0.5454545454545454,
          0.6060606060606061,
          0.6666666666666667,
          0.7272727272727273,
          0.7878787878787878,
          0.8484848484848485,
          0.9090909090909092,
          0.9696969696969697,
          1.0303030303030303,
          1.0909090909090908,
          1.1515151515151516,
          1.2121212121212122,
          1.2727272727272727,
          1.3333333333333335,
          1.393939393939394,
          1.4545454545454546,
          1.5151515151515151,
          1.5757575757575757,
          1.6363636363636365,
          1.696969696969697,
          1.7575757575757576,
          1.8181818181818183,
          1.878787878787879,
          1.9393939393939394,
          2,
          2.0606060606060606,
          2.121212121212121,
          2.1818181818181817,
          2.2424242424242427,
          2.303030303030303,
          2.3636363636363638,
          2.4242424242424243,
          2.484848484848485,
          2.5454545454545454,
          2.606060606060606,
          2.666666666666667,
          2.7272727272727275,
          2.787878787878788,
          2.8484848484848486,
          2.909090909090909,
          2.9696969696969697,
          3.0303030303030303,
          3.090909090909091,
          3.1515151515151514,
          3.2121212121212124,
          3.272727272727273,
          3.3333333333333335,
          3.393939393939394,
          3.4545454545454546,
          3.515151515151515,
          3.5757575757575757,
          3.6363636363636367,
          3.6969696969696972,
          3.757575757575758,
          3.8181818181818183,
          3.878787878787879,
          3.9393939393939394,
          4,
          4.0606060606060606,
          4.121212121212121,
          4.181818181818182,
          4.242424242424242,
          4.303030303030303,
          4.363636363636363,
          4.424242424242425,
          4.484848484848485,
          4.545454545454546,
          4.606060606060606,
          4.666666666666667,
          4.7272727272727275,
          4.787878787878788,
          4.848484848484849,
          4.909090909090909,
          4.96969696969697,
          5.03030303030303,
          5.090909090909091,
          5.151515151515151,
          5.212121212121212,
          5.2727272727272725,
          5.333333333333334,
          5.3939393939393945,
          5.454545454545455,
          5.515151515151516,
          5.575757575757576,
          5.636363636363637,
          5.696969696969697,
          5.757575757575758,
          5.818181818181818,
          5.878787878787879,
          5.9393939393939394,
          6
         ],
         "y": [
          2,
          1.9275335492248116,
          1.5998570448183158,
          0.9207862693459719,
          0.13452380333354474,
          -0.6096974411447051,
          -1.25418286777096,
          -1.6315066370137474,
          -1.7217982150242048,
          -1.7017244189024154,
          -1.6700302474907762,
          -1.5781988180295607,
          -1.3221327967571772,
          -0.901343926760748,
          -0.43467524475434816,
          0.0333016047943252,
          0.4800108723105486,
          0.8920927536479553,
          1.214756118762398,
          1.3976879378663665,
          1.4526259468738039,
          1.442954381495077,
          1.4255813903787993,
          1.4058384328175886,
          1.3634898177258625,
          1.2656436700418774,
          1.0961820520354395,
          0.8499439650980141,
          0.5618549067897671,
          0.2695346188327875,
          -0.012056756456724438,
          -0.2829946508022103,
          -0.5426981536329409,
          -0.7807976141308778,
          -0.9787535690145659,
          -1.1181971450955797,
          -1.1968382524249763,
          -1.2267293145417821,
          -1.2283350858746076,
          -1.218765884881333,
          -1.2078792877577502,
          -1.1963374565586955,
          -1.1774103359119226,
          -1.141801919424842,
          -1.077975227353395,
          -0.9751755680981544,
          -0.8419064674965777,
          -0.6880491640024515,
          -0.5168102709651442,
          -0.3401496232304454,
          -0.16530749978610298,
          0.005556626513781082,
          0.17276168702259204,
          0.33603112235417243,
          0.49268488969345886,
          0.638604801718343,
          0.7682671719934847,
          0.874651204642688,
          0.9526223884504221,
          1.0037377147474822,
          1.0319086021907513,
          1.0429442550419017,
          1.0436966253868256,
          1.0389965088528312,
          1.0325389420030622,
          1.0260403141806687,
          1.019261502236434,
          1.0108523737125277,
          0.9969858233937599,
          0.9738343045870753,
          0.9382587066429772,
          0.8878083549555266,
          0.8206244809423205,
          0.7369476629506833,
          0.642993497115746,
          0.5437848911571765,
          0.4418973555949667,
          0.3374590037494313,
          0.23079499325036934,
          0.12531703523257906,
          0.021437293621688613,
          -0.08065804417819616,
          -0.18093061802559007,
          -0.27950095292857546,
          -0.3759127275752592,
          -0.468353399568596,
          -0.5549384291647691,
          -0.6338487704533912,
          -0.703330871357506,
          -0.7616966736335846,
          -0.8074055644946889,
          -0.8407267922843258,
          -0.8629886140673411,
          -0.8757361353043331,
          -0.8808875729908584,
          -0.8807342556574316,
          -0.8779406233695258,
          -0.8746929064874464,
          -0.8708418604020798,
          -0.8668445571525387
         ]
        },
        {
         "name": "y",
         "type": "scatter",
         "uid": "df780fc7-bbda-4bc8-8df6-a22ee8a3c397",
         "x": [
          0,
          0.06060606060606061,
          0.12121212121212122,
          0.18181818181818182,
          0.24242424242424243,
          0.30303030303030304,
          0.36363636363636365,
          0.42424242424242425,
          0.48484848484848486,
          0.5454545454545454,
          0.6060606060606061,
          0.6666666666666667,
          0.7272727272727273,
          0.7878787878787878,
          0.8484848484848485,
          0.9090909090909092,
          0.9696969696969697,
          1.0303030303030303,
          1.0909090909090908,
          1.1515151515151516,
          1.2121212121212122,
          1.2727272727272727,
          1.3333333333333335,
          1.393939393939394,
          1.4545454545454546,
          1.5151515151515151,
          1.5757575757575757,
          1.6363636363636365,
          1.696969696969697,
          1.7575757575757576,
          1.8181818181818183,
          1.878787878787879,
          1.9393939393939394,
          2,
          2.0606060606060606,
          2.121212121212121,
          2.1818181818181817,
          2.2424242424242427,
          2.303030303030303,
          2.3636363636363638,
          2.4242424242424243,
          2.484848484848485,
          2.5454545454545454,
          2.606060606060606,
          2.666666666666667,
          2.7272727272727275,
          2.787878787878788,
          2.8484848484848486,
          2.909090909090909,
          2.9696969696969697,
          3.0303030303030303,
          3.090909090909091,
          3.1515151515151514,
          3.2121212121212124,
          3.272727272727273,
          3.3333333333333335,
          3.393939393939394,
          3.4545454545454546,
          3.515151515151515,
          3.5757575757575757,
          3.6363636363636367,
          3.6969696969696972,
          3.757575757575758,
          3.8181818181818183,
          3.878787878787879,
          3.9393939393939394,
          4,
          4.0606060606060606,
          4.121212121212121,
          4.181818181818182,
          4.242424242424242,
          4.303030303030303,
          4.363636363636363,
          4.424242424242425,
          4.484848484848485,
          4.545454545454546,
          4.606060606060606,
          4.666666666666667,
          4.7272727272727275,
          4.787878787878788,
          4.848484848484849,
          4.909090909090909,
          4.96969696969697,
          5.03030303030303,
          5.090909090909091,
          5.151515151515151,
          5.212121212121212,
          5.2727272727272725,
          5.333333333333334,
          5.3939393939393945,
          5.454545454545455,
          5.515151515151516,
          5.575757575757576,
          5.636363636363637,
          5.696969696969697,
          5.757575757575758,
          5.818181818181818,
          5.878787878787879,
          5.9393939393939394,
          6
         ],
         "y": [
          0,
          0.92767359733138,
          1.6236364059159534,
          1.862495009616343,
          1.8504436673479705,
          1.8081410065121932,
          1.6600196417184159,
          1.243670285934535,
          0.6826977146142301,
          0.07091957709316621,
          -0.5107038961187252,
          -1.0359023775911775,
          -1.4099581570148587,
          -1.565496848360753,
          -1.578321127714461,
          -1.55709036625239,
          -1.531462473814801,
          -1.46802226109182,
          -1.3022818384538941,
          -1.013045125470994,
          -0.6547969109119751,
          -0.2887396242047247,
          0.06917728133035839,
          0.4133164572216731,
          0.7366200521580439,
          1.0115590116871076,
          1.2074401560237296,
          1.311827536282643,
          1.3431532130964434,
          1.333728263665205,
          1.3141961040300532,
          1.3001045438837755,
          1.2772040129255096,
          1.2288143619793583,
          1.1329244683279451,
          0.9845881125287094,
          0.7970889159854662,
          0.579076054204053,
          0.3533507804545602,
          0.13087223162864448,
          -0.08559930661923389,
          -0.2965485463303953,
          -0.49917257392901715,
          -0.6872691120536587,
          -0.8513520664221781,
          -0.9790578885944551,
          -1.0650959389864738,
          -1.112759699373835,
          -1.1312602416241362,
          -1.1323073347811754,
          -1.1256495247438258,
          -1.1171701345137144,
          -1.1088461415713071,
          -1.0986293624408012,
          -1.0812645722408296,
          -1.0510948019025106,
          -1.0022011766235353,
          -0.9279655880744159,
          -0.8279965013987982,
          -0.7141617734943242,
          -0.5930192773554054,
          -0.4642161384973843,
          -0.3264255006637625,
          -0.18926607662208678,
          -0.054426584030468794,
          0.07763254079945348,
          0.20720037126359214,
          0.33428922626695684,
          0.45673934646600184,
          0.5717828319199982,
          0.676329426307424,
          0.766966516925964,
          0.8400065028878141,
          0.8936522292463625,
          0.9293736519282503,
          0.9497259991742929,
          0.9581867472486649,
          0.9591556204389011,
          0.9566556605096217,
          0.9520829033232334,
          0.9469360079588964,
          0.9418666121558459,
          0.9366459630639052,
          0.9302899187959699,
          0.9215355794315356,
          0.9077830447182627,
          0.886745761676608,
          0.8567303875078388,
          0.8166367895940315,
          0.765958045498074,
          0.7047609401737821,
          0.6343121146515425,
          0.5578098308056048,
          0.47798245296343117,
          0.3967437904861758,
          0.31519309776868404,
          0.23361507423949268,
          0.15205358796334403,
          0.07144622160728176,
          -0.00812771806672058
         ]
        }
       ],
       "layout": {}
      },
      "text/html": [
       "<div id=\"6be1e119-8cdd-4893-a1c8-ee78924cfe50\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"6be1e119-8cdd-4893-a1c8-ee78924cfe50\", [{\"name\": \"x\", \"x\": [0.0, 0.06060606060606061, 0.12121212121212122, 0.18181818181818182, 0.24242424242424243, 0.30303030303030304, 0.36363636363636365, 0.42424242424242425, 0.48484848484848486, 0.5454545454545454, 0.6060606060606061, 0.6666666666666667, 0.7272727272727273, 0.7878787878787878, 0.8484848484848485, 0.9090909090909092, 0.9696969696969697, 1.0303030303030303, 1.0909090909090908, 1.1515151515151516, 1.2121212121212122, 1.2727272727272727, 1.3333333333333335, 1.393939393939394, 1.4545454545454546, 1.5151515151515151, 1.5757575757575757, 1.6363636363636365, 1.696969696969697, 1.7575757575757576, 1.8181818181818183, 1.878787878787879, 1.9393939393939394, 2.0, 2.0606060606060606, 2.121212121212121, 2.1818181818181817, 2.2424242424242427, 2.303030303030303, 2.3636363636363638, 2.4242424242424243, 2.484848484848485, 2.5454545454545454, 2.606060606060606, 2.666666666666667, 2.7272727272727275, 2.787878787878788, 2.8484848484848486, 2.909090909090909, 2.9696969696969697, 3.0303030303030303, 3.090909090909091, 3.1515151515151514, 3.2121212121212124, 3.272727272727273, 3.3333333333333335, 3.393939393939394, 3.4545454545454546, 3.515151515151515, 3.5757575757575757, 3.6363636363636367, 3.6969696969696972, 3.757575757575758, 3.8181818181818183, 3.878787878787879, 3.9393939393939394, 4.0, 4.0606060606060606, 4.121212121212121, 4.181818181818182, 4.242424242424242, 4.303030303030303, 4.363636363636363, 4.424242424242425, 4.484848484848485, 4.545454545454546, 4.606060606060606, 4.666666666666667, 4.7272727272727275, 4.787878787878788, 4.848484848484849, 4.909090909090909, 4.96969696969697, 5.03030303030303, 5.090909090909091, 5.151515151515151, 5.212121212121212, 5.2727272727272725, 5.333333333333334, 5.3939393939393945, 5.454545454545455, 5.515151515151516, 5.575757575757576, 5.636363636363637, 5.696969696969697, 5.757575757575758, 5.818181818181818, 5.878787878787879, 5.9393939393939394, 6.0], \"y\": [2.0, 1.9275335492248116, 1.5998570448183158, 0.9207862693459719, 0.13452380333354474, -0.6096974411447051, -1.25418286777096, -1.6315066370137474, -1.7217982150242048, -1.7017244189024154, -1.6700302474907762, -1.5781988180295607, -1.3221327967571772, -0.901343926760748, -0.43467524475434816, 0.0333016047943252, 0.4800108723105486, 0.8920927536479553, 1.214756118762398, 1.3976879378663665, 1.4526259468738039, 1.442954381495077, 1.4255813903787993, 1.4058384328175886, 1.3634898177258625, 1.2656436700418774, 1.0961820520354395, 0.8499439650980141, 0.5618549067897671, 0.2695346188327875, -0.012056756456724438, -0.2829946508022103, -0.5426981536329409, -0.7807976141308778, -0.9787535690145659, -1.1181971450955797, -1.1968382524249763, -1.2267293145417821, -1.2283350858746076, -1.218765884881333, -1.2078792877577502, -1.1963374565586955, -1.1774103359119226, -1.141801919424842, -1.077975227353395, -0.9751755680981544, -0.8419064674965777, -0.6880491640024515, -0.5168102709651442, -0.3401496232304454, -0.16530749978610298, 0.005556626513781082, 0.17276168702259204, 0.33603112235417243, 0.49268488969345886, 0.638604801718343, 0.7682671719934847, 0.874651204642688, 0.9526223884504221, 1.0037377147474822, 1.0319086021907513, 1.0429442550419017, 1.0436966253868256, 1.0389965088528312, 1.0325389420030622, 1.0260403141806687, 1.019261502236434, 1.0108523737125277, 0.9969858233937599, 0.9738343045870753, 0.9382587066429772, 0.8878083549555266, 0.8206244809423205, 0.7369476629506833, 0.642993497115746, 0.5437848911571765, 0.4418973555949667, 0.3374590037494313, 0.23079499325036934, 0.12531703523257906, 0.021437293621688613, -0.08065804417819616, -0.18093061802559007, -0.27950095292857546, -0.3759127275752592, -0.468353399568596, -0.5549384291647691, -0.6338487704533912, -0.703330871357506, -0.7616966736335846, -0.8074055644946889, -0.8407267922843258, -0.8629886140673411, -0.8757361353043331, -0.8808875729908584, -0.8807342556574316, -0.8779406233695258, -0.8746929064874464, -0.8708418604020798, -0.8668445571525387], \"type\": \"scatter\", \"uid\": \"e4fc882e-aeab-4446-8f39-61b6ec2c6c71\"}, {\"name\": \"y\", \"x\": [0.0, 0.06060606060606061, 0.12121212121212122, 0.18181818181818182, 0.24242424242424243, 0.30303030303030304, 0.36363636363636365, 0.42424242424242425, 0.48484848484848486, 0.5454545454545454, 0.6060606060606061, 0.6666666666666667, 0.7272727272727273, 0.7878787878787878, 0.8484848484848485, 0.9090909090909092, 0.9696969696969697, 1.0303030303030303, 1.0909090909090908, 1.1515151515151516, 1.2121212121212122, 1.2727272727272727, 1.3333333333333335, 1.393939393939394, 1.4545454545454546, 1.5151515151515151, 1.5757575757575757, 1.6363636363636365, 1.696969696969697, 1.7575757575757576, 1.8181818181818183, 1.878787878787879, 1.9393939393939394, 2.0, 2.0606060606060606, 2.121212121212121, 2.1818181818181817, 2.2424242424242427, 2.303030303030303, 2.3636363636363638, 2.4242424242424243, 2.484848484848485, 2.5454545454545454, 2.606060606060606, 2.666666666666667, 2.7272727272727275, 2.787878787878788, 2.8484848484848486, 2.909090909090909, 2.9696969696969697, 3.0303030303030303, 3.090909090909091, 3.1515151515151514, 3.2121212121212124, 3.272727272727273, 3.3333333333333335, 3.393939393939394, 3.4545454545454546, 3.515151515151515, 3.5757575757575757, 3.6363636363636367, 3.6969696969696972, 3.757575757575758, 3.8181818181818183, 3.878787878787879, 3.9393939393939394, 4.0, 4.0606060606060606, 4.121212121212121, 4.181818181818182, 4.242424242424242, 4.303030303030303, 4.363636363636363, 4.424242424242425, 4.484848484848485, 4.545454545454546, 4.606060606060606, 4.666666666666667, 4.7272727272727275, 4.787878787878788, 4.848484848484849, 4.909090909090909, 4.96969696969697, 5.03030303030303, 5.090909090909091, 5.151515151515151, 5.212121212121212, 5.2727272727272725, 5.333333333333334, 5.3939393939393945, 5.454545454545455, 5.515151515151516, 5.575757575757576, 5.636363636363637, 5.696969696969697, 5.757575757575758, 5.818181818181818, 5.878787878787879, 5.9393939393939394, 6.0], \"y\": [0.0, 0.92767359733138, 1.6236364059159534, 1.862495009616343, 1.8504436673479705, 1.8081410065121932, 1.6600196417184159, 1.243670285934535, 0.6826977146142301, 0.07091957709316621, -0.5107038961187252, -1.0359023775911775, -1.4099581570148587, -1.565496848360753, -1.578321127714461, -1.55709036625239, -1.531462473814801, -1.46802226109182, -1.3022818384538941, -1.013045125470994, -0.6547969109119751, -0.2887396242047247, 0.06917728133035839, 0.4133164572216731, 0.7366200521580439, 1.0115590116871076, 1.2074401560237296, 1.311827536282643, 1.3431532130964434, 1.333728263665205, 1.3141961040300532, 1.3001045438837755, 1.2772040129255096, 1.2288143619793583, 1.1329244683279451, 0.9845881125287094, 0.7970889159854662, 0.579076054204053, 0.3533507804545602, 0.13087223162864448, -0.08559930661923389, -0.2965485463303953, -0.49917257392901715, -0.6872691120536587, -0.8513520664221781, -0.9790578885944551, -1.0650959389864738, -1.112759699373835, -1.1312602416241362, -1.1323073347811754, -1.1256495247438258, -1.1171701345137144, -1.1088461415713071, -1.0986293624408012, -1.0812645722408296, -1.0510948019025106, -1.0022011766235353, -0.9279655880744159, -0.8279965013987982, -0.7141617734943242, -0.5930192773554054, -0.4642161384973843, -0.3264255006637625, -0.18926607662208678, -0.054426584030468794, 0.07763254079945348, 0.20720037126359214, 0.33428922626695684, 0.45673934646600184, 0.5717828319199982, 0.676329426307424, 0.766966516925964, 0.8400065028878141, 0.8936522292463625, 0.9293736519282503, 0.9497259991742929, 0.9581867472486649, 0.9591556204389011, 0.9566556605096217, 0.9520829033232334, 0.9469360079588964, 0.9418666121558459, 0.9366459630639052, 0.9302899187959699, 0.9215355794315356, 0.9077830447182627, 0.886745761676608, 0.8567303875078388, 0.8166367895940315, 0.765958045498074, 0.7047609401737821, 0.6343121146515425, 0.5578098308056048, 0.47798245296343117, 0.3967437904861758, 0.31519309776868404, 0.23361507423949268, 0.15205358796334403, 0.07144622160728176, -0.00812771806672058], \"type\": \"scatter\", \"uid\": \"3b6bbe3c-f91e-48b0-be4e-ddfbd9dfe613\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"6be1e119-8cdd-4893-a1c8-ee78924cfe50\"));});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"6be1e119-8cdd-4893-a1c8-ee78924cfe50\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"6be1e119-8cdd-4893-a1c8-ee78924cfe50\", [{\"name\": \"x\", \"x\": [0.0, 0.06060606060606061, 0.12121212121212122, 0.18181818181818182, 0.24242424242424243, 0.30303030303030304, 0.36363636363636365, 0.42424242424242425, 0.48484848484848486, 0.5454545454545454, 0.6060606060606061, 0.6666666666666667, 0.7272727272727273, 0.7878787878787878, 0.8484848484848485, 0.9090909090909092, 0.9696969696969697, 1.0303030303030303, 1.0909090909090908, 1.1515151515151516, 1.2121212121212122, 1.2727272727272727, 1.3333333333333335, 1.393939393939394, 1.4545454545454546, 1.5151515151515151, 1.5757575757575757, 1.6363636363636365, 1.696969696969697, 1.7575757575757576, 1.8181818181818183, 1.878787878787879, 1.9393939393939394, 2.0, 2.0606060606060606, 2.121212121212121, 2.1818181818181817, 2.2424242424242427, 2.303030303030303, 2.3636363636363638, 2.4242424242424243, 2.484848484848485, 2.5454545454545454, 2.606060606060606, 2.666666666666667, 2.7272727272727275, 2.787878787878788, 2.8484848484848486, 2.909090909090909, 2.9696969696969697, 3.0303030303030303, 3.090909090909091, 3.1515151515151514, 3.2121212121212124, 3.272727272727273, 3.3333333333333335, 3.393939393939394, 3.4545454545454546, 3.515151515151515, 3.5757575757575757, 3.6363636363636367, 3.6969696969696972, 3.757575757575758, 3.8181818181818183, 3.878787878787879, 3.9393939393939394, 4.0, 4.0606060606060606, 4.121212121212121, 4.181818181818182, 4.242424242424242, 4.303030303030303, 4.363636363636363, 4.424242424242425, 4.484848484848485, 4.545454545454546, 4.606060606060606, 4.666666666666667, 4.7272727272727275, 4.787878787878788, 4.848484848484849, 4.909090909090909, 4.96969696969697, 5.03030303030303, 5.090909090909091, 5.151515151515151, 5.212121212121212, 5.2727272727272725, 5.333333333333334, 5.3939393939393945, 5.454545454545455, 5.515151515151516, 5.575757575757576, 5.636363636363637, 5.696969696969697, 5.757575757575758, 5.818181818181818, 5.878787878787879, 5.9393939393939394, 6.0], \"y\": [2.0, 1.9275335492248116, 1.5998570448183158, 0.9207862693459719, 0.13452380333354474, -0.6096974411447051, -1.25418286777096, -1.6315066370137474, -1.7217982150242048, -1.7017244189024154, -1.6700302474907762, -1.5781988180295607, -1.3221327967571772, -0.901343926760748, -0.43467524475434816, 0.0333016047943252, 0.4800108723105486, 0.8920927536479553, 1.214756118762398, 1.3976879378663665, 1.4526259468738039, 1.442954381495077, 1.4255813903787993, 1.4058384328175886, 1.3634898177258625, 1.2656436700418774, 1.0961820520354395, 0.8499439650980141, 0.5618549067897671, 0.2695346188327875, -0.012056756456724438, -0.2829946508022103, -0.5426981536329409, -0.7807976141308778, -0.9787535690145659, -1.1181971450955797, -1.1968382524249763, -1.2267293145417821, -1.2283350858746076, -1.218765884881333, -1.2078792877577502, -1.1963374565586955, -1.1774103359119226, -1.141801919424842, -1.077975227353395, -0.9751755680981544, -0.8419064674965777, -0.6880491640024515, -0.5168102709651442, -0.3401496232304454, -0.16530749978610298, 0.005556626513781082, 0.17276168702259204, 0.33603112235417243, 0.49268488969345886, 0.638604801718343, 0.7682671719934847, 0.874651204642688, 0.9526223884504221, 1.0037377147474822, 1.0319086021907513, 1.0429442550419017, 1.0436966253868256, 1.0389965088528312, 1.0325389420030622, 1.0260403141806687, 1.019261502236434, 1.0108523737125277, 0.9969858233937599, 0.9738343045870753, 0.9382587066429772, 0.8878083549555266, 0.8206244809423205, 0.7369476629506833, 0.642993497115746, 0.5437848911571765, 0.4418973555949667, 0.3374590037494313, 0.23079499325036934, 0.12531703523257906, 0.021437293621688613, -0.08065804417819616, -0.18093061802559007, -0.27950095292857546, -0.3759127275752592, -0.468353399568596, -0.5549384291647691, -0.6338487704533912, -0.703330871357506, -0.7616966736335846, -0.8074055644946889, -0.8407267922843258, -0.8629886140673411, -0.8757361353043331, -0.8808875729908584, -0.8807342556574316, -0.8779406233695258, -0.8746929064874464, -0.8708418604020798, -0.8668445571525387], \"type\": \"scatter\", \"uid\": \"e4fc882e-aeab-4446-8f39-61b6ec2c6c71\"}, {\"name\": \"y\", \"x\": [0.0, 0.06060606060606061, 0.12121212121212122, 0.18181818181818182, 0.24242424242424243, 0.30303030303030304, 0.36363636363636365, 0.42424242424242425, 0.48484848484848486, 0.5454545454545454, 0.6060606060606061, 0.6666666666666667, 0.7272727272727273, 0.7878787878787878, 0.8484848484848485, 0.9090909090909092, 0.9696969696969697, 1.0303030303030303, 1.0909090909090908, 1.1515151515151516, 1.2121212121212122, 1.2727272727272727, 1.3333333333333335, 1.393939393939394, 1.4545454545454546, 1.5151515151515151, 1.5757575757575757, 1.6363636363636365, 1.696969696969697, 1.7575757575757576, 1.8181818181818183, 1.878787878787879, 1.9393939393939394, 2.0, 2.0606060606060606, 2.121212121212121, 2.1818181818181817, 2.2424242424242427, 2.303030303030303, 2.3636363636363638, 2.4242424242424243, 2.484848484848485, 2.5454545454545454, 2.606060606060606, 2.666666666666667, 2.7272727272727275, 2.787878787878788, 2.8484848484848486, 2.909090909090909, 2.9696969696969697, 3.0303030303030303, 3.090909090909091, 3.1515151515151514, 3.2121212121212124, 3.272727272727273, 3.3333333333333335, 3.393939393939394, 3.4545454545454546, 3.515151515151515, 3.5757575757575757, 3.6363636363636367, 3.6969696969696972, 3.757575757575758, 3.8181818181818183, 3.878787878787879, 3.9393939393939394, 4.0, 4.0606060606060606, 4.121212121212121, 4.181818181818182, 4.242424242424242, 4.303030303030303, 4.363636363636363, 4.424242424242425, 4.484848484848485, 4.545454545454546, 4.606060606060606, 4.666666666666667, 4.7272727272727275, 4.787878787878788, 4.848484848484849, 4.909090909090909, 4.96969696969697, 5.03030303030303, 5.090909090909091, 5.151515151515151, 5.212121212121212, 5.2727272727272725, 5.333333333333334, 5.3939393939393945, 5.454545454545455, 5.515151515151516, 5.575757575757576, 5.636363636363637, 5.696969696969697, 5.757575757575758, 5.818181818181818, 5.878787878787879, 5.9393939393939394, 6.0], \"y\": [0.0, 0.92767359733138, 1.6236364059159534, 1.862495009616343, 1.8504436673479705, 1.8081410065121932, 1.6600196417184159, 1.243670285934535, 0.6826977146142301, 0.07091957709316621, -0.5107038961187252, -1.0359023775911775, -1.4099581570148587, -1.565496848360753, -1.578321127714461, -1.55709036625239, -1.531462473814801, -1.46802226109182, -1.3022818384538941, -1.013045125470994, -0.6547969109119751, -0.2887396242047247, 0.06917728133035839, 0.4133164572216731, 0.7366200521580439, 1.0115590116871076, 1.2074401560237296, 1.311827536282643, 1.3431532130964434, 1.333728263665205, 1.3141961040300532, 1.3001045438837755, 1.2772040129255096, 1.2288143619793583, 1.1329244683279451, 0.9845881125287094, 0.7970889159854662, 0.579076054204053, 0.3533507804545602, 0.13087223162864448, -0.08559930661923389, -0.2965485463303953, -0.49917257392901715, -0.6872691120536587, -0.8513520664221781, -0.9790578885944551, -1.0650959389864738, -1.112759699373835, -1.1312602416241362, -1.1323073347811754, -1.1256495247438258, -1.1171701345137144, -1.1088461415713071, -1.0986293624408012, -1.0812645722408296, -1.0510948019025106, -1.0022011766235353, -0.9279655880744159, -0.8279965013987982, -0.7141617734943242, -0.5930192773554054, -0.4642161384973843, -0.3264255006637625, -0.18926607662208678, -0.054426584030468794, 0.07763254079945348, 0.20720037126359214, 0.33428922626695684, 0.45673934646600184, 0.5717828319199982, 0.676329426307424, 0.766966516925964, 0.8400065028878141, 0.8936522292463625, 0.9293736519282503, 0.9497259991742929, 0.9581867472486649, 0.9591556204389011, 0.9566556605096217, 0.9520829033232334, 0.9469360079588964, 0.9418666121558459, 0.9366459630639052, 0.9302899187959699, 0.9215355794315356, 0.9077830447182627, 0.886745761676608, 0.8567303875078388, 0.8166367895940315, 0.765958045498074, 0.7047609401737821, 0.6343121146515425, 0.5578098308056048, 0.47798245296343117, 0.3967437904861758, 0.31519309776868404, 0.23361507423949268, 0.15205358796334403, 0.07144622160728176, -0.00812771806672058], \"type\": \"scatter\", \"uid\": \"3b6bbe3c-f91e-48b0-be4e-ddfbd9dfe613\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"6be1e119-8cdd-4893-a1c8-ee78924cfe50\"));});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "linkText": "Export to plot.ly",
        "plotlyServerURL": "https://plot.ly",
        "showLink": false
       },
       "data": [
        {
         "type": "scatter",
         "uid": "a71959c7-8523-4ebe-83fc-a7f5fa02c0e0",
         "x": [
          2,
          1.9275335492248116,
          1.5998570448183158,
          0.9207862693459719,
          0.13452380333354474,
          -0.6096974411447051,
          -1.25418286777096,
          -1.6315066370137474,
          -1.7217982150242048,
          -1.7017244189024154,
          -1.6700302474907762,
          -1.5781988180295607,
          -1.3221327967571772,
          -0.901343926760748,
          -0.43467524475434816,
          0.0333016047943252,
          0.4800108723105486,
          0.8920927536479553,
          1.214756118762398,
          1.3976879378663665,
          1.4526259468738039,
          1.442954381495077,
          1.4255813903787993,
          1.4058384328175886,
          1.3634898177258625,
          1.2656436700418774,
          1.0961820520354395,
          0.8499439650980141,
          0.5618549067897671,
          0.2695346188327875,
          -0.012056756456724438,
          -0.2829946508022103,
          -0.5426981536329409,
          -0.7807976141308778,
          -0.9787535690145659,
          -1.1181971450955797,
          -1.1968382524249763,
          -1.2267293145417821,
          -1.2283350858746076,
          -1.218765884881333,
          -1.2078792877577502,
          -1.1963374565586955,
          -1.1774103359119226,
          -1.141801919424842,
          -1.077975227353395,
          -0.9751755680981544,
          -0.8419064674965777,
          -0.6880491640024515,
          -0.5168102709651442,
          -0.3401496232304454,
          -0.16530749978610298,
          0.005556626513781082,
          0.17276168702259204,
          0.33603112235417243,
          0.49268488969345886,
          0.638604801718343,
          0.7682671719934847,
          0.874651204642688,
          0.9526223884504221,
          1.0037377147474822,
          1.0319086021907513,
          1.0429442550419017,
          1.0436966253868256,
          1.0389965088528312,
          1.0325389420030622,
          1.0260403141806687,
          1.019261502236434,
          1.0108523737125277,
          0.9969858233937599,
          0.9738343045870753,
          0.9382587066429772,
          0.8878083549555266,
          0.8206244809423205,
          0.7369476629506833,
          0.642993497115746,
          0.5437848911571765,
          0.4418973555949667,
          0.3374590037494313,
          0.23079499325036934,
          0.12531703523257906,
          0.021437293621688613,
          -0.08065804417819616,
          -0.18093061802559007,
          -0.27950095292857546,
          -0.3759127275752592,
          -0.468353399568596,
          -0.5549384291647691,
          -0.6338487704533912,
          -0.703330871357506,
          -0.7616966736335846,
          -0.8074055644946889,
          -0.8407267922843258,
          -0.8629886140673411,
          -0.8757361353043331,
          -0.8808875729908584,
          -0.8807342556574316,
          -0.8779406233695258,
          -0.8746929064874464,
          -0.8708418604020798,
          -0.8668445571525387
         ],
         "y": [
          0,
          0.92767359733138,
          1.6236364059159534,
          1.862495009616343,
          1.8504436673479705,
          1.8081410065121932,
          1.6600196417184159,
          1.243670285934535,
          0.6826977146142301,
          0.07091957709316621,
          -0.5107038961187252,
          -1.0359023775911775,
          -1.4099581570148587,
          -1.565496848360753,
          -1.578321127714461,
          -1.55709036625239,
          -1.531462473814801,
          -1.46802226109182,
          -1.3022818384538941,
          -1.013045125470994,
          -0.6547969109119751,
          -0.2887396242047247,
          0.06917728133035839,
          0.4133164572216731,
          0.7366200521580439,
          1.0115590116871076,
          1.2074401560237296,
          1.311827536282643,
          1.3431532130964434,
          1.333728263665205,
          1.3141961040300532,
          1.3001045438837755,
          1.2772040129255096,
          1.2288143619793583,
          1.1329244683279451,
          0.9845881125287094,
          0.7970889159854662,
          0.579076054204053,
          0.3533507804545602,
          0.13087223162864448,
          -0.08559930661923389,
          -0.2965485463303953,
          -0.49917257392901715,
          -0.6872691120536587,
          -0.8513520664221781,
          -0.9790578885944551,
          -1.0650959389864738,
          -1.112759699373835,
          -1.1312602416241362,
          -1.1323073347811754,
          -1.1256495247438258,
          -1.1171701345137144,
          -1.1088461415713071,
          -1.0986293624408012,
          -1.0812645722408296,
          -1.0510948019025106,
          -1.0022011766235353,
          -0.9279655880744159,
          -0.8279965013987982,
          -0.7141617734943242,
          -0.5930192773554054,
          -0.4642161384973843,
          -0.3264255006637625,
          -0.18926607662208678,
          -0.054426584030468794,
          0.07763254079945348,
          0.20720037126359214,
          0.33428922626695684,
          0.45673934646600184,
          0.5717828319199982,
          0.676329426307424,
          0.766966516925964,
          0.8400065028878141,
          0.8936522292463625,
          0.9293736519282503,
          0.9497259991742929,
          0.9581867472486649,
          0.9591556204389011,
          0.9566556605096217,
          0.9520829033232334,
          0.9469360079588964,
          0.9418666121558459,
          0.9366459630639052,
          0.9302899187959699,
          0.9215355794315356,
          0.9077830447182627,
          0.886745761676608,
          0.8567303875078388,
          0.8166367895940315,
          0.765958045498074,
          0.7047609401737821,
          0.6343121146515425,
          0.5578098308056048,
          0.47798245296343117,
          0.3967437904861758,
          0.31519309776868404,
          0.23361507423949268,
          0.15205358796334403,
          0.07144622160728176,
          -0.00812771806672058
         ]
        }
       ],
       "layout": {}
      },
      "text/html": [
       "<div id=\"9ff3cfe4-4888-4c37-bbad-126487c5beea\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"9ff3cfe4-4888-4c37-bbad-126487c5beea\", [{\"x\": [2.0, 1.9275335492248116, 1.5998570448183158, 0.9207862693459719, 0.13452380333354474, -0.6096974411447051, -1.25418286777096, -1.6315066370137474, -1.7217982150242048, -1.7017244189024154, -1.6700302474907762, -1.5781988180295607, -1.3221327967571772, -0.901343926760748, -0.43467524475434816, 0.0333016047943252, 0.4800108723105486, 0.8920927536479553, 1.214756118762398, 1.3976879378663665, 1.4526259468738039, 1.442954381495077, 1.4255813903787993, 1.4058384328175886, 1.3634898177258625, 1.2656436700418774, 1.0961820520354395, 0.8499439650980141, 0.5618549067897671, 0.2695346188327875, -0.012056756456724438, -0.2829946508022103, -0.5426981536329409, -0.7807976141308778, -0.9787535690145659, -1.1181971450955797, -1.1968382524249763, -1.2267293145417821, -1.2283350858746076, -1.218765884881333, -1.2078792877577502, -1.1963374565586955, -1.1774103359119226, -1.141801919424842, -1.077975227353395, -0.9751755680981544, -0.8419064674965777, -0.6880491640024515, -0.5168102709651442, -0.3401496232304454, -0.16530749978610298, 0.005556626513781082, 0.17276168702259204, 0.33603112235417243, 0.49268488969345886, 0.638604801718343, 0.7682671719934847, 0.874651204642688, 0.9526223884504221, 1.0037377147474822, 1.0319086021907513, 1.0429442550419017, 1.0436966253868256, 1.0389965088528312, 1.0325389420030622, 1.0260403141806687, 1.019261502236434, 1.0108523737125277, 0.9969858233937599, 0.9738343045870753, 0.9382587066429772, 0.8878083549555266, 0.8206244809423205, 0.7369476629506833, 0.642993497115746, 0.5437848911571765, 0.4418973555949667, 0.3374590037494313, 0.23079499325036934, 0.12531703523257906, 0.021437293621688613, -0.08065804417819616, -0.18093061802559007, -0.27950095292857546, -0.3759127275752592, -0.468353399568596, -0.5549384291647691, -0.6338487704533912, -0.703330871357506, -0.7616966736335846, -0.8074055644946889, -0.8407267922843258, -0.8629886140673411, -0.8757361353043331, -0.8808875729908584, -0.8807342556574316, -0.8779406233695258, -0.8746929064874464, -0.8708418604020798, -0.8668445571525387], \"y\": [0.0, 0.92767359733138, 1.6236364059159534, 1.862495009616343, 1.8504436673479705, 1.8081410065121932, 1.6600196417184159, 1.243670285934535, 0.6826977146142301, 0.07091957709316621, -0.5107038961187252, -1.0359023775911775, -1.4099581570148587, -1.565496848360753, -1.578321127714461, -1.55709036625239, -1.531462473814801, -1.46802226109182, -1.3022818384538941, -1.013045125470994, -0.6547969109119751, -0.2887396242047247, 0.06917728133035839, 0.4133164572216731, 0.7366200521580439, 1.0115590116871076, 1.2074401560237296, 1.311827536282643, 1.3431532130964434, 1.333728263665205, 1.3141961040300532, 1.3001045438837755, 1.2772040129255096, 1.2288143619793583, 1.1329244683279451, 0.9845881125287094, 0.7970889159854662, 0.579076054204053, 0.3533507804545602, 0.13087223162864448, -0.08559930661923389, -0.2965485463303953, -0.49917257392901715, -0.6872691120536587, -0.8513520664221781, -0.9790578885944551, -1.0650959389864738, -1.112759699373835, -1.1312602416241362, -1.1323073347811754, -1.1256495247438258, -1.1171701345137144, -1.1088461415713071, -1.0986293624408012, -1.0812645722408296, -1.0510948019025106, -1.0022011766235353, -0.9279655880744159, -0.8279965013987982, -0.7141617734943242, -0.5930192773554054, -0.4642161384973843, -0.3264255006637625, -0.18926607662208678, -0.054426584030468794, 0.07763254079945348, 0.20720037126359214, 0.33428922626695684, 0.45673934646600184, 0.5717828319199982, 0.676329426307424, 0.766966516925964, 0.8400065028878141, 0.8936522292463625, 0.9293736519282503, 0.9497259991742929, 0.9581867472486649, 0.9591556204389011, 0.9566556605096217, 0.9520829033232334, 0.9469360079588964, 0.9418666121558459, 0.9366459630639052, 0.9302899187959699, 0.9215355794315356, 0.9077830447182627, 0.886745761676608, 0.8567303875078388, 0.8166367895940315, 0.765958045498074, 0.7047609401737821, 0.6343121146515425, 0.5578098308056048, 0.47798245296343117, 0.3967437904861758, 0.31519309776868404, 0.23361507423949268, 0.15205358796334403, 0.07144622160728176, -0.00812771806672058], \"type\": \"scatter\", \"uid\": \"fae995ae-40a3-49dc-8770-127aef13f969\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"9ff3cfe4-4888-4c37-bbad-126487c5beea\"));});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"9ff3cfe4-4888-4c37-bbad-126487c5beea\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"9ff3cfe4-4888-4c37-bbad-126487c5beea\", [{\"x\": [2.0, 1.9275335492248116, 1.5998570448183158, 0.9207862693459719, 0.13452380333354474, -0.6096974411447051, -1.25418286777096, -1.6315066370137474, -1.7217982150242048, -1.7017244189024154, -1.6700302474907762, -1.5781988180295607, -1.3221327967571772, -0.901343926760748, -0.43467524475434816, 0.0333016047943252, 0.4800108723105486, 0.8920927536479553, 1.214756118762398, 1.3976879378663665, 1.4526259468738039, 1.442954381495077, 1.4255813903787993, 1.4058384328175886, 1.3634898177258625, 1.2656436700418774, 1.0961820520354395, 0.8499439650980141, 0.5618549067897671, 0.2695346188327875, -0.012056756456724438, -0.2829946508022103, -0.5426981536329409, -0.7807976141308778, -0.9787535690145659, -1.1181971450955797, -1.1968382524249763, -1.2267293145417821, -1.2283350858746076, -1.218765884881333, -1.2078792877577502, -1.1963374565586955, -1.1774103359119226, -1.141801919424842, -1.077975227353395, -0.9751755680981544, -0.8419064674965777, -0.6880491640024515, -0.5168102709651442, -0.3401496232304454, -0.16530749978610298, 0.005556626513781082, 0.17276168702259204, 0.33603112235417243, 0.49268488969345886, 0.638604801718343, 0.7682671719934847, 0.874651204642688, 0.9526223884504221, 1.0037377147474822, 1.0319086021907513, 1.0429442550419017, 1.0436966253868256, 1.0389965088528312, 1.0325389420030622, 1.0260403141806687, 1.019261502236434, 1.0108523737125277, 0.9969858233937599, 0.9738343045870753, 0.9382587066429772, 0.8878083549555266, 0.8206244809423205, 0.7369476629506833, 0.642993497115746, 0.5437848911571765, 0.4418973555949667, 0.3374590037494313, 0.23079499325036934, 0.12531703523257906, 0.021437293621688613, -0.08065804417819616, -0.18093061802559007, -0.27950095292857546, -0.3759127275752592, -0.468353399568596, -0.5549384291647691, -0.6338487704533912, -0.703330871357506, -0.7616966736335846, -0.8074055644946889, -0.8407267922843258, -0.8629886140673411, -0.8757361353043331, -0.8808875729908584, -0.8807342556574316, -0.8779406233695258, -0.8746929064874464, -0.8708418604020798, -0.8668445571525387], \"y\": [0.0, 0.92767359733138, 1.6236364059159534, 1.862495009616343, 1.8504436673479705, 1.8081410065121932, 1.6600196417184159, 1.243670285934535, 0.6826977146142301, 0.07091957709316621, -0.5107038961187252, -1.0359023775911775, -1.4099581570148587, -1.565496848360753, -1.578321127714461, -1.55709036625239, -1.531462473814801, -1.46802226109182, -1.3022818384538941, -1.013045125470994, -0.6547969109119751, -0.2887396242047247, 0.06917728133035839, 0.4133164572216731, 0.7366200521580439, 1.0115590116871076, 1.2074401560237296, 1.311827536282643, 1.3431532130964434, 1.333728263665205, 1.3141961040300532, 1.3001045438837755, 1.2772040129255096, 1.2288143619793583, 1.1329244683279451, 0.9845881125287094, 0.7970889159854662, 0.579076054204053, 0.3533507804545602, 0.13087223162864448, -0.08559930661923389, -0.2965485463303953, -0.49917257392901715, -0.6872691120536587, -0.8513520664221781, -0.9790578885944551, -1.0650959389864738, -1.112759699373835, -1.1312602416241362, -1.1323073347811754, -1.1256495247438258, -1.1171701345137144, -1.1088461415713071, -1.0986293624408012, -1.0812645722408296, -1.0510948019025106, -1.0022011766235353, -0.9279655880744159, -0.8279965013987982, -0.7141617734943242, -0.5930192773554054, -0.4642161384973843, -0.3264255006637625, -0.18926607662208678, -0.054426584030468794, 0.07763254079945348, 0.20720037126359214, 0.33428922626695684, 0.45673934646600184, 0.5717828319199982, 0.676329426307424, 0.766966516925964, 0.8400065028878141, 0.8936522292463625, 0.9293736519282503, 0.9497259991742929, 0.9581867472486649, 0.9591556204389011, 0.9566556605096217, 0.9520829033232334, 0.9469360079588964, 0.9418666121558459, 0.9366459630639052, 0.9302899187959699, 0.9215355794315356, 0.9077830447182627, 0.886745761676608, 0.8567303875078388, 0.8166367895940315, 0.765958045498074, 0.7047609401737821, 0.6343121146515425, 0.5578098308056048, 0.47798245296343117, 0.3967437904861758, 0.31519309776868404, 0.23361507423949268, 0.15205358796334403, 0.07144622160728176, -0.00812771806672058], \"type\": \"scatter\", \"uid\": \"fae995ae-40a3-49dc-8770-127aef13f969\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"9ff3cfe4-4888-4c37-bbad-126487c5beea\"));});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "zs, info = tf.contrib.integrate.odeint(lambda z0, ts: tf.tensordot(z0**3, A, axes=1), z0, ts,\n",
    "                                method=\"dopri5\", options={\"max_num_steps\":10, \"first_step\": 0.1}, rtol = 0.001,\n",
    "                                full_output=True)\n",
    "zs = zs.numpy()\n",
    "xs = zs[:, 0]\n",
    "ys = zs[:, 1]\n",
    "iplot([go.Scatter(x=ts, y=xs, name=\"x\"), go.Scatter(x=ts, y=ys, name=\"y\")])\n",
    "iplot([go.Scatter(x=xs, y=ys)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Scipy ODE Solver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "linkText": "Export to plot.ly",
        "plotlyServerURL": "https://plot.ly",
        "showLink": false
       },
       "data": [
        {
         "name": "x",
         "type": "scatter",
         "uid": "0cac984c-b034-4eed-8f77-984d8bf65c5c",
         "x": [
          0,
          0.06060606060606061,
          0.12121212121212122,
          0.18181818181818182,
          0.24242424242424243,
          0.30303030303030304,
          0.36363636363636365,
          0.42424242424242425,
          0.48484848484848486,
          0.5454545454545454,
          0.6060606060606061,
          0.6666666666666667,
          0.7272727272727273,
          0.7878787878787878,
          0.8484848484848485,
          0.9090909090909092,
          0.9696969696969697,
          1.0303030303030303,
          1.0909090909090908,
          1.1515151515151516,
          1.2121212121212122,
          1.2727272727272727,
          1.3333333333333335,
          1.393939393939394,
          1.4545454545454546,
          1.5151515151515151,
          1.5757575757575757,
          1.6363636363636365,
          1.696969696969697,
          1.7575757575757576,
          1.8181818181818183,
          1.878787878787879,
          1.9393939393939394,
          2,
          2.0606060606060606,
          2.121212121212121,
          2.1818181818181817,
          2.2424242424242427,
          2.303030303030303,
          2.3636363636363638,
          2.4242424242424243,
          2.484848484848485,
          2.5454545454545454,
          2.606060606060606,
          2.666666666666667,
          2.7272727272727275,
          2.787878787878788,
          2.8484848484848486,
          2.909090909090909,
          2.9696969696969697,
          3.0303030303030303,
          3.090909090909091,
          3.1515151515151514,
          3.2121212121212124,
          3.272727272727273,
          3.3333333333333335,
          3.393939393939394,
          3.4545454545454546,
          3.515151515151515,
          3.5757575757575757,
          3.6363636363636367,
          3.6969696969696972,
          3.757575757575758,
          3.8181818181818183,
          3.878787878787879,
          3.9393939393939394,
          4,
          4.0606060606060606,
          4.121212121212121,
          4.181818181818182,
          4.242424242424242,
          4.303030303030303,
          4.363636363636363,
          4.424242424242425,
          4.484848484848485,
          4.545454545454546,
          4.606060606060606,
          4.666666666666667,
          4.7272727272727275,
          4.787878787878788,
          4.848484848484849,
          4.909090909090909,
          4.96969696969697,
          5.03030303030303,
          5.090909090909091,
          5.151515151515151,
          5.212121212121212,
          5.2727272727272725,
          5.333333333333334,
          5.3939393939393945,
          5.454545454545455,
          5.515151515151516,
          5.575757575757576,
          5.636363636363637,
          5.696969696969697,
          5.757575757575758,
          5.818181818181818,
          5.878787878787879,
          5.9393939393939394,
          6
         ],
         "y": [
          2,
          1.9285373714028342,
          1.601104367867955,
          0.9032253373890189,
          0.12226440995856018,
          -0.616152506793763,
          -1.2569160225766454,
          -1.631102663173404,
          -1.716655996540273,
          -1.6952775715140853,
          -1.6623170891768728,
          -1.5686931419451788,
          -1.3100016111362691,
          -0.8887037395657653,
          -0.4126330334956034,
          0.05321024695495329,
          0.4981754501197791,
          0.9060395627706298,
          1.2231061668130039,
          1.3984836499938114,
          1.4514488764532503,
          1.4461671600939257,
          1.4285930930306656,
          1.408041164759553,
          1.3634401509702723,
          1.258176733921232,
          1.069686708334913,
          0.813529560698556,
          0.5272056023710875,
          0.2385483933236769,
          -0.042417962587647844,
          -0.3144752036919671,
          -0.5746910503758015,
          -0.8114062628576733,
          -1.0043998937279062,
          -1.1360488565734772,
          -1.205403955167543,
          -1.2287664218277128,
          -1.2275269893083047,
          -1.2177404426274663,
          -1.206917122186534,
          -1.1946317862874565,
          -1.1739434608185018,
          -1.1335403029956341,
          -1.0619083111053418,
          -0.9532647670850372,
          -0.8113551159422893,
          -0.6472081733138771,
          -0.47293294302501243,
          -0.29709837463031746,
          -0.1238716675454302,
          0.04557660629262953,
          0.2111855803669549,
          0.37236593846510824,
          0.5267591791125672,
          0.669620171386869,
          0.794340641393042,
          0.8944548128824078,
          0.9663919571705629,
          1.011166707633886,
          1.0337935409046548,
          1.041104687002145,
          1.0395766447424966,
          1.0340867813943715,
          1.0275466030435731,
          1.0209874144697157,
          1.0137731705250905,
          1.0038291632099972,
          0.987943009955437,
          0.9622779086746256,
          0.9231976826890829,
          0.8682965562043136,
          0.7972388073272193,
          0.7119279582679571,
          0.6158588993101382,
          0.5130220628279911,
          0.4069329778902411,
          0.30012682887769665,
          0.19410829401541932,
          0.08956976745400544,
          -0.013298993172405946,
          -0.11450746597972528,
          -0.21399952646094372,
          -0.3114314587971726,
          -0.40600025937489503,
          -0.4963508303707573,
          -0.5806126106277416,
          -0.6566142362811292,
          -0.7222751032857833,
          -0.7760799885276684,
          -0.8174643972938972,
          -0.8469504460764985,
          -0.8659892169342667,
          -0.8766034774528556,
          -0.8809867080771778,
          -0.8811808418712176,
          -0.8788794914100785,
          -0.875342998856256,
          -0.8713865658481453,
          -0.8674041974220982
         ]
        },
        {
         "name": "y",
         "type": "scatter",
         "uid": "12607cde-887c-4950-a73b-59205494454e",
         "x": [
          0,
          0.06060606060606061,
          0.12121212121212122,
          0.18181818181818182,
          0.24242424242424243,
          0.30303030303030304,
          0.36363636363636365,
          0.42424242424242425,
          0.48484848484848486,
          0.5454545454545454,
          0.6060606060606061,
          0.6666666666666667,
          0.7272727272727273,
          0.7878787878787878,
          0.8484848484848485,
          0.9090909090909092,
          0.9696969696969697,
          1.0303030303030303,
          1.0909090909090908,
          1.1515151515151516,
          1.2121212121212122,
          1.2727272727272727,
          1.3333333333333335,
          1.393939393939394,
          1.4545454545454546,
          1.5151515151515151,
          1.5757575757575757,
          1.6363636363636365,
          1.696969696969697,
          1.7575757575757576,
          1.8181818181818183,
          1.878787878787879,
          1.9393939393939394,
          2,
          2.0606060606060606,
          2.121212121212121,
          2.1818181818181817,
          2.2424242424242427,
          2.303030303030303,
          2.3636363636363638,
          2.4242424242424243,
          2.484848484848485,
          2.5454545454545454,
          2.606060606060606,
          2.666666666666667,
          2.7272727272727275,
          2.787878787878788,
          2.8484848484848486,
          2.909090909090909,
          2.9696969696969697,
          3.0303030303030303,
          3.090909090909091,
          3.1515151515151514,
          3.2121212121212124,
          3.272727272727273,
          3.3333333333333335,
          3.393939393939394,
          3.4545454545454546,
          3.515151515151515,
          3.5757575757575757,
          3.6363636363636367,
          3.6969696969696972,
          3.757575757575758,
          3.8181818181818183,
          3.878787878787879,
          3.9393939393939394,
          4,
          4.0606060606060606,
          4.121212121212121,
          4.181818181818182,
          4.242424242424242,
          4.303030303030303,
          4.363636363636363,
          4.424242424242425,
          4.484848484848485,
          4.545454545454546,
          4.606060606060606,
          4.666666666666667,
          4.7272727272727275,
          4.787878787878788,
          4.848484848484849,
          4.909090909090909,
          4.96969696969697,
          5.03030303030303,
          5.090909090909091,
          5.151515151515151,
          5.212121212121212,
          5.2727272727272725,
          5.333333333333334,
          5.3939393939393945,
          5.454545454545455,
          5.515151515151516,
          5.575757575757576,
          5.636363636363637,
          5.696969696969697,
          5.757575757575758,
          5.818181818181818,
          5.878787878787879,
          5.9393939393939394,
          6
         ],
         "y": [
          0,
          0.9275448325420768,
          1.6252700330371528,
          1.8595825564303345,
          1.8462229799162784,
          1.8032366504665134,
          1.654850126167924,
          1.2420859948843215,
          0.6500446642053311,
          0.045186729136523704,
          -0.5293808596012912,
          -1.0467786871313218,
          -1.4116199039357176,
          -1.562175760571004,
          -1.5764441810825647,
          -1.5550164411023637,
          -1.5285014339222076,
          -1.4616043361426325,
          -1.291505224891791,
          -1.0000827824475937,
          -0.6400449652065053,
          -0.2689412160255923,
          0.09113476397468115,
          0.4373639467714392,
          0.761003644410565,
          1.0339499480600653,
          1.220822680163824,
          1.3126948946943262,
          1.3368361746660284,
          1.330169274647251,
          1.3164670961001288,
          1.3017623788703534,
          1.2770614754174923,
          1.2231106733033337,
          1.1201144316983378,
          0.9617133373745834,
          0.7607333528390774,
          0.5388915745589706,
          0.31315916295167495,
          0.09141342236007063,
          -0.12456283247173233,
          -0.33450913640796603,
          -0.5359050064294689,
          -0.7216127723114438,
          -0.8802482849426211,
          -1.000623444492499,
          -1.0782106487307939,
          -1.1179905490555937,
          -1.1311902796232456,
          -1.1296900778981762,
          -1.1223488567491111,
          -1.1139175303919728,
          -1.1052708654704486,
          -1.0939493963422573,
          -1.0748091148231402,
          -1.0411243544048727,
          -0.9865305321520321,
          -0.9075172357574051,
          -0.8051458500149421,
          -0.6846257598701961,
          -0.5529834915172145,
          -0.4165434361830663,
          -0.27959838141171955,
          -0.1443592757302281,
          -0.011579224635494732,
          0.11867379015958941,
          0.24632212404621642,
          0.3707085943559691,
          0.49017895345271467,
          0.6019199820028384,
          0.7022265788225424,
          0.7872991219011463,
          0.8543812837238588,
          0.9027128641654808,
          0.9337703189076347,
          0.9507088508175557,
          0.9573993793257143,
          0.9575500253355517,
          0.9541648868248808,
          0.9493307661572116,
          0.944206221139383,
          0.939093970374526,
          0.933529966735517,
          0.9263742217723451,
          0.9159258281662173,
          0.9001048012852292,
          0.8767389999196389,
          0.8439545746477302,
          0.8005959053336364,
          0.7465332557001076,
          0.6827182384082245,
          0.6109470103180275,
          0.5334311581292164,
          0.45235209669634685,
          0.3695418171493111,
          0.2863395156099453,
          0.20359478685896584,
          0.12175746216775195,
          0.041001303842690545,
          -0.03865033076267496
         ]
        }
       ],
       "layout": {}
      },
      "text/html": [
       "<div id=\"9690f851-87ba-4365-9f8d-dee8f63e14e3\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"9690f851-87ba-4365-9f8d-dee8f63e14e3\", [{\"name\": \"x\", \"x\": [0.0, 0.06060606060606061, 0.12121212121212122, 0.18181818181818182, 0.24242424242424243, 0.30303030303030304, 0.36363636363636365, 0.42424242424242425, 0.48484848484848486, 0.5454545454545454, 0.6060606060606061, 0.6666666666666667, 0.7272727272727273, 0.7878787878787878, 0.8484848484848485, 0.9090909090909092, 0.9696969696969697, 1.0303030303030303, 1.0909090909090908, 1.1515151515151516, 1.2121212121212122, 1.2727272727272727, 1.3333333333333335, 1.393939393939394, 1.4545454545454546, 1.5151515151515151, 1.5757575757575757, 1.6363636363636365, 1.696969696969697, 1.7575757575757576, 1.8181818181818183, 1.878787878787879, 1.9393939393939394, 2.0, 2.0606060606060606, 2.121212121212121, 2.1818181818181817, 2.2424242424242427, 2.303030303030303, 2.3636363636363638, 2.4242424242424243, 2.484848484848485, 2.5454545454545454, 2.606060606060606, 2.666666666666667, 2.7272727272727275, 2.787878787878788, 2.8484848484848486, 2.909090909090909, 2.9696969696969697, 3.0303030303030303, 3.090909090909091, 3.1515151515151514, 3.2121212121212124, 3.272727272727273, 3.3333333333333335, 3.393939393939394, 3.4545454545454546, 3.515151515151515, 3.5757575757575757, 3.6363636363636367, 3.6969696969696972, 3.757575757575758, 3.8181818181818183, 3.878787878787879, 3.9393939393939394, 4.0, 4.0606060606060606, 4.121212121212121, 4.181818181818182, 4.242424242424242, 4.303030303030303, 4.363636363636363, 4.424242424242425, 4.484848484848485, 4.545454545454546, 4.606060606060606, 4.666666666666667, 4.7272727272727275, 4.787878787878788, 4.848484848484849, 4.909090909090909, 4.96969696969697, 5.03030303030303, 5.090909090909091, 5.151515151515151, 5.212121212121212, 5.2727272727272725, 5.333333333333334, 5.3939393939393945, 5.454545454545455, 5.515151515151516, 5.575757575757576, 5.636363636363637, 5.696969696969697, 5.757575757575758, 5.818181818181818, 5.878787878787879, 5.9393939393939394, 6.0], \"y\": [2.0, 1.9285373714028342, 1.601104367867955, 0.9032253373890189, 0.12226440995856018, -0.616152506793763, -1.2569160225766454, -1.631102663173404, -1.716655996540273, -1.6952775715140853, -1.6623170891768728, -1.5686931419451788, -1.3100016111362691, -0.8887037395657653, -0.4126330334956034, 0.05321024695495329, 0.4981754501197791, 0.9060395627706298, 1.2231061668130039, 1.3984836499938114, 1.4514488764532503, 1.4461671600939257, 1.4285930930306656, 1.408041164759553, 1.3634401509702723, 1.258176733921232, 1.069686708334913, 0.813529560698556, 0.5272056023710875, 0.2385483933236769, -0.042417962587647844, -0.3144752036919671, -0.5746910503758015, -0.8114062628576733, -1.0043998937279062, -1.1360488565734772, -1.205403955167543, -1.2287664218277128, -1.2275269893083047, -1.2177404426274663, -1.206917122186534, -1.1946317862874565, -1.1739434608185018, -1.1335403029956341, -1.0619083111053418, -0.9532647670850372, -0.8113551159422893, -0.6472081733138771, -0.47293294302501243, -0.29709837463031746, -0.1238716675454302, 0.04557660629262953, 0.2111855803669549, 0.37236593846510824, 0.5267591791125672, 0.669620171386869, 0.794340641393042, 0.8944548128824078, 0.9663919571705629, 1.011166707633886, 1.0337935409046548, 1.041104687002145, 1.0395766447424966, 1.0340867813943715, 1.0275466030435731, 1.0209874144697157, 1.0137731705250905, 1.0038291632099972, 0.987943009955437, 0.9622779086746256, 0.9231976826890829, 0.8682965562043136, 0.7972388073272193, 0.7119279582679571, 0.6158588993101382, 0.5130220628279911, 0.4069329778902411, 0.30012682887769665, 0.19410829401541932, 0.08956976745400544, -0.013298993172405946, -0.11450746597972528, -0.21399952646094372, -0.3114314587971726, -0.40600025937489503, -0.4963508303707573, -0.5806126106277416, -0.6566142362811292, -0.7222751032857833, -0.7760799885276684, -0.8174643972938972, -0.8469504460764985, -0.8659892169342667, -0.8766034774528556, -0.8809867080771778, -0.8811808418712176, -0.8788794914100785, -0.875342998856256, -0.8713865658481453, -0.8674041974220982], \"type\": \"scatter\", \"uid\": \"fa824901-82cf-492d-a4dc-1b8e3fe2f057\"}, {\"name\": \"y\", \"x\": [0.0, 0.06060606060606061, 0.12121212121212122, 0.18181818181818182, 0.24242424242424243, 0.30303030303030304, 0.36363636363636365, 0.42424242424242425, 0.48484848484848486, 0.5454545454545454, 0.6060606060606061, 0.6666666666666667, 0.7272727272727273, 0.7878787878787878, 0.8484848484848485, 0.9090909090909092, 0.9696969696969697, 1.0303030303030303, 1.0909090909090908, 1.1515151515151516, 1.2121212121212122, 1.2727272727272727, 1.3333333333333335, 1.393939393939394, 1.4545454545454546, 1.5151515151515151, 1.5757575757575757, 1.6363636363636365, 1.696969696969697, 1.7575757575757576, 1.8181818181818183, 1.878787878787879, 1.9393939393939394, 2.0, 2.0606060606060606, 2.121212121212121, 2.1818181818181817, 2.2424242424242427, 2.303030303030303, 2.3636363636363638, 2.4242424242424243, 2.484848484848485, 2.5454545454545454, 2.606060606060606, 2.666666666666667, 2.7272727272727275, 2.787878787878788, 2.8484848484848486, 2.909090909090909, 2.9696969696969697, 3.0303030303030303, 3.090909090909091, 3.1515151515151514, 3.2121212121212124, 3.272727272727273, 3.3333333333333335, 3.393939393939394, 3.4545454545454546, 3.515151515151515, 3.5757575757575757, 3.6363636363636367, 3.6969696969696972, 3.757575757575758, 3.8181818181818183, 3.878787878787879, 3.9393939393939394, 4.0, 4.0606060606060606, 4.121212121212121, 4.181818181818182, 4.242424242424242, 4.303030303030303, 4.363636363636363, 4.424242424242425, 4.484848484848485, 4.545454545454546, 4.606060606060606, 4.666666666666667, 4.7272727272727275, 4.787878787878788, 4.848484848484849, 4.909090909090909, 4.96969696969697, 5.03030303030303, 5.090909090909091, 5.151515151515151, 5.212121212121212, 5.2727272727272725, 5.333333333333334, 5.3939393939393945, 5.454545454545455, 5.515151515151516, 5.575757575757576, 5.636363636363637, 5.696969696969697, 5.757575757575758, 5.818181818181818, 5.878787878787879, 5.9393939393939394, 6.0], \"y\": [0.0, 0.9275448325420768, 1.6252700330371528, 1.8595825564303345, 1.8462229799162784, 1.8032366504665134, 1.654850126167924, 1.2420859948843215, 0.6500446642053311, 0.045186729136523704, -0.5293808596012912, -1.0467786871313218, -1.4116199039357176, -1.562175760571004, -1.5764441810825647, -1.5550164411023637, -1.5285014339222076, -1.4616043361426325, -1.291505224891791, -1.0000827824475937, -0.6400449652065053, -0.2689412160255923, 0.09113476397468115, 0.4373639467714392, 0.761003644410565, 1.0339499480600653, 1.220822680163824, 1.3126948946943262, 1.3368361746660284, 1.330169274647251, 1.3164670961001288, 1.3017623788703534, 1.2770614754174923, 1.2231106733033337, 1.1201144316983378, 0.9617133373745834, 0.7607333528390774, 0.5388915745589706, 0.31315916295167495, 0.09141342236007063, -0.12456283247173233, -0.33450913640796603, -0.5359050064294689, -0.7216127723114438, -0.8802482849426211, -1.000623444492499, -1.0782106487307939, -1.1179905490555937, -1.1311902796232456, -1.1296900778981762, -1.1223488567491111, -1.1139175303919728, -1.1052708654704486, -1.0939493963422573, -1.0748091148231402, -1.0411243544048727, -0.9865305321520321, -0.9075172357574051, -0.8051458500149421, -0.6846257598701961, -0.5529834915172145, -0.4165434361830663, -0.27959838141171955, -0.1443592757302281, -0.011579224635494732, 0.11867379015958941, 0.24632212404621642, 0.3707085943559691, 0.49017895345271467, 0.6019199820028384, 0.7022265788225424, 0.7872991219011463, 0.8543812837238588, 0.9027128641654808, 0.9337703189076347, 0.9507088508175557, 0.9573993793257143, 0.9575500253355517, 0.9541648868248808, 0.9493307661572116, 0.944206221139383, 0.939093970374526, 0.933529966735517, 0.9263742217723451, 0.9159258281662173, 0.9001048012852292, 0.8767389999196389, 0.8439545746477302, 0.8005959053336364, 0.7465332557001076, 0.6827182384082245, 0.6109470103180275, 0.5334311581292164, 0.45235209669634685, 0.3695418171493111, 0.2863395156099453, 0.20359478685896584, 0.12175746216775195, 0.041001303842690545, -0.03865033076267496], \"type\": \"scatter\", \"uid\": \"6fe7517f-4454-4b98-ade4-0df9db8a8e5e\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"9690f851-87ba-4365-9f8d-dee8f63e14e3\"));});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"9690f851-87ba-4365-9f8d-dee8f63e14e3\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"9690f851-87ba-4365-9f8d-dee8f63e14e3\", [{\"name\": \"x\", \"x\": [0.0, 0.06060606060606061, 0.12121212121212122, 0.18181818181818182, 0.24242424242424243, 0.30303030303030304, 0.36363636363636365, 0.42424242424242425, 0.48484848484848486, 0.5454545454545454, 0.6060606060606061, 0.6666666666666667, 0.7272727272727273, 0.7878787878787878, 0.8484848484848485, 0.9090909090909092, 0.9696969696969697, 1.0303030303030303, 1.0909090909090908, 1.1515151515151516, 1.2121212121212122, 1.2727272727272727, 1.3333333333333335, 1.393939393939394, 1.4545454545454546, 1.5151515151515151, 1.5757575757575757, 1.6363636363636365, 1.696969696969697, 1.7575757575757576, 1.8181818181818183, 1.878787878787879, 1.9393939393939394, 2.0, 2.0606060606060606, 2.121212121212121, 2.1818181818181817, 2.2424242424242427, 2.303030303030303, 2.3636363636363638, 2.4242424242424243, 2.484848484848485, 2.5454545454545454, 2.606060606060606, 2.666666666666667, 2.7272727272727275, 2.787878787878788, 2.8484848484848486, 2.909090909090909, 2.9696969696969697, 3.0303030303030303, 3.090909090909091, 3.1515151515151514, 3.2121212121212124, 3.272727272727273, 3.3333333333333335, 3.393939393939394, 3.4545454545454546, 3.515151515151515, 3.5757575757575757, 3.6363636363636367, 3.6969696969696972, 3.757575757575758, 3.8181818181818183, 3.878787878787879, 3.9393939393939394, 4.0, 4.0606060606060606, 4.121212121212121, 4.181818181818182, 4.242424242424242, 4.303030303030303, 4.363636363636363, 4.424242424242425, 4.484848484848485, 4.545454545454546, 4.606060606060606, 4.666666666666667, 4.7272727272727275, 4.787878787878788, 4.848484848484849, 4.909090909090909, 4.96969696969697, 5.03030303030303, 5.090909090909091, 5.151515151515151, 5.212121212121212, 5.2727272727272725, 5.333333333333334, 5.3939393939393945, 5.454545454545455, 5.515151515151516, 5.575757575757576, 5.636363636363637, 5.696969696969697, 5.757575757575758, 5.818181818181818, 5.878787878787879, 5.9393939393939394, 6.0], \"y\": [2.0, 1.9285373714028342, 1.601104367867955, 0.9032253373890189, 0.12226440995856018, -0.616152506793763, -1.2569160225766454, -1.631102663173404, -1.716655996540273, -1.6952775715140853, -1.6623170891768728, -1.5686931419451788, -1.3100016111362691, -0.8887037395657653, -0.4126330334956034, 0.05321024695495329, 0.4981754501197791, 0.9060395627706298, 1.2231061668130039, 1.3984836499938114, 1.4514488764532503, 1.4461671600939257, 1.4285930930306656, 1.408041164759553, 1.3634401509702723, 1.258176733921232, 1.069686708334913, 0.813529560698556, 0.5272056023710875, 0.2385483933236769, -0.042417962587647844, -0.3144752036919671, -0.5746910503758015, -0.8114062628576733, -1.0043998937279062, -1.1360488565734772, -1.205403955167543, -1.2287664218277128, -1.2275269893083047, -1.2177404426274663, -1.206917122186534, -1.1946317862874565, -1.1739434608185018, -1.1335403029956341, -1.0619083111053418, -0.9532647670850372, -0.8113551159422893, -0.6472081733138771, -0.47293294302501243, -0.29709837463031746, -0.1238716675454302, 0.04557660629262953, 0.2111855803669549, 0.37236593846510824, 0.5267591791125672, 0.669620171386869, 0.794340641393042, 0.8944548128824078, 0.9663919571705629, 1.011166707633886, 1.0337935409046548, 1.041104687002145, 1.0395766447424966, 1.0340867813943715, 1.0275466030435731, 1.0209874144697157, 1.0137731705250905, 1.0038291632099972, 0.987943009955437, 0.9622779086746256, 0.9231976826890829, 0.8682965562043136, 0.7972388073272193, 0.7119279582679571, 0.6158588993101382, 0.5130220628279911, 0.4069329778902411, 0.30012682887769665, 0.19410829401541932, 0.08956976745400544, -0.013298993172405946, -0.11450746597972528, -0.21399952646094372, -0.3114314587971726, -0.40600025937489503, -0.4963508303707573, -0.5806126106277416, -0.6566142362811292, -0.7222751032857833, -0.7760799885276684, -0.8174643972938972, -0.8469504460764985, -0.8659892169342667, -0.8766034774528556, -0.8809867080771778, -0.8811808418712176, -0.8788794914100785, -0.875342998856256, -0.8713865658481453, -0.8674041974220982], \"type\": \"scatter\", \"uid\": \"fa824901-82cf-492d-a4dc-1b8e3fe2f057\"}, {\"name\": \"y\", \"x\": [0.0, 0.06060606060606061, 0.12121212121212122, 0.18181818181818182, 0.24242424242424243, 0.30303030303030304, 0.36363636363636365, 0.42424242424242425, 0.48484848484848486, 0.5454545454545454, 0.6060606060606061, 0.6666666666666667, 0.7272727272727273, 0.7878787878787878, 0.8484848484848485, 0.9090909090909092, 0.9696969696969697, 1.0303030303030303, 1.0909090909090908, 1.1515151515151516, 1.2121212121212122, 1.2727272727272727, 1.3333333333333335, 1.393939393939394, 1.4545454545454546, 1.5151515151515151, 1.5757575757575757, 1.6363636363636365, 1.696969696969697, 1.7575757575757576, 1.8181818181818183, 1.878787878787879, 1.9393939393939394, 2.0, 2.0606060606060606, 2.121212121212121, 2.1818181818181817, 2.2424242424242427, 2.303030303030303, 2.3636363636363638, 2.4242424242424243, 2.484848484848485, 2.5454545454545454, 2.606060606060606, 2.666666666666667, 2.7272727272727275, 2.787878787878788, 2.8484848484848486, 2.909090909090909, 2.9696969696969697, 3.0303030303030303, 3.090909090909091, 3.1515151515151514, 3.2121212121212124, 3.272727272727273, 3.3333333333333335, 3.393939393939394, 3.4545454545454546, 3.515151515151515, 3.5757575757575757, 3.6363636363636367, 3.6969696969696972, 3.757575757575758, 3.8181818181818183, 3.878787878787879, 3.9393939393939394, 4.0, 4.0606060606060606, 4.121212121212121, 4.181818181818182, 4.242424242424242, 4.303030303030303, 4.363636363636363, 4.424242424242425, 4.484848484848485, 4.545454545454546, 4.606060606060606, 4.666666666666667, 4.7272727272727275, 4.787878787878788, 4.848484848484849, 4.909090909090909, 4.96969696969697, 5.03030303030303, 5.090909090909091, 5.151515151515151, 5.212121212121212, 5.2727272727272725, 5.333333333333334, 5.3939393939393945, 5.454545454545455, 5.515151515151516, 5.575757575757576, 5.636363636363637, 5.696969696969697, 5.757575757575758, 5.818181818181818, 5.878787878787879, 5.9393939393939394, 6.0], \"y\": [0.0, 0.9275448325420768, 1.6252700330371528, 1.8595825564303345, 1.8462229799162784, 1.8032366504665134, 1.654850126167924, 1.2420859948843215, 0.6500446642053311, 0.045186729136523704, -0.5293808596012912, -1.0467786871313218, -1.4116199039357176, -1.562175760571004, -1.5764441810825647, -1.5550164411023637, -1.5285014339222076, -1.4616043361426325, -1.291505224891791, -1.0000827824475937, -0.6400449652065053, -0.2689412160255923, 0.09113476397468115, 0.4373639467714392, 0.761003644410565, 1.0339499480600653, 1.220822680163824, 1.3126948946943262, 1.3368361746660284, 1.330169274647251, 1.3164670961001288, 1.3017623788703534, 1.2770614754174923, 1.2231106733033337, 1.1201144316983378, 0.9617133373745834, 0.7607333528390774, 0.5388915745589706, 0.31315916295167495, 0.09141342236007063, -0.12456283247173233, -0.33450913640796603, -0.5359050064294689, -0.7216127723114438, -0.8802482849426211, -1.000623444492499, -1.0782106487307939, -1.1179905490555937, -1.1311902796232456, -1.1296900778981762, -1.1223488567491111, -1.1139175303919728, -1.1052708654704486, -1.0939493963422573, -1.0748091148231402, -1.0411243544048727, -0.9865305321520321, -0.9075172357574051, -0.8051458500149421, -0.6846257598701961, -0.5529834915172145, -0.4165434361830663, -0.27959838141171955, -0.1443592757302281, -0.011579224635494732, 0.11867379015958941, 0.24632212404621642, 0.3707085943559691, 0.49017895345271467, 0.6019199820028384, 0.7022265788225424, 0.7872991219011463, 0.8543812837238588, 0.9027128641654808, 0.9337703189076347, 0.9507088508175557, 0.9573993793257143, 0.9575500253355517, 0.9541648868248808, 0.9493307661572116, 0.944206221139383, 0.939093970374526, 0.933529966735517, 0.9263742217723451, 0.9159258281662173, 0.9001048012852292, 0.8767389999196389, 0.8439545746477302, 0.8005959053336364, 0.7465332557001076, 0.6827182384082245, 0.6109470103180275, 0.5334311581292164, 0.45235209669634685, 0.3695418171493111, 0.2863395156099453, 0.20359478685896584, 0.12175746216775195, 0.041001303842690545, -0.03865033076267496], \"type\": \"scatter\", \"uid\": \"6fe7517f-4454-4b98-ade4-0df9db8a8e5e\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"9690f851-87ba-4365-9f8d-dee8f63e14e3\"));});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "linkText": "Export to plot.ly",
        "plotlyServerURL": "https://plot.ly",
        "showLink": false
       },
       "data": [
        {
         "type": "scatter",
         "uid": "26a5a04f-0f20-40e0-b72a-d6a0b8581da7",
         "x": [
          2,
          1.9285373714028342,
          1.601104367867955,
          0.9032253373890189,
          0.12226440995856018,
          -0.616152506793763,
          -1.2569160225766454,
          -1.631102663173404,
          -1.716655996540273,
          -1.6952775715140853,
          -1.6623170891768728,
          -1.5686931419451788,
          -1.3100016111362691,
          -0.8887037395657653,
          -0.4126330334956034,
          0.05321024695495329,
          0.4981754501197791,
          0.9060395627706298,
          1.2231061668130039,
          1.3984836499938114,
          1.4514488764532503,
          1.4461671600939257,
          1.4285930930306656,
          1.408041164759553,
          1.3634401509702723,
          1.258176733921232,
          1.069686708334913,
          0.813529560698556,
          0.5272056023710875,
          0.2385483933236769,
          -0.042417962587647844,
          -0.3144752036919671,
          -0.5746910503758015,
          -0.8114062628576733,
          -1.0043998937279062,
          -1.1360488565734772,
          -1.205403955167543,
          -1.2287664218277128,
          -1.2275269893083047,
          -1.2177404426274663,
          -1.206917122186534,
          -1.1946317862874565,
          -1.1739434608185018,
          -1.1335403029956341,
          -1.0619083111053418,
          -0.9532647670850372,
          -0.8113551159422893,
          -0.6472081733138771,
          -0.47293294302501243,
          -0.29709837463031746,
          -0.1238716675454302,
          0.04557660629262953,
          0.2111855803669549,
          0.37236593846510824,
          0.5267591791125672,
          0.669620171386869,
          0.794340641393042,
          0.8944548128824078,
          0.9663919571705629,
          1.011166707633886,
          1.0337935409046548,
          1.041104687002145,
          1.0395766447424966,
          1.0340867813943715,
          1.0275466030435731,
          1.0209874144697157,
          1.0137731705250905,
          1.0038291632099972,
          0.987943009955437,
          0.9622779086746256,
          0.9231976826890829,
          0.8682965562043136,
          0.7972388073272193,
          0.7119279582679571,
          0.6158588993101382,
          0.5130220628279911,
          0.4069329778902411,
          0.30012682887769665,
          0.19410829401541932,
          0.08956976745400544,
          -0.013298993172405946,
          -0.11450746597972528,
          -0.21399952646094372,
          -0.3114314587971726,
          -0.40600025937489503,
          -0.4963508303707573,
          -0.5806126106277416,
          -0.6566142362811292,
          -0.7222751032857833,
          -0.7760799885276684,
          -0.8174643972938972,
          -0.8469504460764985,
          -0.8659892169342667,
          -0.8766034774528556,
          -0.8809867080771778,
          -0.8811808418712176,
          -0.8788794914100785,
          -0.875342998856256,
          -0.8713865658481453,
          -0.8674041974220982
         ],
         "y": [
          0,
          0.9275448325420768,
          1.6252700330371528,
          1.8595825564303345,
          1.8462229799162784,
          1.8032366504665134,
          1.654850126167924,
          1.2420859948843215,
          0.6500446642053311,
          0.045186729136523704,
          -0.5293808596012912,
          -1.0467786871313218,
          -1.4116199039357176,
          -1.562175760571004,
          -1.5764441810825647,
          -1.5550164411023637,
          -1.5285014339222076,
          -1.4616043361426325,
          -1.291505224891791,
          -1.0000827824475937,
          -0.6400449652065053,
          -0.2689412160255923,
          0.09113476397468115,
          0.4373639467714392,
          0.761003644410565,
          1.0339499480600653,
          1.220822680163824,
          1.3126948946943262,
          1.3368361746660284,
          1.330169274647251,
          1.3164670961001288,
          1.3017623788703534,
          1.2770614754174923,
          1.2231106733033337,
          1.1201144316983378,
          0.9617133373745834,
          0.7607333528390774,
          0.5388915745589706,
          0.31315916295167495,
          0.09141342236007063,
          -0.12456283247173233,
          -0.33450913640796603,
          -0.5359050064294689,
          -0.7216127723114438,
          -0.8802482849426211,
          -1.000623444492499,
          -1.0782106487307939,
          -1.1179905490555937,
          -1.1311902796232456,
          -1.1296900778981762,
          -1.1223488567491111,
          -1.1139175303919728,
          -1.1052708654704486,
          -1.0939493963422573,
          -1.0748091148231402,
          -1.0411243544048727,
          -0.9865305321520321,
          -0.9075172357574051,
          -0.8051458500149421,
          -0.6846257598701961,
          -0.5529834915172145,
          -0.4165434361830663,
          -0.27959838141171955,
          -0.1443592757302281,
          -0.011579224635494732,
          0.11867379015958941,
          0.24632212404621642,
          0.3707085943559691,
          0.49017895345271467,
          0.6019199820028384,
          0.7022265788225424,
          0.7872991219011463,
          0.8543812837238588,
          0.9027128641654808,
          0.9337703189076347,
          0.9507088508175557,
          0.9573993793257143,
          0.9575500253355517,
          0.9541648868248808,
          0.9493307661572116,
          0.944206221139383,
          0.939093970374526,
          0.933529966735517,
          0.9263742217723451,
          0.9159258281662173,
          0.9001048012852292,
          0.8767389999196389,
          0.8439545746477302,
          0.8005959053336364,
          0.7465332557001076,
          0.6827182384082245,
          0.6109470103180275,
          0.5334311581292164,
          0.45235209669634685,
          0.3695418171493111,
          0.2863395156099453,
          0.20359478685896584,
          0.12175746216775195,
          0.041001303842690545,
          -0.03865033076267496
         ]
        }
       ],
       "layout": {}
      },
      "text/html": [
       "<div id=\"bb7c881d-1689-4d29-bdc9-a0fd70d7733b\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"bb7c881d-1689-4d29-bdc9-a0fd70d7733b\", [{\"x\": [2.0, 1.9285373714028342, 1.601104367867955, 0.9032253373890189, 0.12226440995856018, -0.616152506793763, -1.2569160225766454, -1.631102663173404, -1.716655996540273, -1.6952775715140853, -1.6623170891768728, -1.5686931419451788, -1.3100016111362691, -0.8887037395657653, -0.4126330334956034, 0.05321024695495329, 0.4981754501197791, 0.9060395627706298, 1.2231061668130039, 1.3984836499938114, 1.4514488764532503, 1.4461671600939257, 1.4285930930306656, 1.408041164759553, 1.3634401509702723, 1.258176733921232, 1.069686708334913, 0.813529560698556, 0.5272056023710875, 0.2385483933236769, -0.042417962587647844, -0.3144752036919671, -0.5746910503758015, -0.8114062628576733, -1.0043998937279062, -1.1360488565734772, -1.205403955167543, -1.2287664218277128, -1.2275269893083047, -1.2177404426274663, -1.206917122186534, -1.1946317862874565, -1.1739434608185018, -1.1335403029956341, -1.0619083111053418, -0.9532647670850372, -0.8113551159422893, -0.6472081733138771, -0.47293294302501243, -0.29709837463031746, -0.1238716675454302, 0.04557660629262953, 0.2111855803669549, 0.37236593846510824, 0.5267591791125672, 0.669620171386869, 0.794340641393042, 0.8944548128824078, 0.9663919571705629, 1.011166707633886, 1.0337935409046548, 1.041104687002145, 1.0395766447424966, 1.0340867813943715, 1.0275466030435731, 1.0209874144697157, 1.0137731705250905, 1.0038291632099972, 0.987943009955437, 0.9622779086746256, 0.9231976826890829, 0.8682965562043136, 0.7972388073272193, 0.7119279582679571, 0.6158588993101382, 0.5130220628279911, 0.4069329778902411, 0.30012682887769665, 0.19410829401541932, 0.08956976745400544, -0.013298993172405946, -0.11450746597972528, -0.21399952646094372, -0.3114314587971726, -0.40600025937489503, -0.4963508303707573, -0.5806126106277416, -0.6566142362811292, -0.7222751032857833, -0.7760799885276684, -0.8174643972938972, -0.8469504460764985, -0.8659892169342667, -0.8766034774528556, -0.8809867080771778, -0.8811808418712176, -0.8788794914100785, -0.875342998856256, -0.8713865658481453, -0.8674041974220982], \"y\": [0.0, 0.9275448325420768, 1.6252700330371528, 1.8595825564303345, 1.8462229799162784, 1.8032366504665134, 1.654850126167924, 1.2420859948843215, 0.6500446642053311, 0.045186729136523704, -0.5293808596012912, -1.0467786871313218, -1.4116199039357176, -1.562175760571004, -1.5764441810825647, -1.5550164411023637, -1.5285014339222076, -1.4616043361426325, -1.291505224891791, -1.0000827824475937, -0.6400449652065053, -0.2689412160255923, 0.09113476397468115, 0.4373639467714392, 0.761003644410565, 1.0339499480600653, 1.220822680163824, 1.3126948946943262, 1.3368361746660284, 1.330169274647251, 1.3164670961001288, 1.3017623788703534, 1.2770614754174923, 1.2231106733033337, 1.1201144316983378, 0.9617133373745834, 0.7607333528390774, 0.5388915745589706, 0.31315916295167495, 0.09141342236007063, -0.12456283247173233, -0.33450913640796603, -0.5359050064294689, -0.7216127723114438, -0.8802482849426211, -1.000623444492499, -1.0782106487307939, -1.1179905490555937, -1.1311902796232456, -1.1296900778981762, -1.1223488567491111, -1.1139175303919728, -1.1052708654704486, -1.0939493963422573, -1.0748091148231402, -1.0411243544048727, -0.9865305321520321, -0.9075172357574051, -0.8051458500149421, -0.6846257598701961, -0.5529834915172145, -0.4165434361830663, -0.27959838141171955, -0.1443592757302281, -0.011579224635494732, 0.11867379015958941, 0.24632212404621642, 0.3707085943559691, 0.49017895345271467, 0.6019199820028384, 0.7022265788225424, 0.7872991219011463, 0.8543812837238588, 0.9027128641654808, 0.9337703189076347, 0.9507088508175557, 0.9573993793257143, 0.9575500253355517, 0.9541648868248808, 0.9493307661572116, 0.944206221139383, 0.939093970374526, 0.933529966735517, 0.9263742217723451, 0.9159258281662173, 0.9001048012852292, 0.8767389999196389, 0.8439545746477302, 0.8005959053336364, 0.7465332557001076, 0.6827182384082245, 0.6109470103180275, 0.5334311581292164, 0.45235209669634685, 0.3695418171493111, 0.2863395156099453, 0.20359478685896584, 0.12175746216775195, 0.041001303842690545, -0.03865033076267496], \"type\": \"scatter\", \"uid\": \"064ad92f-3886-4f0e-aed7-df56c8e73d3a\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"bb7c881d-1689-4d29-bdc9-a0fd70d7733b\"));});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"bb7c881d-1689-4d29-bdc9-a0fd70d7733b\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"bb7c881d-1689-4d29-bdc9-a0fd70d7733b\", [{\"x\": [2.0, 1.9285373714028342, 1.601104367867955, 0.9032253373890189, 0.12226440995856018, -0.616152506793763, -1.2569160225766454, -1.631102663173404, -1.716655996540273, -1.6952775715140853, -1.6623170891768728, -1.5686931419451788, -1.3100016111362691, -0.8887037395657653, -0.4126330334956034, 0.05321024695495329, 0.4981754501197791, 0.9060395627706298, 1.2231061668130039, 1.3984836499938114, 1.4514488764532503, 1.4461671600939257, 1.4285930930306656, 1.408041164759553, 1.3634401509702723, 1.258176733921232, 1.069686708334913, 0.813529560698556, 0.5272056023710875, 0.2385483933236769, -0.042417962587647844, -0.3144752036919671, -0.5746910503758015, -0.8114062628576733, -1.0043998937279062, -1.1360488565734772, -1.205403955167543, -1.2287664218277128, -1.2275269893083047, -1.2177404426274663, -1.206917122186534, -1.1946317862874565, -1.1739434608185018, -1.1335403029956341, -1.0619083111053418, -0.9532647670850372, -0.8113551159422893, -0.6472081733138771, -0.47293294302501243, -0.29709837463031746, -0.1238716675454302, 0.04557660629262953, 0.2111855803669549, 0.37236593846510824, 0.5267591791125672, 0.669620171386869, 0.794340641393042, 0.8944548128824078, 0.9663919571705629, 1.011166707633886, 1.0337935409046548, 1.041104687002145, 1.0395766447424966, 1.0340867813943715, 1.0275466030435731, 1.0209874144697157, 1.0137731705250905, 1.0038291632099972, 0.987943009955437, 0.9622779086746256, 0.9231976826890829, 0.8682965562043136, 0.7972388073272193, 0.7119279582679571, 0.6158588993101382, 0.5130220628279911, 0.4069329778902411, 0.30012682887769665, 0.19410829401541932, 0.08956976745400544, -0.013298993172405946, -0.11450746597972528, -0.21399952646094372, -0.3114314587971726, -0.40600025937489503, -0.4963508303707573, -0.5806126106277416, -0.6566142362811292, -0.7222751032857833, -0.7760799885276684, -0.8174643972938972, -0.8469504460764985, -0.8659892169342667, -0.8766034774528556, -0.8809867080771778, -0.8811808418712176, -0.8788794914100785, -0.875342998856256, -0.8713865658481453, -0.8674041974220982], \"y\": [0.0, 0.9275448325420768, 1.6252700330371528, 1.8595825564303345, 1.8462229799162784, 1.8032366504665134, 1.654850126167924, 1.2420859948843215, 0.6500446642053311, 0.045186729136523704, -0.5293808596012912, -1.0467786871313218, -1.4116199039357176, -1.562175760571004, -1.5764441810825647, -1.5550164411023637, -1.5285014339222076, -1.4616043361426325, -1.291505224891791, -1.0000827824475937, -0.6400449652065053, -0.2689412160255923, 0.09113476397468115, 0.4373639467714392, 0.761003644410565, 1.0339499480600653, 1.220822680163824, 1.3126948946943262, 1.3368361746660284, 1.330169274647251, 1.3164670961001288, 1.3017623788703534, 1.2770614754174923, 1.2231106733033337, 1.1201144316983378, 0.9617133373745834, 0.7607333528390774, 0.5388915745589706, 0.31315916295167495, 0.09141342236007063, -0.12456283247173233, -0.33450913640796603, -0.5359050064294689, -0.7216127723114438, -0.8802482849426211, -1.000623444492499, -1.0782106487307939, -1.1179905490555937, -1.1311902796232456, -1.1296900778981762, -1.1223488567491111, -1.1139175303919728, -1.1052708654704486, -1.0939493963422573, -1.0748091148231402, -1.0411243544048727, -0.9865305321520321, -0.9075172357574051, -0.8051458500149421, -0.6846257598701961, -0.5529834915172145, -0.4165434361830663, -0.27959838141171955, -0.1443592757302281, -0.011579224635494732, 0.11867379015958941, 0.24632212404621642, 0.3707085943559691, 0.49017895345271467, 0.6019199820028384, 0.7022265788225424, 0.7872991219011463, 0.8543812837238588, 0.9027128641654808, 0.9337703189076347, 0.9507088508175557, 0.9573993793257143, 0.9575500253355517, 0.9541648868248808, 0.9493307661572116, 0.944206221139383, 0.939093970374526, 0.933529966735517, 0.9263742217723451, 0.9159258281662173, 0.9001048012852292, 0.8767389999196389, 0.8439545746477302, 0.8005959053336364, 0.7465332557001076, 0.6827182384082245, 0.6109470103180275, 0.5334311581292164, 0.45235209669634685, 0.3695418171493111, 0.2863395156099453, 0.20359478685896584, 0.12175746216775195, 0.041001303842690545, -0.03865033076267496], \"type\": \"scatter\", \"uid\": \"064ad92f-3886-4f0e-aed7-df56c8e73d3a\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"bb7c881d-1689-4d29-bdc9-a0fd70d7733b\"));});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "zs = scipy.integrate.odeint(damped_oscillator_2d, z0, ts, args=(A,))\n",
    "\n",
    "xs = zs[:, 0]\n",
    "ys = zs[:, 1]\n",
    "iplot([go.Scatter(x=ts, y=xs, name=\"x\"), go.Scatter(x=ts, y=ys, name=\"y\")])\n",
    "iplot([go.Scatter(x=xs, y=ys)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Helpers to call ODE Solvers from TF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ode_tf(f, z0, ts, args):\n",
    "    \"\"\"calls Full TF ODE Runge-Kutta Dopri5 Solver\n",
    "    \n",
    "    Args:\n",
    "        f: a function of type (z0, ts, args) -> ys\n",
    "            where z0, the initial value\n",
    "                  ts, the time values\n",
    "                  args, the supplementary args\n",
    "\n",
    "        z0: initial value\n",
    "        ts: time values on which to solve equation\n",
    "        args: supplementary arsg\n",
    "        \n",
    "    Returns:\n",
    "        array of solutions computed by ODE Solver\n",
    "    Please note, TF solver params are hard-coded here but they could be managed in a cleaner way\n",
    "    \"\"\"\n",
    "    # current TF odeint implementation doesn't manage args so we need this lambda\n",
    "    zs, info = tf.contrib.integrate.odeint(\n",
    "        lambda z0, ts: f(z0, ts, args), z0, ts,\n",
    "        method=\"dopri5\",\n",
    "        options={\"max_num_steps\":1000, \"first_step\": 0.001},\n",
    "        rtol = 0.001,\n",
    "        full_output=True\n",
    "    )\n",
    "    return zs\n",
    "\n",
    "def ode_scipy(f, z0, ts, args):\n",
    "    \"\"\"calls Python Scipy ODE Solver from TF\n",
    "    \n",
    "    Args:\n",
    "        f: a function of type (z0, ts, args) -> ys\n",
    "            where z0, the initial value\n",
    "                  ts, the time values\n",
    "                  args, the supplementary args\n",
    "\n",
    "        y-z0: initial value\n",
    "        ts: time values on which to solve equation\n",
    "        args: supplementary args\n",
    "\n",
    "    Returns:\n",
    "        array of solutions computed by ODE Solver\n",
    "    \"\"\"\n",
    "    # managing args as None... this is ugly but simplifies a bit later code\n",
    "    if args is not None:\n",
    "        r = tf.py_func(\n",
    "            lambda z0, ts, args: scipy.integrate.odeint(f, z0, ts, args=(args,)),\n",
    "            [z0, ts, args], z0.dtype)\n",
    "    else:\n",
    "        r = tf.py_func(\n",
    "            lambda z0, ts: scipy.integrate.odeint(f, z0, ts, args=(None,)),\n",
    "            [z0, ts], z0.dtype)\n",
    "    return r\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lotka–Volterra equations (the funny predators/preys or rabbits/foxes equations)\n",
    "\n",
    "> This sample was taken from article https://julialang.org/blog/2019/01/fluxdiffeq\n",
    "\n",
    "https://en.wikipedia.org/wiki/Lotka%E2%80%93Volterra_equations\n",
    "\n",
    "$\n",
    "\\begin{aligned}\n",
    "{\\frac {dx}{dt}} & = \\alpha x - \\beta xy, \\\\\n",
    "{\\frac {dy}{dt}} & = \\delta xy - \\gamma y\n",
    "\\end{aligned}\n",
    "$\n",
    "\n",
    "where\n",
    "\n",
    "$x$ is the number of preys (for example, rabbits);\n",
    "$y$ is the number of predators (for example, foxes);\n",
    "\n",
    "${\\frac {dy}{dt}}$ and ${\\frac {dx}{dt}}$ represent the instantaneous growth rates of the two populations;\n",
    "\n",
    "$t$ represents time\n",
    "\n",
    "$α, β, γ, δ$ are positive real parameters describing the interaction of the two species.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Numpy or eager-mode TF\n",
    "def lotka_volterra(z, ts, args):\n",
    "    x, y = z[0], z[1]\n",
    "    α, β, δ, γ = args[0], args[1], args[2], args[3]\n",
    "    dx = α*x - β*x*y\n",
    "    dy = -δ*y + γ*x*y\n",
    "    return tf.stack([dx, dy], axis=0)\n",
    "\n",
    "# Tensorflow Variables\n",
    "D = 2\n",
    "# time interval from 0 to 10 divided in 100 steps\n",
    "N = 100\n",
    "ts = tf.constant(np.linspace(0., 10.0, N))\n",
    "# initial arguments\n",
    "args = tf.Variable(tf.constant([1.5, 1.0, 3.0, 1.0], shape=(4,), dtype=ts.dtype))\n",
    "z0 = tf.Variable(tf.constant([1.0,1.0], shape=(2,), dtype=ts.dtype))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TF-RungeKutta ODE Solver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "linkText": "Export to plot.ly",
        "plotlyServerURL": "https://plot.ly",
        "showLink": false
       },
       "data": [
        {
         "name": "x",
         "type": "scatter",
         "uid": "57f1e07f-48ee-40d9-a2b6-e87a992002df",
         "x": [
          0,
          0.10101010101010101,
          0.20202020202020202,
          0.30303030303030304,
          0.40404040404040403,
          0.5050505050505051,
          0.6060606060606061,
          0.7070707070707071,
          0.8080808080808081,
          0.9090909090909091,
          1.0101010101010102,
          1.1111111111111112,
          1.2121212121212122,
          1.3131313131313131,
          1.4141414141414141,
          1.5151515151515151,
          1.6161616161616161,
          1.7171717171717171,
          1.8181818181818181,
          1.9191919191919191,
          2.0202020202020203,
          2.121212121212121,
          2.2222222222222223,
          2.323232323232323,
          2.4242424242424243,
          2.525252525252525,
          2.6262626262626263,
          2.727272727272727,
          2.8282828282828283,
          2.929292929292929,
          3.0303030303030303,
          3.131313131313131,
          3.2323232323232323,
          3.3333333333333335,
          3.4343434343434343,
          3.5353535353535355,
          3.6363636363636362,
          3.7373737373737375,
          3.8383838383838382,
          3.9393939393939394,
          4.040404040404041,
          4.141414141414141,
          4.242424242424242,
          4.343434343434343,
          4.444444444444445,
          4.545454545454545,
          4.646464646464646,
          4.747474747474747,
          4.848484848484849,
          4.94949494949495,
          5.05050505050505,
          5.151515151515151,
          5.252525252525253,
          5.353535353535354,
          5.454545454545454,
          5.555555555555555,
          5.656565656565657,
          5.757575757575758,
          5.858585858585858,
          5.959595959595959,
          6.0606060606060606,
          6.161616161616162,
          6.262626262626262,
          6.363636363636363,
          6.4646464646464645,
          6.565656565656566,
          6.666666666666667,
          6.767676767676767,
          6.8686868686868685,
          6.96969696969697,
          7.070707070707071,
          7.171717171717171,
          7.2727272727272725,
          7.373737373737374,
          7.474747474747475,
          7.575757575757575,
          7.6767676767676765,
          7.777777777777778,
          7.878787878787879,
          7.979797979797979,
          8.080808080808081,
          8.181818181818182,
          8.282828282828282,
          8.383838383838384,
          8.484848484848484,
          8.585858585858587,
          8.686868686868687,
          8.787878787878787,
          8.88888888888889,
          8.98989898989899,
          9.09090909090909,
          9.191919191919192,
          9.292929292929292,
          9.393939393939394,
          9.494949494949495,
          9.595959595959595,
          9.696969696969697,
          9.797979797979798,
          9.8989898989899,
          10
         ],
         "y": [
          1,
          1.061807053318688,
          1.145408152082289,
          1.2514936667618204,
          1.382402727024627,
          1.540324118858284,
          1.7272861559723265,
          1.945156679798251,
          2.196408566606793,
          2.4860145424435753,
          2.818434840870422,
          3.196925406251669,
          3.6235416188424057,
          4.0991382947884745,
          4.62133335158341,
          5.181647091781896,
          5.761500027114934,
          6.308725767841675,
          6.735976975851553,
          6.920323644032519,
          6.696557966250137,
          5.968926115336271,
          4.855280353296202,
          3.675511857566862,
          2.6897683308027944,
          1.9985375970034487,
          1.5540630475511819,
          1.2831977856530787,
          1.1126772453856806,
          1.0145341708284379,
          0.9639023899234396,
          0.9504882992353505,
          0.9683457377881893,
          1.0115246860408835,
          1.0780506021105003,
          1.1666392988318872,
          1.277579624153188,
          1.4128818689991496,
          1.5748503208007834,
          1.7659844055942244,
          1.9889786880207279,
          2.2466348466862254,
          2.542602025064851,
          2.882071762205804,
          3.2685868203425548,
          3.7037161547381032,
          4.187054913684991,
          4.716000282661653,
          5.279437416630268,
          5.854405801264264,
          6.384916942554239,
          6.776359124704928,
          6.895521853870157,
          6.5851736625239665,
          5.778074674866982,
          4.637025691280423,
          3.468928077108298,
          2.532936877253989,
          1.8947195063462554,
          1.4879884800337813,
          1.2378613082920311,
          1.0842692098782192,
          0.9970821693519221,
          0.9539252089584267,
          0.9473502935079327,
          0.9703304411933735,
          1.0181169401963994,
          1.088854996413232,
          1.1814480126133402,
          1.296676711092822,
          1.4366642215025767,
          1.6037493905419014,
          1.8004601971920677,
          2.029513752716314,
          2.2937141888423334,
          2.5973244834316405,
          2.945291964333588,
          3.340776097269539,
          3.7850759909325378,
          4.277630396987309,
          4.814983492492444,
          5.384504225476172,
          5.959528801018875,
          6.477205961046247,
          6.835781204824413,
          6.894933545936323,
          6.498236922908888,
          5.609419062813758,
          4.435223775619042,
          3.289326965524876,
          2.4047539852304185,
          1.8095307742486502,
          1.4347602213277777,
          1.2059514801019686,
          1.0650265016051608,
          0.9870809603748563,
          0.9521225427713154,
          0.9517880700186899,
          0.9791729106894487,
          1.0311129102480758
         ]
        },
        {
         "name": "y",
         "type": "scatter",
         "uid": "bc4b1ad4-94f1-4e10-9423-5fbf996608fe",
         "x": [
          0,
          0.10101010101010101,
          0.20202020202020202,
          0.30303030303030304,
          0.40404040404040403,
          0.5050505050505051,
          0.6060606060606061,
          0.7070707070707071,
          0.8080808080808081,
          0.9090909090909091,
          1.0101010101010102,
          1.1111111111111112,
          1.2121212121212122,
          1.3131313131313131,
          1.4141414141414141,
          1.5151515151515151,
          1.6161616161616161,
          1.7171717171717171,
          1.8181818181818181,
          1.9191919191919191,
          2.0202020202020203,
          2.121212121212121,
          2.2222222222222223,
          2.323232323232323,
          2.4242424242424243,
          2.525252525252525,
          2.6262626262626263,
          2.727272727272727,
          2.8282828282828283,
          2.929292929292929,
          3.0303030303030303,
          3.131313131313131,
          3.2323232323232323,
          3.3333333333333335,
          3.4343434343434343,
          3.5353535353535355,
          3.6363636363636362,
          3.7373737373737375,
          3.8383838383838382,
          3.9393939393939394,
          4.040404040404041,
          4.141414141414141,
          4.242424242424242,
          4.343434343434343,
          4.444444444444445,
          4.545454545454545,
          4.646464646464646,
          4.747474747474747,
          4.848484848484849,
          4.94949494949495,
          5.05050505050505,
          5.151515151515151,
          5.252525252525253,
          5.353535353535354,
          5.454545454545454,
          5.555555555555555,
          5.656565656565657,
          5.757575757575758,
          5.858585858585858,
          5.959595959595959,
          6.0606060606060606,
          6.161616161616162,
          6.262626262626262,
          6.363636363636363,
          6.4646464646464645,
          6.565656565656566,
          6.666666666666667,
          6.767676767676767,
          6.8686868686868685,
          6.96969696969697,
          7.070707070707071,
          7.171717171717171,
          7.2727272727272725,
          7.373737373737374,
          7.474747474747475,
          7.575757575757575,
          7.6767676767676765,
          7.777777777777778,
          7.878787878787879,
          7.979797979797979,
          8.080808080808081,
          8.181818181818182,
          8.282828282828282,
          8.383838383838384,
          8.484848484848484,
          8.585858585858587,
          8.686868686868687,
          8.787878787878787,
          8.88888888888889,
          8.98989898989899,
          9.09090909090909,
          9.191919191919192,
          9.292929292929292,
          9.393939393939394,
          9.494949494949495,
          9.595959595959595,
          9.696969696969697,
          9.797979797979798,
          9.8989898989899,
          10
         ],
         "y": [
          1,
          0.8194779946594071,
          0.6769761068104436,
          0.5649942997342504,
          0.47649230332538395,
          0.4063822874493165,
          0.3515376038120329,
          0.3107927859600305,
          0.2830456388909482,
          0.26500731809697947,
          0.2556885483605984,
          0.2554480924491077,
          0.2659701062039867,
          0.29026413854089095,
          0.333487934162219,
          0.40470752040644703,
          0.5188000623698279,
          0.7052680287423514,
          1.0088452457308028,
          1.488378592599668,
          2.196984089990479,
          3.0917560303845817,
          3.9551792167335784,
          4.492259504662691,
          4.570189585959295,
          4.265913874376103,
          3.7577366748706247,
          3.1908471694544316,
          2.6576537569553196,
          2.1843033299205263,
          1.7841644186582941,
          1.4532382343208106,
          1.1824166118569455,
          0.9647924837307708,
          0.7913278118358653,
          0.6543607261963258,
          0.5470804649708603,
          0.4630933171316783,
          0.39736906789565885,
          0.3463580652423448,
          0.30799121991394485,
          0.2815076233281379,
          0.26520721954168175,
          0.25762888314334614,
          0.25905779366122234,
          0.27148903866740604,
          0.29862761377799757,
          0.34592102856899964,
          0.42284297012467786,
          0.548314959703641,
          0.7541871348193039,
          1.0857938103504312,
          1.6019090153540527,
          2.347087887132778,
          3.2500616182413165,
          4.066311516539782,
          4.53485638244743,
          4.535402020067465,
          4.178273697238685,
          3.6502232312883685,
          3.086345632526418,
          2.56134912688668,
          2.100637685126049,
          1.7143039329680214,
          1.394953029710587,
          1.134909434534885,
          0.9263527175910217,
          0.7604673121084548,
          0.6297908472581991,
          0.5274932354269619,
          0.44738322232337935,
          0.3847210633005533,
          0.33625997671333385,
          0.3002461439183234,
          0.27591930926278896,
          0.261238077301631,
          0.2551448601416579,
          0.2582646477794929,
          0.2728773160832124,
          0.3029176267923462,
          0.3542301539987484,
          0.4377772742611726,
          0.5738862954973636,
          0.7967805117018197,
          1.1550869059767341,
          1.710596567457325,
          2.4961538912971166,
          3.412049481736945,
          4.189897393925115,
          4.580476109729426,
          4.504155699517714,
          4.104124966508637,
          3.5594888299612975,
          2.9964157448589375,
          2.4805740244946155,
          2.031274217603796,
          1.6555569590152095,
          1.3464560022040521,
          1.09618095556177,
          0.8956363459306874
         ]
        }
       ],
       "layout": {}
      },
      "text/html": [
       "<div id=\"ec454abe-78cd-4964-875b-d9891b73199a\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"ec454abe-78cd-4964-875b-d9891b73199a\", [{\"name\": \"x\", \"x\": [0.0, 0.10101010101010101, 0.20202020202020202, 0.30303030303030304, 0.40404040404040403, 0.5050505050505051, 0.6060606060606061, 0.7070707070707071, 0.8080808080808081, 0.9090909090909091, 1.0101010101010102, 1.1111111111111112, 1.2121212121212122, 1.3131313131313131, 1.4141414141414141, 1.5151515151515151, 1.6161616161616161, 1.7171717171717171, 1.8181818181818181, 1.9191919191919191, 2.0202020202020203, 2.121212121212121, 2.2222222222222223, 2.323232323232323, 2.4242424242424243, 2.525252525252525, 2.6262626262626263, 2.727272727272727, 2.8282828282828283, 2.929292929292929, 3.0303030303030303, 3.131313131313131, 3.2323232323232323, 3.3333333333333335, 3.4343434343434343, 3.5353535353535355, 3.6363636363636362, 3.7373737373737375, 3.8383838383838382, 3.9393939393939394, 4.040404040404041, 4.141414141414141, 4.242424242424242, 4.343434343434343, 4.444444444444445, 4.545454545454545, 4.646464646464646, 4.747474747474747, 4.848484848484849, 4.94949494949495, 5.05050505050505, 5.151515151515151, 5.252525252525253, 5.353535353535354, 5.454545454545454, 5.555555555555555, 5.656565656565657, 5.757575757575758, 5.858585858585858, 5.959595959595959, 6.0606060606060606, 6.161616161616162, 6.262626262626262, 6.363636363636363, 6.4646464646464645, 6.565656565656566, 6.666666666666667, 6.767676767676767, 6.8686868686868685, 6.96969696969697, 7.070707070707071, 7.171717171717171, 7.2727272727272725, 7.373737373737374, 7.474747474747475, 7.575757575757575, 7.6767676767676765, 7.777777777777778, 7.878787878787879, 7.979797979797979, 8.080808080808081, 8.181818181818182, 8.282828282828282, 8.383838383838384, 8.484848484848484, 8.585858585858587, 8.686868686868687, 8.787878787878787, 8.88888888888889, 8.98989898989899, 9.09090909090909, 9.191919191919192, 9.292929292929292, 9.393939393939394, 9.494949494949495, 9.595959595959595, 9.696969696969697, 9.797979797979798, 9.8989898989899, 10.0], \"y\": [1.0, 1.061807053318688, 1.145408152082289, 1.2514936667618204, 1.382402727024627, 1.540324118858284, 1.7272861559723265, 1.945156679798251, 2.196408566606793, 2.4860145424435753, 2.818434840870422, 3.196925406251669, 3.6235416188424057, 4.0991382947884745, 4.62133335158341, 5.181647091781896, 5.761500027114934, 6.308725767841675, 6.735976975851553, 6.920323644032519, 6.696557966250137, 5.968926115336271, 4.855280353296202, 3.675511857566862, 2.6897683308027944, 1.9985375970034487, 1.5540630475511819, 1.2831977856530787, 1.1126772453856806, 1.0145341708284379, 0.9639023899234396, 0.9504882992353505, 0.9683457377881893, 1.0115246860408835, 1.0780506021105003, 1.1666392988318872, 1.277579624153188, 1.4128818689991496, 1.5748503208007834, 1.7659844055942244, 1.9889786880207279, 2.2466348466862254, 2.542602025064851, 2.882071762205804, 3.2685868203425548, 3.7037161547381032, 4.187054913684991, 4.716000282661653, 5.279437416630268, 5.854405801264264, 6.384916942554239, 6.776359124704928, 6.895521853870157, 6.5851736625239665, 5.778074674866982, 4.637025691280423, 3.468928077108298, 2.532936877253989, 1.8947195063462554, 1.4879884800337813, 1.2378613082920311, 1.0842692098782192, 0.9970821693519221, 0.9539252089584267, 0.9473502935079327, 0.9703304411933735, 1.0181169401963994, 1.088854996413232, 1.1814480126133402, 1.296676711092822, 1.4366642215025767, 1.6037493905419014, 1.8004601971920677, 2.029513752716314, 2.2937141888423334, 2.5973244834316405, 2.945291964333588, 3.340776097269539, 3.7850759909325378, 4.277630396987309, 4.814983492492444, 5.384504225476172, 5.959528801018875, 6.477205961046247, 6.835781204824413, 6.894933545936323, 6.498236922908888, 5.609419062813758, 4.435223775619042, 3.289326965524876, 2.4047539852304185, 1.8095307742486502, 1.4347602213277777, 1.2059514801019686, 1.0650265016051608, 0.9870809603748563, 0.9521225427713154, 0.9517880700186899, 0.9791729106894487, 1.0311129102480758], \"type\": \"scatter\", \"uid\": \"7f0a0aa2-d0e5-430a-89ff-ba739f270e7e\"}, {\"name\": \"y\", \"x\": [0.0, 0.10101010101010101, 0.20202020202020202, 0.30303030303030304, 0.40404040404040403, 0.5050505050505051, 0.6060606060606061, 0.7070707070707071, 0.8080808080808081, 0.9090909090909091, 1.0101010101010102, 1.1111111111111112, 1.2121212121212122, 1.3131313131313131, 1.4141414141414141, 1.5151515151515151, 1.6161616161616161, 1.7171717171717171, 1.8181818181818181, 1.9191919191919191, 2.0202020202020203, 2.121212121212121, 2.2222222222222223, 2.323232323232323, 2.4242424242424243, 2.525252525252525, 2.6262626262626263, 2.727272727272727, 2.8282828282828283, 2.929292929292929, 3.0303030303030303, 3.131313131313131, 3.2323232323232323, 3.3333333333333335, 3.4343434343434343, 3.5353535353535355, 3.6363636363636362, 3.7373737373737375, 3.8383838383838382, 3.9393939393939394, 4.040404040404041, 4.141414141414141, 4.242424242424242, 4.343434343434343, 4.444444444444445, 4.545454545454545, 4.646464646464646, 4.747474747474747, 4.848484848484849, 4.94949494949495, 5.05050505050505, 5.151515151515151, 5.252525252525253, 5.353535353535354, 5.454545454545454, 5.555555555555555, 5.656565656565657, 5.757575757575758, 5.858585858585858, 5.959595959595959, 6.0606060606060606, 6.161616161616162, 6.262626262626262, 6.363636363636363, 6.4646464646464645, 6.565656565656566, 6.666666666666667, 6.767676767676767, 6.8686868686868685, 6.96969696969697, 7.070707070707071, 7.171717171717171, 7.2727272727272725, 7.373737373737374, 7.474747474747475, 7.575757575757575, 7.6767676767676765, 7.777777777777778, 7.878787878787879, 7.979797979797979, 8.080808080808081, 8.181818181818182, 8.282828282828282, 8.383838383838384, 8.484848484848484, 8.585858585858587, 8.686868686868687, 8.787878787878787, 8.88888888888889, 8.98989898989899, 9.09090909090909, 9.191919191919192, 9.292929292929292, 9.393939393939394, 9.494949494949495, 9.595959595959595, 9.696969696969697, 9.797979797979798, 9.8989898989899, 10.0], \"y\": [1.0, 0.8194779946594071, 0.6769761068104436, 0.5649942997342504, 0.47649230332538395, 0.4063822874493165, 0.3515376038120329, 0.3107927859600305, 0.2830456388909482, 0.26500731809697947, 0.2556885483605984, 0.2554480924491077, 0.2659701062039867, 0.29026413854089095, 0.333487934162219, 0.40470752040644703, 0.5188000623698279, 0.7052680287423514, 1.0088452457308028, 1.488378592599668, 2.196984089990479, 3.0917560303845817, 3.9551792167335784, 4.492259504662691, 4.570189585959295, 4.265913874376103, 3.7577366748706247, 3.1908471694544316, 2.6576537569553196, 2.1843033299205263, 1.7841644186582941, 1.4532382343208106, 1.1824166118569455, 0.9647924837307708, 0.7913278118358653, 0.6543607261963258, 0.5470804649708603, 0.4630933171316783, 0.39736906789565885, 0.3463580652423448, 0.30799121991394485, 0.2815076233281379, 0.26520721954168175, 0.25762888314334614, 0.25905779366122234, 0.27148903866740604, 0.29862761377799757, 0.34592102856899964, 0.42284297012467786, 0.548314959703641, 0.7541871348193039, 1.0857938103504312, 1.6019090153540527, 2.347087887132778, 3.2500616182413165, 4.066311516539782, 4.53485638244743, 4.535402020067465, 4.178273697238685, 3.6502232312883685, 3.086345632526418, 2.56134912688668, 2.100637685126049, 1.7143039329680214, 1.394953029710587, 1.134909434534885, 0.9263527175910217, 0.7604673121084548, 0.6297908472581991, 0.5274932354269619, 0.44738322232337935, 0.3847210633005533, 0.33625997671333385, 0.3002461439183234, 0.27591930926278896, 0.261238077301631, 0.2551448601416579, 0.2582646477794929, 0.2728773160832124, 0.3029176267923462, 0.3542301539987484, 0.4377772742611726, 0.5738862954973636, 0.7967805117018197, 1.1550869059767341, 1.710596567457325, 2.4961538912971166, 3.412049481736945, 4.189897393925115, 4.580476109729426, 4.504155699517714, 4.104124966508637, 3.5594888299612975, 2.9964157448589375, 2.4805740244946155, 2.031274217603796, 1.6555569590152095, 1.3464560022040521, 1.09618095556177, 0.8956363459306874], \"type\": \"scatter\", \"uid\": \"d1e4cc27-c866-4afb-981c-b55dfc1f4d63\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"ec454abe-78cd-4964-875b-d9891b73199a\"));});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"ec454abe-78cd-4964-875b-d9891b73199a\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"ec454abe-78cd-4964-875b-d9891b73199a\", [{\"name\": \"x\", \"x\": [0.0, 0.10101010101010101, 0.20202020202020202, 0.30303030303030304, 0.40404040404040403, 0.5050505050505051, 0.6060606060606061, 0.7070707070707071, 0.8080808080808081, 0.9090909090909091, 1.0101010101010102, 1.1111111111111112, 1.2121212121212122, 1.3131313131313131, 1.4141414141414141, 1.5151515151515151, 1.6161616161616161, 1.7171717171717171, 1.8181818181818181, 1.9191919191919191, 2.0202020202020203, 2.121212121212121, 2.2222222222222223, 2.323232323232323, 2.4242424242424243, 2.525252525252525, 2.6262626262626263, 2.727272727272727, 2.8282828282828283, 2.929292929292929, 3.0303030303030303, 3.131313131313131, 3.2323232323232323, 3.3333333333333335, 3.4343434343434343, 3.5353535353535355, 3.6363636363636362, 3.7373737373737375, 3.8383838383838382, 3.9393939393939394, 4.040404040404041, 4.141414141414141, 4.242424242424242, 4.343434343434343, 4.444444444444445, 4.545454545454545, 4.646464646464646, 4.747474747474747, 4.848484848484849, 4.94949494949495, 5.05050505050505, 5.151515151515151, 5.252525252525253, 5.353535353535354, 5.454545454545454, 5.555555555555555, 5.656565656565657, 5.757575757575758, 5.858585858585858, 5.959595959595959, 6.0606060606060606, 6.161616161616162, 6.262626262626262, 6.363636363636363, 6.4646464646464645, 6.565656565656566, 6.666666666666667, 6.767676767676767, 6.8686868686868685, 6.96969696969697, 7.070707070707071, 7.171717171717171, 7.2727272727272725, 7.373737373737374, 7.474747474747475, 7.575757575757575, 7.6767676767676765, 7.777777777777778, 7.878787878787879, 7.979797979797979, 8.080808080808081, 8.181818181818182, 8.282828282828282, 8.383838383838384, 8.484848484848484, 8.585858585858587, 8.686868686868687, 8.787878787878787, 8.88888888888889, 8.98989898989899, 9.09090909090909, 9.191919191919192, 9.292929292929292, 9.393939393939394, 9.494949494949495, 9.595959595959595, 9.696969696969697, 9.797979797979798, 9.8989898989899, 10.0], \"y\": [1.0, 1.061807053318688, 1.145408152082289, 1.2514936667618204, 1.382402727024627, 1.540324118858284, 1.7272861559723265, 1.945156679798251, 2.196408566606793, 2.4860145424435753, 2.818434840870422, 3.196925406251669, 3.6235416188424057, 4.0991382947884745, 4.62133335158341, 5.181647091781896, 5.761500027114934, 6.308725767841675, 6.735976975851553, 6.920323644032519, 6.696557966250137, 5.968926115336271, 4.855280353296202, 3.675511857566862, 2.6897683308027944, 1.9985375970034487, 1.5540630475511819, 1.2831977856530787, 1.1126772453856806, 1.0145341708284379, 0.9639023899234396, 0.9504882992353505, 0.9683457377881893, 1.0115246860408835, 1.0780506021105003, 1.1666392988318872, 1.277579624153188, 1.4128818689991496, 1.5748503208007834, 1.7659844055942244, 1.9889786880207279, 2.2466348466862254, 2.542602025064851, 2.882071762205804, 3.2685868203425548, 3.7037161547381032, 4.187054913684991, 4.716000282661653, 5.279437416630268, 5.854405801264264, 6.384916942554239, 6.776359124704928, 6.895521853870157, 6.5851736625239665, 5.778074674866982, 4.637025691280423, 3.468928077108298, 2.532936877253989, 1.8947195063462554, 1.4879884800337813, 1.2378613082920311, 1.0842692098782192, 0.9970821693519221, 0.9539252089584267, 0.9473502935079327, 0.9703304411933735, 1.0181169401963994, 1.088854996413232, 1.1814480126133402, 1.296676711092822, 1.4366642215025767, 1.6037493905419014, 1.8004601971920677, 2.029513752716314, 2.2937141888423334, 2.5973244834316405, 2.945291964333588, 3.340776097269539, 3.7850759909325378, 4.277630396987309, 4.814983492492444, 5.384504225476172, 5.959528801018875, 6.477205961046247, 6.835781204824413, 6.894933545936323, 6.498236922908888, 5.609419062813758, 4.435223775619042, 3.289326965524876, 2.4047539852304185, 1.8095307742486502, 1.4347602213277777, 1.2059514801019686, 1.0650265016051608, 0.9870809603748563, 0.9521225427713154, 0.9517880700186899, 0.9791729106894487, 1.0311129102480758], \"type\": \"scatter\", \"uid\": \"7f0a0aa2-d0e5-430a-89ff-ba739f270e7e\"}, {\"name\": \"y\", \"x\": [0.0, 0.10101010101010101, 0.20202020202020202, 0.30303030303030304, 0.40404040404040403, 0.5050505050505051, 0.6060606060606061, 0.7070707070707071, 0.8080808080808081, 0.9090909090909091, 1.0101010101010102, 1.1111111111111112, 1.2121212121212122, 1.3131313131313131, 1.4141414141414141, 1.5151515151515151, 1.6161616161616161, 1.7171717171717171, 1.8181818181818181, 1.9191919191919191, 2.0202020202020203, 2.121212121212121, 2.2222222222222223, 2.323232323232323, 2.4242424242424243, 2.525252525252525, 2.6262626262626263, 2.727272727272727, 2.8282828282828283, 2.929292929292929, 3.0303030303030303, 3.131313131313131, 3.2323232323232323, 3.3333333333333335, 3.4343434343434343, 3.5353535353535355, 3.6363636363636362, 3.7373737373737375, 3.8383838383838382, 3.9393939393939394, 4.040404040404041, 4.141414141414141, 4.242424242424242, 4.343434343434343, 4.444444444444445, 4.545454545454545, 4.646464646464646, 4.747474747474747, 4.848484848484849, 4.94949494949495, 5.05050505050505, 5.151515151515151, 5.252525252525253, 5.353535353535354, 5.454545454545454, 5.555555555555555, 5.656565656565657, 5.757575757575758, 5.858585858585858, 5.959595959595959, 6.0606060606060606, 6.161616161616162, 6.262626262626262, 6.363636363636363, 6.4646464646464645, 6.565656565656566, 6.666666666666667, 6.767676767676767, 6.8686868686868685, 6.96969696969697, 7.070707070707071, 7.171717171717171, 7.2727272727272725, 7.373737373737374, 7.474747474747475, 7.575757575757575, 7.6767676767676765, 7.777777777777778, 7.878787878787879, 7.979797979797979, 8.080808080808081, 8.181818181818182, 8.282828282828282, 8.383838383838384, 8.484848484848484, 8.585858585858587, 8.686868686868687, 8.787878787878787, 8.88888888888889, 8.98989898989899, 9.09090909090909, 9.191919191919192, 9.292929292929292, 9.393939393939394, 9.494949494949495, 9.595959595959595, 9.696969696969697, 9.797979797979798, 9.8989898989899, 10.0], \"y\": [1.0, 0.8194779946594071, 0.6769761068104436, 0.5649942997342504, 0.47649230332538395, 0.4063822874493165, 0.3515376038120329, 0.3107927859600305, 0.2830456388909482, 0.26500731809697947, 0.2556885483605984, 0.2554480924491077, 0.2659701062039867, 0.29026413854089095, 0.333487934162219, 0.40470752040644703, 0.5188000623698279, 0.7052680287423514, 1.0088452457308028, 1.488378592599668, 2.196984089990479, 3.0917560303845817, 3.9551792167335784, 4.492259504662691, 4.570189585959295, 4.265913874376103, 3.7577366748706247, 3.1908471694544316, 2.6576537569553196, 2.1843033299205263, 1.7841644186582941, 1.4532382343208106, 1.1824166118569455, 0.9647924837307708, 0.7913278118358653, 0.6543607261963258, 0.5470804649708603, 0.4630933171316783, 0.39736906789565885, 0.3463580652423448, 0.30799121991394485, 0.2815076233281379, 0.26520721954168175, 0.25762888314334614, 0.25905779366122234, 0.27148903866740604, 0.29862761377799757, 0.34592102856899964, 0.42284297012467786, 0.548314959703641, 0.7541871348193039, 1.0857938103504312, 1.6019090153540527, 2.347087887132778, 3.2500616182413165, 4.066311516539782, 4.53485638244743, 4.535402020067465, 4.178273697238685, 3.6502232312883685, 3.086345632526418, 2.56134912688668, 2.100637685126049, 1.7143039329680214, 1.394953029710587, 1.134909434534885, 0.9263527175910217, 0.7604673121084548, 0.6297908472581991, 0.5274932354269619, 0.44738322232337935, 0.3847210633005533, 0.33625997671333385, 0.3002461439183234, 0.27591930926278896, 0.261238077301631, 0.2551448601416579, 0.2582646477794929, 0.2728773160832124, 0.3029176267923462, 0.3542301539987484, 0.4377772742611726, 0.5738862954973636, 0.7967805117018197, 1.1550869059767341, 1.710596567457325, 2.4961538912971166, 3.412049481736945, 4.189897393925115, 4.580476109729426, 4.504155699517714, 4.104124966508637, 3.5594888299612975, 2.9964157448589375, 2.4805740244946155, 2.031274217603796, 1.6555569590152095, 1.3464560022040521, 1.09618095556177, 0.8956363459306874], \"type\": \"scatter\", \"uid\": \"d1e4cc27-c866-4afb-981c-b55dfc1f4d63\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"ec454abe-78cd-4964-875b-d9891b73199a\"));});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "linkText": "Export to plot.ly",
        "plotlyServerURL": "https://plot.ly",
        "showLink": false
       },
       "data": [
        {
         "type": "scatter",
         "uid": "0654c1cc-fb09-4130-85fd-d7ca0d52cb47",
         "x": [
          1,
          1.061807053318688,
          1.145408152082289,
          1.2514936667618204,
          1.382402727024627,
          1.540324118858284,
          1.7272861559723265,
          1.945156679798251,
          2.196408566606793,
          2.4860145424435753,
          2.818434840870422,
          3.196925406251669,
          3.6235416188424057,
          4.0991382947884745,
          4.62133335158341,
          5.181647091781896,
          5.761500027114934,
          6.308725767841675,
          6.735976975851553,
          6.920323644032519,
          6.696557966250137,
          5.968926115336271,
          4.855280353296202,
          3.675511857566862,
          2.6897683308027944,
          1.9985375970034487,
          1.5540630475511819,
          1.2831977856530787,
          1.1126772453856806,
          1.0145341708284379,
          0.9639023899234396,
          0.9504882992353505,
          0.9683457377881893,
          1.0115246860408835,
          1.0780506021105003,
          1.1666392988318872,
          1.277579624153188,
          1.4128818689991496,
          1.5748503208007834,
          1.7659844055942244,
          1.9889786880207279,
          2.2466348466862254,
          2.542602025064851,
          2.882071762205804,
          3.2685868203425548,
          3.7037161547381032,
          4.187054913684991,
          4.716000282661653,
          5.279437416630268,
          5.854405801264264,
          6.384916942554239,
          6.776359124704928,
          6.895521853870157,
          6.5851736625239665,
          5.778074674866982,
          4.637025691280423,
          3.468928077108298,
          2.532936877253989,
          1.8947195063462554,
          1.4879884800337813,
          1.2378613082920311,
          1.0842692098782192,
          0.9970821693519221,
          0.9539252089584267,
          0.9473502935079327,
          0.9703304411933735,
          1.0181169401963994,
          1.088854996413232,
          1.1814480126133402,
          1.296676711092822,
          1.4366642215025767,
          1.6037493905419014,
          1.8004601971920677,
          2.029513752716314,
          2.2937141888423334,
          2.5973244834316405,
          2.945291964333588,
          3.340776097269539,
          3.7850759909325378,
          4.277630396987309,
          4.814983492492444,
          5.384504225476172,
          5.959528801018875,
          6.477205961046247,
          6.835781204824413,
          6.894933545936323,
          6.498236922908888,
          5.609419062813758,
          4.435223775619042,
          3.289326965524876,
          2.4047539852304185,
          1.8095307742486502,
          1.4347602213277777,
          1.2059514801019686,
          1.0650265016051608,
          0.9870809603748563,
          0.9521225427713154,
          0.9517880700186899,
          0.9791729106894487,
          1.0311129102480758
         ],
         "y": [
          1,
          0.8194779946594071,
          0.6769761068104436,
          0.5649942997342504,
          0.47649230332538395,
          0.4063822874493165,
          0.3515376038120329,
          0.3107927859600305,
          0.2830456388909482,
          0.26500731809697947,
          0.2556885483605984,
          0.2554480924491077,
          0.2659701062039867,
          0.29026413854089095,
          0.333487934162219,
          0.40470752040644703,
          0.5188000623698279,
          0.7052680287423514,
          1.0088452457308028,
          1.488378592599668,
          2.196984089990479,
          3.0917560303845817,
          3.9551792167335784,
          4.492259504662691,
          4.570189585959295,
          4.265913874376103,
          3.7577366748706247,
          3.1908471694544316,
          2.6576537569553196,
          2.1843033299205263,
          1.7841644186582941,
          1.4532382343208106,
          1.1824166118569455,
          0.9647924837307708,
          0.7913278118358653,
          0.6543607261963258,
          0.5470804649708603,
          0.4630933171316783,
          0.39736906789565885,
          0.3463580652423448,
          0.30799121991394485,
          0.2815076233281379,
          0.26520721954168175,
          0.25762888314334614,
          0.25905779366122234,
          0.27148903866740604,
          0.29862761377799757,
          0.34592102856899964,
          0.42284297012467786,
          0.548314959703641,
          0.7541871348193039,
          1.0857938103504312,
          1.6019090153540527,
          2.347087887132778,
          3.2500616182413165,
          4.066311516539782,
          4.53485638244743,
          4.535402020067465,
          4.178273697238685,
          3.6502232312883685,
          3.086345632526418,
          2.56134912688668,
          2.100637685126049,
          1.7143039329680214,
          1.394953029710587,
          1.134909434534885,
          0.9263527175910217,
          0.7604673121084548,
          0.6297908472581991,
          0.5274932354269619,
          0.44738322232337935,
          0.3847210633005533,
          0.33625997671333385,
          0.3002461439183234,
          0.27591930926278896,
          0.261238077301631,
          0.2551448601416579,
          0.2582646477794929,
          0.2728773160832124,
          0.3029176267923462,
          0.3542301539987484,
          0.4377772742611726,
          0.5738862954973636,
          0.7967805117018197,
          1.1550869059767341,
          1.710596567457325,
          2.4961538912971166,
          3.412049481736945,
          4.189897393925115,
          4.580476109729426,
          4.504155699517714,
          4.104124966508637,
          3.5594888299612975,
          2.9964157448589375,
          2.4805740244946155,
          2.031274217603796,
          1.6555569590152095,
          1.3464560022040521,
          1.09618095556177,
          0.8956363459306874
         ]
        }
       ],
       "layout": {}
      },
      "text/html": [
       "<div id=\"20a4f452-2d23-4cdd-8cbd-16abe9ce435d\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"20a4f452-2d23-4cdd-8cbd-16abe9ce435d\", [{\"x\": [1.0, 1.061807053318688, 1.145408152082289, 1.2514936667618204, 1.382402727024627, 1.540324118858284, 1.7272861559723265, 1.945156679798251, 2.196408566606793, 2.4860145424435753, 2.818434840870422, 3.196925406251669, 3.6235416188424057, 4.0991382947884745, 4.62133335158341, 5.181647091781896, 5.761500027114934, 6.308725767841675, 6.735976975851553, 6.920323644032519, 6.696557966250137, 5.968926115336271, 4.855280353296202, 3.675511857566862, 2.6897683308027944, 1.9985375970034487, 1.5540630475511819, 1.2831977856530787, 1.1126772453856806, 1.0145341708284379, 0.9639023899234396, 0.9504882992353505, 0.9683457377881893, 1.0115246860408835, 1.0780506021105003, 1.1666392988318872, 1.277579624153188, 1.4128818689991496, 1.5748503208007834, 1.7659844055942244, 1.9889786880207279, 2.2466348466862254, 2.542602025064851, 2.882071762205804, 3.2685868203425548, 3.7037161547381032, 4.187054913684991, 4.716000282661653, 5.279437416630268, 5.854405801264264, 6.384916942554239, 6.776359124704928, 6.895521853870157, 6.5851736625239665, 5.778074674866982, 4.637025691280423, 3.468928077108298, 2.532936877253989, 1.8947195063462554, 1.4879884800337813, 1.2378613082920311, 1.0842692098782192, 0.9970821693519221, 0.9539252089584267, 0.9473502935079327, 0.9703304411933735, 1.0181169401963994, 1.088854996413232, 1.1814480126133402, 1.296676711092822, 1.4366642215025767, 1.6037493905419014, 1.8004601971920677, 2.029513752716314, 2.2937141888423334, 2.5973244834316405, 2.945291964333588, 3.340776097269539, 3.7850759909325378, 4.277630396987309, 4.814983492492444, 5.384504225476172, 5.959528801018875, 6.477205961046247, 6.835781204824413, 6.894933545936323, 6.498236922908888, 5.609419062813758, 4.435223775619042, 3.289326965524876, 2.4047539852304185, 1.8095307742486502, 1.4347602213277777, 1.2059514801019686, 1.0650265016051608, 0.9870809603748563, 0.9521225427713154, 0.9517880700186899, 0.9791729106894487, 1.0311129102480758], \"y\": [1.0, 0.8194779946594071, 0.6769761068104436, 0.5649942997342504, 0.47649230332538395, 0.4063822874493165, 0.3515376038120329, 0.3107927859600305, 0.2830456388909482, 0.26500731809697947, 0.2556885483605984, 0.2554480924491077, 0.2659701062039867, 0.29026413854089095, 0.333487934162219, 0.40470752040644703, 0.5188000623698279, 0.7052680287423514, 1.0088452457308028, 1.488378592599668, 2.196984089990479, 3.0917560303845817, 3.9551792167335784, 4.492259504662691, 4.570189585959295, 4.265913874376103, 3.7577366748706247, 3.1908471694544316, 2.6576537569553196, 2.1843033299205263, 1.7841644186582941, 1.4532382343208106, 1.1824166118569455, 0.9647924837307708, 0.7913278118358653, 0.6543607261963258, 0.5470804649708603, 0.4630933171316783, 0.39736906789565885, 0.3463580652423448, 0.30799121991394485, 0.2815076233281379, 0.26520721954168175, 0.25762888314334614, 0.25905779366122234, 0.27148903866740604, 0.29862761377799757, 0.34592102856899964, 0.42284297012467786, 0.548314959703641, 0.7541871348193039, 1.0857938103504312, 1.6019090153540527, 2.347087887132778, 3.2500616182413165, 4.066311516539782, 4.53485638244743, 4.535402020067465, 4.178273697238685, 3.6502232312883685, 3.086345632526418, 2.56134912688668, 2.100637685126049, 1.7143039329680214, 1.394953029710587, 1.134909434534885, 0.9263527175910217, 0.7604673121084548, 0.6297908472581991, 0.5274932354269619, 0.44738322232337935, 0.3847210633005533, 0.33625997671333385, 0.3002461439183234, 0.27591930926278896, 0.261238077301631, 0.2551448601416579, 0.2582646477794929, 0.2728773160832124, 0.3029176267923462, 0.3542301539987484, 0.4377772742611726, 0.5738862954973636, 0.7967805117018197, 1.1550869059767341, 1.710596567457325, 2.4961538912971166, 3.412049481736945, 4.189897393925115, 4.580476109729426, 4.504155699517714, 4.104124966508637, 3.5594888299612975, 2.9964157448589375, 2.4805740244946155, 2.031274217603796, 1.6555569590152095, 1.3464560022040521, 1.09618095556177, 0.8956363459306874], \"type\": \"scatter\", \"uid\": \"fc8123b5-183c-40f0-b626-f4acb3e75d06\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"20a4f452-2d23-4cdd-8cbd-16abe9ce435d\"));});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"20a4f452-2d23-4cdd-8cbd-16abe9ce435d\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"20a4f452-2d23-4cdd-8cbd-16abe9ce435d\", [{\"x\": [1.0, 1.061807053318688, 1.145408152082289, 1.2514936667618204, 1.382402727024627, 1.540324118858284, 1.7272861559723265, 1.945156679798251, 2.196408566606793, 2.4860145424435753, 2.818434840870422, 3.196925406251669, 3.6235416188424057, 4.0991382947884745, 4.62133335158341, 5.181647091781896, 5.761500027114934, 6.308725767841675, 6.735976975851553, 6.920323644032519, 6.696557966250137, 5.968926115336271, 4.855280353296202, 3.675511857566862, 2.6897683308027944, 1.9985375970034487, 1.5540630475511819, 1.2831977856530787, 1.1126772453856806, 1.0145341708284379, 0.9639023899234396, 0.9504882992353505, 0.9683457377881893, 1.0115246860408835, 1.0780506021105003, 1.1666392988318872, 1.277579624153188, 1.4128818689991496, 1.5748503208007834, 1.7659844055942244, 1.9889786880207279, 2.2466348466862254, 2.542602025064851, 2.882071762205804, 3.2685868203425548, 3.7037161547381032, 4.187054913684991, 4.716000282661653, 5.279437416630268, 5.854405801264264, 6.384916942554239, 6.776359124704928, 6.895521853870157, 6.5851736625239665, 5.778074674866982, 4.637025691280423, 3.468928077108298, 2.532936877253989, 1.8947195063462554, 1.4879884800337813, 1.2378613082920311, 1.0842692098782192, 0.9970821693519221, 0.9539252089584267, 0.9473502935079327, 0.9703304411933735, 1.0181169401963994, 1.088854996413232, 1.1814480126133402, 1.296676711092822, 1.4366642215025767, 1.6037493905419014, 1.8004601971920677, 2.029513752716314, 2.2937141888423334, 2.5973244834316405, 2.945291964333588, 3.340776097269539, 3.7850759909325378, 4.277630396987309, 4.814983492492444, 5.384504225476172, 5.959528801018875, 6.477205961046247, 6.835781204824413, 6.894933545936323, 6.498236922908888, 5.609419062813758, 4.435223775619042, 3.289326965524876, 2.4047539852304185, 1.8095307742486502, 1.4347602213277777, 1.2059514801019686, 1.0650265016051608, 0.9870809603748563, 0.9521225427713154, 0.9517880700186899, 0.9791729106894487, 1.0311129102480758], \"y\": [1.0, 0.8194779946594071, 0.6769761068104436, 0.5649942997342504, 0.47649230332538395, 0.4063822874493165, 0.3515376038120329, 0.3107927859600305, 0.2830456388909482, 0.26500731809697947, 0.2556885483605984, 0.2554480924491077, 0.2659701062039867, 0.29026413854089095, 0.333487934162219, 0.40470752040644703, 0.5188000623698279, 0.7052680287423514, 1.0088452457308028, 1.488378592599668, 2.196984089990479, 3.0917560303845817, 3.9551792167335784, 4.492259504662691, 4.570189585959295, 4.265913874376103, 3.7577366748706247, 3.1908471694544316, 2.6576537569553196, 2.1843033299205263, 1.7841644186582941, 1.4532382343208106, 1.1824166118569455, 0.9647924837307708, 0.7913278118358653, 0.6543607261963258, 0.5470804649708603, 0.4630933171316783, 0.39736906789565885, 0.3463580652423448, 0.30799121991394485, 0.2815076233281379, 0.26520721954168175, 0.25762888314334614, 0.25905779366122234, 0.27148903866740604, 0.29862761377799757, 0.34592102856899964, 0.42284297012467786, 0.548314959703641, 0.7541871348193039, 1.0857938103504312, 1.6019090153540527, 2.347087887132778, 3.2500616182413165, 4.066311516539782, 4.53485638244743, 4.535402020067465, 4.178273697238685, 3.6502232312883685, 3.086345632526418, 2.56134912688668, 2.100637685126049, 1.7143039329680214, 1.394953029710587, 1.134909434534885, 0.9263527175910217, 0.7604673121084548, 0.6297908472581991, 0.5274932354269619, 0.44738322232337935, 0.3847210633005533, 0.33625997671333385, 0.3002461439183234, 0.27591930926278896, 0.261238077301631, 0.2551448601416579, 0.2582646477794929, 0.2728773160832124, 0.3029176267923462, 0.3542301539987484, 0.4377772742611726, 0.5738862954973636, 0.7967805117018197, 1.1550869059767341, 1.710596567457325, 2.4961538912971166, 3.412049481736945, 4.189897393925115, 4.580476109729426, 4.504155699517714, 4.104124966508637, 3.5594888299612975, 2.9964157448589375, 2.4805740244946155, 2.031274217603796, 1.6555569590152095, 1.3464560022040521, 1.09618095556177, 0.8956363459306874], \"type\": \"scatter\", \"uid\": \"fc8123b5-183c-40f0-b626-f4acb3e75d06\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"20a4f452-2d23-4cdd-8cbd-16abe9ce435d\"));});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sol = ode_tf(lotka_volterra, z0, ts, args)\n",
    "xs = sol[:, 0].numpy()\n",
    "ys = sol[:, 1].numpy()\n",
    "\n",
    "iplot([go.Scatter(x=ts.numpy(), y=xs, name=\"x\"), go.Scatter(x=ts.numpy(), y=ys, name=\"y\")])\n",
    "\n",
    "iplot([go.Scatter(x=xs, y=ys)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Scipy ODE Solver"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "linkText": "Export to plot.ly",
        "plotlyServerURL": "https://plot.ly",
        "showLink": false
       },
       "data": [
        {
         "name": "x",
         "type": "scatter",
         "uid": "fa1b5cda-7c61-47d1-b460-26e79af62dfa",
         "x": [
          0,
          0.10101010101010101,
          0.20202020202020202,
          0.30303030303030304,
          0.40404040404040403,
          0.5050505050505051,
          0.6060606060606061,
          0.7070707070707071,
          0.8080808080808081,
          0.9090909090909091,
          1.0101010101010102,
          1.1111111111111112,
          1.2121212121212122,
          1.3131313131313131,
          1.4141414141414141,
          1.5151515151515151,
          1.6161616161616161,
          1.7171717171717171,
          1.8181818181818181,
          1.9191919191919191,
          2.0202020202020203,
          2.121212121212121,
          2.2222222222222223,
          2.323232323232323,
          2.4242424242424243,
          2.525252525252525,
          2.6262626262626263,
          2.727272727272727,
          2.8282828282828283,
          2.929292929292929,
          3.0303030303030303,
          3.131313131313131,
          3.2323232323232323,
          3.3333333333333335,
          3.4343434343434343,
          3.5353535353535355,
          3.6363636363636362,
          3.7373737373737375,
          3.8383838383838382,
          3.9393939393939394,
          4.040404040404041,
          4.141414141414141,
          4.242424242424242,
          4.343434343434343,
          4.444444444444445,
          4.545454545454545,
          4.646464646464646,
          4.747474747474747,
          4.848484848484849,
          4.94949494949495,
          5.05050505050505,
          5.151515151515151,
          5.252525252525253,
          5.353535353535354,
          5.454545454545454,
          5.555555555555555,
          5.656565656565657,
          5.757575757575758,
          5.858585858585858,
          5.959595959595959,
          6.0606060606060606,
          6.161616161616162,
          6.262626262626262,
          6.363636363636363,
          6.4646464646464645,
          6.565656565656566,
          6.666666666666667,
          6.767676767676767,
          6.8686868686868685,
          6.96969696969697,
          7.070707070707071,
          7.171717171717171,
          7.2727272727272725,
          7.373737373737374,
          7.474747474747475,
          7.575757575757575,
          7.6767676767676765,
          7.777777777777778,
          7.878787878787879,
          7.979797979797979,
          8.080808080808081,
          8.181818181818182,
          8.282828282828282,
          8.383838383838384,
          8.484848484848484,
          8.585858585858587,
          8.686868686868687,
          8.787878787878787,
          8.88888888888889,
          8.98989898989899,
          9.09090909090909,
          9.191919191919192,
          9.292929292929292,
          9.393939393939394,
          9.494949494949495,
          9.595959595959595,
          9.696969696969697,
          9.797979797979798,
          9.8989898989899,
          10
         ],
         "y": [
          1,
          1.0618070307440879,
          1.145929472156516,
          1.2527145963594501,
          1.3833490466432634,
          1.5396941281147056,
          1.724183472649697,
          1.939748377151537,
          2.1897428861562735,
          2.4778380798749042,
          2.8078443136636517,
          3.1833949608806322,
          3.607377400523907,
          4.080909231590283,
          4.601493623685861,
          5.159707458965986,
          5.733352730028728,
          6.277667864218449,
          6.711268256890407,
          6.904440011008827,
          6.696753452844413,
          5.990466584023633,
          4.8939220593127315,
          3.7148466638106155,
          2.729784663546461,
          2.0285430515831777,
          1.570359150497045,
          1.2832935384326256,
          1.108978575142571,
          1.0090137863093822,
          0.9600300753674814,
          0.9482538396282152,
          0.9657635486283048,
          1.0082214348283722,
          1.0735540255610745,
          1.161191486553061,
          1.2716277786722856,
          1.4061643998446829,
          1.566758349065195,
          1.7559261065098757,
          1.97667078782143,
          2.232404516602338,
          2.526834648288896,
          2.8637694588411318,
          3.24677132330882,
          3.678532912045141,
          4.15975391112831,
          4.6871180033650175,
          5.249665253763894,
          5.822418285193855,
          6.355871516354743,
          6.761535075926115,
          6.902317317568296,
          6.619041840013945,
          5.837627979344146,
          4.7052557342537265,
          3.5425772356645586,
          2.600872715355647,
          1.9422658460972155,
          1.5156417390917318,
          1.2496262471975523,
          1.0890844960110397,
          0.9983794051927775,
          0.9559714644362712,
          0.9492107891192526,
          0.9708593556727112,
          1.017004696881,
          1.0858497126545796,
          1.1770149810027473,
          1.2911349743510399,
          1.4296224031318614,
          1.5945276272544242,
          1.7884482394273178,
          2.014457298600624,
          2.2760223046651262,
          2.576882205763647,
          2.920834842100241,
          3.3113566908414485,
          3.7509186483749555,
          4.239753932401369,
          4.773639359021834,
          5.339923742320558,
          5.910592029305686,
          6.431024175751646,
          6.80532700591946,
          6.889567231625224,
          6.528521146565962,
          5.675691614931862,
          4.516033832849433,
          3.3762118380970882,
          2.4791768096856948,
          1.8617413193330443,
          1.4648511457410303,
          1.218523826500176,
          1.070887680453135,
          0.9889208660549728,
          0.9527830406322477,
          0.9508631564864141,
          0.9765562488887533,
          1.026345986742193
         ]
        },
        {
         "name": "y",
         "type": "scatter",
         "uid": "ee08ba24-d77c-42a3-910a-8b65c7b99a93",
         "x": [
          0,
          0.10101010101010101,
          0.20202020202020202,
          0.30303030303030304,
          0.40404040404040403,
          0.5050505050505051,
          0.6060606060606061,
          0.7070707070707071,
          0.8080808080808081,
          0.9090909090909091,
          1.0101010101010102,
          1.1111111111111112,
          1.2121212121212122,
          1.3131313131313131,
          1.4141414141414141,
          1.5151515151515151,
          1.6161616161616161,
          1.7171717171717171,
          1.8181818181818181,
          1.9191919191919191,
          2.0202020202020203,
          2.121212121212121,
          2.2222222222222223,
          2.323232323232323,
          2.4242424242424243,
          2.525252525252525,
          2.6262626262626263,
          2.727272727272727,
          2.8282828282828283,
          2.929292929292929,
          3.0303030303030303,
          3.131313131313131,
          3.2323232323232323,
          3.3333333333333335,
          3.4343434343434343,
          3.5353535353535355,
          3.6363636363636362,
          3.7373737373737375,
          3.8383838383838382,
          3.9393939393939394,
          4.040404040404041,
          4.141414141414141,
          4.242424242424242,
          4.343434343434343,
          4.444444444444445,
          4.545454545454545,
          4.646464646464646,
          4.747474747474747,
          4.848484848484849,
          4.94949494949495,
          5.05050505050505,
          5.151515151515151,
          5.252525252525253,
          5.353535353535354,
          5.454545454545454,
          5.555555555555555,
          5.656565656565657,
          5.757575757575758,
          5.858585858585858,
          5.959595959595959,
          6.0606060606060606,
          6.161616161616162,
          6.262626262626262,
          6.363636363636363,
          6.4646464646464645,
          6.565656565656566,
          6.666666666666667,
          6.767676767676767,
          6.8686868686868685,
          6.96969696969697,
          7.070707070707071,
          7.171717171717171,
          7.2727272727272725,
          7.373737373737374,
          7.474747474747475,
          7.575757575757575,
          7.6767676767676765,
          7.777777777777778,
          7.878787878787879,
          7.979797979797979,
          8.080808080808081,
          8.181818181818182,
          8.282828282828282,
          8.383838383838384,
          8.484848484848484,
          8.585858585858587,
          8.686868686868687,
          8.787878787878787,
          8.88888888888889,
          8.98989898989899,
          9.09090909090909,
          9.191919191919192,
          9.292929292929292,
          9.393939393939394,
          9.494949494949495,
          9.595959595959595,
          9.696969696969697,
          9.797979797979798,
          9.8989898989899,
          10
         ],
         "y": [
          1,
          0.8194780403487356,
          0.6765127495203546,
          0.56389609625139,
          0.47569009444784355,
          0.4071325800587527,
          0.3544978931327249,
          0.314959533062111,
          0.28648013542973505,
          0.26774597865042593,
          0.25816360856341236,
          0.2579469322984345,
          0.2683456371868822,
          0.2921095397315125,
          0.334365038476902,
          0.40421382585143045,
          0.5175376495783515,
          0.7014782538147918,
          0.9998115904859729,
          1.472756585332035,
          2.1706622911770497,
          3.0554113343515357,
          3.918697776219176,
          4.467360890577506,
          4.558310218292185,
          4.271279240131057,
          3.7769131206229396,
          3.2182521631044367,
          2.680143728991083,
          2.2018555967821105,
          1.7956319505706138,
          1.4599931723519706,
          1.1874938082423045,
          0.9688092307014377,
          0.7947195968702216,
          0.6569688097587096,
          0.5485506507019844,
          0.4637201203050939,
          0.3978875266488159,
          0.34747716515413507,
          0.3097929131106958,
          0.2829141618797309,
          0.2656386861700612,
          0.25749131489744953,
          0.2588290499185519,
          0.2710986463789994,
          0.297351479415145,
          0.3432085961054161,
          0.41861286580473106,
          0.5408741465495848,
          0.7394100482842035,
          1.0609248631027752,
          1.5669357286059675,
          2.3004949471113316,
          3.200132383150966,
          4.032281636926246,
          4.512143034717381,
          4.534110127315921,
          4.202993815954185,
          3.6905273622907466,
          3.130873733257954,
          2.600456891443095,
          2.1331597386259635,
          1.7383489995018377,
          1.4132098749796922,
          1.149799832236806,
          0.9387169289160194,
          0.770855364134301,
          0.6381461518623532,
          0.5337849090322841,
          0.4522168437293153,
          0.3890211974881503,
          0.3407688527285771,
          0.30489146466507755,
          0.2795837015467116,
          0.26375502463334416,
          0.25705104616219393,
          0.2599779816361807,
          0.2741920513039548,
          0.30306934424872284,
          0.3527678263747656,
          0.4341456815433788,
          0.5660592029906896,
          0.7803431610075672,
          1.1266160072758133,
          1.666942453409209,
          2.434839845971714,
          3.3433386865995858,
          4.136722414776079,
          4.545135706853778,
          4.501075720158733,
          4.130329191092171,
          3.603099553142873,
          3.0443025150264105,
          2.5223765874635244,
          2.066277455199996,
          1.682796538125196,
          1.3679539360912996,
          1.1133977095213605,
          0.9096901889173676
         ]
        }
       ],
       "layout": {}
      },
      "text/html": [
       "<div id=\"3946221b-f857-4e5f-9d56-c92797f1ce39\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"3946221b-f857-4e5f-9d56-c92797f1ce39\", [{\"name\": \"x\", \"x\": [0.0, 0.10101010101010101, 0.20202020202020202, 0.30303030303030304, 0.40404040404040403, 0.5050505050505051, 0.6060606060606061, 0.7070707070707071, 0.8080808080808081, 0.9090909090909091, 1.0101010101010102, 1.1111111111111112, 1.2121212121212122, 1.3131313131313131, 1.4141414141414141, 1.5151515151515151, 1.6161616161616161, 1.7171717171717171, 1.8181818181818181, 1.9191919191919191, 2.0202020202020203, 2.121212121212121, 2.2222222222222223, 2.323232323232323, 2.4242424242424243, 2.525252525252525, 2.6262626262626263, 2.727272727272727, 2.8282828282828283, 2.929292929292929, 3.0303030303030303, 3.131313131313131, 3.2323232323232323, 3.3333333333333335, 3.4343434343434343, 3.5353535353535355, 3.6363636363636362, 3.7373737373737375, 3.8383838383838382, 3.9393939393939394, 4.040404040404041, 4.141414141414141, 4.242424242424242, 4.343434343434343, 4.444444444444445, 4.545454545454545, 4.646464646464646, 4.747474747474747, 4.848484848484849, 4.94949494949495, 5.05050505050505, 5.151515151515151, 5.252525252525253, 5.353535353535354, 5.454545454545454, 5.555555555555555, 5.656565656565657, 5.757575757575758, 5.858585858585858, 5.959595959595959, 6.0606060606060606, 6.161616161616162, 6.262626262626262, 6.363636363636363, 6.4646464646464645, 6.565656565656566, 6.666666666666667, 6.767676767676767, 6.8686868686868685, 6.96969696969697, 7.070707070707071, 7.171717171717171, 7.2727272727272725, 7.373737373737374, 7.474747474747475, 7.575757575757575, 7.6767676767676765, 7.777777777777778, 7.878787878787879, 7.979797979797979, 8.080808080808081, 8.181818181818182, 8.282828282828282, 8.383838383838384, 8.484848484848484, 8.585858585858587, 8.686868686868687, 8.787878787878787, 8.88888888888889, 8.98989898989899, 9.09090909090909, 9.191919191919192, 9.292929292929292, 9.393939393939394, 9.494949494949495, 9.595959595959595, 9.696969696969697, 9.797979797979798, 9.8989898989899, 10.0], \"y\": [1.0, 1.0618070307440879, 1.145929472156516, 1.2527145963594501, 1.3833490466432634, 1.5396941281147056, 1.724183472649697, 1.939748377151537, 2.1897428861562735, 2.4778380798749042, 2.8078443136636517, 3.1833949608806322, 3.607377400523907, 4.080909231590283, 4.601493623685861, 5.159707458965986, 5.733352730028728, 6.277667864218449, 6.711268256890407, 6.904440011008827, 6.696753452844413, 5.990466584023633, 4.8939220593127315, 3.7148466638106155, 2.729784663546461, 2.0285430515831777, 1.570359150497045, 1.2832935384326256, 1.108978575142571, 1.0090137863093822, 0.9600300753674814, 0.9482538396282152, 0.9657635486283048, 1.0082214348283722, 1.0735540255610745, 1.161191486553061, 1.2716277786722856, 1.4061643998446829, 1.566758349065195, 1.7559261065098757, 1.97667078782143, 2.232404516602338, 2.526834648288896, 2.8637694588411318, 3.24677132330882, 3.678532912045141, 4.15975391112831, 4.6871180033650175, 5.249665253763894, 5.822418285193855, 6.355871516354743, 6.761535075926115, 6.902317317568296, 6.619041840013945, 5.837627979344146, 4.7052557342537265, 3.5425772356645586, 2.600872715355647, 1.9422658460972155, 1.5156417390917318, 1.2496262471975523, 1.0890844960110397, 0.9983794051927775, 0.9559714644362712, 0.9492107891192526, 0.9708593556727112, 1.017004696881, 1.0858497126545796, 1.1770149810027473, 1.2911349743510399, 1.4296224031318614, 1.5945276272544242, 1.7884482394273178, 2.014457298600624, 2.2760223046651262, 2.576882205763647, 2.920834842100241, 3.3113566908414485, 3.7509186483749555, 4.239753932401369, 4.773639359021834, 5.339923742320558, 5.910592029305686, 6.431024175751646, 6.80532700591946, 6.889567231625224, 6.528521146565962, 5.675691614931862, 4.516033832849433, 3.3762118380970882, 2.4791768096856948, 1.8617413193330443, 1.4648511457410303, 1.218523826500176, 1.070887680453135, 0.9889208660549728, 0.9527830406322477, 0.9508631564864141, 0.9765562488887533, 1.026345986742193], \"type\": \"scatter\", \"uid\": \"23cf4e7f-1893-4e74-b6c9-46bdfe59044a\"}, {\"name\": \"y\", \"x\": [0.0, 0.10101010101010101, 0.20202020202020202, 0.30303030303030304, 0.40404040404040403, 0.5050505050505051, 0.6060606060606061, 0.7070707070707071, 0.8080808080808081, 0.9090909090909091, 1.0101010101010102, 1.1111111111111112, 1.2121212121212122, 1.3131313131313131, 1.4141414141414141, 1.5151515151515151, 1.6161616161616161, 1.7171717171717171, 1.8181818181818181, 1.9191919191919191, 2.0202020202020203, 2.121212121212121, 2.2222222222222223, 2.323232323232323, 2.4242424242424243, 2.525252525252525, 2.6262626262626263, 2.727272727272727, 2.8282828282828283, 2.929292929292929, 3.0303030303030303, 3.131313131313131, 3.2323232323232323, 3.3333333333333335, 3.4343434343434343, 3.5353535353535355, 3.6363636363636362, 3.7373737373737375, 3.8383838383838382, 3.9393939393939394, 4.040404040404041, 4.141414141414141, 4.242424242424242, 4.343434343434343, 4.444444444444445, 4.545454545454545, 4.646464646464646, 4.747474747474747, 4.848484848484849, 4.94949494949495, 5.05050505050505, 5.151515151515151, 5.252525252525253, 5.353535353535354, 5.454545454545454, 5.555555555555555, 5.656565656565657, 5.757575757575758, 5.858585858585858, 5.959595959595959, 6.0606060606060606, 6.161616161616162, 6.262626262626262, 6.363636363636363, 6.4646464646464645, 6.565656565656566, 6.666666666666667, 6.767676767676767, 6.8686868686868685, 6.96969696969697, 7.070707070707071, 7.171717171717171, 7.2727272727272725, 7.373737373737374, 7.474747474747475, 7.575757575757575, 7.6767676767676765, 7.777777777777778, 7.878787878787879, 7.979797979797979, 8.080808080808081, 8.181818181818182, 8.282828282828282, 8.383838383838384, 8.484848484848484, 8.585858585858587, 8.686868686868687, 8.787878787878787, 8.88888888888889, 8.98989898989899, 9.09090909090909, 9.191919191919192, 9.292929292929292, 9.393939393939394, 9.494949494949495, 9.595959595959595, 9.696969696969697, 9.797979797979798, 9.8989898989899, 10.0], \"y\": [1.0, 0.8194780403487356, 0.6765127495203546, 0.56389609625139, 0.47569009444784355, 0.4071325800587527, 0.3544978931327249, 0.314959533062111, 0.28648013542973505, 0.26774597865042593, 0.25816360856341236, 0.2579469322984345, 0.2683456371868822, 0.2921095397315125, 0.334365038476902, 0.40421382585143045, 0.5175376495783515, 0.7014782538147918, 0.9998115904859729, 1.472756585332035, 2.1706622911770497, 3.0554113343515357, 3.918697776219176, 4.467360890577506, 4.558310218292185, 4.271279240131057, 3.7769131206229396, 3.2182521631044367, 2.680143728991083, 2.2018555967821105, 1.7956319505706138, 1.4599931723519706, 1.1874938082423045, 0.9688092307014377, 0.7947195968702216, 0.6569688097587096, 0.5485506507019844, 0.4637201203050939, 0.3978875266488159, 0.34747716515413507, 0.3097929131106958, 0.2829141618797309, 0.2656386861700612, 0.25749131489744953, 0.2588290499185519, 0.2710986463789994, 0.297351479415145, 0.3432085961054161, 0.41861286580473106, 0.5408741465495848, 0.7394100482842035, 1.0609248631027752, 1.5669357286059675, 2.3004949471113316, 3.200132383150966, 4.032281636926246, 4.512143034717381, 4.534110127315921, 4.202993815954185, 3.6905273622907466, 3.130873733257954, 2.600456891443095, 2.1331597386259635, 1.7383489995018377, 1.4132098749796922, 1.149799832236806, 0.9387169289160194, 0.770855364134301, 0.6381461518623532, 0.5337849090322841, 0.4522168437293153, 0.3890211974881503, 0.3407688527285771, 0.30489146466507755, 0.2795837015467116, 0.26375502463334416, 0.25705104616219393, 0.2599779816361807, 0.2741920513039548, 0.30306934424872284, 0.3527678263747656, 0.4341456815433788, 0.5660592029906896, 0.7803431610075672, 1.1266160072758133, 1.666942453409209, 2.434839845971714, 3.3433386865995858, 4.136722414776079, 4.545135706853778, 4.501075720158733, 4.130329191092171, 3.603099553142873, 3.0443025150264105, 2.5223765874635244, 2.066277455199996, 1.682796538125196, 1.3679539360912996, 1.1133977095213605, 0.9096901889173676], \"type\": \"scatter\", \"uid\": \"d7ace71e-5bbd-417c-9639-3e5119d67a86\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"3946221b-f857-4e5f-9d56-c92797f1ce39\"));});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"3946221b-f857-4e5f-9d56-c92797f1ce39\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"3946221b-f857-4e5f-9d56-c92797f1ce39\", [{\"name\": \"x\", \"x\": [0.0, 0.10101010101010101, 0.20202020202020202, 0.30303030303030304, 0.40404040404040403, 0.5050505050505051, 0.6060606060606061, 0.7070707070707071, 0.8080808080808081, 0.9090909090909091, 1.0101010101010102, 1.1111111111111112, 1.2121212121212122, 1.3131313131313131, 1.4141414141414141, 1.5151515151515151, 1.6161616161616161, 1.7171717171717171, 1.8181818181818181, 1.9191919191919191, 2.0202020202020203, 2.121212121212121, 2.2222222222222223, 2.323232323232323, 2.4242424242424243, 2.525252525252525, 2.6262626262626263, 2.727272727272727, 2.8282828282828283, 2.929292929292929, 3.0303030303030303, 3.131313131313131, 3.2323232323232323, 3.3333333333333335, 3.4343434343434343, 3.5353535353535355, 3.6363636363636362, 3.7373737373737375, 3.8383838383838382, 3.9393939393939394, 4.040404040404041, 4.141414141414141, 4.242424242424242, 4.343434343434343, 4.444444444444445, 4.545454545454545, 4.646464646464646, 4.747474747474747, 4.848484848484849, 4.94949494949495, 5.05050505050505, 5.151515151515151, 5.252525252525253, 5.353535353535354, 5.454545454545454, 5.555555555555555, 5.656565656565657, 5.757575757575758, 5.858585858585858, 5.959595959595959, 6.0606060606060606, 6.161616161616162, 6.262626262626262, 6.363636363636363, 6.4646464646464645, 6.565656565656566, 6.666666666666667, 6.767676767676767, 6.8686868686868685, 6.96969696969697, 7.070707070707071, 7.171717171717171, 7.2727272727272725, 7.373737373737374, 7.474747474747475, 7.575757575757575, 7.6767676767676765, 7.777777777777778, 7.878787878787879, 7.979797979797979, 8.080808080808081, 8.181818181818182, 8.282828282828282, 8.383838383838384, 8.484848484848484, 8.585858585858587, 8.686868686868687, 8.787878787878787, 8.88888888888889, 8.98989898989899, 9.09090909090909, 9.191919191919192, 9.292929292929292, 9.393939393939394, 9.494949494949495, 9.595959595959595, 9.696969696969697, 9.797979797979798, 9.8989898989899, 10.0], \"y\": [1.0, 1.0618070307440879, 1.145929472156516, 1.2527145963594501, 1.3833490466432634, 1.5396941281147056, 1.724183472649697, 1.939748377151537, 2.1897428861562735, 2.4778380798749042, 2.8078443136636517, 3.1833949608806322, 3.607377400523907, 4.080909231590283, 4.601493623685861, 5.159707458965986, 5.733352730028728, 6.277667864218449, 6.711268256890407, 6.904440011008827, 6.696753452844413, 5.990466584023633, 4.8939220593127315, 3.7148466638106155, 2.729784663546461, 2.0285430515831777, 1.570359150497045, 1.2832935384326256, 1.108978575142571, 1.0090137863093822, 0.9600300753674814, 0.9482538396282152, 0.9657635486283048, 1.0082214348283722, 1.0735540255610745, 1.161191486553061, 1.2716277786722856, 1.4061643998446829, 1.566758349065195, 1.7559261065098757, 1.97667078782143, 2.232404516602338, 2.526834648288896, 2.8637694588411318, 3.24677132330882, 3.678532912045141, 4.15975391112831, 4.6871180033650175, 5.249665253763894, 5.822418285193855, 6.355871516354743, 6.761535075926115, 6.902317317568296, 6.619041840013945, 5.837627979344146, 4.7052557342537265, 3.5425772356645586, 2.600872715355647, 1.9422658460972155, 1.5156417390917318, 1.2496262471975523, 1.0890844960110397, 0.9983794051927775, 0.9559714644362712, 0.9492107891192526, 0.9708593556727112, 1.017004696881, 1.0858497126545796, 1.1770149810027473, 1.2911349743510399, 1.4296224031318614, 1.5945276272544242, 1.7884482394273178, 2.014457298600624, 2.2760223046651262, 2.576882205763647, 2.920834842100241, 3.3113566908414485, 3.7509186483749555, 4.239753932401369, 4.773639359021834, 5.339923742320558, 5.910592029305686, 6.431024175751646, 6.80532700591946, 6.889567231625224, 6.528521146565962, 5.675691614931862, 4.516033832849433, 3.3762118380970882, 2.4791768096856948, 1.8617413193330443, 1.4648511457410303, 1.218523826500176, 1.070887680453135, 0.9889208660549728, 0.9527830406322477, 0.9508631564864141, 0.9765562488887533, 1.026345986742193], \"type\": \"scatter\", \"uid\": \"23cf4e7f-1893-4e74-b6c9-46bdfe59044a\"}, {\"name\": \"y\", \"x\": [0.0, 0.10101010101010101, 0.20202020202020202, 0.30303030303030304, 0.40404040404040403, 0.5050505050505051, 0.6060606060606061, 0.7070707070707071, 0.8080808080808081, 0.9090909090909091, 1.0101010101010102, 1.1111111111111112, 1.2121212121212122, 1.3131313131313131, 1.4141414141414141, 1.5151515151515151, 1.6161616161616161, 1.7171717171717171, 1.8181818181818181, 1.9191919191919191, 2.0202020202020203, 2.121212121212121, 2.2222222222222223, 2.323232323232323, 2.4242424242424243, 2.525252525252525, 2.6262626262626263, 2.727272727272727, 2.8282828282828283, 2.929292929292929, 3.0303030303030303, 3.131313131313131, 3.2323232323232323, 3.3333333333333335, 3.4343434343434343, 3.5353535353535355, 3.6363636363636362, 3.7373737373737375, 3.8383838383838382, 3.9393939393939394, 4.040404040404041, 4.141414141414141, 4.242424242424242, 4.343434343434343, 4.444444444444445, 4.545454545454545, 4.646464646464646, 4.747474747474747, 4.848484848484849, 4.94949494949495, 5.05050505050505, 5.151515151515151, 5.252525252525253, 5.353535353535354, 5.454545454545454, 5.555555555555555, 5.656565656565657, 5.757575757575758, 5.858585858585858, 5.959595959595959, 6.0606060606060606, 6.161616161616162, 6.262626262626262, 6.363636363636363, 6.4646464646464645, 6.565656565656566, 6.666666666666667, 6.767676767676767, 6.8686868686868685, 6.96969696969697, 7.070707070707071, 7.171717171717171, 7.2727272727272725, 7.373737373737374, 7.474747474747475, 7.575757575757575, 7.6767676767676765, 7.777777777777778, 7.878787878787879, 7.979797979797979, 8.080808080808081, 8.181818181818182, 8.282828282828282, 8.383838383838384, 8.484848484848484, 8.585858585858587, 8.686868686868687, 8.787878787878787, 8.88888888888889, 8.98989898989899, 9.09090909090909, 9.191919191919192, 9.292929292929292, 9.393939393939394, 9.494949494949495, 9.595959595959595, 9.696969696969697, 9.797979797979798, 9.8989898989899, 10.0], \"y\": [1.0, 0.8194780403487356, 0.6765127495203546, 0.56389609625139, 0.47569009444784355, 0.4071325800587527, 0.3544978931327249, 0.314959533062111, 0.28648013542973505, 0.26774597865042593, 0.25816360856341236, 0.2579469322984345, 0.2683456371868822, 0.2921095397315125, 0.334365038476902, 0.40421382585143045, 0.5175376495783515, 0.7014782538147918, 0.9998115904859729, 1.472756585332035, 2.1706622911770497, 3.0554113343515357, 3.918697776219176, 4.467360890577506, 4.558310218292185, 4.271279240131057, 3.7769131206229396, 3.2182521631044367, 2.680143728991083, 2.2018555967821105, 1.7956319505706138, 1.4599931723519706, 1.1874938082423045, 0.9688092307014377, 0.7947195968702216, 0.6569688097587096, 0.5485506507019844, 0.4637201203050939, 0.3978875266488159, 0.34747716515413507, 0.3097929131106958, 0.2829141618797309, 0.2656386861700612, 0.25749131489744953, 0.2588290499185519, 0.2710986463789994, 0.297351479415145, 0.3432085961054161, 0.41861286580473106, 0.5408741465495848, 0.7394100482842035, 1.0609248631027752, 1.5669357286059675, 2.3004949471113316, 3.200132383150966, 4.032281636926246, 4.512143034717381, 4.534110127315921, 4.202993815954185, 3.6905273622907466, 3.130873733257954, 2.600456891443095, 2.1331597386259635, 1.7383489995018377, 1.4132098749796922, 1.149799832236806, 0.9387169289160194, 0.770855364134301, 0.6381461518623532, 0.5337849090322841, 0.4522168437293153, 0.3890211974881503, 0.3407688527285771, 0.30489146466507755, 0.2795837015467116, 0.26375502463334416, 0.25705104616219393, 0.2599779816361807, 0.2741920513039548, 0.30306934424872284, 0.3527678263747656, 0.4341456815433788, 0.5660592029906896, 0.7803431610075672, 1.1266160072758133, 1.666942453409209, 2.434839845971714, 3.3433386865995858, 4.136722414776079, 4.545135706853778, 4.501075720158733, 4.130329191092171, 3.603099553142873, 3.0443025150264105, 2.5223765874635244, 2.066277455199996, 1.682796538125196, 1.3679539360912996, 1.1133977095213605, 0.9096901889173676], \"type\": \"scatter\", \"uid\": \"d7ace71e-5bbd-417c-9639-3e5119d67a86\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"3946221b-f857-4e5f-9d56-c92797f1ce39\"));});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "linkText": "Export to plot.ly",
        "plotlyServerURL": "https://plot.ly",
        "showLink": false
       },
       "data": [
        {
         "type": "scatter",
         "uid": "4b68730a-9e18-4c2b-aa5e-84c62a69ed7d",
         "x": [
          1,
          1.0618070307440879,
          1.145929472156516,
          1.2527145963594501,
          1.3833490466432634,
          1.5396941281147056,
          1.724183472649697,
          1.939748377151537,
          2.1897428861562735,
          2.4778380798749042,
          2.8078443136636517,
          3.1833949608806322,
          3.607377400523907,
          4.080909231590283,
          4.601493623685861,
          5.159707458965986,
          5.733352730028728,
          6.277667864218449,
          6.711268256890407,
          6.904440011008827,
          6.696753452844413,
          5.990466584023633,
          4.8939220593127315,
          3.7148466638106155,
          2.729784663546461,
          2.0285430515831777,
          1.570359150497045,
          1.2832935384326256,
          1.108978575142571,
          1.0090137863093822,
          0.9600300753674814,
          0.9482538396282152,
          0.9657635486283048,
          1.0082214348283722,
          1.0735540255610745,
          1.161191486553061,
          1.2716277786722856,
          1.4061643998446829,
          1.566758349065195,
          1.7559261065098757,
          1.97667078782143,
          2.232404516602338,
          2.526834648288896,
          2.8637694588411318,
          3.24677132330882,
          3.678532912045141,
          4.15975391112831,
          4.6871180033650175,
          5.249665253763894,
          5.822418285193855,
          6.355871516354743,
          6.761535075926115,
          6.902317317568296,
          6.619041840013945,
          5.837627979344146,
          4.7052557342537265,
          3.5425772356645586,
          2.600872715355647,
          1.9422658460972155,
          1.5156417390917318,
          1.2496262471975523,
          1.0890844960110397,
          0.9983794051927775,
          0.9559714644362712,
          0.9492107891192526,
          0.9708593556727112,
          1.017004696881,
          1.0858497126545796,
          1.1770149810027473,
          1.2911349743510399,
          1.4296224031318614,
          1.5945276272544242,
          1.7884482394273178,
          2.014457298600624,
          2.2760223046651262,
          2.576882205763647,
          2.920834842100241,
          3.3113566908414485,
          3.7509186483749555,
          4.239753932401369,
          4.773639359021834,
          5.339923742320558,
          5.910592029305686,
          6.431024175751646,
          6.80532700591946,
          6.889567231625224,
          6.528521146565962,
          5.675691614931862,
          4.516033832849433,
          3.3762118380970882,
          2.4791768096856948,
          1.8617413193330443,
          1.4648511457410303,
          1.218523826500176,
          1.070887680453135,
          0.9889208660549728,
          0.9527830406322477,
          0.9508631564864141,
          0.9765562488887533,
          1.026345986742193
         ],
         "y": [
          1,
          0.8194780403487356,
          0.6765127495203546,
          0.56389609625139,
          0.47569009444784355,
          0.4071325800587527,
          0.3544978931327249,
          0.314959533062111,
          0.28648013542973505,
          0.26774597865042593,
          0.25816360856341236,
          0.2579469322984345,
          0.2683456371868822,
          0.2921095397315125,
          0.334365038476902,
          0.40421382585143045,
          0.5175376495783515,
          0.7014782538147918,
          0.9998115904859729,
          1.472756585332035,
          2.1706622911770497,
          3.0554113343515357,
          3.918697776219176,
          4.467360890577506,
          4.558310218292185,
          4.271279240131057,
          3.7769131206229396,
          3.2182521631044367,
          2.680143728991083,
          2.2018555967821105,
          1.7956319505706138,
          1.4599931723519706,
          1.1874938082423045,
          0.9688092307014377,
          0.7947195968702216,
          0.6569688097587096,
          0.5485506507019844,
          0.4637201203050939,
          0.3978875266488159,
          0.34747716515413507,
          0.3097929131106958,
          0.2829141618797309,
          0.2656386861700612,
          0.25749131489744953,
          0.2588290499185519,
          0.2710986463789994,
          0.297351479415145,
          0.3432085961054161,
          0.41861286580473106,
          0.5408741465495848,
          0.7394100482842035,
          1.0609248631027752,
          1.5669357286059675,
          2.3004949471113316,
          3.200132383150966,
          4.032281636926246,
          4.512143034717381,
          4.534110127315921,
          4.202993815954185,
          3.6905273622907466,
          3.130873733257954,
          2.600456891443095,
          2.1331597386259635,
          1.7383489995018377,
          1.4132098749796922,
          1.149799832236806,
          0.9387169289160194,
          0.770855364134301,
          0.6381461518623532,
          0.5337849090322841,
          0.4522168437293153,
          0.3890211974881503,
          0.3407688527285771,
          0.30489146466507755,
          0.2795837015467116,
          0.26375502463334416,
          0.25705104616219393,
          0.2599779816361807,
          0.2741920513039548,
          0.30306934424872284,
          0.3527678263747656,
          0.4341456815433788,
          0.5660592029906896,
          0.7803431610075672,
          1.1266160072758133,
          1.666942453409209,
          2.434839845971714,
          3.3433386865995858,
          4.136722414776079,
          4.545135706853778,
          4.501075720158733,
          4.130329191092171,
          3.603099553142873,
          3.0443025150264105,
          2.5223765874635244,
          2.066277455199996,
          1.682796538125196,
          1.3679539360912996,
          1.1133977095213605,
          0.9096901889173676
         ]
        }
       ],
       "layout": {}
      },
      "text/html": [
       "<div id=\"28f6de18-42a8-4aad-9413-fcc459fd9398\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"28f6de18-42a8-4aad-9413-fcc459fd9398\", [{\"x\": [1.0, 1.0618070307440879, 1.145929472156516, 1.2527145963594501, 1.3833490466432634, 1.5396941281147056, 1.724183472649697, 1.939748377151537, 2.1897428861562735, 2.4778380798749042, 2.8078443136636517, 3.1833949608806322, 3.607377400523907, 4.080909231590283, 4.601493623685861, 5.159707458965986, 5.733352730028728, 6.277667864218449, 6.711268256890407, 6.904440011008827, 6.696753452844413, 5.990466584023633, 4.8939220593127315, 3.7148466638106155, 2.729784663546461, 2.0285430515831777, 1.570359150497045, 1.2832935384326256, 1.108978575142571, 1.0090137863093822, 0.9600300753674814, 0.9482538396282152, 0.9657635486283048, 1.0082214348283722, 1.0735540255610745, 1.161191486553061, 1.2716277786722856, 1.4061643998446829, 1.566758349065195, 1.7559261065098757, 1.97667078782143, 2.232404516602338, 2.526834648288896, 2.8637694588411318, 3.24677132330882, 3.678532912045141, 4.15975391112831, 4.6871180033650175, 5.249665253763894, 5.822418285193855, 6.355871516354743, 6.761535075926115, 6.902317317568296, 6.619041840013945, 5.837627979344146, 4.7052557342537265, 3.5425772356645586, 2.600872715355647, 1.9422658460972155, 1.5156417390917318, 1.2496262471975523, 1.0890844960110397, 0.9983794051927775, 0.9559714644362712, 0.9492107891192526, 0.9708593556727112, 1.017004696881, 1.0858497126545796, 1.1770149810027473, 1.2911349743510399, 1.4296224031318614, 1.5945276272544242, 1.7884482394273178, 2.014457298600624, 2.2760223046651262, 2.576882205763647, 2.920834842100241, 3.3113566908414485, 3.7509186483749555, 4.239753932401369, 4.773639359021834, 5.339923742320558, 5.910592029305686, 6.431024175751646, 6.80532700591946, 6.889567231625224, 6.528521146565962, 5.675691614931862, 4.516033832849433, 3.3762118380970882, 2.4791768096856948, 1.8617413193330443, 1.4648511457410303, 1.218523826500176, 1.070887680453135, 0.9889208660549728, 0.9527830406322477, 0.9508631564864141, 0.9765562488887533, 1.026345986742193], \"y\": [1.0, 0.8194780403487356, 0.6765127495203546, 0.56389609625139, 0.47569009444784355, 0.4071325800587527, 0.3544978931327249, 0.314959533062111, 0.28648013542973505, 0.26774597865042593, 0.25816360856341236, 0.2579469322984345, 0.2683456371868822, 0.2921095397315125, 0.334365038476902, 0.40421382585143045, 0.5175376495783515, 0.7014782538147918, 0.9998115904859729, 1.472756585332035, 2.1706622911770497, 3.0554113343515357, 3.918697776219176, 4.467360890577506, 4.558310218292185, 4.271279240131057, 3.7769131206229396, 3.2182521631044367, 2.680143728991083, 2.2018555967821105, 1.7956319505706138, 1.4599931723519706, 1.1874938082423045, 0.9688092307014377, 0.7947195968702216, 0.6569688097587096, 0.5485506507019844, 0.4637201203050939, 0.3978875266488159, 0.34747716515413507, 0.3097929131106958, 0.2829141618797309, 0.2656386861700612, 0.25749131489744953, 0.2588290499185519, 0.2710986463789994, 0.297351479415145, 0.3432085961054161, 0.41861286580473106, 0.5408741465495848, 0.7394100482842035, 1.0609248631027752, 1.5669357286059675, 2.3004949471113316, 3.200132383150966, 4.032281636926246, 4.512143034717381, 4.534110127315921, 4.202993815954185, 3.6905273622907466, 3.130873733257954, 2.600456891443095, 2.1331597386259635, 1.7383489995018377, 1.4132098749796922, 1.149799832236806, 0.9387169289160194, 0.770855364134301, 0.6381461518623532, 0.5337849090322841, 0.4522168437293153, 0.3890211974881503, 0.3407688527285771, 0.30489146466507755, 0.2795837015467116, 0.26375502463334416, 0.25705104616219393, 0.2599779816361807, 0.2741920513039548, 0.30306934424872284, 0.3527678263747656, 0.4341456815433788, 0.5660592029906896, 0.7803431610075672, 1.1266160072758133, 1.666942453409209, 2.434839845971714, 3.3433386865995858, 4.136722414776079, 4.545135706853778, 4.501075720158733, 4.130329191092171, 3.603099553142873, 3.0443025150264105, 2.5223765874635244, 2.066277455199996, 1.682796538125196, 1.3679539360912996, 1.1133977095213605, 0.9096901889173676], \"type\": \"scatter\", \"uid\": \"7a7f138f-5c00-452b-a506-3150b57a4ba0\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"28f6de18-42a8-4aad-9413-fcc459fd9398\"));});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"28f6de18-42a8-4aad-9413-fcc459fd9398\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"28f6de18-42a8-4aad-9413-fcc459fd9398\", [{\"x\": [1.0, 1.0618070307440879, 1.145929472156516, 1.2527145963594501, 1.3833490466432634, 1.5396941281147056, 1.724183472649697, 1.939748377151537, 2.1897428861562735, 2.4778380798749042, 2.8078443136636517, 3.1833949608806322, 3.607377400523907, 4.080909231590283, 4.601493623685861, 5.159707458965986, 5.733352730028728, 6.277667864218449, 6.711268256890407, 6.904440011008827, 6.696753452844413, 5.990466584023633, 4.8939220593127315, 3.7148466638106155, 2.729784663546461, 2.0285430515831777, 1.570359150497045, 1.2832935384326256, 1.108978575142571, 1.0090137863093822, 0.9600300753674814, 0.9482538396282152, 0.9657635486283048, 1.0082214348283722, 1.0735540255610745, 1.161191486553061, 1.2716277786722856, 1.4061643998446829, 1.566758349065195, 1.7559261065098757, 1.97667078782143, 2.232404516602338, 2.526834648288896, 2.8637694588411318, 3.24677132330882, 3.678532912045141, 4.15975391112831, 4.6871180033650175, 5.249665253763894, 5.822418285193855, 6.355871516354743, 6.761535075926115, 6.902317317568296, 6.619041840013945, 5.837627979344146, 4.7052557342537265, 3.5425772356645586, 2.600872715355647, 1.9422658460972155, 1.5156417390917318, 1.2496262471975523, 1.0890844960110397, 0.9983794051927775, 0.9559714644362712, 0.9492107891192526, 0.9708593556727112, 1.017004696881, 1.0858497126545796, 1.1770149810027473, 1.2911349743510399, 1.4296224031318614, 1.5945276272544242, 1.7884482394273178, 2.014457298600624, 2.2760223046651262, 2.576882205763647, 2.920834842100241, 3.3113566908414485, 3.7509186483749555, 4.239753932401369, 4.773639359021834, 5.339923742320558, 5.910592029305686, 6.431024175751646, 6.80532700591946, 6.889567231625224, 6.528521146565962, 5.675691614931862, 4.516033832849433, 3.3762118380970882, 2.4791768096856948, 1.8617413193330443, 1.4648511457410303, 1.218523826500176, 1.070887680453135, 0.9889208660549728, 0.9527830406322477, 0.9508631564864141, 0.9765562488887533, 1.026345986742193], \"y\": [1.0, 0.8194780403487356, 0.6765127495203546, 0.56389609625139, 0.47569009444784355, 0.4071325800587527, 0.3544978931327249, 0.314959533062111, 0.28648013542973505, 0.26774597865042593, 0.25816360856341236, 0.2579469322984345, 0.2683456371868822, 0.2921095397315125, 0.334365038476902, 0.40421382585143045, 0.5175376495783515, 0.7014782538147918, 0.9998115904859729, 1.472756585332035, 2.1706622911770497, 3.0554113343515357, 3.918697776219176, 4.467360890577506, 4.558310218292185, 4.271279240131057, 3.7769131206229396, 3.2182521631044367, 2.680143728991083, 2.2018555967821105, 1.7956319505706138, 1.4599931723519706, 1.1874938082423045, 0.9688092307014377, 0.7947195968702216, 0.6569688097587096, 0.5485506507019844, 0.4637201203050939, 0.3978875266488159, 0.34747716515413507, 0.3097929131106958, 0.2829141618797309, 0.2656386861700612, 0.25749131489744953, 0.2588290499185519, 0.2710986463789994, 0.297351479415145, 0.3432085961054161, 0.41861286580473106, 0.5408741465495848, 0.7394100482842035, 1.0609248631027752, 1.5669357286059675, 2.3004949471113316, 3.200132383150966, 4.032281636926246, 4.512143034717381, 4.534110127315921, 4.202993815954185, 3.6905273622907466, 3.130873733257954, 2.600456891443095, 2.1331597386259635, 1.7383489995018377, 1.4132098749796922, 1.149799832236806, 0.9387169289160194, 0.770855364134301, 0.6381461518623532, 0.5337849090322841, 0.4522168437293153, 0.3890211974881503, 0.3407688527285771, 0.30489146466507755, 0.2795837015467116, 0.26375502463334416, 0.25705104616219393, 0.2599779816361807, 0.2741920513039548, 0.30306934424872284, 0.3527678263747656, 0.4341456815433788, 0.5660592029906896, 0.7803431610075672, 1.1266160072758133, 1.666942453409209, 2.434839845971714, 3.3433386865995858, 4.136722414776079, 4.545135706853778, 4.501075720158733, 4.130329191092171, 3.603099553142873, 3.0443025150264105, 2.5223765874635244, 2.066277455199996, 1.682796538125196, 1.3679539360912996, 1.1133977095213605, 0.9096901889173676], \"type\": \"scatter\", \"uid\": \"7a7f138f-5c00-452b-a506-3150b57a4ba0\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"28f6de18-42a8-4aad-9413-fcc459fd9398\"));});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sol = ode_scipy(lotka_volterra, z0, ts, args)\n",
    "xs = sol[:, 0].numpy()\n",
    "ys = sol[:, 1].numpy()\n",
    "\n",
    "iplot([go.Scatter(x=ts.numpy(), y=xs, name=\"x\"), go.Scatter(x=ts.numpy(), y=ys, name=\"y\")])\n",
    "\n",
    "iplot([go.Scatter(x=xs, y=ys)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The ugly Augmented State in Tensorfow Eager-mode + GradientTape\n",
    "\n",
    "This is an implementation of the augmented-state gradient computed by ODE-Solver code sample found in Neural ODE paper made in pure TF 1.x in eager-mode and using Gradient-Tape.\n",
    "\n",
    "To be clear, that was a true pain to implement it and it's still not stable IMHO...\n",
    "\n",
    "I don't pretend this is nice or idiomatic code but it works now and enables experimentations..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def flatten(params):\n",
    "    \"\"\"flatten a list of tensors/arrays into a flat tensor and returns also the unflatten function\n",
    "    able to rebuild original list from flattenned tensor\n",
    "    \"\"\"\n",
    "    if len(params) == 0:\n",
    "        def unflatten(flat_params):\n",
    "            return flat_params\n",
    "        return np.array(params), unflatten\n",
    "    \n",
    "    original_shapes = []\n",
    "    flat_params = []\n",
    "    default_dtype = params[0].dtype\n",
    "    for p in params:\n",
    "        sh = p.shape\n",
    "        original_shapes.append(sh)\n",
    "        flat_sh = 1\n",
    "        for s in sh:\n",
    "            flat_sh *= s\n",
    "\n",
    "        res = tf.reshape(p, (flat_sh,))\n",
    "        res = tf.cast(res, default_dtype)\n",
    "        flat_params.append(res)\n",
    "    \n",
    "    def unflatten(flat_params):\n",
    "        cur_idx = 0\n",
    "        params = []\n",
    "        for sh in original_shapes:\n",
    "            flat_sh = 1\n",
    "            for s in sh:\n",
    "                flat_sh *= s\n",
    "            param = flat_params[cur_idx:cur_idx + flat_sh]\n",
    "            param = tf.reshape(param, sh)\n",
    "            cur_idx += int(flat_sh)\n",
    "            params.append(param)\n",
    "            \n",
    "        return params\n",
    "            \n",
    "    flat_params = tf.concat(flat_params, axis=0)\n",
    "    return flat_params, unflatten\n",
    "    \n",
    "    \n",
    "# Directly inspired from function implemented with autograd at the end of the paper\n",
    "# Extended from \"Scalable Inference of Ordinary Differential\n",
    "# Equation Models of Biochemical Processes\", Sec. 2.4.2\n",
    "# Fabian Froehlich, Carolin Loos, Jan Hasenauer, 2017\n",
    "# https://arxiv.org/abs/1711.08079\n",
    "def grad_odeint(z_ts, func, ts, args, T, D, ode_solver, variables = None):\n",
    "    \"\"\"\n",
    "    This code is still ugly and not very robust IMHO...\n",
    "    It's full of reshape that might be unneeded but at least I know what I manipulate without compilted types\n",
    "    \n",
    "    Args:\n",
    "        z_ts: the values with ODE Solver in forward path on ts time values\n",
    "        func: the adjoint gradient function to integrate with ODE Solver\n",
    "        ts: the discretized time values\n",
    "        args: the derivable arguments of func\n",
    "        ode_solver: the ode_solver function (z0, ts, args) -> zs\n",
    "        variables: the hidden TF/Keras tensor variables (separated from args to be manageable as they have their own life cycle I'm not sure to understand yet)\n",
    "\n",
    "    Returns:\n",
    "        grad: the custom augmentate-state TF gradient function\n",
    "    \"\"\"\n",
    "    # let's store\n",
    "    if variables is not None:\n",
    "        flat_variables, unflatten_variables = flatten(variables)\n",
    "\n",
    "    def unpack_D_2D(x, D):\n",
    "        \"\"\"simple extract \"\"\"\n",
    "        x = tf.squeeze(x)\n",
    "        return (\n",
    "            x[0:D],\n",
    "            x[D:2 * D]\n",
    "        )\n",
    "        \n",
    "    # augmented_state = [z_t, a_z_t, a_t, a_args, a_vars]\n",
    "    def a_gradient_aug(augmented_state, t, args):\n",
    "        augmented_state = tf.convert_to_tensor(augmented_state, dtype=z_ts.dtype)\n",
    "        t = tf.convert_to_tensor(t, dtype=ts.dtype)\n",
    "        t = tf.reshape(t, (1,))\n",
    "        args = tf.convert_to_tensor(args, dtype=args.dtype)\n",
    "        z_t, a_z_t = unpack_D_2D(augmented_state, z_ts.shape[1])\n",
    "\n",
    "        # the internal gradient-tape computing second derivative\n",
    "        with tf.GradientTape() as gg:\n",
    "            gg.watch(args)                \n",
    "            gg.watch(a_z_t)\n",
    "            gg.watch(t)\n",
    "            gg.watch(z_t)\n",
    "            dz_dt = func(z_t, t, args)\n",
    "\n",
    "        if variables is not None:\n",
    "            da_z_dt, da_t_dt, da_args_dargs, da_dvars = gg.gradient(\n",
    "                dz_dt, [z_t, t, args, variables], output_gradients = -a_z_t)\n",
    "        else:\n",
    "            da_z_dt, da_t_dt, da_args_dargs = gg.gradient(\n",
    "                dz_dt, [z_t, t, args], output_gradients = -a_z_t)\n",
    "            da_dvars = None\n",
    "        \n",
    "        if da_z_dt is None:\n",
    "            da_z_dt = tf.zeros_like(z_t, dtype=z_t.dtype)\n",
    "        if da_t_dt is None:\n",
    "            da_t_dt = tf.zeros_like(t, dtype=t.dtype)\n",
    "        if da_args_dargs is None:\n",
    "            da_args_dargs = tf.zeros_like(args, dtype=args.dtype)\n",
    "        if da_dvars is None:\n",
    "            da_dvars = tf.zeros_like((1,))\n",
    "        else:\n",
    "            da_dvars, _ = flatten(da_dvars)\n",
    "        r, _ = flatten([dz_dt, da_z_dt, da_t_dt, da_args_dargs, da_dvars])\n",
    "        return r\n",
    "\n",
    "    def grad(gt):\n",
    "        \"\"\"the TF custom gradient function\"\"\"\n",
    "        \n",
    "        # initialization of backward gradient computation starting at the end of time interval\n",
    "        dl_dz_t = gt[-1, :]\n",
    "        dl_dt0 = tf.constant([0.], shape=(1,), dtype=ts.dtype)\n",
    "        dl_dargs = tf.zeros_like(args, dtype=ts.dtype)\n",
    "        if variables is not None:\n",
    "            dl_dvars = tf.zeros_like(flat_variables, dtype=flat_variables.dtype)\n",
    "        else:\n",
    "            dl_dvars = tf.zeros((1,))\n",
    "        dl_dt_list = []\n",
    "\n",
    "        # compute gradient back-propagation by ODE solver calls going backward in time\n",
    "        for i in range(T - 1, 0, -1):\n",
    "            # Compute effect of moving back in time (just porting formulas found in paper)\n",
    "            ztsi = z_ts[i, :]\n",
    "            gti = gt[i, :]\n",
    "            # Compute the adjoint gradient of loss at current time\n",
    "            fi = func(ztsi, ts[i], args)\n",
    "            dl_dt = tf.tensordot(fi, tf.transpose(gti), axes=1) #, axes=[[1], [0]])\n",
    "            dl_dt = tf.reshape(dl_dt, (1,))\n",
    "            dl_dt_list.append(dl_dt)\n",
    "            dl_dt0 = dl_dt0 - dl_dt\n",
    "            dl_dt0 = tf.reshape(dl_dt0, shape=(1,))\n",
    "\n",
    "            # build augmented state\n",
    "            aug_z0, unflatten_aug = flatten([ztsi, dl_dz_t, dl_dt0, dl_dargs, dl_dvars])\n",
    "            # Run augmented system backwards to the previous observation using ODE_Solver\n",
    "            cur_ts = tf.reverse(tf.slice(ts, [i-1], [2]), axis=[0])\n",
    "            a_aug = ode_solver(lambda aug, t, args: a_gradient_aug(aug, t, args), aug_z0, cur_ts, args)\n",
    "\n",
    "            # get at ts[i - 1]\n",
    "            a_aug = a_aug[1]\n",
    "            _, dl_dz_t, dl_dt0, dl_dargs, dl_dvars = unflatten_aug(a_aug)\n",
    "\n",
    "            # Add gradient from current output.\n",
    "            dl_dz_t = dl_dz_t + gt[i - 1, :]\n",
    "\n",
    "        dl_dt0 = tf.reshape(dl_dt0, (1,))\n",
    "        dl_dt_list.append(dl_dt0)\n",
    "        dl_dts = tf.stack(dl_dt_list, axis=0)\n",
    "        dl_dts = tf.reverse(dl_dts, axis=[0])\n",
    "\n",
    "        # output z_t is not needed but can't put None in TF tensors\n",
    "        n_o_n_e = tf.zeros_like(z_ts) #, dtype=z_ts.dtype)\n",
    "        \n",
    "        if variables is not None:\n",
    "            return [n_o_n_e, dl_dz_t, dl_dts, dl_dargs], unflatten_variables(dl_dvars)\n",
    "        else:\n",
    "            return [n_o_n_e, dl_dz_t, dl_dts, dl_dargs], []\n",
    "    return grad\n",
    "\n",
    "def tf_ode_grad(f, N, D, ode_solver):\n",
    "    \"\"\"The plug in custom_gradient TF mechanism\"\"\"\n",
    "    @tf.custom_gradient\n",
    "    def custom_grad(z0, ts, args):\n",
    "        z0 = tf.squeeze(z0)\n",
    "        # forward values computation with ODE Solver\n",
    "        ys = ode_solver(f, z0, ts, args)\n",
    "        def grad_fn(dr, variables=None):\n",
    "            # backward gradient computation using backward ODE Solver\n",
    "            (_, dl_dz_t, dl_dts, dl_dargs), dl_dvars = grad_odeint(ys, f, ts, args, N, D, ode_solver, variables)(dr)\n",
    "            return [dl_dz_t, dl_dts, dl_dargs], dl_dvars\n",
    "        return ys, grad_fn\n",
    "    return custom_grad\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A gradient-descent Neural-ODE optimizer based on loss function\n",
    "\n",
    "The following function optimizes iteratively a function defined by its derivative f using an ODE Solver, a ground_truth and a time interval sampling.\n",
    "\n",
    "The derivative function is parameterized by a set of custom arguments and/or a set of hidden Tensorflow variables (the weights & bias in Keras mode for example). Those parameters are those we want to learn during training by gradient descent controlled by an Adam optimizer.\n",
    "\n",
    "I had to separate Tensorflow variables from other arguments because they are managed differently in TF `@custom_gradient`. That was the deepest pain during this study because there is no doc or samples about those features except very basic cases which were useless to me. Anyway, by forcing this separation, I could reach a runnable state that seems to optimize my parameters as you'll see below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython import display\n",
    "\n",
    "def loss_L2(y_true, y_pred):\n",
    "    \"\"\"The classic derivable least-square loss function\"\"\"\n",
    "    return tf.reduce_sum(tf.square(y_true - y_pred))\n",
    "\n",
    "def optimize(f, ode_solver, loss_fn, labels, z0, ts,\n",
    "             args=None, variables=None,\n",
    "             epochs=25,\n",
    "             x_axis_min=-0.25, x_axis_max=10.25,\n",
    "             y_axis_min=-0.25, y_axis_max=7.0,\n",
    "             x2_axis_min=-0.25, x2_axis_max=10.25,\n",
    "             y2_axis_min=-0.25, y2_axis_max=7.0,\n",
    "             learning_rate=0.1, epsilon=1e-7, output_dir=\"images\", start = 0):\n",
    "    \"\"\"A sample optimization process that trie to optimize args/variables parameters of a function\n",
    "    defined by its derivative f integrated with ODE Solver according to a loss function loss_fn, groundtruth labels,\n",
    "    initial input value and time interval sampling using Adam optimizer (with plots & image saving)\"\"\"\n",
    "    def custom_grad(z0, ts, the_args):\n",
    "        with tf.GradientTape() as tape:\n",
    "            tape.watch(z0)\n",
    "            tape.watch(the_args)\n",
    "            ode = tf_ode_grad(f, N, D, ode_solver)(z0, ts, the_args)\n",
    "            loss = loss_fn(labels, ode)\n",
    "\n",
    "        # managing stupidly the abscence of args or variables...\n",
    "        if variables is not None:\n",
    "            grads, grad_vars = tape.gradient(loss, [[the_args], variables])\n",
    "        else:\n",
    "            grads = tape.gradient(loss, [the_args])\n",
    "            grad_vars = []\n",
    "\n",
    "        the_grads = []\n",
    "        for grad in grads:\n",
    "            grad = tf.reshape(grad, the_args.shape)\n",
    "            grad = tf.cast(grad, the_args.dtype)\n",
    "            the_grads.append(grad)\n",
    "            \n",
    "        the_grad_vars = []\n",
    "        for i, grad in enumerate(grad_vars):\n",
    "            the_grad_vars.append(tf.cast(grad, variables[i].dtype))\n",
    "            \n",
    "        return loss, ode, the_grads, the_grad_vars\n",
    "\n",
    "    # Adam Optimizer\n",
    "    optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate,\n",
    "                                       epsilon=epsilon)\n",
    "\n",
    "    # plotting layout\n",
    "    layout = go.Layout(\n",
    "        xaxis=dict(\n",
    "            range=[x_axis_min, x_axis_max],\n",
    "        ),\n",
    "        yaxis=dict(\n",
    "            range=[y_axis_min, y_axis_max],\n",
    "            domain=[0.55, 1.0],\n",
    "        ),\n",
    "        xaxis2=dict(\n",
    "            range=[x2_axis_min, x2_axis_max],\n",
    "        ),\n",
    "        yaxis2=dict(\n",
    "            range=[y2_axis_min, y2_axis_max],\n",
    "            domain=[0, 0.45],  \n",
    "        ),\n",
    "        title=\"\",\n",
    "        height=680, width=1024\n",
    "    )\n",
    "    \n",
    "    # no args, replacing it by a fake tf.Variable for now... to be removed later\n",
    "    if args is None:\n",
    "        var_args = tf.Variable(0, dtype=tf.float64)\n",
    "    else:\n",
    "        var_args = args\n",
    "    \n",
    "    # the training loop\n",
    "    global_step = tf.Variable(start)\n",
    "    best_loss = 1e400\n",
    "    best_zss = []\n",
    "\n",
    "    if not os.path.exists(output_dir):\n",
    "        os.mkdir(output_dir)\n",
    "    \n",
    "    for epoch in range(epochs):\n",
    "        loss, ode, grads, grad_vars = custom_grad(z0, ts, var_args)\n",
    "\n",
    "        print(\"loss:\", loss)\n",
    "        #print(\"ode:\", ode)\n",
    "        #print(\"grads:\", grads)\n",
    "\n",
    "        grad_params = list(zip(grads, [var_args]))\n",
    "        if variables is not None:\n",
    "            grad_params.extend(list(zip(grad_vars, variables)))\n",
    "                    \n",
    "        optimizer.apply_gradients(grad_params, global_step)\n",
    "        \n",
    "        #loss, ode, grads, grad_vars = custom_grad(z0, ts, var_args)\n",
    "        ys = ode.numpy()\n",
    "        last_ts = ts.numpy()\n",
    "        last_ys = ys\n",
    "        \n",
    "        if loss < best_loss:\n",
    "            best_loss = loss\n",
    "            best_var_args = var_args\n",
    "            #best_ts = ts.numpy()\n",
    "            best_zs = ys\n",
    "            best_zss.append(best_zs)\n",
    "\n",
    "        # More Plotting\n",
    "        display.clear_output(wait=True)\n",
    "        data = []\n",
    "        \n",
    "        for zs in best_zss[-1:]:\n",
    "            data.extend([\n",
    "                go.Scatter(x=ts.numpy(), y=zs[:, 0], name=\"best_x\",\n",
    "                           mode = 'lines+markers',\n",
    "                           line = dict(\n",
    "                                color = ('rgb(34, 119, 158)'),\n",
    "                                width = 1)),\n",
    "                go.Scatter(x=ts.numpy(), y=zs[:, 1], name=\"best_y\",\n",
    "                           mode = 'lines+markers',\n",
    "                           line = dict(\n",
    "                                color = ('rgb(201, 79, 14)'),\n",
    "                                width = 1)),\n",
    "                go.Scatter(x=zs[:, 0], y=zs[:, 1], name=\"best_xy\",\n",
    "                           xaxis=\"x2\", yaxis=\"y2\",\n",
    "                           mode = 'lines+markers',\n",
    "                           line = dict(\n",
    "                                color = ('rgb(8, 173, 99)'),\n",
    "                                width = 1)),\n",
    "            ])\n",
    "            \n",
    "        data.extend([            \n",
    "            go.Scatter(x=ts.numpy(), y=labels.numpy()[:, 0], name=\"groundtruth_x\",\n",
    "                       line = dict(\n",
    "                            color = ('rgb(34, 119, 158)'),\n",
    "                            width = 2,\n",
    "                            dash = 'dot')),\n",
    "            go.Scatter(x=ts.numpy(), y=labels.numpy()[:, 1], name=\"groundtruth_y\",\n",
    "                       line = dict(\n",
    "                            color = ('rgb(201, 79, 14)'),\n",
    "                            width = 2,\n",
    "                            dash = 'dot')),\n",
    "            go.Scatter(x=labels.numpy()[:, 0], y=labels.numpy()[:, 1], name=\"groundtruth_xy\",\n",
    "                       xaxis=\"x2\", yaxis=\"y2\",\n",
    "                       line = dict(\n",
    "                            color = ('rgb(173, 7, 7)'),\n",
    "                            width = 2,\n",
    "                            dash = 'dot')),\n",
    "        ])\n",
    "        if loss >= best_loss:\n",
    "            data.extend([                \n",
    "                go.Scatter(x=ts.numpy(), y=last_ys[:, 0], name=\"current_x\",\n",
    "                           line = dict(\n",
    "                                    color = ('rgb(24, 24, 24)'),\n",
    "                                    width = 1,\n",
    "                                    dash = 'dash')),\n",
    "                go.Scatter(x=ts.numpy(), y=last_ys[:, 1], name=\"current_y\",\n",
    "                           line = dict(\n",
    "                                    color = ('rgb(24, 24, 24)'),\n",
    "                                    width = 1,\n",
    "                                    dash = 'dash')),\n",
    "                go.Scatter(x=last_ys[:, 0], y=last_ys[:, 1], name=\"current_xy\",\n",
    "                           xaxis=\"x2\", yaxis=\"y2\",\n",
    "                           line = dict(\n",
    "                                color = ('rgb(24, 24, 24)'),\n",
    "                                width = 2,\n",
    "                                dash = 'dash')),\n",
    "            ])\n",
    "\n",
    "        layout.title = f\"Step {global_step.numpy()}<br>Best Loss {best_loss.numpy()}<br>Current Loss {loss.numpy()}\"\n",
    "        fig = go.Figure(data=data, layout=layout)\n",
    "        iplot(fig)\n",
    "\n",
    "        pio.write_image(fig, f\"{output_dir}/step_{global_step.numpy()}.png\")\n",
    "\n",
    "        #print(f\"Step: {global_step.numpy()}, Loss: {loss.numpy()}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimizing lotka_volterra\n",
    "\n",
    "Let's optimize lotka_volterra parameters using our optimization function with a dumb loss function that is the squared distance of the output values from 1 to check it works."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scipy ODE Solver"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tensorflow Variables\n",
    "# the dimension of our input vector [x, y] => 2\n",
    "D = 2\n",
    "# time interval sampling\n",
    "N = 25\n",
    "ts = tf.constant(np.linspace(0., 10.0, N))\n",
    "# the arguments we want to optimize\n",
    "args = tf.Variable(tf.constant([1.5, 1.0, 3.0, 1.0], shape=(4,), dtype=ts.dtype))\n",
    "z0 = tf.Variable(tf.constant([1.0,1.0], shape=(2,), dtype=ts.dtype))\n",
    "\n",
    "def ode_scipy_solver(f, z0, ts, args):\n",
    "    r = ode_scipy(f, z0, ts, args)\n",
    "    r = tf.cast(r, z0.dtype)\n",
    "    return r\n",
    "\n",
    "ground_truth_ones = tf.ones(shape=(N, D), dtype=z0.dtype)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training Video"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<video width=\"640\" height=\"480\" controls>\n",
       "  <source src=\"lotka_volterra_optim_scipy/video.mp4\" type=\"video/mp4\">\n",
       "</video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "\n",
    "HTML(\"\"\"\n",
    "<video width=\"640\" height=\"480\" controls>\n",
    "  <source src=\"lotka_volterra_optim_scipy/video.mp4\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Running Optimization by yourself"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "linkText": "Export to plot.ly",
        "plotlyServerURL": "https://plot.ly",
        "showLink": false
       },
       "data": [
        {
         "line": {
          "color": "rgb(34, 119, 158)",
          "width": 1
         },
         "mode": "lines+markers",
         "name": "best_x",
         "type": "scatter",
         "uid": "9ebe9e11-bbe4-4204-8cd4-25f2ec4197e4",
         "x": [
          0,
          0.4166666666666667,
          0.8333333333333334,
          1.25,
          1.6666666666666667,
          2.0833333333333335,
          2.5,
          2.916666666666667,
          3.3333333333333335,
          3.75,
          4.166666666666667,
          4.583333333333334,
          5,
          5.416666666666667,
          5.833333333333334,
          6.25,
          6.666666666666667,
          7.083333333333334,
          7.5,
          7.916666666666667,
          8.333333333333334,
          8.75,
          9.166666666666668,
          9.583333333333334,
          10
         ],
         "y": [
          1,
          0.9946109821746748,
          0.9975093325168041,
          1.0066779892487072,
          1.0158140782874718,
          1.0184570928355554,
          1.0126420021399425,
          1.0025373510794588,
          0.9953109552792863,
          0.995914225891269,
          1.003879239590604,
          1.0137490585799174,
          1.0186090483879588,
          1.014918874448412,
          1.0053022411266337,
          0.9966238356612817,
          0.9948944471929325,
          1.0012309144372078,
          1.011292745135055,
          1.0180893085876028,
          1.016730447076507,
          1.0081424789583326,
          0.9984814392159677,
          0.9945049293261299,
          0.9988749773099755
         ]
        },
        {
         "line": {
          "color": "rgb(201, 79, 14)",
          "width": 1
         },
         "mode": "lines+markers",
         "name": "best_y",
         "type": "scatter",
         "uid": "2b8ae7a5-dfd8-4d68-a90c-2f38e06d545b",
         "x": [
          0,
          0.4166666666666667,
          0.8333333333333334,
          1.25,
          1.6666666666666667,
          2.0833333333333335,
          2.5,
          2.916666666666667,
          3.3333333333333335,
          3.75,
          4.166666666666667,
          4.583333333333334,
          5,
          5.416666666666667,
          5.833333333333334,
          6.25,
          6.666666666666667,
          7.083333333333334,
          7.5,
          7.916666666666667,
          8.333333333333334,
          8.75,
          9.166666666666668,
          9.583333333333334,
          10
         ],
         "y": [
          1,
          0.9914853953160562,
          0.981916696483938,
          0.9778896222660368,
          0.9821800877100572,
          0.9919084628722946,
          1.0002615619643,
          1.0012481140530367,
          0.994211310507671,
          0.9842070236278863,
          0.978178245055648,
          0.9802545829481121,
          0.989081693611526,
          0.9985376770462829,
          1.0018666901309434,
          0.9967065348231313,
          0.9868091726014248,
          0.9791002271767438,
          0.9788584654256741,
          0.9863060843836177,
          0.9963327122880606,
          1.001818434483738,
          0.9988318714335261,
          0.9895837858368035,
          0.9806056670831349
         ]
        },
        {
         "line": {
          "color": "rgb(8, 173, 99)",
          "width": 1
         },
         "mode": "lines+markers",
         "name": "best_xy",
         "type": "scatter",
         "uid": "fa701640-b8cf-4d93-bcec-23e0363f9234",
         "x": [
          1,
          0.9946109821746748,
          0.9975093325168041,
          1.0066779892487072,
          1.0158140782874718,
          1.0184570928355554,
          1.0126420021399425,
          1.0025373510794588,
          0.9953109552792863,
          0.995914225891269,
          1.003879239590604,
          1.0137490585799174,
          1.0186090483879588,
          1.014918874448412,
          1.0053022411266337,
          0.9966238356612817,
          0.9948944471929325,
          1.0012309144372078,
          1.011292745135055,
          1.0180893085876028,
          1.016730447076507,
          1.0081424789583326,
          0.9984814392159677,
          0.9945049293261299,
          0.9988749773099755
         ],
         "xaxis": "x2",
         "y": [
          1,
          0.9914853953160562,
          0.981916696483938,
          0.9778896222660368,
          0.9821800877100572,
          0.9919084628722946,
          1.0002615619643,
          1.0012481140530367,
          0.994211310507671,
          0.9842070236278863,
          0.978178245055648,
          0.9802545829481121,
          0.989081693611526,
          0.9985376770462829,
          1.0018666901309434,
          0.9967065348231313,
          0.9868091726014248,
          0.9791002271767438,
          0.9788584654256741,
          0.9863060843836177,
          0.9963327122880606,
          1.001818434483738,
          0.9988318714335261,
          0.9895837858368035,
          0.9806056670831349
         ],
         "yaxis": "y2"
        },
        {
         "line": {
          "color": "rgb(34, 119, 158)",
          "dash": "dot",
          "width": 2
         },
         "name": "groundtruth_x",
         "type": "scatter",
         "uid": "1f34addd-b703-4aeb-b15a-ce3acad16505",
         "x": [
          0,
          0.4166666666666667,
          0.8333333333333334,
          1.25,
          1.6666666666666667,
          2.0833333333333335,
          2.5,
          2.916666666666667,
          3.3333333333333335,
          3.75,
          4.166666666666667,
          4.583333333333334,
          5,
          5.416666666666667,
          5.833333333333334,
          6.25,
          6.666666666666667,
          7.083333333333334,
          7.5,
          7.916666666666667,
          8.333333333333334,
          8.75,
          9.166666666666668,
          9.583333333333334,
          10
         ],
         "y": [
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1
         ]
        },
        {
         "line": {
          "color": "rgb(201, 79, 14)",
          "dash": "dot",
          "width": 2
         },
         "name": "groundtruth_y",
         "type": "scatter",
         "uid": "7d44825a-ea60-4cf8-b6bc-d1e5da1945cb",
         "x": [
          0,
          0.4166666666666667,
          0.8333333333333334,
          1.25,
          1.6666666666666667,
          2.0833333333333335,
          2.5,
          2.916666666666667,
          3.3333333333333335,
          3.75,
          4.166666666666667,
          4.583333333333334,
          5,
          5.416666666666667,
          5.833333333333334,
          6.25,
          6.666666666666667,
          7.083333333333334,
          7.5,
          7.916666666666667,
          8.333333333333334,
          8.75,
          9.166666666666668,
          9.583333333333334,
          10
         ],
         "y": [
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1
         ]
        },
        {
         "line": {
          "color": "rgb(173, 7, 7)",
          "dash": "dot",
          "width": 2
         },
         "name": "groundtruth_xy",
         "type": "scatter",
         "uid": "a42a42ec-e2ce-475f-b3bf-7d3e2956c687",
         "x": [
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1
         ],
         "xaxis": "x2",
         "y": [
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1
         ],
         "yaxis": "y2"
        },
        {
         "line": {
          "color": "rgb(24, 24, 24)",
          "dash": "dash",
          "width": 1
         },
         "name": "current_x",
         "type": "scatter",
         "uid": "aca5b0ce-d8e5-49d9-81e5-8ef2e853394e",
         "x": [
          0,
          0.4166666666666667,
          0.8333333333333334,
          1.25,
          1.6666666666666667,
          2.0833333333333335,
          2.5,
          2.916666666666667,
          3.3333333333333335,
          3.75,
          4.166666666666667,
          4.583333333333334,
          5,
          5.416666666666667,
          5.833333333333334,
          6.25,
          6.666666666666667,
          7.083333333333334,
          7.5,
          7.916666666666667,
          8.333333333333334,
          8.75,
          9.166666666666668,
          9.583333333333334,
          10
         ],
         "y": [
          1,
          1.0117713176740013,
          1.0300400350849734,
          1.0418947821442424,
          1.0384167475453314,
          1.0220589929329797,
          1.00498974015452,
          0.9993248119836119,
          1.0087430239915052,
          1.0267409502809268,
          1.040713033361885,
          1.040283721664198,
          1.0255813654725876,
          1.0075783980233675,
          0.9992004988146628,
          1.0060296164251512,
          1.0232972709463908,
          1.0390017884127052,
          1.0416236726431536,
          1.0289844113195854,
          1.0105223748276346,
          0.999631885119563,
          1.0036983252880305,
          1.0197992659413884,
          1.0368105402739054
         ]
        },
        {
         "line": {
          "color": "rgb(24, 24, 24)",
          "dash": "dash",
          "width": 1
         },
         "name": "current_y",
         "type": "scatter",
         "uid": "6068be72-a79f-4235-9324-47620d6c7033",
         "x": [
          0,
          0.4166666666666667,
          0.8333333333333334,
          1.25,
          1.6666666666666667,
          2.0833333333333335,
          2.5,
          2.916666666666667,
          3.3333333333333335,
          3.75,
          4.166666666666667,
          4.583333333333334,
          5,
          5.416666666666667,
          5.833333333333334,
          6.25,
          6.666666666666667,
          7.083333333333334,
          7.5,
          7.916666666666667,
          8.333333333333334,
          8.75,
          9.166666666666668,
          9.583333333333334,
          10
         ],
         "y": [
          1,
          0.9863428420042899,
          0.9864722401445015,
          1.0004914128935396,
          1.0186950365828926,
          1.027704086271057,
          1.0208273962262011,
          1.0034081342724925,
          0.988013330931901,
          0.9852543067215627,
          0.9971745902503322,
          1.0156379039315484,
          1.0271924700876494,
          1.0232121255638968,
          1.0068928745611894,
          0.9901454106172497,
          0.9845737819236131,
          0.9941011034093465,
          1.0123279510351868,
          1.0261072640874267,
          1.0251444511438081,
          1.0103646435807363,
          0.9926861846769144,
          0.9844452222939443,
          0.9913494422841072
         ]
        },
        {
         "line": {
          "color": "rgb(24, 24, 24)",
          "dash": "dash",
          "width": 2
         },
         "name": "current_xy",
         "type": "scatter",
         "uid": "9b900117-64a9-40de-a390-5ec45f3cb71d",
         "x": [
          1,
          1.0117713176740013,
          1.0300400350849734,
          1.0418947821442424,
          1.0384167475453314,
          1.0220589929329797,
          1.00498974015452,
          0.9993248119836119,
          1.0087430239915052,
          1.0267409502809268,
          1.040713033361885,
          1.040283721664198,
          1.0255813654725876,
          1.0075783980233675,
          0.9992004988146628,
          1.0060296164251512,
          1.0232972709463908,
          1.0390017884127052,
          1.0416236726431536,
          1.0289844113195854,
          1.0105223748276346,
          0.999631885119563,
          1.0036983252880305,
          1.0197992659413884,
          1.0368105402739054
         ],
         "xaxis": "x2",
         "y": [
          1,
          0.9863428420042899,
          0.9864722401445015,
          1.0004914128935396,
          1.0186950365828926,
          1.027704086271057,
          1.0208273962262011,
          1.0034081342724925,
          0.988013330931901,
          0.9852543067215627,
          0.9971745902503322,
          1.0156379039315484,
          1.0271924700876494,
          1.0232121255638968,
          1.0068928745611894,
          0.9901454106172497,
          0.9845737819236131,
          0.9941011034093465,
          1.0123279510351868,
          1.0261072640874267,
          1.0251444511438081,
          1.0103646435807363,
          0.9926861846769144,
          0.9844452222939443,
          0.9913494422841072
         ],
         "yaxis": "y2"
        }
       ],
       "layout": {
        "height": 680,
        "title": {
         "text": "Step 80<br>Best Loss 0.006856044573044227<br>Current Loss 0.022297664451166526"
        },
        "width": 1024,
        "xaxis": {
         "range": [
          -0.25,
          10.25
         ]
        },
        "xaxis2": {
         "range": [
          -0.25,
          10.25
         ]
        },
        "yaxis": {
         "domain": [
          0.55,
          1
         ],
         "range": [
          -0.25,
          7
         ]
        },
        "yaxis2": {
         "domain": [
          0,
          0.45
         ],
         "range": [
          -0.25,
          7
         ]
        }
       }
      },
      "text/html": [
       "<div id=\"07136567-341a-4e48-9dc9-89c8cb0ac3ba\" style=\"height: 680px; width: 1024px;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"07136567-341a-4e48-9dc9-89c8cb0ac3ba\", [{\"line\": {\"color\": \"rgb(34, 119, 158)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_x\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 0.9946109821746748, 0.9975093325168041, 1.0066779892487072, 1.0158140782874718, 1.0184570928355554, 1.0126420021399425, 1.0025373510794588, 0.9953109552792863, 0.995914225891269, 1.003879239590604, 1.0137490585799174, 1.0186090483879588, 1.014918874448412, 1.0053022411266337, 0.9966238356612817, 0.9948944471929325, 1.0012309144372078, 1.011292745135055, 1.0180893085876028, 1.016730447076507, 1.0081424789583326, 0.9984814392159677, 0.9945049293261299, 0.9988749773099755], \"type\": \"scatter\", \"uid\": \"9ebe9e11-bbe4-4204-8cd4-25f2ec4197e4\"}, {\"line\": {\"color\": \"rgb(201, 79, 14)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_y\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 0.9914853953160562, 0.981916696483938, 0.9778896222660368, 0.9821800877100572, 0.9919084628722946, 1.0002615619643, 1.0012481140530367, 0.994211310507671, 0.9842070236278863, 0.978178245055648, 0.9802545829481121, 0.989081693611526, 0.9985376770462829, 1.0018666901309434, 0.9967065348231313, 0.9868091726014248, 0.9791002271767438, 0.9788584654256741, 0.9863060843836177, 0.9963327122880606, 1.001818434483738, 0.9988318714335261, 0.9895837858368035, 0.9806056670831349], \"type\": \"scatter\", \"uid\": \"2b8ae7a5-dfd8-4d68-a90c-2f38e06d545b\"}, {\"line\": {\"color\": \"rgb(8, 173, 99)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_xy\", \"x\": [1.0, 0.9946109821746748, 0.9975093325168041, 1.0066779892487072, 1.0158140782874718, 1.0184570928355554, 1.0126420021399425, 1.0025373510794588, 0.9953109552792863, 0.995914225891269, 1.003879239590604, 1.0137490585799174, 1.0186090483879588, 1.014918874448412, 1.0053022411266337, 0.9966238356612817, 0.9948944471929325, 1.0012309144372078, 1.011292745135055, 1.0180893085876028, 1.016730447076507, 1.0081424789583326, 0.9984814392159677, 0.9945049293261299, 0.9988749773099755], \"xaxis\": \"x2\", \"y\": [1.0, 0.9914853953160562, 0.981916696483938, 0.9778896222660368, 0.9821800877100572, 0.9919084628722946, 1.0002615619643, 1.0012481140530367, 0.994211310507671, 0.9842070236278863, 0.978178245055648, 0.9802545829481121, 0.989081693611526, 0.9985376770462829, 1.0018666901309434, 0.9967065348231313, 0.9868091726014248, 0.9791002271767438, 0.9788584654256741, 0.9863060843836177, 0.9963327122880606, 1.001818434483738, 0.9988318714335261, 0.9895837858368035, 0.9806056670831349], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"fa701640-b8cf-4d93-bcec-23e0363f9234\"}, {\"line\": {\"color\": \"rgb(34, 119, 158)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_x\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], \"type\": \"scatter\", \"uid\": \"1f34addd-b703-4aeb-b15a-ce3acad16505\"}, {\"line\": {\"color\": \"rgb(201, 79, 14)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_y\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], \"type\": \"scatter\", \"uid\": \"7d44825a-ea60-4cf8-b6bc-d1e5da1945cb\"}, {\"line\": {\"color\": \"rgb(173, 7, 7)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_xy\", \"x\": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], \"xaxis\": \"x2\", \"y\": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"a42a42ec-e2ce-475f-b3bf-7d3e2956c687\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 1}, \"name\": \"current_x\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 1.0117713176740013, 1.0300400350849734, 1.0418947821442424, 1.0384167475453314, 1.0220589929329797, 1.00498974015452, 0.9993248119836119, 1.0087430239915052, 1.0267409502809268, 1.040713033361885, 1.040283721664198, 1.0255813654725876, 1.0075783980233675, 0.9992004988146628, 1.0060296164251512, 1.0232972709463908, 1.0390017884127052, 1.0416236726431536, 1.0289844113195854, 1.0105223748276346, 0.999631885119563, 1.0036983252880305, 1.0197992659413884, 1.0368105402739054], \"type\": \"scatter\", \"uid\": \"aca5b0ce-d8e5-49d9-81e5-8ef2e853394e\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 1}, \"name\": \"current_y\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 0.9863428420042899, 0.9864722401445015, 1.0004914128935396, 1.0186950365828926, 1.027704086271057, 1.0208273962262011, 1.0034081342724925, 0.988013330931901, 0.9852543067215627, 0.9971745902503322, 1.0156379039315484, 1.0271924700876494, 1.0232121255638968, 1.0068928745611894, 0.9901454106172497, 0.9845737819236131, 0.9941011034093465, 1.0123279510351868, 1.0261072640874267, 1.0251444511438081, 1.0103646435807363, 0.9926861846769144, 0.9844452222939443, 0.9913494422841072], \"type\": \"scatter\", \"uid\": \"6068be72-a79f-4235-9324-47620d6c7033\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 2}, \"name\": \"current_xy\", \"x\": [1.0, 1.0117713176740013, 1.0300400350849734, 1.0418947821442424, 1.0384167475453314, 1.0220589929329797, 1.00498974015452, 0.9993248119836119, 1.0087430239915052, 1.0267409502809268, 1.040713033361885, 1.040283721664198, 1.0255813654725876, 1.0075783980233675, 0.9992004988146628, 1.0060296164251512, 1.0232972709463908, 1.0390017884127052, 1.0416236726431536, 1.0289844113195854, 1.0105223748276346, 0.999631885119563, 1.0036983252880305, 1.0197992659413884, 1.0368105402739054], \"xaxis\": \"x2\", \"y\": [1.0, 0.9863428420042899, 0.9864722401445015, 1.0004914128935396, 1.0186950365828926, 1.027704086271057, 1.0208273962262011, 1.0034081342724925, 0.988013330931901, 0.9852543067215627, 0.9971745902503322, 1.0156379039315484, 1.0271924700876494, 1.0232121255638968, 1.0068928745611894, 0.9901454106172497, 0.9845737819236131, 0.9941011034093465, 1.0123279510351868, 1.0261072640874267, 1.0251444511438081, 1.0103646435807363, 0.9926861846769144, 0.9844452222939443, 0.9913494422841072], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"9b900117-64a9-40de-a390-5ec45f3cb71d\"}], {\"height\": 680, \"title\": {\"text\": \"Step 80<br>Best Loss 0.006856044573044227<br>Current Loss 0.022297664451166526\"}, \"width\": 1024, \"xaxis\": {\"range\": [-0.25, 10.25]}, \"yaxis\": {\"domain\": [0.55, 1.0], \"range\": [-0.25, 7.0]}, \"xaxis2\": {\"range\": [-0.25, 10.25]}, \"yaxis2\": {\"domain\": [0, 0.45], \"range\": [-0.25, 7.0]}}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"07136567-341a-4e48-9dc9-89c8cb0ac3ba\" style=\"height: 680px; width: 1024px;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"07136567-341a-4e48-9dc9-89c8cb0ac3ba\", [{\"line\": {\"color\": \"rgb(34, 119, 158)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_x\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 0.9946109821746748, 0.9975093325168041, 1.0066779892487072, 1.0158140782874718, 1.0184570928355554, 1.0126420021399425, 1.0025373510794588, 0.9953109552792863, 0.995914225891269, 1.003879239590604, 1.0137490585799174, 1.0186090483879588, 1.014918874448412, 1.0053022411266337, 0.9966238356612817, 0.9948944471929325, 1.0012309144372078, 1.011292745135055, 1.0180893085876028, 1.016730447076507, 1.0081424789583326, 0.9984814392159677, 0.9945049293261299, 0.9988749773099755], \"type\": \"scatter\", \"uid\": \"9ebe9e11-bbe4-4204-8cd4-25f2ec4197e4\"}, {\"line\": {\"color\": \"rgb(201, 79, 14)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_y\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 0.9914853953160562, 0.981916696483938, 0.9778896222660368, 0.9821800877100572, 0.9919084628722946, 1.0002615619643, 1.0012481140530367, 0.994211310507671, 0.9842070236278863, 0.978178245055648, 0.9802545829481121, 0.989081693611526, 0.9985376770462829, 1.0018666901309434, 0.9967065348231313, 0.9868091726014248, 0.9791002271767438, 0.9788584654256741, 0.9863060843836177, 0.9963327122880606, 1.001818434483738, 0.9988318714335261, 0.9895837858368035, 0.9806056670831349], \"type\": \"scatter\", \"uid\": \"2b8ae7a5-dfd8-4d68-a90c-2f38e06d545b\"}, {\"line\": {\"color\": \"rgb(8, 173, 99)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_xy\", \"x\": [1.0, 0.9946109821746748, 0.9975093325168041, 1.0066779892487072, 1.0158140782874718, 1.0184570928355554, 1.0126420021399425, 1.0025373510794588, 0.9953109552792863, 0.995914225891269, 1.003879239590604, 1.0137490585799174, 1.0186090483879588, 1.014918874448412, 1.0053022411266337, 0.9966238356612817, 0.9948944471929325, 1.0012309144372078, 1.011292745135055, 1.0180893085876028, 1.016730447076507, 1.0081424789583326, 0.9984814392159677, 0.9945049293261299, 0.9988749773099755], \"xaxis\": \"x2\", \"y\": [1.0, 0.9914853953160562, 0.981916696483938, 0.9778896222660368, 0.9821800877100572, 0.9919084628722946, 1.0002615619643, 1.0012481140530367, 0.994211310507671, 0.9842070236278863, 0.978178245055648, 0.9802545829481121, 0.989081693611526, 0.9985376770462829, 1.0018666901309434, 0.9967065348231313, 0.9868091726014248, 0.9791002271767438, 0.9788584654256741, 0.9863060843836177, 0.9963327122880606, 1.001818434483738, 0.9988318714335261, 0.9895837858368035, 0.9806056670831349], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"fa701640-b8cf-4d93-bcec-23e0363f9234\"}, {\"line\": {\"color\": \"rgb(34, 119, 158)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_x\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], \"type\": \"scatter\", \"uid\": \"1f34addd-b703-4aeb-b15a-ce3acad16505\"}, {\"line\": {\"color\": \"rgb(201, 79, 14)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_y\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], \"type\": \"scatter\", \"uid\": \"7d44825a-ea60-4cf8-b6bc-d1e5da1945cb\"}, {\"line\": {\"color\": \"rgb(173, 7, 7)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_xy\", \"x\": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], \"xaxis\": \"x2\", \"y\": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"a42a42ec-e2ce-475f-b3bf-7d3e2956c687\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 1}, \"name\": \"current_x\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 1.0117713176740013, 1.0300400350849734, 1.0418947821442424, 1.0384167475453314, 1.0220589929329797, 1.00498974015452, 0.9993248119836119, 1.0087430239915052, 1.0267409502809268, 1.040713033361885, 1.040283721664198, 1.0255813654725876, 1.0075783980233675, 0.9992004988146628, 1.0060296164251512, 1.0232972709463908, 1.0390017884127052, 1.0416236726431536, 1.0289844113195854, 1.0105223748276346, 0.999631885119563, 1.0036983252880305, 1.0197992659413884, 1.0368105402739054], \"type\": \"scatter\", \"uid\": \"aca5b0ce-d8e5-49d9-81e5-8ef2e853394e\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 1}, \"name\": \"current_y\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 0.9863428420042899, 0.9864722401445015, 1.0004914128935396, 1.0186950365828926, 1.027704086271057, 1.0208273962262011, 1.0034081342724925, 0.988013330931901, 0.9852543067215627, 0.9971745902503322, 1.0156379039315484, 1.0271924700876494, 1.0232121255638968, 1.0068928745611894, 0.9901454106172497, 0.9845737819236131, 0.9941011034093465, 1.0123279510351868, 1.0261072640874267, 1.0251444511438081, 1.0103646435807363, 0.9926861846769144, 0.9844452222939443, 0.9913494422841072], \"type\": \"scatter\", \"uid\": \"6068be72-a79f-4235-9324-47620d6c7033\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 2}, \"name\": \"current_xy\", \"x\": [1.0, 1.0117713176740013, 1.0300400350849734, 1.0418947821442424, 1.0384167475453314, 1.0220589929329797, 1.00498974015452, 0.9993248119836119, 1.0087430239915052, 1.0267409502809268, 1.040713033361885, 1.040283721664198, 1.0255813654725876, 1.0075783980233675, 0.9992004988146628, 1.0060296164251512, 1.0232972709463908, 1.0390017884127052, 1.0416236726431536, 1.0289844113195854, 1.0105223748276346, 0.999631885119563, 1.0036983252880305, 1.0197992659413884, 1.0368105402739054], \"xaxis\": \"x2\", \"y\": [1.0, 0.9863428420042899, 0.9864722401445015, 1.0004914128935396, 1.0186950365828926, 1.027704086271057, 1.0208273962262011, 1.0034081342724925, 0.988013330931901, 0.9852543067215627, 0.9971745902503322, 1.0156379039315484, 1.0271924700876494, 1.0232121255638968, 1.0068928745611894, 0.9901454106172497, 0.9845737819236131, 0.9941011034093465, 1.0123279510351868, 1.0261072640874267, 1.0251444511438081, 1.0103646435807363, 0.9926861846769144, 0.9844452222939443, 0.9913494422841072], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"9b900117-64a9-40de-a390-5ec45f3cb71d\"}], {\"height\": 680, \"title\": {\"text\": \"Step 80<br>Best Loss 0.006856044573044227<br>Current Loss 0.022297664451166526\"}, \"width\": 1024, \"xaxis\": {\"range\": [-0.25, 10.25]}, \"yaxis\": {\"domain\": [0.55, 1.0], \"range\": [-0.25, 7.0]}, \"xaxis2\": {\"range\": [-0.25, 10.25]}, \"yaxis2\": {\"domain\": [0, 0.45], \"range\": [-0.25, 7.0]}}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "optimize(lotka_volterra, ode_scipy_solver, loss_L2,\n",
    "         ground_truth_ones, z0, ts, args=args, variables=None, epochs=80,\n",
    "         y_axis_min=-0.25, y_axis_max=7.0,\n",
    "         x_axis_min=-0.25, x_axis_max=10.25,\n",
    "         output_dir=\"lotka_volterra_optim_scipy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TF Runge-Kutta ODE Solver (Diverging)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tensorflow Variables\n",
    "# the dimension of our input vector [x, y] => 2\n",
    "D = 2\n",
    "# time interval sampling\n",
    "N = 25\n",
    "ts = tf.constant(np.linspace(0., 10.0, N))\n",
    "# the arguments we want to optimize\n",
    "args = tf.Variable(tf.constant([1.5, 1.0, 3.0, 1.0], shape=(4,), dtype=ts.dtype))\n",
    "\n",
    "# the initial input value\n",
    "z0 = tf.Variable(tf.constant([1.0,1.0], shape=(2,), dtype=ts.dtype))\n",
    "\n",
    "def ode_tf_solver(f, z0, ts, args):\n",
    "    ts = tf.contrib.framework.sort(ts)\n",
    "    r = ode_tf(f, z0, ts, args)\n",
    "    r = tf.cast(r, z0.dtype)\n",
    "    return r\n",
    "\n",
    "# our ground_truth (all 1.0)\n",
    "ground_truth_ones = tf.ones(shape=(N, D), dtype=z0.dtype)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training Video\n",
    "\n",
    ">Look how it diverges at a first period of optimization.\n",
    ">This is due to computational instability of ODE Solver Runge-Kutta Dopri5 method.\n",
    ">That's why in next parts of this notebook, I'll use on Scipy LSODA implementation that is more much stable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<video width=\"640\" height=\"480\" controls>\n",
       "  <source src=\"lotka_volterra_optim_tf/video.mp4\" type=\"video/mp4\">\n",
       "</video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "\n",
    "HTML(\"\"\"\n",
    "<video width=\"640\" height=\"480\" controls>\n",
    "  <source src=\"lotka_volterra_optim_tf/video.mp4\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Running Training by yourself"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "linkText": "Export to plot.ly",
        "plotlyServerURL": "https://plot.ly",
        "showLink": false
       },
       "data": [
        {
         "line": {
          "color": "rgb(34, 119, 158)",
          "width": 1
         },
         "mode": "lines+markers",
         "name": "best_x",
         "type": "scatter",
         "uid": "df2ab195-bf34-4670-baa9-91e2b5791a5a",
         "x": [
          0,
          0.4166666666666667,
          0.8333333333333334,
          1.25,
          1.6666666666666667,
          2.0833333333333335,
          2.5,
          2.916666666666667,
          3.3333333333333335,
          3.75,
          4.166666666666667,
          4.583333333333334,
          5,
          5.416666666666667,
          5.833333333333334,
          6.25,
          6.666666666666667,
          7.083333333333334,
          7.5,
          7.916666666666667,
          8.333333333333334,
          8.75,
          9.166666666666668,
          9.583333333333334,
          10
         ],
         "y": [
          1,
          0.5469861409879883,
          0.4623286544537798,
          0.4526677958312663,
          0.4662688341077791,
          0.48848162824025926,
          0.5146538531149325,
          0.5432568000403785,
          0.5738164051059214,
          0.6062271273290543,
          0.640516462393733,
          0.6767628641021278,
          0.7150669512758806,
          0.7555414579376466,
          0.7983078476305732,
          0.8434953271664158,
          0.8912407412893579,
          0.9416888032860075,
          0.9949924646995688,
          1.0513133529320395,
          1.110822251320456,
          1.1736996132552628,
          1.240136107748398,
          1.3103331973757084,
          1.3845037456516083
         ]
        },
        {
         "line": {
          "color": "rgb(201, 79, 14)",
          "width": 1
         },
         "mode": "lines+markers",
         "name": "best_y",
         "type": "scatter",
         "uid": "cd6349e1-6248-4ea9-a366-c5f4c996c1b3",
         "x": [
          0,
          0.4166666666666667,
          0.8333333333333334,
          1.25,
          1.6666666666666667,
          2.0833333333333335,
          2.5,
          2.916666666666667,
          3.3333333333333335,
          3.75,
          4.166666666666667,
          4.583333333333334,
          5,
          5.416666666666667,
          5.833333333333334,
          6.25,
          6.666666666666667,
          7.083333333333334,
          7.5,
          7.916666666666667,
          8.333333333333334,
          8.75,
          9.166666666666668,
          9.583333333333334,
          10
         ],
         "y": [
          1,
          0.3602348484841792,
          0.12104091734062972,
          0.040288902510318095,
          0.013436212909988372,
          0.004506260536340903,
          0.001522929741763787,
          0.0005192483769064531,
          0.000178747349231545,
          6.216671368853992e-05,
          2.185736416232387e-05,
          7.773682552775823e-06,
          2.7984762192557576e-06,
          1.0204082027134862e-06,
          3.771329684189433e-07,
          1.4138906203008268e-07,
          5.3813826838374516e-08,
          2.0811758369952916e-08,
          8.185858906830021e-09,
          3.2778308957567594e-09,
          1.3375641527345213e-09,
          5.565837347585952e-10,
          2.3610849320297715e-10,
          1.0237534429825102e-10,
          4.581466846371736e-11
         ]
        },
        {
         "line": {
          "color": "rgb(8, 173, 99)",
          "width": 1
         },
         "mode": "lines+markers",
         "name": "best_xy",
         "type": "scatter",
         "uid": "23b5bad1-1dca-462e-b1c9-2c12ececd438",
         "x": [
          1,
          0.5469861409879883,
          0.4623286544537798,
          0.4526677958312663,
          0.4662688341077791,
          0.48848162824025926,
          0.5146538531149325,
          0.5432568000403785,
          0.5738164051059214,
          0.6062271273290543,
          0.640516462393733,
          0.6767628641021278,
          0.7150669512758806,
          0.7555414579376466,
          0.7983078476305732,
          0.8434953271664158,
          0.8912407412893579,
          0.9416888032860075,
          0.9949924646995688,
          1.0513133529320395,
          1.110822251320456,
          1.1736996132552628,
          1.240136107748398,
          1.3103331973757084,
          1.3845037456516083
         ],
         "xaxis": "x2",
         "y": [
          1,
          0.3602348484841792,
          0.12104091734062972,
          0.040288902510318095,
          0.013436212909988372,
          0.004506260536340903,
          0.001522929741763787,
          0.0005192483769064531,
          0.000178747349231545,
          6.216671368853992e-05,
          2.185736416232387e-05,
          7.773682552775823e-06,
          2.7984762192557576e-06,
          1.0204082027134862e-06,
          3.771329684189433e-07,
          1.4138906203008268e-07,
          5.3813826838374516e-08,
          2.0811758369952916e-08,
          8.185858906830021e-09,
          3.2778308957567594e-09,
          1.3375641527345213e-09,
          5.565837347585952e-10,
          2.3610849320297715e-10,
          1.0237534429825102e-10,
          4.581466846371736e-11
         ],
         "yaxis": "y2"
        },
        {
         "line": {
          "color": "rgb(34, 119, 158)",
          "dash": "dot",
          "width": 2
         },
         "name": "groundtruth_x",
         "type": "scatter",
         "uid": "7f69b4bd-3aec-4069-9576-af1dc07221e5",
         "x": [
          0,
          0.4166666666666667,
          0.8333333333333334,
          1.25,
          1.6666666666666667,
          2.0833333333333335,
          2.5,
          2.916666666666667,
          3.3333333333333335,
          3.75,
          4.166666666666667,
          4.583333333333334,
          5,
          5.416666666666667,
          5.833333333333334,
          6.25,
          6.666666666666667,
          7.083333333333334,
          7.5,
          7.916666666666667,
          8.333333333333334,
          8.75,
          9.166666666666668,
          9.583333333333334,
          10
         ],
         "y": [
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1
         ]
        },
        {
         "line": {
          "color": "rgb(201, 79, 14)",
          "dash": "dot",
          "width": 2
         },
         "name": "groundtruth_y",
         "type": "scatter",
         "uid": "a91f7560-3ff5-4936-811f-bbeecf1be8b3",
         "x": [
          0,
          0.4166666666666667,
          0.8333333333333334,
          1.25,
          1.6666666666666667,
          2.0833333333333335,
          2.5,
          2.916666666666667,
          3.3333333333333335,
          3.75,
          4.166666666666667,
          4.583333333333334,
          5,
          5.416666666666667,
          5.833333333333334,
          6.25,
          6.666666666666667,
          7.083333333333334,
          7.5,
          7.916666666666667,
          8.333333333333334,
          8.75,
          9.166666666666668,
          9.583333333333334,
          10
         ],
         "y": [
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1
         ]
        },
        {
         "line": {
          "color": "rgb(173, 7, 7)",
          "dash": "dot",
          "width": 2
         },
         "name": "groundtruth_xy",
         "type": "scatter",
         "uid": "caa562c7-d09b-4468-9b59-5241622bf6f2",
         "x": [
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1
         ],
         "xaxis": "x2",
         "y": [
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1,
          1
         ],
         "yaxis": "y2"
        },
        {
         "line": {
          "color": "rgb(24, 24, 24)",
          "dash": "dash",
          "width": 1
         },
         "name": "current_x",
         "type": "scatter",
         "uid": "51feedb8-e20f-45e4-93ac-86745b6f93af",
         "x": [
          0,
          0.4166666666666667,
          0.8333333333333334,
          1.25,
          1.6666666666666667,
          2.0833333333333335,
          2.5,
          2.916666666666667,
          3.3333333333333335,
          3.75,
          4.166666666666667,
          4.583333333333334,
          5,
          5.416666666666667,
          5.833333333333334,
          6.25,
          6.666666666666667,
          7.083333333333334,
          7.5,
          7.916666666666667,
          8.333333333333334,
          8.75,
          9.166666666666668,
          9.583333333333334,
          10
         ],
         "y": [
          1,
          0.7337795429591185,
          0.7758382384828106,
          0.9100954571979445,
          1.0996463408033195,
          1.3400469269078639,
          1.6371341722459787,
          2.0016426626220984,
          2.4479229458239775,
          2.9939591619785966,
          3.661906935982877,
          4.47892588160893,
          5.478264866697493,
          6.700575696827205,
          8.195635872005138,
          10.024246777837735,
          12.260857570689051,
          14.99635557867744,
          18.341374724959902,
          22.42363398970523,
          27.213219192265697,
          24.685173910242014,
          2.9934470415287606,
          0.8669382254636931,
          0.698902712235221
         ]
        },
        {
         "line": {
          "color": "rgb(24, 24, 24)",
          "dash": "dash",
          "width": 1
         },
         "name": "current_y",
         "type": "scatter",
         "uid": "48c86c8a-55c4-4af0-a32e-e60febca3e9d",
         "x": [
          0,
          0.4166666666666667,
          0.8333333333333334,
          1.25,
          1.6666666666666667,
          2.0833333333333335,
          2.5,
          2.916666666666667,
          3.3333333333333335,
          3.75,
          4.166666666666667,
          4.583333333333334,
          5,
          5.416666666666667,
          5.833333333333334,
          6.25,
          6.666666666666667,
          7.083333333333334,
          7.5,
          7.916666666666667,
          8.333333333333334,
          8.75,
          9.166666666666668,
          9.583333333333334,
          10
         ],
         "y": [
          1,
          0.2874407376688374,
          0.08167007770507945,
          0.023596314399726284,
          0.007029377251898545,
          0.002179339444101176,
          0.000709212144325492,
          0.00024484701951870867,
          9.09518479535417e-05,
          3.696176644911109e-05,
          1.676333544291322e-05,
          8.695060343488018e-06,
          5.316125414361768e-06,
          3.98034021384626e-06,
          3.8127666514015385e-06,
          4.918194170876157e-06,
          9.237797327599345e-06,
          2.701466297050206e-05,
          0.000137138211042569,
          0.0013705944658042534,
          0.0302926286127583,
          1.221939990755047,
          2.524896839880646,
          0.8266843090826341,
          0.23459333774400026
         ]
        },
        {
         "line": {
          "color": "rgb(24, 24, 24)",
          "dash": "dash",
          "width": 2
         },
         "name": "current_xy",
         "type": "scatter",
         "uid": "fdf2fbde-4ecd-445c-86d9-b24f447fb889",
         "x": [
          1,
          0.7337795429591185,
          0.7758382384828106,
          0.9100954571979445,
          1.0996463408033195,
          1.3400469269078639,
          1.6371341722459787,
          2.0016426626220984,
          2.4479229458239775,
          2.9939591619785966,
          3.661906935982877,
          4.47892588160893,
          5.478264866697493,
          6.700575696827205,
          8.195635872005138,
          10.024246777837735,
          12.260857570689051,
          14.99635557867744,
          18.341374724959902,
          22.42363398970523,
          27.213219192265697,
          24.685173910242014,
          2.9934470415287606,
          0.8669382254636931,
          0.698902712235221
         ],
         "xaxis": "x2",
         "y": [
          1,
          0.2874407376688374,
          0.08167007770507945,
          0.023596314399726284,
          0.007029377251898545,
          0.002179339444101176,
          0.000709212144325492,
          0.00024484701951870867,
          9.09518479535417e-05,
          3.696176644911109e-05,
          1.676333544291322e-05,
          8.695060343488018e-06,
          5.316125414361768e-06,
          3.98034021384626e-06,
          3.8127666514015385e-06,
          4.918194170876157e-06,
          9.237797327599345e-06,
          2.701466297050206e-05,
          0.000137138211042569,
          0.0013705944658042534,
          0.0302926286127583,
          1.221939990755047,
          2.524896839880646,
          0.8266843090826341,
          0.23459333774400026
         ],
         "yaxis": "y2"
        }
       ],
       "layout": {
        "height": 680,
        "title": {
         "text": "Step 150<br>Best Loss 25.985853594762723<br>Current Loss 2569.504940671772"
        },
        "width": 1024,
        "xaxis": {
         "range": [
          -0.25,
          10.25
         ]
        },
        "xaxis2": {
         "range": [
          -0.25,
          10.25
         ]
        },
        "yaxis": {
         "domain": [
          0.55,
          1
         ],
         "range": [
          -0.25,
          7
         ]
        },
        "yaxis2": {
         "domain": [
          0,
          0.45
         ],
         "range": [
          -0.25,
          7
         ]
        }
       }
      },
      "text/html": [
       "<div id=\"ffc227be-90d4-478f-955f-84a96a583bd2\" style=\"height: 680px; width: 1024px;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"ffc227be-90d4-478f-955f-84a96a583bd2\", [{\"line\": {\"color\": \"rgb(34, 119, 158)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_x\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 0.5469861409879883, 0.4623286544537798, 0.4526677958312663, 0.4662688341077791, 0.48848162824025926, 0.5146538531149325, 0.5432568000403785, 0.5738164051059214, 0.6062271273290543, 0.640516462393733, 0.6767628641021278, 0.7150669512758806, 0.7555414579376466, 0.7983078476305732, 0.8434953271664158, 0.8912407412893579, 0.9416888032860075, 0.9949924646995688, 1.0513133529320395, 1.110822251320456, 1.1736996132552628, 1.240136107748398, 1.3103331973757084, 1.3845037456516083], \"type\": \"scatter\", \"uid\": \"df2ab195-bf34-4670-baa9-91e2b5791a5a\"}, {\"line\": {\"color\": \"rgb(201, 79, 14)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_y\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 0.3602348484841792, 0.12104091734062972, 0.040288902510318095, 0.013436212909988372, 0.004506260536340903, 0.001522929741763787, 0.0005192483769064531, 0.000178747349231545, 6.216671368853992e-05, 2.185736416232387e-05, 7.773682552775823e-06, 2.7984762192557576e-06, 1.0204082027134862e-06, 3.771329684189433e-07, 1.4138906203008268e-07, 5.3813826838374516e-08, 2.0811758369952916e-08, 8.185858906830021e-09, 3.2778308957567594e-09, 1.3375641527345213e-09, 5.565837347585952e-10, 2.3610849320297715e-10, 1.0237534429825102e-10, 4.581466846371736e-11], \"type\": \"scatter\", \"uid\": \"cd6349e1-6248-4ea9-a366-c5f4c996c1b3\"}, {\"line\": {\"color\": \"rgb(8, 173, 99)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_xy\", \"x\": [1.0, 0.5469861409879883, 0.4623286544537798, 0.4526677958312663, 0.4662688341077791, 0.48848162824025926, 0.5146538531149325, 0.5432568000403785, 0.5738164051059214, 0.6062271273290543, 0.640516462393733, 0.6767628641021278, 0.7150669512758806, 0.7555414579376466, 0.7983078476305732, 0.8434953271664158, 0.8912407412893579, 0.9416888032860075, 0.9949924646995688, 1.0513133529320395, 1.110822251320456, 1.1736996132552628, 1.240136107748398, 1.3103331973757084, 1.3845037456516083], \"xaxis\": \"x2\", \"y\": [1.0, 0.3602348484841792, 0.12104091734062972, 0.040288902510318095, 0.013436212909988372, 0.004506260536340903, 0.001522929741763787, 0.0005192483769064531, 0.000178747349231545, 6.216671368853992e-05, 2.185736416232387e-05, 7.773682552775823e-06, 2.7984762192557576e-06, 1.0204082027134862e-06, 3.771329684189433e-07, 1.4138906203008268e-07, 5.3813826838374516e-08, 2.0811758369952916e-08, 8.185858906830021e-09, 3.2778308957567594e-09, 1.3375641527345213e-09, 5.565837347585952e-10, 2.3610849320297715e-10, 1.0237534429825102e-10, 4.581466846371736e-11], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"23b5bad1-1dca-462e-b1c9-2c12ececd438\"}, {\"line\": {\"color\": \"rgb(34, 119, 158)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_x\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], \"type\": \"scatter\", \"uid\": \"7f69b4bd-3aec-4069-9576-af1dc07221e5\"}, {\"line\": {\"color\": \"rgb(201, 79, 14)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_y\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], \"type\": \"scatter\", \"uid\": \"a91f7560-3ff5-4936-811f-bbeecf1be8b3\"}, {\"line\": {\"color\": \"rgb(173, 7, 7)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_xy\", \"x\": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], \"xaxis\": \"x2\", \"y\": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"caa562c7-d09b-4468-9b59-5241622bf6f2\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 1}, \"name\": \"current_x\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 0.7337795429591185, 0.7758382384828106, 0.9100954571979445, 1.0996463408033195, 1.3400469269078639, 1.6371341722459787, 2.0016426626220984, 2.4479229458239775, 2.9939591619785966, 3.661906935982877, 4.47892588160893, 5.478264866697493, 6.700575696827205, 8.195635872005138, 10.024246777837735, 12.260857570689051, 14.99635557867744, 18.341374724959902, 22.42363398970523, 27.213219192265697, 24.685173910242014, 2.9934470415287606, 0.8669382254636931, 0.698902712235221], \"type\": \"scatter\", \"uid\": \"51feedb8-e20f-45e4-93ac-86745b6f93af\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 1}, \"name\": \"current_y\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 0.2874407376688374, 0.08167007770507945, 0.023596314399726284, 0.007029377251898545, 0.002179339444101176, 0.000709212144325492, 0.00024484701951870867, 9.09518479535417e-05, 3.696176644911109e-05, 1.676333544291322e-05, 8.695060343488018e-06, 5.316125414361768e-06, 3.98034021384626e-06, 3.8127666514015385e-06, 4.918194170876157e-06, 9.237797327599345e-06, 2.701466297050206e-05, 0.000137138211042569, 0.0013705944658042534, 0.0302926286127583, 1.221939990755047, 2.524896839880646, 0.8266843090826341, 0.23459333774400026], \"type\": \"scatter\", \"uid\": \"48c86c8a-55c4-4af0-a32e-e60febca3e9d\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 2}, \"name\": \"current_xy\", \"x\": [1.0, 0.7337795429591185, 0.7758382384828106, 0.9100954571979445, 1.0996463408033195, 1.3400469269078639, 1.6371341722459787, 2.0016426626220984, 2.4479229458239775, 2.9939591619785966, 3.661906935982877, 4.47892588160893, 5.478264866697493, 6.700575696827205, 8.195635872005138, 10.024246777837735, 12.260857570689051, 14.99635557867744, 18.341374724959902, 22.42363398970523, 27.213219192265697, 24.685173910242014, 2.9934470415287606, 0.8669382254636931, 0.698902712235221], \"xaxis\": \"x2\", \"y\": [1.0, 0.2874407376688374, 0.08167007770507945, 0.023596314399726284, 0.007029377251898545, 0.002179339444101176, 0.000709212144325492, 0.00024484701951870867, 9.09518479535417e-05, 3.696176644911109e-05, 1.676333544291322e-05, 8.695060343488018e-06, 5.316125414361768e-06, 3.98034021384626e-06, 3.8127666514015385e-06, 4.918194170876157e-06, 9.237797327599345e-06, 2.701466297050206e-05, 0.000137138211042569, 0.0013705944658042534, 0.0302926286127583, 1.221939990755047, 2.524896839880646, 0.8266843090826341, 0.23459333774400026], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"fdf2fbde-4ecd-445c-86d9-b24f447fb889\"}], {\"height\": 680, \"title\": {\"text\": \"Step 150<br>Best Loss 25.985853594762723<br>Current Loss 2569.504940671772\"}, \"width\": 1024, \"xaxis\": {\"range\": [-0.25, 10.25]}, \"yaxis\": {\"domain\": [0.55, 1.0], \"range\": [-0.25, 7.0]}, \"xaxis2\": {\"range\": [-0.25, 10.25]}, \"yaxis2\": {\"domain\": [0, 0.45], \"range\": [-0.25, 7.0]}}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"ffc227be-90d4-478f-955f-84a96a583bd2\" style=\"height: 680px; width: 1024px;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"ffc227be-90d4-478f-955f-84a96a583bd2\", [{\"line\": {\"color\": \"rgb(34, 119, 158)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_x\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 0.5469861409879883, 0.4623286544537798, 0.4526677958312663, 0.4662688341077791, 0.48848162824025926, 0.5146538531149325, 0.5432568000403785, 0.5738164051059214, 0.6062271273290543, 0.640516462393733, 0.6767628641021278, 0.7150669512758806, 0.7555414579376466, 0.7983078476305732, 0.8434953271664158, 0.8912407412893579, 0.9416888032860075, 0.9949924646995688, 1.0513133529320395, 1.110822251320456, 1.1736996132552628, 1.240136107748398, 1.3103331973757084, 1.3845037456516083], \"type\": \"scatter\", \"uid\": \"df2ab195-bf34-4670-baa9-91e2b5791a5a\"}, {\"line\": {\"color\": \"rgb(201, 79, 14)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_y\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 0.3602348484841792, 0.12104091734062972, 0.040288902510318095, 0.013436212909988372, 0.004506260536340903, 0.001522929741763787, 0.0005192483769064531, 0.000178747349231545, 6.216671368853992e-05, 2.185736416232387e-05, 7.773682552775823e-06, 2.7984762192557576e-06, 1.0204082027134862e-06, 3.771329684189433e-07, 1.4138906203008268e-07, 5.3813826838374516e-08, 2.0811758369952916e-08, 8.185858906830021e-09, 3.2778308957567594e-09, 1.3375641527345213e-09, 5.565837347585952e-10, 2.3610849320297715e-10, 1.0237534429825102e-10, 4.581466846371736e-11], \"type\": \"scatter\", \"uid\": \"cd6349e1-6248-4ea9-a366-c5f4c996c1b3\"}, {\"line\": {\"color\": \"rgb(8, 173, 99)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_xy\", \"x\": [1.0, 0.5469861409879883, 0.4623286544537798, 0.4526677958312663, 0.4662688341077791, 0.48848162824025926, 0.5146538531149325, 0.5432568000403785, 0.5738164051059214, 0.6062271273290543, 0.640516462393733, 0.6767628641021278, 0.7150669512758806, 0.7555414579376466, 0.7983078476305732, 0.8434953271664158, 0.8912407412893579, 0.9416888032860075, 0.9949924646995688, 1.0513133529320395, 1.110822251320456, 1.1736996132552628, 1.240136107748398, 1.3103331973757084, 1.3845037456516083], \"xaxis\": \"x2\", \"y\": [1.0, 0.3602348484841792, 0.12104091734062972, 0.040288902510318095, 0.013436212909988372, 0.004506260536340903, 0.001522929741763787, 0.0005192483769064531, 0.000178747349231545, 6.216671368853992e-05, 2.185736416232387e-05, 7.773682552775823e-06, 2.7984762192557576e-06, 1.0204082027134862e-06, 3.771329684189433e-07, 1.4138906203008268e-07, 5.3813826838374516e-08, 2.0811758369952916e-08, 8.185858906830021e-09, 3.2778308957567594e-09, 1.3375641527345213e-09, 5.565837347585952e-10, 2.3610849320297715e-10, 1.0237534429825102e-10, 4.581466846371736e-11], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"23b5bad1-1dca-462e-b1c9-2c12ececd438\"}, {\"line\": {\"color\": \"rgb(34, 119, 158)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_x\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], \"type\": \"scatter\", \"uid\": \"7f69b4bd-3aec-4069-9576-af1dc07221e5\"}, {\"line\": {\"color\": \"rgb(201, 79, 14)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_y\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], \"type\": \"scatter\", \"uid\": \"a91f7560-3ff5-4936-811f-bbeecf1be8b3\"}, {\"line\": {\"color\": \"rgb(173, 7, 7)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_xy\", \"x\": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], \"xaxis\": \"x2\", \"y\": [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"caa562c7-d09b-4468-9b59-5241622bf6f2\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 1}, \"name\": \"current_x\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 0.7337795429591185, 0.7758382384828106, 0.9100954571979445, 1.0996463408033195, 1.3400469269078639, 1.6371341722459787, 2.0016426626220984, 2.4479229458239775, 2.9939591619785966, 3.661906935982877, 4.47892588160893, 5.478264866697493, 6.700575696827205, 8.195635872005138, 10.024246777837735, 12.260857570689051, 14.99635557867744, 18.341374724959902, 22.42363398970523, 27.213219192265697, 24.685173910242014, 2.9934470415287606, 0.8669382254636931, 0.698902712235221], \"type\": \"scatter\", \"uid\": \"51feedb8-e20f-45e4-93ac-86745b6f93af\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 1}, \"name\": \"current_y\", \"x\": [0.0, 0.4166666666666667, 0.8333333333333334, 1.25, 1.6666666666666667, 2.0833333333333335, 2.5, 2.916666666666667, 3.3333333333333335, 3.75, 4.166666666666667, 4.583333333333334, 5.0, 5.416666666666667, 5.833333333333334, 6.25, 6.666666666666667, 7.083333333333334, 7.5, 7.916666666666667, 8.333333333333334, 8.75, 9.166666666666668, 9.583333333333334, 10.0], \"y\": [1.0, 0.2874407376688374, 0.08167007770507945, 0.023596314399726284, 0.007029377251898545, 0.002179339444101176, 0.000709212144325492, 0.00024484701951870867, 9.09518479535417e-05, 3.696176644911109e-05, 1.676333544291322e-05, 8.695060343488018e-06, 5.316125414361768e-06, 3.98034021384626e-06, 3.8127666514015385e-06, 4.918194170876157e-06, 9.237797327599345e-06, 2.701466297050206e-05, 0.000137138211042569, 0.0013705944658042534, 0.0302926286127583, 1.221939990755047, 2.524896839880646, 0.8266843090826341, 0.23459333774400026], \"type\": \"scatter\", \"uid\": \"48c86c8a-55c4-4af0-a32e-e60febca3e9d\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 2}, \"name\": \"current_xy\", \"x\": [1.0, 0.7337795429591185, 0.7758382384828106, 0.9100954571979445, 1.0996463408033195, 1.3400469269078639, 1.6371341722459787, 2.0016426626220984, 2.4479229458239775, 2.9939591619785966, 3.661906935982877, 4.47892588160893, 5.478264866697493, 6.700575696827205, 8.195635872005138, 10.024246777837735, 12.260857570689051, 14.99635557867744, 18.341374724959902, 22.42363398970523, 27.213219192265697, 24.685173910242014, 2.9934470415287606, 0.8669382254636931, 0.698902712235221], \"xaxis\": \"x2\", \"y\": [1.0, 0.2874407376688374, 0.08167007770507945, 0.023596314399726284, 0.007029377251898545, 0.002179339444101176, 0.000709212144325492, 0.00024484701951870867, 9.09518479535417e-05, 3.696176644911109e-05, 1.676333544291322e-05, 8.695060343488018e-06, 5.316125414361768e-06, 3.98034021384626e-06, 3.8127666514015385e-06, 4.918194170876157e-06, 9.237797327599345e-06, 2.701466297050206e-05, 0.000137138211042569, 0.0013705944658042534, 0.0302926286127583, 1.221939990755047, 2.524896839880646, 0.8266843090826341, 0.23459333774400026], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"fdf2fbde-4ecd-445c-86d9-b24f447fb889\"}], {\"height\": 680, \"title\": {\"text\": \"Step 150<br>Best Loss 25.985853594762723<br>Current Loss 2569.504940671772\"}, \"width\": 1024, \"xaxis\": {\"range\": [-0.25, 10.25]}, \"yaxis\": {\"domain\": [0.55, 1.0], \"range\": [-0.25, 7.0]}, \"xaxis2\": {\"range\": [-0.25, 10.25]}, \"yaxis2\": {\"domain\": [0, 0.45], \"range\": [-0.25, 7.0]}}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "optimize(lotka_volterra, ode_tf_solver, loss_L2,\n",
    "         ground_truth_ones, z0, ts, args=args, variables=None, epochs=150,\n",
    "         y_axis_min=-0.25, y_axis_max=7.0,\n",
    "         x_axis_min=-0.25, x_axis_max=10.25,\n",
    "         output_dir=\"lotka_volterra_optim_tf\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final Step: learning time-series with Neural ODE\n",
    "\n",
    "Finally, let's try to learn some time-series function with our neural ODE to check paper assertions are not just imaginary.\n",
    "\n",
    "We are going to model derivative of our function with a classic one-layer dense neural layer as shown in https://julialang.org/blog/2019/01/fluxdiffeq.\n",
    "\n",
    "Then we compute our ground-truth using ODE-Solver from true derivative, decide to use L2 loss.\n",
    "\n",
    "Finally, we launch iterative optimization process using ODE augmented state gradient descent and observe what happens...\n",
    "\n",
    "I won't launch it with Runge-Kutta as it is much less stable than scipy LSODA... tensors are flowing from GPU to CPU but it's OK for current experimentations...\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning a piece of lotka_volterra curve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "D = 2\n",
    "N = 20\n",
    "ts = tf.constant(np.linspace(0., 3.5, N))\n",
    "args = tf.Variable(tf.constant([1.5, 1.0, 3.0, 1.0], shape=(4,), dtype=ts.dtype))\n",
    "z0 = tf.Variable(tf.constant([1.0, 1.0], shape=(2,), dtype=ts.dtype))\n",
    "\n",
    "model = tf.keras.Sequential([\n",
    "  tf.keras.layers.Dense(25, activation=tf.nn.relu, input_shape=(1, D)),  # input shape required\n",
    "  tf.keras.layers.Dense(D)\n",
    "])\n",
    "\n",
    "def ode_scipy_solver(f, z0, ts, args):\n",
    "    r = ode_scipy(f, z0, ts, args)\n",
    "    r = tf.cast(r, z0.dtype)\n",
    "    return r\n",
    "\n",
    "def predict_fn(y, ts, args):\n",
    "    y2 = tf.reshape(y, [1, D])\n",
    "    y2 = tf.cast(y2, tf.float32)\n",
    "    r = model(y2)\n",
    "    r = tf.squeeze(r)\n",
    "    r = tf.cast(r, y.dtype)\n",
    "    return r\n",
    "\n",
    "ground_truth = ode_scipy(lotka_volterra, z0, ts, args)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For the lazy, video of training\n",
    "\n",
    "Run next cell to see video of a 500 epochs training I did on CPU/GPU (starting at epoch 80 because the rest was about the same).\n",
    "It's quite interesting to see how it learns the curve starting from a kind of average curve and then trying to fit curves progressively...\n",
    "\n",
    "Conclusion: Neural ODE is able to learn a tiny time-series (and certainly overfit a lot on it ;))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<video width=\"800\" height=\"600\" controls>\n",
       "  <source src=\"ode_dense_lotka_volterra/video.mp4\" type=\"video/mp4\">\n",
       "</video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "\n",
    "HTML(\"\"\"\n",
    "<video width=\"800\" height=\"600\" controls>\n",
    "  <source src=\"ode_dense_lotka_volterra/video.mp4\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Running the experiment by yourself\n",
    "\n",
    "> On a 1080TI, it takes around 3h for 1000 epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "linkText": "Export to plot.ly",
        "plotlyServerURL": "https://plot.ly",
        "showLink": false
       },
       "data": [
        {
         "line": {
          "color": "rgb(34, 119, 158)",
          "width": 1
         },
         "mode": "lines+markers",
         "name": "best_x",
         "type": "scatter",
         "uid": "58b5ec85-e256-4e30-9426-ea5728d65105",
         "x": [
          0,
          0.18421052631578946,
          0.3684210526315789,
          0.5526315789473684,
          0.7368421052631579,
          0.9210526315789473,
          1.1052631578947367,
          1.2894736842105263,
          1.4736842105263157,
          1.657894736842105,
          1.8421052631578947,
          2.026315789473684,
          2.2105263157894735,
          2.394736842105263,
          2.5789473684210527,
          2.763157894736842,
          2.9473684210526314,
          3.131578947368421,
          3.31578947368421,
          3.5
         ],
         "y": [
          1,
          1.1755901853819541,
          1.3956450781676293,
          1.6354558338646423,
          1.9407131519972247,
          2.400253696324886,
          3.0333885676598182,
          3.861676448146876,
          4.9093531429981905,
          6.060548457596613,
          6.758444020373528,
          6.5248919054295245,
          5.118528800044711,
          3.136228525384952,
          1.6249190793125918,
          0.9809208080671945,
          0.9174299636625924,
          1.0330181993423149,
          1.1625200045493267,
          1.3528257966065576
         ]
        },
        {
         "line": {
          "color": "rgb(201, 79, 14)",
          "width": 1
         },
         "mode": "lines+markers",
         "name": "best_y",
         "type": "scatter",
         "uid": "762c0202-a34c-4d91-ad5a-e58d65256f9a",
         "x": [
          0,
          0.18421052631578946,
          0.3684210526315789,
          0.5526315789473684,
          0.7368421052631579,
          0.9210526315789473,
          1.1052631578947367,
          1.2894736842105263,
          1.4736842105263157,
          1.657894736842105,
          1.8421052631578947,
          2.026315789473684,
          2.2105263157894735,
          2.394736842105263,
          2.5789473684210527,
          2.763157894736842,
          2.9473684210526314,
          3.131578947368421,
          3.31578947368421,
          3.5
         ],
         "y": [
          1,
          0.7180437183577189,
          0.5222963092798567,
          0.35996607896938,
          0.22095375967087919,
          0.13678908000334467,
          0.11433708690096386,
          0.1612618891272612,
          0.2861793034059818,
          0.5934998305722174,
          1.3214631271457367,
          2.4224059479048172,
          3.6825755841001153,
          4.522138679695391,
          4.1734489537542165,
          3.095683500689061,
          2.039841890017843,
          1.3775663666847708,
          0.9891630015578474,
          0.737146604224775
         ]
        },
        {
         "line": {
          "color": "rgb(8, 173, 99)",
          "width": 1
         },
         "mode": "lines+markers",
         "name": "best_xy",
         "type": "scatter",
         "uid": "447100f1-a8f5-47be-9011-8e116d0f8efc",
         "x": [
          1,
          1.1755901853819541,
          1.3956450781676293,
          1.6354558338646423,
          1.9407131519972247,
          2.400253696324886,
          3.0333885676598182,
          3.861676448146876,
          4.9093531429981905,
          6.060548457596613,
          6.758444020373528,
          6.5248919054295245,
          5.118528800044711,
          3.136228525384952,
          1.6249190793125918,
          0.9809208080671945,
          0.9174299636625924,
          1.0330181993423149,
          1.1625200045493267,
          1.3528257966065576
         ],
         "xaxis": "x2",
         "y": [
          1,
          0.7180437183577189,
          0.5222963092798567,
          0.35996607896938,
          0.22095375967087919,
          0.13678908000334467,
          0.11433708690096386,
          0.1612618891272612,
          0.2861793034059818,
          0.5934998305722174,
          1.3214631271457367,
          2.4224059479048172,
          3.6825755841001153,
          4.522138679695391,
          4.1734489537542165,
          3.095683500689061,
          2.039841890017843,
          1.3775663666847708,
          0.9891630015578474,
          0.737146604224775
         ],
         "yaxis": "y2"
        },
        {
         "line": {
          "color": "rgb(34, 119, 158)",
          "dash": "dot",
          "width": 2
         },
         "name": "groundtruth_x",
         "type": "scatter",
         "uid": "77f723ec-3d9f-45bc-bf2a-0484d77642fc",
         "x": [
          0,
          0.18421052631578946,
          0.3684210526315789,
          0.5526315789473684,
          0.7368421052631579,
          0.9210526315789473,
          1.1052631578947367,
          1.2894736842105263,
          1.4736842105263157,
          1.657894736842105,
          1.8421052631578947,
          2.026315789473684,
          2.2105263157894735,
          2.394736842105263,
          2.5789473684210527,
          2.763157894736842,
          2.9473684210526314,
          3.131578947368421,
          3.31578947368421,
          3.5
         ],
         "y": [
          1,
          1.129473699791744,
          1.334457327793724,
          1.6229238330274667,
          2.0096927957237023,
          2.514656122528785,
          3.160353801381743,
          3.9656091906332263,
          4.927034498506909,
          5.965442821227219,
          6.784579833661564,
          6.668346000200168,
          5.031745107406269,
          2.9886735473377124,
          1.7593685481141117,
          1.2108291289378594,
          0.9970006169678738,
          0.9482640252325972,
          0.999169952029643,
          1.1279693039045784
         ]
        },
        {
         "line": {
          "color": "rgb(201, 79, 14)",
          "dash": "dot",
          "width": 2
         },
         "name": "groundtruth_y",
         "type": "scatter",
         "uid": "5dc8a8c8-48a2-420e-91e6-7b212e75e150",
         "x": [
          0,
          0.18421052631578946,
          0.3684210526315789,
          0.5526315789473684,
          0.7368421052631579,
          0.9210526315789473,
          1.1052631578947367,
          1.2894736842105263,
          1.4736842105263157,
          1.657894736842105,
          1.8421052631578947,
          2.026315789473684,
          2.2105263157894735,
          2.394736842105263,
          2.5789473684210527,
          2.763157894736842,
          2.9473684210526314,
          3.131578947368421,
          3.31578947368421,
          3.5
         ],
         "y": [
          1,
          0.6993275761764033,
          0.5043299534524412,
          0.38054740372930423,
          0.3054886548953127,
          0.2661373073873576,
          0.2576878146763418,
          0.2851245655677185,
          0.37137943640403503,
          0.5829025114533225,
          1.0936310630805497,
          2.220089593474913,
          3.8297555854271876,
          4.577564853099683,
          4.023856051174704,
          3.0219280338880323,
          2.1237884168805654,
          1.4591971302511102,
          1.0033419073845884,
          0.7015136364427325
         ]
        },
        {
         "line": {
          "color": "rgb(173, 7, 7)",
          "dash": "dot",
          "width": 2
         },
         "name": "groundtruth_xy",
         "type": "scatter",
         "uid": "99664dc2-df22-4af8-b8a2-3444a8d09ac9",
         "x": [
          1,
          1.129473699791744,
          1.334457327793724,
          1.6229238330274667,
          2.0096927957237023,
          2.514656122528785,
          3.160353801381743,
          3.9656091906332263,
          4.927034498506909,
          5.965442821227219,
          6.784579833661564,
          6.668346000200168,
          5.031745107406269,
          2.9886735473377124,
          1.7593685481141117,
          1.2108291289378594,
          0.9970006169678738,
          0.9482640252325972,
          0.999169952029643,
          1.1279693039045784
         ],
         "xaxis": "x2",
         "y": [
          1,
          0.6993275761764033,
          0.5043299534524412,
          0.38054740372930423,
          0.3054886548953127,
          0.2661373073873576,
          0.2576878146763418,
          0.2851245655677185,
          0.37137943640403503,
          0.5829025114533225,
          1.0936310630805497,
          2.220089593474913,
          3.8297555854271876,
          4.577564853099683,
          4.023856051174704,
          3.0219280338880323,
          2.1237884168805654,
          1.4591971302511102,
          1.0033419073845884,
          0.7015136364427325
         ],
         "yaxis": "y2"
        },
        {
         "line": {
          "color": "rgb(24, 24, 24)",
          "dash": "dash",
          "width": 1
         },
         "name": "current_x",
         "type": "scatter",
         "uid": "63a84177-d841-400d-a5a3-2c8a41bb4a59",
         "x": [
          0,
          0.18421052631578946,
          0.3684210526315789,
          0.5526315789473684,
          0.7368421052631579,
          0.9210526315789473,
          1.1052631578947367,
          1.2894736842105263,
          1.4736842105263157,
          1.657894736842105,
          1.8421052631578947,
          2.026315789473684,
          2.2105263157894735,
          2.394736842105263,
          2.5789473684210527,
          2.763157894736842,
          2.9473684210526314,
          3.131578947368421,
          3.31578947368421,
          3.5
         ],
         "y": [
          1,
          1.1755901853819541,
          1.3956450781676293,
          1.6354558338646423,
          1.9407131519972247,
          2.400253696324886,
          3.0333885676598182,
          3.861676448146876,
          4.9093531429981905,
          6.060548457596613,
          6.758444020373528,
          6.5248919054295245,
          5.118528800044711,
          3.136228525384952,
          1.6249190793125918,
          0.9809208080671945,
          0.9174299636625924,
          1.0330181993423149,
          1.1625200045493267,
          1.3528257966065576
         ]
        },
        {
         "line": {
          "color": "rgb(24, 24, 24)",
          "dash": "dash",
          "width": 1
         },
         "name": "current_y",
         "type": "scatter",
         "uid": "3fd430c4-a80d-43f6-88fe-b7a6fa2fb84c",
         "x": [
          0,
          0.18421052631578946,
          0.3684210526315789,
          0.5526315789473684,
          0.7368421052631579,
          0.9210526315789473,
          1.1052631578947367,
          1.2894736842105263,
          1.4736842105263157,
          1.657894736842105,
          1.8421052631578947,
          2.026315789473684,
          2.2105263157894735,
          2.394736842105263,
          2.5789473684210527,
          2.763157894736842,
          2.9473684210526314,
          3.131578947368421,
          3.31578947368421,
          3.5
         ],
         "y": [
          1,
          0.7180437183577189,
          0.5222963092798567,
          0.35996607896938,
          0.22095375967087919,
          0.13678908000334467,
          0.11433708690096386,
          0.1612618891272612,
          0.2861793034059818,
          0.5934998305722174,
          1.3214631271457367,
          2.4224059479048172,
          3.6825755841001153,
          4.522138679695391,
          4.1734489537542165,
          3.095683500689061,
          2.039841890017843,
          1.3775663666847708,
          0.9891630015578474,
          0.737146604224775
         ]
        },
        {
         "line": {
          "color": "rgb(24, 24, 24)",
          "dash": "dash",
          "width": 2
         },
         "name": "current_xy",
         "type": "scatter",
         "uid": "fc3b03a4-d9e6-4ce8-99ae-a5a9baa808d3",
         "x": [
          1,
          1.1755901853819541,
          1.3956450781676293,
          1.6354558338646423,
          1.9407131519972247,
          2.400253696324886,
          3.0333885676598182,
          3.861676448146876,
          4.9093531429981905,
          6.060548457596613,
          6.758444020373528,
          6.5248919054295245,
          5.118528800044711,
          3.136228525384952,
          1.6249190793125918,
          0.9809208080671945,
          0.9174299636625924,
          1.0330181993423149,
          1.1625200045493267,
          1.3528257966065576
         ],
         "xaxis": "x2",
         "y": [
          1,
          0.7180437183577189,
          0.5222963092798567,
          0.35996607896938,
          0.22095375967087919,
          0.13678908000334467,
          0.11433708690096386,
          0.1612618891272612,
          0.2861793034059818,
          0.5934998305722174,
          1.3214631271457367,
          2.4224059479048172,
          3.6825755841001153,
          4.522138679695391,
          4.1734489537542165,
          3.095683500689061,
          2.039841890017843,
          1.3775663666847708,
          0.9891630015578474,
          0.737146604224775
         ],
         "yaxis": "y2"
        }
       ],
       "layout": {
        "height": 680,
        "title": {
         "text": "Step 500<br>Best Loss 0.5012218749663192<br>Current Loss 0.5012218749663192"
        },
        "width": 1024,
        "xaxis": {
         "range": [
          -0.25,
          3.75
         ]
        },
        "xaxis2": {
         "range": [
          -0.25,
          8
         ]
        },
        "yaxis": {
         "domain": [
          0.55,
          1
         ],
         "range": [
          -0.25,
          8
         ]
        },
        "yaxis2": {
         "domain": [
          0,
          0.45
         ],
         "range": [
          -0.25,
          8
         ]
        }
       }
      },
      "text/html": [
       "<div id=\"8ca6ddd3-416c-4953-9cea-33e9932f1cf8\" style=\"height: 680px; width: 1024px;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"8ca6ddd3-416c-4953-9cea-33e9932f1cf8\", [{\"line\": {\"color\": \"rgb(34, 119, 158)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_x\", \"x\": [0.0, 0.18421052631578946, 0.3684210526315789, 0.5526315789473684, 0.7368421052631579, 0.9210526315789473, 1.1052631578947367, 1.2894736842105263, 1.4736842105263157, 1.657894736842105, 1.8421052631578947, 2.026315789473684, 2.2105263157894735, 2.394736842105263, 2.5789473684210527, 2.763157894736842, 2.9473684210526314, 3.131578947368421, 3.31578947368421, 3.5], \"y\": [1.0, 1.1755901853819541, 1.3956450781676293, 1.6354558338646423, 1.9407131519972247, 2.400253696324886, 3.0333885676598182, 3.861676448146876, 4.9093531429981905, 6.060548457596613, 6.758444020373528, 6.5248919054295245, 5.118528800044711, 3.136228525384952, 1.6249190793125918, 0.9809208080671945, 0.9174299636625924, 1.0330181993423149, 1.1625200045493267, 1.3528257966065576], \"type\": \"scatter\", \"uid\": \"58b5ec85-e256-4e30-9426-ea5728d65105\"}, {\"line\": {\"color\": \"rgb(201, 79, 14)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_y\", \"x\": [0.0, 0.18421052631578946, 0.3684210526315789, 0.5526315789473684, 0.7368421052631579, 0.9210526315789473, 1.1052631578947367, 1.2894736842105263, 1.4736842105263157, 1.657894736842105, 1.8421052631578947, 2.026315789473684, 2.2105263157894735, 2.394736842105263, 2.5789473684210527, 2.763157894736842, 2.9473684210526314, 3.131578947368421, 3.31578947368421, 3.5], \"y\": [1.0, 0.7180437183577189, 0.5222963092798567, 0.35996607896938, 0.22095375967087919, 0.13678908000334467, 0.11433708690096386, 0.1612618891272612, 0.2861793034059818, 0.5934998305722174, 1.3214631271457367, 2.4224059479048172, 3.6825755841001153, 4.522138679695391, 4.1734489537542165, 3.095683500689061, 2.039841890017843, 1.3775663666847708, 0.9891630015578474, 0.737146604224775], \"type\": \"scatter\", \"uid\": \"762c0202-a34c-4d91-ad5a-e58d65256f9a\"}, {\"line\": {\"color\": \"rgb(8, 173, 99)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_xy\", \"x\": [1.0, 1.1755901853819541, 1.3956450781676293, 1.6354558338646423, 1.9407131519972247, 2.400253696324886, 3.0333885676598182, 3.861676448146876, 4.9093531429981905, 6.060548457596613, 6.758444020373528, 6.5248919054295245, 5.118528800044711, 3.136228525384952, 1.6249190793125918, 0.9809208080671945, 0.9174299636625924, 1.0330181993423149, 1.1625200045493267, 1.3528257966065576], \"xaxis\": \"x2\", \"y\": [1.0, 0.7180437183577189, 0.5222963092798567, 0.35996607896938, 0.22095375967087919, 0.13678908000334467, 0.11433708690096386, 0.1612618891272612, 0.2861793034059818, 0.5934998305722174, 1.3214631271457367, 2.4224059479048172, 3.6825755841001153, 4.522138679695391, 4.1734489537542165, 3.095683500689061, 2.039841890017843, 1.3775663666847708, 0.9891630015578474, 0.737146604224775], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"447100f1-a8f5-47be-9011-8e116d0f8efc\"}, {\"line\": {\"color\": \"rgb(34, 119, 158)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_x\", \"x\": [0.0, 0.18421052631578946, 0.3684210526315789, 0.5526315789473684, 0.7368421052631579, 0.9210526315789473, 1.1052631578947367, 1.2894736842105263, 1.4736842105263157, 1.657894736842105, 1.8421052631578947, 2.026315789473684, 2.2105263157894735, 2.394736842105263, 2.5789473684210527, 2.763157894736842, 2.9473684210526314, 3.131578947368421, 3.31578947368421, 3.5], \"y\": [1.0, 1.129473699791744, 1.334457327793724, 1.6229238330274667, 2.0096927957237023, 2.514656122528785, 3.160353801381743, 3.9656091906332263, 4.927034498506909, 5.965442821227219, 6.784579833661564, 6.668346000200168, 5.031745107406269, 2.9886735473377124, 1.7593685481141117, 1.2108291289378594, 0.9970006169678738, 0.9482640252325972, 0.999169952029643, 1.1279693039045784], \"type\": \"scatter\", \"uid\": \"77f723ec-3d9f-45bc-bf2a-0484d77642fc\"}, {\"line\": {\"color\": \"rgb(201, 79, 14)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_y\", \"x\": [0.0, 0.18421052631578946, 0.3684210526315789, 0.5526315789473684, 0.7368421052631579, 0.9210526315789473, 1.1052631578947367, 1.2894736842105263, 1.4736842105263157, 1.657894736842105, 1.8421052631578947, 2.026315789473684, 2.2105263157894735, 2.394736842105263, 2.5789473684210527, 2.763157894736842, 2.9473684210526314, 3.131578947368421, 3.31578947368421, 3.5], \"y\": [1.0, 0.6993275761764033, 0.5043299534524412, 0.38054740372930423, 0.3054886548953127, 0.2661373073873576, 0.2576878146763418, 0.2851245655677185, 0.37137943640403503, 0.5829025114533225, 1.0936310630805497, 2.220089593474913, 3.8297555854271876, 4.577564853099683, 4.023856051174704, 3.0219280338880323, 2.1237884168805654, 1.4591971302511102, 1.0033419073845884, 0.7015136364427325], \"type\": \"scatter\", \"uid\": \"5dc8a8c8-48a2-420e-91e6-7b212e75e150\"}, {\"line\": {\"color\": \"rgb(173, 7, 7)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_xy\", \"x\": [1.0, 1.129473699791744, 1.334457327793724, 1.6229238330274667, 2.0096927957237023, 2.514656122528785, 3.160353801381743, 3.9656091906332263, 4.927034498506909, 5.965442821227219, 6.784579833661564, 6.668346000200168, 5.031745107406269, 2.9886735473377124, 1.7593685481141117, 1.2108291289378594, 0.9970006169678738, 0.9482640252325972, 0.999169952029643, 1.1279693039045784], \"xaxis\": \"x2\", \"y\": [1.0, 0.6993275761764033, 0.5043299534524412, 0.38054740372930423, 0.3054886548953127, 0.2661373073873576, 0.2576878146763418, 0.2851245655677185, 0.37137943640403503, 0.5829025114533225, 1.0936310630805497, 2.220089593474913, 3.8297555854271876, 4.577564853099683, 4.023856051174704, 3.0219280338880323, 2.1237884168805654, 1.4591971302511102, 1.0033419073845884, 0.7015136364427325], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"99664dc2-df22-4af8-b8a2-3444a8d09ac9\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 1}, \"name\": \"current_x\", \"x\": [0.0, 0.18421052631578946, 0.3684210526315789, 0.5526315789473684, 0.7368421052631579, 0.9210526315789473, 1.1052631578947367, 1.2894736842105263, 1.4736842105263157, 1.657894736842105, 1.8421052631578947, 2.026315789473684, 2.2105263157894735, 2.394736842105263, 2.5789473684210527, 2.763157894736842, 2.9473684210526314, 3.131578947368421, 3.31578947368421, 3.5], \"y\": [1.0, 1.1755901853819541, 1.3956450781676293, 1.6354558338646423, 1.9407131519972247, 2.400253696324886, 3.0333885676598182, 3.861676448146876, 4.9093531429981905, 6.060548457596613, 6.758444020373528, 6.5248919054295245, 5.118528800044711, 3.136228525384952, 1.6249190793125918, 0.9809208080671945, 0.9174299636625924, 1.0330181993423149, 1.1625200045493267, 1.3528257966065576], \"type\": \"scatter\", \"uid\": \"63a84177-d841-400d-a5a3-2c8a41bb4a59\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 1}, \"name\": \"current_y\", \"x\": [0.0, 0.18421052631578946, 0.3684210526315789, 0.5526315789473684, 0.7368421052631579, 0.9210526315789473, 1.1052631578947367, 1.2894736842105263, 1.4736842105263157, 1.657894736842105, 1.8421052631578947, 2.026315789473684, 2.2105263157894735, 2.394736842105263, 2.5789473684210527, 2.763157894736842, 2.9473684210526314, 3.131578947368421, 3.31578947368421, 3.5], \"y\": [1.0, 0.7180437183577189, 0.5222963092798567, 0.35996607896938, 0.22095375967087919, 0.13678908000334467, 0.11433708690096386, 0.1612618891272612, 0.2861793034059818, 0.5934998305722174, 1.3214631271457367, 2.4224059479048172, 3.6825755841001153, 4.522138679695391, 4.1734489537542165, 3.095683500689061, 2.039841890017843, 1.3775663666847708, 0.9891630015578474, 0.737146604224775], \"type\": \"scatter\", \"uid\": \"3fd430c4-a80d-43f6-88fe-b7a6fa2fb84c\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 2}, \"name\": \"current_xy\", \"x\": [1.0, 1.1755901853819541, 1.3956450781676293, 1.6354558338646423, 1.9407131519972247, 2.400253696324886, 3.0333885676598182, 3.861676448146876, 4.9093531429981905, 6.060548457596613, 6.758444020373528, 6.5248919054295245, 5.118528800044711, 3.136228525384952, 1.6249190793125918, 0.9809208080671945, 0.9174299636625924, 1.0330181993423149, 1.1625200045493267, 1.3528257966065576], \"xaxis\": \"x2\", \"y\": [1.0, 0.7180437183577189, 0.5222963092798567, 0.35996607896938, 0.22095375967087919, 0.13678908000334467, 0.11433708690096386, 0.1612618891272612, 0.2861793034059818, 0.5934998305722174, 1.3214631271457367, 2.4224059479048172, 3.6825755841001153, 4.522138679695391, 4.1734489537542165, 3.095683500689061, 2.039841890017843, 1.3775663666847708, 0.9891630015578474, 0.737146604224775], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"fc3b03a4-d9e6-4ce8-99ae-a5a9baa808d3\"}], {\"height\": 680, \"title\": {\"text\": \"Step 500<br>Best Loss 0.5012218749663192<br>Current Loss 0.5012218749663192\"}, \"width\": 1024, \"xaxis\": {\"range\": [-0.25, 3.75]}, \"yaxis\": {\"domain\": [0.55, 1.0], \"range\": [-0.25, 8.0]}, \"xaxis2\": {\"range\": [-0.25, 8.0]}, \"yaxis2\": {\"domain\": [0, 0.45], \"range\": [-0.25, 8.0]}}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"8ca6ddd3-416c-4953-9cea-33e9932f1cf8\" style=\"height: 680px; width: 1024px;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"8ca6ddd3-416c-4953-9cea-33e9932f1cf8\", [{\"line\": {\"color\": \"rgb(34, 119, 158)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_x\", \"x\": [0.0, 0.18421052631578946, 0.3684210526315789, 0.5526315789473684, 0.7368421052631579, 0.9210526315789473, 1.1052631578947367, 1.2894736842105263, 1.4736842105263157, 1.657894736842105, 1.8421052631578947, 2.026315789473684, 2.2105263157894735, 2.394736842105263, 2.5789473684210527, 2.763157894736842, 2.9473684210526314, 3.131578947368421, 3.31578947368421, 3.5], \"y\": [1.0, 1.1755901853819541, 1.3956450781676293, 1.6354558338646423, 1.9407131519972247, 2.400253696324886, 3.0333885676598182, 3.861676448146876, 4.9093531429981905, 6.060548457596613, 6.758444020373528, 6.5248919054295245, 5.118528800044711, 3.136228525384952, 1.6249190793125918, 0.9809208080671945, 0.9174299636625924, 1.0330181993423149, 1.1625200045493267, 1.3528257966065576], \"type\": \"scatter\", \"uid\": \"58b5ec85-e256-4e30-9426-ea5728d65105\"}, {\"line\": {\"color\": \"rgb(201, 79, 14)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_y\", \"x\": [0.0, 0.18421052631578946, 0.3684210526315789, 0.5526315789473684, 0.7368421052631579, 0.9210526315789473, 1.1052631578947367, 1.2894736842105263, 1.4736842105263157, 1.657894736842105, 1.8421052631578947, 2.026315789473684, 2.2105263157894735, 2.394736842105263, 2.5789473684210527, 2.763157894736842, 2.9473684210526314, 3.131578947368421, 3.31578947368421, 3.5], \"y\": [1.0, 0.7180437183577189, 0.5222963092798567, 0.35996607896938, 0.22095375967087919, 0.13678908000334467, 0.11433708690096386, 0.1612618891272612, 0.2861793034059818, 0.5934998305722174, 1.3214631271457367, 2.4224059479048172, 3.6825755841001153, 4.522138679695391, 4.1734489537542165, 3.095683500689061, 2.039841890017843, 1.3775663666847708, 0.9891630015578474, 0.737146604224775], \"type\": \"scatter\", \"uid\": \"762c0202-a34c-4d91-ad5a-e58d65256f9a\"}, {\"line\": {\"color\": \"rgb(8, 173, 99)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_xy\", \"x\": [1.0, 1.1755901853819541, 1.3956450781676293, 1.6354558338646423, 1.9407131519972247, 2.400253696324886, 3.0333885676598182, 3.861676448146876, 4.9093531429981905, 6.060548457596613, 6.758444020373528, 6.5248919054295245, 5.118528800044711, 3.136228525384952, 1.6249190793125918, 0.9809208080671945, 0.9174299636625924, 1.0330181993423149, 1.1625200045493267, 1.3528257966065576], \"xaxis\": \"x2\", \"y\": [1.0, 0.7180437183577189, 0.5222963092798567, 0.35996607896938, 0.22095375967087919, 0.13678908000334467, 0.11433708690096386, 0.1612618891272612, 0.2861793034059818, 0.5934998305722174, 1.3214631271457367, 2.4224059479048172, 3.6825755841001153, 4.522138679695391, 4.1734489537542165, 3.095683500689061, 2.039841890017843, 1.3775663666847708, 0.9891630015578474, 0.737146604224775], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"447100f1-a8f5-47be-9011-8e116d0f8efc\"}, {\"line\": {\"color\": \"rgb(34, 119, 158)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_x\", \"x\": [0.0, 0.18421052631578946, 0.3684210526315789, 0.5526315789473684, 0.7368421052631579, 0.9210526315789473, 1.1052631578947367, 1.2894736842105263, 1.4736842105263157, 1.657894736842105, 1.8421052631578947, 2.026315789473684, 2.2105263157894735, 2.394736842105263, 2.5789473684210527, 2.763157894736842, 2.9473684210526314, 3.131578947368421, 3.31578947368421, 3.5], \"y\": [1.0, 1.129473699791744, 1.334457327793724, 1.6229238330274667, 2.0096927957237023, 2.514656122528785, 3.160353801381743, 3.9656091906332263, 4.927034498506909, 5.965442821227219, 6.784579833661564, 6.668346000200168, 5.031745107406269, 2.9886735473377124, 1.7593685481141117, 1.2108291289378594, 0.9970006169678738, 0.9482640252325972, 0.999169952029643, 1.1279693039045784], \"type\": \"scatter\", \"uid\": \"77f723ec-3d9f-45bc-bf2a-0484d77642fc\"}, {\"line\": {\"color\": \"rgb(201, 79, 14)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_y\", \"x\": [0.0, 0.18421052631578946, 0.3684210526315789, 0.5526315789473684, 0.7368421052631579, 0.9210526315789473, 1.1052631578947367, 1.2894736842105263, 1.4736842105263157, 1.657894736842105, 1.8421052631578947, 2.026315789473684, 2.2105263157894735, 2.394736842105263, 2.5789473684210527, 2.763157894736842, 2.9473684210526314, 3.131578947368421, 3.31578947368421, 3.5], \"y\": [1.0, 0.6993275761764033, 0.5043299534524412, 0.38054740372930423, 0.3054886548953127, 0.2661373073873576, 0.2576878146763418, 0.2851245655677185, 0.37137943640403503, 0.5829025114533225, 1.0936310630805497, 2.220089593474913, 3.8297555854271876, 4.577564853099683, 4.023856051174704, 3.0219280338880323, 2.1237884168805654, 1.4591971302511102, 1.0033419073845884, 0.7015136364427325], \"type\": \"scatter\", \"uid\": \"5dc8a8c8-48a2-420e-91e6-7b212e75e150\"}, {\"line\": {\"color\": \"rgb(173, 7, 7)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_xy\", \"x\": [1.0, 1.129473699791744, 1.334457327793724, 1.6229238330274667, 2.0096927957237023, 2.514656122528785, 3.160353801381743, 3.9656091906332263, 4.927034498506909, 5.965442821227219, 6.784579833661564, 6.668346000200168, 5.031745107406269, 2.9886735473377124, 1.7593685481141117, 1.2108291289378594, 0.9970006169678738, 0.9482640252325972, 0.999169952029643, 1.1279693039045784], \"xaxis\": \"x2\", \"y\": [1.0, 0.6993275761764033, 0.5043299534524412, 0.38054740372930423, 0.3054886548953127, 0.2661373073873576, 0.2576878146763418, 0.2851245655677185, 0.37137943640403503, 0.5829025114533225, 1.0936310630805497, 2.220089593474913, 3.8297555854271876, 4.577564853099683, 4.023856051174704, 3.0219280338880323, 2.1237884168805654, 1.4591971302511102, 1.0033419073845884, 0.7015136364427325], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"99664dc2-df22-4af8-b8a2-3444a8d09ac9\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 1}, \"name\": \"current_x\", \"x\": [0.0, 0.18421052631578946, 0.3684210526315789, 0.5526315789473684, 0.7368421052631579, 0.9210526315789473, 1.1052631578947367, 1.2894736842105263, 1.4736842105263157, 1.657894736842105, 1.8421052631578947, 2.026315789473684, 2.2105263157894735, 2.394736842105263, 2.5789473684210527, 2.763157894736842, 2.9473684210526314, 3.131578947368421, 3.31578947368421, 3.5], \"y\": [1.0, 1.1755901853819541, 1.3956450781676293, 1.6354558338646423, 1.9407131519972247, 2.400253696324886, 3.0333885676598182, 3.861676448146876, 4.9093531429981905, 6.060548457596613, 6.758444020373528, 6.5248919054295245, 5.118528800044711, 3.136228525384952, 1.6249190793125918, 0.9809208080671945, 0.9174299636625924, 1.0330181993423149, 1.1625200045493267, 1.3528257966065576], \"type\": \"scatter\", \"uid\": \"63a84177-d841-400d-a5a3-2c8a41bb4a59\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 1}, \"name\": \"current_y\", \"x\": [0.0, 0.18421052631578946, 0.3684210526315789, 0.5526315789473684, 0.7368421052631579, 0.9210526315789473, 1.1052631578947367, 1.2894736842105263, 1.4736842105263157, 1.657894736842105, 1.8421052631578947, 2.026315789473684, 2.2105263157894735, 2.394736842105263, 2.5789473684210527, 2.763157894736842, 2.9473684210526314, 3.131578947368421, 3.31578947368421, 3.5], \"y\": [1.0, 0.7180437183577189, 0.5222963092798567, 0.35996607896938, 0.22095375967087919, 0.13678908000334467, 0.11433708690096386, 0.1612618891272612, 0.2861793034059818, 0.5934998305722174, 1.3214631271457367, 2.4224059479048172, 3.6825755841001153, 4.522138679695391, 4.1734489537542165, 3.095683500689061, 2.039841890017843, 1.3775663666847708, 0.9891630015578474, 0.737146604224775], \"type\": \"scatter\", \"uid\": \"3fd430c4-a80d-43f6-88fe-b7a6fa2fb84c\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 2}, \"name\": \"current_xy\", \"x\": [1.0, 1.1755901853819541, 1.3956450781676293, 1.6354558338646423, 1.9407131519972247, 2.400253696324886, 3.0333885676598182, 3.861676448146876, 4.9093531429981905, 6.060548457596613, 6.758444020373528, 6.5248919054295245, 5.118528800044711, 3.136228525384952, 1.6249190793125918, 0.9809208080671945, 0.9174299636625924, 1.0330181993423149, 1.1625200045493267, 1.3528257966065576], \"xaxis\": \"x2\", \"y\": [1.0, 0.7180437183577189, 0.5222963092798567, 0.35996607896938, 0.22095375967087919, 0.13678908000334467, 0.11433708690096386, 0.1612618891272612, 0.2861793034059818, 0.5934998305722174, 1.3214631271457367, 2.4224059479048172, 3.6825755841001153, 4.522138679695391, 4.1734489537542165, 3.095683500689061, 2.039841890017843, 1.3775663666847708, 0.9891630015578474, 0.737146604224775], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"fc3b03a4-d9e6-4ce8-99ae-a5a9baa808d3\"}], {\"height\": 680, \"title\": {\"text\": \"Step 500<br>Best Loss 0.5012218749663192<br>Current Loss 0.5012218749663192\"}, \"width\": 1024, \"xaxis\": {\"range\": [-0.25, 3.75]}, \"yaxis\": {\"domain\": [0.55, 1.0], \"range\": [-0.25, 8.0]}, \"xaxis2\": {\"range\": [-0.25, 8.0]}, \"yaxis2\": {\"domain\": [0, 0.45], \"range\": [-0.25, 8.0]}}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "optimize(predict_fn, ode_scipy_solver, loss_L2, ground_truth, z0, ts,\n",
    "         args=None, variables=model.variables, epochs=500,\n",
    "         x_axis_min=-0.25, x_axis_max=3.75,\n",
    "         y_axis_min=-0.25, y_axis_max=8.0,\n",
    "         x2_axis_min=-0.25, x2_axis_max=8.0,\n",
    "         y2_axis_min=-0.25, y2_axis_max=8.0,\n",
    "         output_dir=\"ode_dense_lotka_volterra\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Result of training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "linkText": "Export to plot.ly",
        "plotlyServerURL": "https://plot.ly",
        "showLink": false
       },
       "data": [
        {
         "name": "x",
         "type": "scatter",
         "uid": "a3b77ead-e7ed-419e-ae8e-27d8afd16263",
         "x": [
          0,
          0.18421052631578946,
          0.3684210526315789,
          0.5526315789473684,
          0.7368421052631579,
          0.9210526315789473,
          1.1052631578947367,
          1.2894736842105263,
          1.4736842105263157,
          1.657894736842105,
          1.8421052631578947,
          2.026315789473684,
          2.2105263157894735,
          2.394736842105263,
          2.5789473684210527,
          2.763157894736842,
          2.9473684210526314,
          3.131578947368421,
          3.31578947368421,
          3.5
         ],
         "y": [
          1,
          1.175544731324062,
          1.395624524005696,
          1.6355386591862366,
          1.9410271261155205,
          2.400684049551078,
          3.033814793496186,
          3.8619822792775387,
          4.909429840511175,
          6.060312062757879,
          6.758252433375139,
          6.5251457518908325,
          5.11864990191055,
          3.1356482319248093,
          1.6250392970122292,
          0.9812800470192664,
          0.9174860539527148,
          1.0329120479177918,
          1.1622754502167914,
          1.3525373695794878
         ]
        },
        {
         "name": "y",
         "type": "scatter",
         "uid": "44a78861-a137-4f82-a55e-cf9bd7a43371",
         "x": [
          0,
          0.18421052631578946,
          0.3684210526315789,
          0.5526315789473684,
          0.7368421052631579,
          0.9210526315789473,
          1.1052631578947367,
          1.2894736842105263,
          1.4736842105263157,
          1.657894736842105,
          1.8421052631578947,
          2.026315789473684,
          2.2105263157894735,
          2.394736842105263,
          2.5789473684210527,
          2.763157894736842,
          2.9473684210526314,
          3.131578947368421,
          3.31578947368421,
          3.5
         ],
         "y": [
          1,
          0.7180000870743075,
          0.522294797685479,
          0.36010286711776035,
          0.22125751047751954,
          0.13716948074826793,
          0.11469561986500693,
          0.16149462246362875,
          0.2861771900818411,
          0.5932033359273537,
          1.3209144930679768,
          2.4221345901639233,
          3.682996251299118,
          4.522375089912592,
          4.173036017075003,
          3.0956441678414715,
          2.0401321759796938,
          1.3776217881309316,
          0.9890926533069055,
          0.7370117694182944
         ]
        }
       ],
       "layout": {}
      },
      "text/html": [
       "<div id=\"0923b83a-5902-400e-8fe1-27d0bd63c2d4\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"0923b83a-5902-400e-8fe1-27d0bd63c2d4\", [{\"name\": \"x\", \"x\": [0.0, 0.18421052631578946, 0.3684210526315789, 0.5526315789473684, 0.7368421052631579, 0.9210526315789473, 1.1052631578947367, 1.2894736842105263, 1.4736842105263157, 1.657894736842105, 1.8421052631578947, 2.026315789473684, 2.2105263157894735, 2.394736842105263, 2.5789473684210527, 2.763157894736842, 2.9473684210526314, 3.131578947368421, 3.31578947368421, 3.5], \"y\": [1.0, 1.175544731324062, 1.395624524005696, 1.6355386591862366, 1.9410271261155205, 2.400684049551078, 3.033814793496186, 3.8619822792775387, 4.909429840511175, 6.060312062757879, 6.758252433375139, 6.5251457518908325, 5.11864990191055, 3.1356482319248093, 1.6250392970122292, 0.9812800470192664, 0.9174860539527148, 1.0329120479177918, 1.1622754502167914, 1.3525373695794878], \"type\": \"scatter\", \"uid\": \"73bd3174-2ea6-492e-ba5e-d793e12b4456\"}, {\"name\": \"y\", \"x\": [0.0, 0.18421052631578946, 0.3684210526315789, 0.5526315789473684, 0.7368421052631579, 0.9210526315789473, 1.1052631578947367, 1.2894736842105263, 1.4736842105263157, 1.657894736842105, 1.8421052631578947, 2.026315789473684, 2.2105263157894735, 2.394736842105263, 2.5789473684210527, 2.763157894736842, 2.9473684210526314, 3.131578947368421, 3.31578947368421, 3.5], \"y\": [1.0, 0.7180000870743075, 0.522294797685479, 0.36010286711776035, 0.22125751047751954, 0.13716948074826793, 0.11469561986500693, 0.16149462246362875, 0.2861771900818411, 0.5932033359273537, 1.3209144930679768, 2.4221345901639233, 3.682996251299118, 4.522375089912592, 4.173036017075003, 3.0956441678414715, 2.0401321759796938, 1.3776217881309316, 0.9890926533069055, 0.7370117694182944], \"type\": \"scatter\", \"uid\": \"9f4d15be-28c9-4fbc-92c2-50c8ab1814c6\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"0923b83a-5902-400e-8fe1-27d0bd63c2d4\"));});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"0923b83a-5902-400e-8fe1-27d0bd63c2d4\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"0923b83a-5902-400e-8fe1-27d0bd63c2d4\", [{\"name\": \"x\", \"x\": [0.0, 0.18421052631578946, 0.3684210526315789, 0.5526315789473684, 0.7368421052631579, 0.9210526315789473, 1.1052631578947367, 1.2894736842105263, 1.4736842105263157, 1.657894736842105, 1.8421052631578947, 2.026315789473684, 2.2105263157894735, 2.394736842105263, 2.5789473684210527, 2.763157894736842, 2.9473684210526314, 3.131578947368421, 3.31578947368421, 3.5], \"y\": [1.0, 1.175544731324062, 1.395624524005696, 1.6355386591862366, 1.9410271261155205, 2.400684049551078, 3.033814793496186, 3.8619822792775387, 4.909429840511175, 6.060312062757879, 6.758252433375139, 6.5251457518908325, 5.11864990191055, 3.1356482319248093, 1.6250392970122292, 0.9812800470192664, 0.9174860539527148, 1.0329120479177918, 1.1622754502167914, 1.3525373695794878], \"type\": \"scatter\", \"uid\": \"73bd3174-2ea6-492e-ba5e-d793e12b4456\"}, {\"name\": \"y\", \"x\": [0.0, 0.18421052631578946, 0.3684210526315789, 0.5526315789473684, 0.7368421052631579, 0.9210526315789473, 1.1052631578947367, 1.2894736842105263, 1.4736842105263157, 1.657894736842105, 1.8421052631578947, 2.026315789473684, 2.2105263157894735, 2.394736842105263, 2.5789473684210527, 2.763157894736842, 2.9473684210526314, 3.131578947368421, 3.31578947368421, 3.5], \"y\": [1.0, 0.7180000870743075, 0.522294797685479, 0.36010286711776035, 0.22125751047751954, 0.13716948074826793, 0.11469561986500693, 0.16149462246362875, 0.2861771900818411, 0.5932033359273537, 1.3209144930679768, 2.4221345901639233, 3.682996251299118, 4.522375089912592, 4.173036017075003, 3.0956441678414715, 2.0401321759796938, 1.3776217881309316, 0.9890926533069055, 0.7370117694182944], \"type\": \"scatter\", \"uid\": \"9f4d15be-28c9-4fbc-92c2-50c8ab1814c6\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"0923b83a-5902-400e-8fe1-27d0bd63c2d4\"));});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "linkText": "Export to plot.ly",
        "plotlyServerURL": "https://plot.ly",
        "showLink": false
       },
       "data": [
        {
         "type": "scatter",
         "uid": "bebb2f90-ea31-4046-a4ff-ddb3e62a5c81",
         "x": [
          1,
          1.175544731324062,
          1.395624524005696,
          1.6355386591862366,
          1.9410271261155205,
          2.400684049551078,
          3.033814793496186,
          3.8619822792775387,
          4.909429840511175,
          6.060312062757879,
          6.758252433375139,
          6.5251457518908325,
          5.11864990191055,
          3.1356482319248093,
          1.6250392970122292,
          0.9812800470192664,
          0.9174860539527148,
          1.0329120479177918,
          1.1622754502167914,
          1.3525373695794878
         ],
         "y": [
          1,
          0.7180000870743075,
          0.522294797685479,
          0.36010286711776035,
          0.22125751047751954,
          0.13716948074826793,
          0.11469561986500693,
          0.16149462246362875,
          0.2861771900818411,
          0.5932033359273537,
          1.3209144930679768,
          2.4221345901639233,
          3.682996251299118,
          4.522375089912592,
          4.173036017075003,
          3.0956441678414715,
          2.0401321759796938,
          1.3776217881309316,
          0.9890926533069055,
          0.7370117694182944
         ]
        }
       ],
       "layout": {}
      },
      "text/html": [
       "<div id=\"5c84ce58-6db4-44ad-9aac-98add90eae44\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"5c84ce58-6db4-44ad-9aac-98add90eae44\", [{\"x\": [1.0, 1.175544731324062, 1.395624524005696, 1.6355386591862366, 1.9410271261155205, 2.400684049551078, 3.033814793496186, 3.8619822792775387, 4.909429840511175, 6.060312062757879, 6.758252433375139, 6.5251457518908325, 5.11864990191055, 3.1356482319248093, 1.6250392970122292, 0.9812800470192664, 0.9174860539527148, 1.0329120479177918, 1.1622754502167914, 1.3525373695794878], \"y\": [1.0, 0.7180000870743075, 0.522294797685479, 0.36010286711776035, 0.22125751047751954, 0.13716948074826793, 0.11469561986500693, 0.16149462246362875, 0.2861771900818411, 0.5932033359273537, 1.3209144930679768, 2.4221345901639233, 3.682996251299118, 4.522375089912592, 4.173036017075003, 3.0956441678414715, 2.0401321759796938, 1.3776217881309316, 0.9890926533069055, 0.7370117694182944], \"type\": \"scatter\", \"uid\": \"5a82eb0f-4376-407f-aef6-bc2403e63a5d\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"5c84ce58-6db4-44ad-9aac-98add90eae44\"));});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"5c84ce58-6db4-44ad-9aac-98add90eae44\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"5c84ce58-6db4-44ad-9aac-98add90eae44\", [{\"x\": [1.0, 1.175544731324062, 1.395624524005696, 1.6355386591862366, 1.9410271261155205, 2.400684049551078, 3.033814793496186, 3.8619822792775387, 4.909429840511175, 6.060312062757879, 6.758252433375139, 6.5251457518908325, 5.11864990191055, 3.1356482319248093, 1.6250392970122292, 0.9812800470192664, 0.9174860539527148, 1.0329120479177918, 1.1622754502167914, 1.3525373695794878], \"y\": [1.0, 0.7180000870743075, 0.522294797685479, 0.36010286711776035, 0.22125751047751954, 0.13716948074826793, 0.11469561986500693, 0.16149462246362875, 0.2861771900818411, 0.5932033359273537, 1.3209144930679768, 2.4221345901639233, 3.682996251299118, 4.522375089912592, 4.173036017075003, 3.0956441678414715, 2.0401321759796938, 1.3776217881309316, 0.9890926533069055, 0.7370117694182944], \"type\": \"scatter\", \"uid\": \"5a82eb0f-4376-407f-aef6-bc2403e63a5d\"}], {}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script><script type=\"text/javascript\">window.addEventListener(\"resize\", function(){window._Plotly.Plots.resize(document.getElementById(\"5c84ce58-6db4-44ad-9aac-98add90eae44\"));});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "zs = ode_scipy(predict_fn, z0, ts, args)\n",
    "\n",
    "xs = zs[:, 0].numpy()\n",
    "ys = zs[:, 1].numpy()\n",
    "\n",
    "iplot([go.Scatter(x=ts.numpy(), y=xs, name=\"x\"), go.Scatter(x=ts.numpy(), y=ys, name=\"y\")])\n",
    "\n",
    "iplot([go.Scatter(x=xs, y=ys)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning a piece of Damped 2D oscillator curve but not converging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "D = 2\n",
    "N = 50\n",
    "ts = tf.constant(np.linspace(0., 6.0, N))\n",
    "z0 = tf.Variable(tf.constant([2.0, 0.0], shape=(2,), dtype=ts.dtype))\n",
    "\n",
    "args = tf.Variable(tf.constant([\n",
    "    # −0.1x^3 + 2.0y^3\n",
    "    [-0.1, 2.0],\n",
    "    # −2.0x^3 − 0.1y^3\n",
    "    [-2.0, -0.1]], shape=(2, 2), dtype=ts.dtype))\n",
    "\n",
    "\n",
    "model2 = tf.keras.Sequential([\n",
    "  tf.keras.layers.Dense(25, activation=tf.nn.tanh, input_shape=(1, D)),  # input shape required\n",
    "  tf.keras.layers.Dense(D)\n",
    "])\n",
    "\n",
    "def ode_scipy_solver2(f, z0, ts, args):\n",
    "    r = ode_scipy(f, z0, ts, args)\n",
    "    r = tf.cast(r, z0.dtype)\n",
    "    return r\n",
    "\n",
    "def predict_fn2(y, ts, args):\n",
    "    y2 = tf.reshape(y, [1, D])\n",
    "    y2 = tf.cast(y2, tf.float32)\n",
    "    r = model2(y2)\n",
    "    r = tf.squeeze(r)\n",
    "    r = tf.cast(r, y.dtype)\n",
    "    return r\n",
    "\n",
    "ground_truth2 = ode_scipy(damped_oscillator_2d, z0, ts, args)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### For the lazy, video of training\n",
    "\n",
    "Run next cell to see video of a 1000 epochs training I did on CPU/GPU.\n",
    "It's quite interesting to see how it learns first curves but fails to converge on the whole oscillator.\n",
    "I need to test with a bigger Dense network and maybe another time distribution...\n",
    "Anyway, it's fun to see how it learns, isnt' it? ;)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<video width=\"800\" height=\"600\" controls>\n",
       "  <source src=\"ode_dense_damped/video.mp4\" type=\"video/mp4\">\n",
       "</video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "\n",
    "HTML(\"\"\"\n",
    "<video width=\"800\" height=\"600\" controls>\n",
    "  <source src=\"ode_dense_damped/video.mp4\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Running the experiment by yourself\n",
    "\n",
    "> On a 1080TI, it takes around 3h for 1000 epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "linkText": "Export to plot.ly",
        "plotlyServerURL": "https://plot.ly",
        "showLink": false
       },
       "data": [
        {
         "line": {
          "color": "rgb(34, 119, 158)",
          "width": 1
         },
         "mode": "lines+markers",
         "name": "best_x",
         "type": "scatter",
         "uid": "5b80efc4-89a2-499e-af1f-3c80959d6d75",
         "x": [
          0,
          0.12244897959183673,
          0.24489795918367346,
          0.36734693877551017,
          0.4897959183673469,
          0.6122448979591837,
          0.7346938775510203,
          0.8571428571428571,
          0.9795918367346939,
          1.1020408163265305,
          1.2244897959183674,
          1.346938775510204,
          1.4693877551020407,
          1.5918367346938775,
          1.7142857142857142,
          1.836734693877551,
          1.9591836734693877,
          2.0816326530612246,
          2.204081632653061,
          2.326530612244898,
          2.4489795918367347,
          2.571428571428571,
          2.693877551020408,
          2.816326530612245,
          2.9387755102040813,
          3.061224489795918,
          3.183673469387755,
          3.306122448979592,
          3.4285714285714284,
          3.5510204081632653,
          3.673469387755102,
          3.7959183673469385,
          3.9183673469387754,
          4.040816326530612,
          4.163265306122449,
          4.285714285714286,
          4.408163265306122,
          4.530612244897959,
          4.653061224489796,
          4.775510204081632,
          4.8979591836734695,
          5.020408163265306,
          5.142857142857142,
          5.26530612244898,
          5.387755102040816,
          5.5102040816326525,
          5.63265306122449,
          5.755102040816326,
          5.877551020408163,
          6
         ],
         "y": [
          2,
          1.436581963587984,
          0.9696813817069801,
          0.6037645827844602,
          0.3297921045100084,
          0.13105453462244457,
          -0.010232517338564335,
          -0.10937556649944526,
          -0.1782864612071781,
          -0.2257873393913745,
          -0.2582518050705372,
          -0.28022479902923064,
          -0.294922593030897,
          -0.3046088680265031,
          -0.31086953985409943,
          -0.31481027332807776,
          -0.31719782406609276,
          -0.3185605691636698,
          -0.31925991957613686,
          -0.3195408060355251,
          -0.3195673718246092,
          -0.31944804806327165,
          -0.31925312753621327,
          -0.3190270368476306,
          -0.31879686378820005,
          -0.3185782234979382,
          -0.31837928286496364,
          -0.3182034729367215,
          -0.31805137419076124,
          -0.3179218825359452,
          -0.317813006150125,
          -0.3177223756088433,
          -0.3176475526541967,
          -0.31758619192908394,
          -0.31753616311436,
          -0.3174955696739524,
          -0.3174627753689424,
          -0.31743638194744755,
          -0.3174152058966373,
          -0.31739826231258284,
          -0.3173847388545805,
          -0.31737397182441696,
          -0.3173654159623826,
          -0.31735863389935026,
          -0.3173532602614974,
          -0.3173490124570506,
          -0.3173456546024602,
          -0.3173429939602287,
          -0.31734089488693396,
          -0.31733926026226716
         ]
        },
        {
         "line": {
          "color": "rgb(201, 79, 14)",
          "width": 1
         },
         "mode": "lines+markers",
         "name": "best_y",
         "type": "scatter",
         "uid": "748da336-c1ee-4018-931e-7fab0127a954",
         "x": [
          0,
          0.12244897959183673,
          0.24489795918367346,
          0.36734693877551017,
          0.4897959183673469,
          0.6122448979591837,
          0.7346938775510203,
          0.8571428571428571,
          0.9795918367346939,
          1.1020408163265305,
          1.2244897959183674,
          1.346938775510204,
          1.4693877551020407,
          1.5918367346938775,
          1.7142857142857142,
          1.836734693877551,
          1.9591836734693877,
          2.0816326530612246,
          2.204081632653061,
          2.326530612244898,
          2.4489795918367347,
          2.571428571428571,
          2.693877551020408,
          2.816326530612245,
          2.9387755102040813,
          3.061224489795918,
          3.183673469387755,
          3.306122448979592,
          3.4285714285714284,
          3.5510204081632653,
          3.673469387755102,
          3.7959183673469385,
          3.9183673469387754,
          4.040816326530612,
          4.163265306122449,
          4.285714285714286,
          4.408163265306122,
          4.530612244897959,
          4.653061224489796,
          4.775510204081632,
          4.8979591836734695,
          5.020408163265306,
          5.142857142857142,
          5.26530612244898,
          5.387755102040816,
          5.5102040816326525,
          5.63265306122449,
          5.755102040816326,
          5.877551020408163,
          6
         ],
         "y": [
          0,
          0.05654761074620635,
          0.10893378804666826,
          0.15576674239243876,
          0.19617100656924708,
          0.23001622762941615,
          0.25779752325589206,
          0.28031952743733446,
          0.298444605731176,
          0.312966237870432,
          0.32456622691352277,
          0.33381167338445483,
          0.3411667693712719,
          0.3470083087263204,
          0.3516406539341158,
          0.3553088687209171,
          0.35820972901345516,
          0.3605008960176747,
          0.36230840264771436,
          0.3637328085848013,
          0.364854192437121,
          0.36573619015638587,
          0.36642931809935264,
          0.3669735841613463,
          0.3674006478589418,
          0.36773551693828394,
          0.36799793694538896,
          0.3682034538821892,
          0.36836433120600515,
          0.3684902006144745,
          0.36858862927790315,
          0.36866557165899777,
          0.3687256871409768,
          0.3687726485484641,
          0.3688093220756966,
          0.3688379498148002,
          0.36886027603953764,
          0.36887770542867765,
          0.36889130033904133,
          0.368901899331075,
          0.36891016362860296,
          0.36891660044147007,
          0.3689216174866076,
          0.36892552364913217,
          0.36892856820522735,
          0.36893094069992644,
          0.3689327906087841,
          0.3689342399525199,
          0.36893536580804265,
          0.3689362338075601
         ]
        },
        {
         "line": {
          "color": "rgb(8, 173, 99)",
          "width": 1
         },
         "mode": "lines+markers",
         "name": "best_xy",
         "type": "scatter",
         "uid": "5c479130-a35c-4acd-ac9b-dfd8e83bff2e",
         "x": [
          2,
          1.436581963587984,
          0.9696813817069801,
          0.6037645827844602,
          0.3297921045100084,
          0.13105453462244457,
          -0.010232517338564335,
          -0.10937556649944526,
          -0.1782864612071781,
          -0.2257873393913745,
          -0.2582518050705372,
          -0.28022479902923064,
          -0.294922593030897,
          -0.3046088680265031,
          -0.31086953985409943,
          -0.31481027332807776,
          -0.31719782406609276,
          -0.3185605691636698,
          -0.31925991957613686,
          -0.3195408060355251,
          -0.3195673718246092,
          -0.31944804806327165,
          -0.31925312753621327,
          -0.3190270368476306,
          -0.31879686378820005,
          -0.3185782234979382,
          -0.31837928286496364,
          -0.3182034729367215,
          -0.31805137419076124,
          -0.3179218825359452,
          -0.317813006150125,
          -0.3177223756088433,
          -0.3176475526541967,
          -0.31758619192908394,
          -0.31753616311436,
          -0.3174955696739524,
          -0.3174627753689424,
          -0.31743638194744755,
          -0.3174152058966373,
          -0.31739826231258284,
          -0.3173847388545805,
          -0.31737397182441696,
          -0.3173654159623826,
          -0.31735863389935026,
          -0.3173532602614974,
          -0.3173490124570506,
          -0.3173456546024602,
          -0.3173429939602287,
          -0.31734089488693396,
          -0.31733926026226716
         ],
         "xaxis": "x2",
         "y": [
          0,
          0.05654761074620635,
          0.10893378804666826,
          0.15576674239243876,
          0.19617100656924708,
          0.23001622762941615,
          0.25779752325589206,
          0.28031952743733446,
          0.298444605731176,
          0.312966237870432,
          0.32456622691352277,
          0.33381167338445483,
          0.3411667693712719,
          0.3470083087263204,
          0.3516406539341158,
          0.3553088687209171,
          0.35820972901345516,
          0.3605008960176747,
          0.36230840264771436,
          0.3637328085848013,
          0.364854192437121,
          0.36573619015638587,
          0.36642931809935264,
          0.3669735841613463,
          0.3674006478589418,
          0.36773551693828394,
          0.36799793694538896,
          0.3682034538821892,
          0.36836433120600515,
          0.3684902006144745,
          0.36858862927790315,
          0.36866557165899777,
          0.3687256871409768,
          0.3687726485484641,
          0.3688093220756966,
          0.3688379498148002,
          0.36886027603953764,
          0.36887770542867765,
          0.36889130033904133,
          0.368901899331075,
          0.36891016362860296,
          0.36891660044147007,
          0.3689216174866076,
          0.36892552364913217,
          0.36892856820522735,
          0.36893094069992644,
          0.3689327906087841,
          0.3689342399525199,
          0.36893536580804265,
          0.3689362338075601
         ],
         "yaxis": "y2"
        },
        {
         "line": {
          "color": "rgb(34, 119, 158)",
          "dash": "dot",
          "width": 2
         },
         "name": "groundtruth_x",
         "type": "scatter",
         "uid": "7f4be7c1-4a4c-43a5-bcef-c3a63eec6a92",
         "x": [
          0,
          0.12244897959183673,
          0.24489795918367346,
          0.36734693877551017,
          0.4897959183673469,
          0.6122448979591837,
          0.7346938775510203,
          0.8571428571428571,
          0.9795918367346939,
          1.1020408163265305,
          1.2244897959183674,
          1.346938775510204,
          1.4693877551020407,
          1.5918367346938775,
          1.7142857142857142,
          1.836734693877551,
          1.9591836734693877,
          2.0816326530612246,
          2.204081632653061,
          2.326530612244898,
          2.4489795918367347,
          2.571428571428571,
          2.693877551020408,
          2.816326530612245,
          2.9387755102040813,
          3.061224489795918,
          3.183673469387755,
          3.306122448979592,
          3.4285714285714284,
          3.5510204081632653,
          3.673469387755102,
          3.7959183673469385,
          3.9183673469387754,
          4.040816326530612,
          4.163265306122449,
          4.285714285714286,
          4.408163265306122,
          4.530612244897959,
          4.653061224489796,
          4.775510204081632,
          4.8979591836734695,
          5.020408163265306,
          5.142857142857142,
          5.26530612244898,
          5.387755102040816,
          5.5102040816326525,
          5.63265306122449,
          5.755102040816326,
          5.877551020408163,
          6
         ],
         "y": [
          2,
          1.5898879908691512,
          0.09116942641742595,
          -1.2892733499556064,
          -1.7165722278067121,
          -1.6573383762683993,
          -1.2653104863867892,
          -0.3448977664688868,
          0.5682569166303237,
          1.26649086372092,
          1.4530781713391205,
          1.4245766699976528,
          1.344648982234877,
          1.0068703140648665,
          0.4442478277248246,
          -0.12666749748682002,
          -0.6554371459430208,
          -1.057511259522562,
          -1.2181668044692728,
          -1.224144014204532,
          -1.2023429492275,
          -1.159771559853107,
          -1.0177113207432569,
          -0.7362904289965733,
          -0.3866537610747874,
          -0.03693757942190929,
          0.29735850533321767,
          0.6073011803455556,
          0.8548867068620034,
          0.9959188043497199,
          1.0396474121880794,
          1.0363370876827478,
          1.0232723007911548,
          1.0075545529869299,
          0.9714057912086662,
          0.8856605224634109,
          0.7357719346072075,
          0.5386279402870494,
          0.3240740556179248,
          0.11077027099574432,
          -0.09604216601885016,
          -0.29569168161482734,
          -0.4837677418636904,
          -0.6478248104004547,
          -0.7711576939697778,
          -0.8449592922377955,
          -0.8761546988691105,
          -0.8812318472485765,
          -0.8754213849225162,
          -0.8674041731287144
         ]
        },
        {
         "line": {
          "color": "rgb(201, 79, 14)",
          "dash": "dot",
          "width": 2
         },
         "name": "groundtruth_y",
         "type": "scatter",
         "uid": "c578e072-98be-42aa-9a4a-f10b918499d3",
         "x": [
          0,
          0.12244897959183673,
          0.24489795918367346,
          0.36734693877551017,
          0.4897959183673469,
          0.6122448979591837,
          0.7346938775510203,
          0.8571428571428571,
          0.9795918367346939,
          1.1020408163265305,
          1.2244897959183674,
          1.346938775510204,
          1.4693877551020407,
          1.5918367346938775,
          1.7142857142857142,
          1.836734693877551,
          1.9591836734693877,
          2.0816326530612246,
          2.204081632653061,
          2.326530612244898,
          2.4489795918367347,
          2.571428571428571,
          2.693877551020408,
          2.816326530612245,
          2.9387755102040813,
          3.061224489795918,
          3.183673469387755,
          3.306122448979592,
          3.4285714285714284,
          3.5510204081632653,
          3.673469387755102,
          3.7959183673469385,
          3.9183673469387754,
          4.040816326530612,
          4.163265306122449,
          4.285714285714286,
          4.408163265306122,
          4.530612244897959,
          4.653061224489796,
          4.775510204081632,
          4.8979591836734695,
          5.020408163265306,
          5.142857142857142,
          5.26530612244898,
          5.387755102040816,
          5.5102040816326525,
          5.63265306122449,
          5.755102040816326,
          5.877551020408163,
          6
         ],
         "y": [
          0,
          1.6347817545429182,
          1.844674383806059,
          1.6378754737516776,
          0.5998667122634451,
          -0.5858387521501,
          -1.4411780782906123,
          -1.5740080287701013,
          -1.5219753216299392,
          -1.246228749011831,
          -0.563979027922285,
          0.17013422399462957,
          0.8339999042616996,
          1.2538388566248573,
          1.3366957083936806,
          1.3122265055435145,
          1.2637416972353306,
          1.071145121769598,
          0.6805697949460937,
          0.22651994818854593,
          -0.2110414557078644,
          -0.618010917261637,
          -0.9395152275568366,
          -1.1009851626928493,
          -1.13164900965241,
          -1.1180436278539654,
          -1.0998703474931357,
          -1.0584917104036367,
          -0.9444137589948657,
          -0.735567231443964,
          -0.46969440862493866,
          -0.19377068428574815,
          0.07376774145461143,
          0.3305289903480158,
          0.5687452211048643,
          0.7647389541573506,
          0.8916711271367963,
          0.9476595221233626,
          0.9579168432316844,
          0.950365550632471,
          0.940043438875845,
          0.9277165512939757,
          0.9027774758134423,
          0.8485228199192166,
          0.7525208133424066,
          0.6170523010719102,
          0.45738255489858415,
          0.28973124380035203,
          0.12341746107635407,
          -0.038649916191955624
         ]
        },
        {
         "line": {
          "color": "rgb(173, 7, 7)",
          "dash": "dot",
          "width": 2
         },
         "name": "groundtruth_xy",
         "type": "scatter",
         "uid": "f0d8e031-eb66-41fc-afb3-756087438c94",
         "x": [
          2,
          1.5898879908691512,
          0.09116942641742595,
          -1.2892733499556064,
          -1.7165722278067121,
          -1.6573383762683993,
          -1.2653104863867892,
          -0.3448977664688868,
          0.5682569166303237,
          1.26649086372092,
          1.4530781713391205,
          1.4245766699976528,
          1.344648982234877,
          1.0068703140648665,
          0.4442478277248246,
          -0.12666749748682002,
          -0.6554371459430208,
          -1.057511259522562,
          -1.2181668044692728,
          -1.224144014204532,
          -1.2023429492275,
          -1.159771559853107,
          -1.0177113207432569,
          -0.7362904289965733,
          -0.3866537610747874,
          -0.03693757942190929,
          0.29735850533321767,
          0.6073011803455556,
          0.8548867068620034,
          0.9959188043497199,
          1.0396474121880794,
          1.0363370876827478,
          1.0232723007911548,
          1.0075545529869299,
          0.9714057912086662,
          0.8856605224634109,
          0.7357719346072075,
          0.5386279402870494,
          0.3240740556179248,
          0.11077027099574432,
          -0.09604216601885016,
          -0.29569168161482734,
          -0.4837677418636904,
          -0.6478248104004547,
          -0.7711576939697778,
          -0.8449592922377955,
          -0.8761546988691105,
          -0.8812318472485765,
          -0.8754213849225162,
          -0.8674041731287144
         ],
         "xaxis": "x2",
         "y": [
          0,
          1.6347817545429182,
          1.844674383806059,
          1.6378754737516776,
          0.5998667122634451,
          -0.5858387521501,
          -1.4411780782906123,
          -1.5740080287701013,
          -1.5219753216299392,
          -1.246228749011831,
          -0.563979027922285,
          0.17013422399462957,
          0.8339999042616996,
          1.2538388566248573,
          1.3366957083936806,
          1.3122265055435145,
          1.2637416972353306,
          1.071145121769598,
          0.6805697949460937,
          0.22651994818854593,
          -0.2110414557078644,
          -0.618010917261637,
          -0.9395152275568366,
          -1.1009851626928493,
          -1.13164900965241,
          -1.1180436278539654,
          -1.0998703474931357,
          -1.0584917104036367,
          -0.9444137589948657,
          -0.735567231443964,
          -0.46969440862493866,
          -0.19377068428574815,
          0.07376774145461143,
          0.3305289903480158,
          0.5687452211048643,
          0.7647389541573506,
          0.8916711271367963,
          0.9476595221233626,
          0.9579168432316844,
          0.950365550632471,
          0.940043438875845,
          0.9277165512939757,
          0.9027774758134423,
          0.8485228199192166,
          0.7525208133424066,
          0.6170523010719102,
          0.45738255489858415,
          0.28973124380035203,
          0.12341746107635407,
          -0.038649916191955624
         ],
         "yaxis": "y2"
        },
        {
         "line": {
          "color": "rgb(24, 24, 24)",
          "dash": "dash",
          "width": 1
         },
         "name": "current_x",
         "type": "scatter",
         "uid": "9516cbb1-c2d2-4fbc-863f-1d64a9752b66",
         "x": [
          0,
          0.12244897959183673,
          0.24489795918367346,
          0.36734693877551017,
          0.4897959183673469,
          0.6122448979591837,
          0.7346938775510203,
          0.8571428571428571,
          0.9795918367346939,
          1.1020408163265305,
          1.2244897959183674,
          1.346938775510204,
          1.4693877551020407,
          1.5918367346938775,
          1.7142857142857142,
          1.836734693877551,
          1.9591836734693877,
          2.0816326530612246,
          2.204081632653061,
          2.326530612244898,
          2.4489795918367347,
          2.571428571428571,
          2.693877551020408,
          2.816326530612245,
          2.9387755102040813,
          3.061224489795918,
          3.183673469387755,
          3.306122448979592,
          3.4285714285714284,
          3.5510204081632653,
          3.673469387755102,
          3.7959183673469385,
          3.9183673469387754,
          4.040816326530612,
          4.163265306122449,
          4.285714285714286,
          4.408163265306122,
          4.530612244897959,
          4.653061224489796,
          4.775510204081632,
          4.8979591836734695,
          5.020408163265306,
          5.142857142857142,
          5.26530612244898,
          5.387755102040816,
          5.5102040816326525,
          5.63265306122449,
          5.755102040816326,
          5.877551020408163,
          6
         ],
         "y": [
          2,
          1.436581963587984,
          0.9696813817069801,
          0.6037645827844602,
          0.3297921045100084,
          0.13105453462244457,
          -0.010232517338564335,
          -0.10937556649944526,
          -0.1782864612071781,
          -0.2257873393913745,
          -0.2582518050705372,
          -0.28022479902923064,
          -0.294922593030897,
          -0.3046088680265031,
          -0.31086953985409943,
          -0.31481027332807776,
          -0.31719782406609276,
          -0.3185605691636698,
          -0.31925991957613686,
          -0.3195408060355251,
          -0.3195673718246092,
          -0.31944804806327165,
          -0.31925312753621327,
          -0.3190270368476306,
          -0.31879686378820005,
          -0.3185782234979382,
          -0.31837928286496364,
          -0.3182034729367215,
          -0.31805137419076124,
          -0.3179218825359452,
          -0.317813006150125,
          -0.3177223756088433,
          -0.3176475526541967,
          -0.31758619192908394,
          -0.31753616311436,
          -0.3174955696739524,
          -0.3174627753689424,
          -0.31743638194744755,
          -0.3174152058966373,
          -0.31739826231258284,
          -0.3173847388545805,
          -0.31737397182441696,
          -0.3173654159623826,
          -0.31735863389935026,
          -0.3173532602614974,
          -0.3173490124570506,
          -0.3173456546024602,
          -0.3173429939602287,
          -0.31734089488693396,
          -0.31733926026226716
         ]
        },
        {
         "line": {
          "color": "rgb(24, 24, 24)",
          "dash": "dash",
          "width": 1
         },
         "name": "current_y",
         "type": "scatter",
         "uid": "2e28f5ee-22cb-47b2-826c-03a8c21444ed",
         "x": [
          0,
          0.12244897959183673,
          0.24489795918367346,
          0.36734693877551017,
          0.4897959183673469,
          0.6122448979591837,
          0.7346938775510203,
          0.8571428571428571,
          0.9795918367346939,
          1.1020408163265305,
          1.2244897959183674,
          1.346938775510204,
          1.4693877551020407,
          1.5918367346938775,
          1.7142857142857142,
          1.836734693877551,
          1.9591836734693877,
          2.0816326530612246,
          2.204081632653061,
          2.326530612244898,
          2.4489795918367347,
          2.571428571428571,
          2.693877551020408,
          2.816326530612245,
          2.9387755102040813,
          3.061224489795918,
          3.183673469387755,
          3.306122448979592,
          3.4285714285714284,
          3.5510204081632653,
          3.673469387755102,
          3.7959183673469385,
          3.9183673469387754,
          4.040816326530612,
          4.163265306122449,
          4.285714285714286,
          4.408163265306122,
          4.530612244897959,
          4.653061224489796,
          4.775510204081632,
          4.8979591836734695,
          5.020408163265306,
          5.142857142857142,
          5.26530612244898,
          5.387755102040816,
          5.5102040816326525,
          5.63265306122449,
          5.755102040816326,
          5.877551020408163,
          6
         ],
         "y": [
          0,
          0.05654761074620635,
          0.10893378804666826,
          0.15576674239243876,
          0.19617100656924708,
          0.23001622762941615,
          0.25779752325589206,
          0.28031952743733446,
          0.298444605731176,
          0.312966237870432,
          0.32456622691352277,
          0.33381167338445483,
          0.3411667693712719,
          0.3470083087263204,
          0.3516406539341158,
          0.3553088687209171,
          0.35820972901345516,
          0.3605008960176747,
          0.36230840264771436,
          0.3637328085848013,
          0.364854192437121,
          0.36573619015638587,
          0.36642931809935264,
          0.3669735841613463,
          0.3674006478589418,
          0.36773551693828394,
          0.36799793694538896,
          0.3682034538821892,
          0.36836433120600515,
          0.3684902006144745,
          0.36858862927790315,
          0.36866557165899777,
          0.3687256871409768,
          0.3687726485484641,
          0.3688093220756966,
          0.3688379498148002,
          0.36886027603953764,
          0.36887770542867765,
          0.36889130033904133,
          0.368901899331075,
          0.36891016362860296,
          0.36891660044147007,
          0.3689216174866076,
          0.36892552364913217,
          0.36892856820522735,
          0.36893094069992644,
          0.3689327906087841,
          0.3689342399525199,
          0.36893536580804265,
          0.3689362338075601
         ]
        },
        {
         "line": {
          "color": "rgb(24, 24, 24)",
          "dash": "dash",
          "width": 2
         },
         "name": "current_xy",
         "type": "scatter",
         "uid": "79e694c7-7b30-42b4-a01f-f41412e7b34c",
         "x": [
          2,
          1.436581963587984,
          0.9696813817069801,
          0.6037645827844602,
          0.3297921045100084,
          0.13105453462244457,
          -0.010232517338564335,
          -0.10937556649944526,
          -0.1782864612071781,
          -0.2257873393913745,
          -0.2582518050705372,
          -0.28022479902923064,
          -0.294922593030897,
          -0.3046088680265031,
          -0.31086953985409943,
          -0.31481027332807776,
          -0.31719782406609276,
          -0.3185605691636698,
          -0.31925991957613686,
          -0.3195408060355251,
          -0.3195673718246092,
          -0.31944804806327165,
          -0.31925312753621327,
          -0.3190270368476306,
          -0.31879686378820005,
          -0.3185782234979382,
          -0.31837928286496364,
          -0.3182034729367215,
          -0.31805137419076124,
          -0.3179218825359452,
          -0.317813006150125,
          -0.3177223756088433,
          -0.3176475526541967,
          -0.31758619192908394,
          -0.31753616311436,
          -0.3174955696739524,
          -0.3174627753689424,
          -0.31743638194744755,
          -0.3174152058966373,
          -0.31739826231258284,
          -0.3173847388545805,
          -0.31737397182441696,
          -0.3173654159623826,
          -0.31735863389935026,
          -0.3173532602614974,
          -0.3173490124570506,
          -0.3173456546024602,
          -0.3173429939602287,
          -0.31734089488693396,
          -0.31733926026226716
         ],
         "xaxis": "x2",
         "y": [
          0,
          0.05654761074620635,
          0.10893378804666826,
          0.15576674239243876,
          0.19617100656924708,
          0.23001622762941615,
          0.25779752325589206,
          0.28031952743733446,
          0.298444605731176,
          0.312966237870432,
          0.32456622691352277,
          0.33381167338445483,
          0.3411667693712719,
          0.3470083087263204,
          0.3516406539341158,
          0.3553088687209171,
          0.35820972901345516,
          0.3605008960176747,
          0.36230840264771436,
          0.3637328085848013,
          0.364854192437121,
          0.36573619015638587,
          0.36642931809935264,
          0.3669735841613463,
          0.3674006478589418,
          0.36773551693828394,
          0.36799793694538896,
          0.3682034538821892,
          0.36836433120600515,
          0.3684902006144745,
          0.36858862927790315,
          0.36866557165899777,
          0.3687256871409768,
          0.3687726485484641,
          0.3688093220756966,
          0.3688379498148002,
          0.36886027603953764,
          0.36887770542867765,
          0.36889130033904133,
          0.368901899331075,
          0.36891016362860296,
          0.36891660044147007,
          0.3689216174866076,
          0.36892552364913217,
          0.36892856820522735,
          0.36893094069992644,
          0.3689327906087841,
          0.3689342399525199,
          0.36893536580804265,
          0.3689362338075601
         ],
         "yaxis": "y2"
        }
       ],
       "layout": {
        "height": 680,
        "title": {
         "text": "Step 5<br>Best Loss 97.33419046026196<br>Current Loss 97.33419046026196"
        },
        "width": 1024,
        "xaxis": {
         "range": [
          -0.25,
          6.25
         ]
        },
        "xaxis2": {
         "range": [
          -2.5,
          2.5
         ]
        },
        "yaxis": {
         "domain": [
          0.55,
          1
         ],
         "range": [
          -2.5,
          2.5
         ]
        },
        "yaxis2": {
         "domain": [
          0,
          0.45
         ],
         "range": [
          -2.5,
          2.5
         ]
        }
       }
      },
      "text/html": [
       "<div id=\"8ed2c56a-90b9-4ee6-8664-c5cde61b20e7\" style=\"height: 680px; width: 1024px;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"8ed2c56a-90b9-4ee6-8664-c5cde61b20e7\", [{\"line\": {\"color\": \"rgb(34, 119, 158)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_x\", \"x\": [0.0, 0.12244897959183673, 0.24489795918367346, 0.36734693877551017, 0.4897959183673469, 0.6122448979591837, 0.7346938775510203, 0.8571428571428571, 0.9795918367346939, 1.1020408163265305, 1.2244897959183674, 1.346938775510204, 1.4693877551020407, 1.5918367346938775, 1.7142857142857142, 1.836734693877551, 1.9591836734693877, 2.0816326530612246, 2.204081632653061, 2.326530612244898, 2.4489795918367347, 2.571428571428571, 2.693877551020408, 2.816326530612245, 2.9387755102040813, 3.061224489795918, 3.183673469387755, 3.306122448979592, 3.4285714285714284, 3.5510204081632653, 3.673469387755102, 3.7959183673469385, 3.9183673469387754, 4.040816326530612, 4.163265306122449, 4.285714285714286, 4.408163265306122, 4.530612244897959, 4.653061224489796, 4.775510204081632, 4.8979591836734695, 5.020408163265306, 5.142857142857142, 5.26530612244898, 5.387755102040816, 5.5102040816326525, 5.63265306122449, 5.755102040816326, 5.877551020408163, 6.0], \"y\": [2.0, 1.436581963587984, 0.9696813817069801, 0.6037645827844602, 0.3297921045100084, 0.13105453462244457, -0.010232517338564335, -0.10937556649944526, -0.1782864612071781, -0.2257873393913745, -0.2582518050705372, -0.28022479902923064, -0.294922593030897, -0.3046088680265031, -0.31086953985409943, -0.31481027332807776, -0.31719782406609276, -0.3185605691636698, -0.31925991957613686, -0.3195408060355251, -0.3195673718246092, -0.31944804806327165, -0.31925312753621327, -0.3190270368476306, -0.31879686378820005, -0.3185782234979382, -0.31837928286496364, -0.3182034729367215, -0.31805137419076124, -0.3179218825359452, -0.317813006150125, -0.3177223756088433, -0.3176475526541967, -0.31758619192908394, -0.31753616311436, -0.3174955696739524, -0.3174627753689424, -0.31743638194744755, -0.3174152058966373, -0.31739826231258284, -0.3173847388545805, -0.31737397182441696, -0.3173654159623826, -0.31735863389935026, -0.3173532602614974, -0.3173490124570506, -0.3173456546024602, -0.3173429939602287, -0.31734089488693396, -0.31733926026226716], \"type\": \"scatter\", \"uid\": \"5b80efc4-89a2-499e-af1f-3c80959d6d75\"}, {\"line\": {\"color\": \"rgb(201, 79, 14)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_y\", \"x\": [0.0, 0.12244897959183673, 0.24489795918367346, 0.36734693877551017, 0.4897959183673469, 0.6122448979591837, 0.7346938775510203, 0.8571428571428571, 0.9795918367346939, 1.1020408163265305, 1.2244897959183674, 1.346938775510204, 1.4693877551020407, 1.5918367346938775, 1.7142857142857142, 1.836734693877551, 1.9591836734693877, 2.0816326530612246, 2.204081632653061, 2.326530612244898, 2.4489795918367347, 2.571428571428571, 2.693877551020408, 2.816326530612245, 2.9387755102040813, 3.061224489795918, 3.183673469387755, 3.306122448979592, 3.4285714285714284, 3.5510204081632653, 3.673469387755102, 3.7959183673469385, 3.9183673469387754, 4.040816326530612, 4.163265306122449, 4.285714285714286, 4.408163265306122, 4.530612244897959, 4.653061224489796, 4.775510204081632, 4.8979591836734695, 5.020408163265306, 5.142857142857142, 5.26530612244898, 5.387755102040816, 5.5102040816326525, 5.63265306122449, 5.755102040816326, 5.877551020408163, 6.0], \"y\": [0.0, 0.05654761074620635, 0.10893378804666826, 0.15576674239243876, 0.19617100656924708, 0.23001622762941615, 0.25779752325589206, 0.28031952743733446, 0.298444605731176, 0.312966237870432, 0.32456622691352277, 0.33381167338445483, 0.3411667693712719, 0.3470083087263204, 0.3516406539341158, 0.3553088687209171, 0.35820972901345516, 0.3605008960176747, 0.36230840264771436, 0.3637328085848013, 0.364854192437121, 0.36573619015638587, 0.36642931809935264, 0.3669735841613463, 0.3674006478589418, 0.36773551693828394, 0.36799793694538896, 0.3682034538821892, 0.36836433120600515, 0.3684902006144745, 0.36858862927790315, 0.36866557165899777, 0.3687256871409768, 0.3687726485484641, 0.3688093220756966, 0.3688379498148002, 0.36886027603953764, 0.36887770542867765, 0.36889130033904133, 0.368901899331075, 0.36891016362860296, 0.36891660044147007, 0.3689216174866076, 0.36892552364913217, 0.36892856820522735, 0.36893094069992644, 0.3689327906087841, 0.3689342399525199, 0.36893536580804265, 0.3689362338075601], \"type\": \"scatter\", \"uid\": \"748da336-c1ee-4018-931e-7fab0127a954\"}, {\"line\": {\"color\": \"rgb(8, 173, 99)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_xy\", \"x\": [2.0, 1.436581963587984, 0.9696813817069801, 0.6037645827844602, 0.3297921045100084, 0.13105453462244457, -0.010232517338564335, -0.10937556649944526, -0.1782864612071781, -0.2257873393913745, -0.2582518050705372, -0.28022479902923064, -0.294922593030897, -0.3046088680265031, -0.31086953985409943, -0.31481027332807776, -0.31719782406609276, -0.3185605691636698, -0.31925991957613686, -0.3195408060355251, -0.3195673718246092, -0.31944804806327165, -0.31925312753621327, -0.3190270368476306, -0.31879686378820005, -0.3185782234979382, -0.31837928286496364, -0.3182034729367215, -0.31805137419076124, -0.3179218825359452, -0.317813006150125, -0.3177223756088433, -0.3176475526541967, -0.31758619192908394, -0.31753616311436, -0.3174955696739524, -0.3174627753689424, -0.31743638194744755, -0.3174152058966373, -0.31739826231258284, -0.3173847388545805, -0.31737397182441696, -0.3173654159623826, -0.31735863389935026, -0.3173532602614974, -0.3173490124570506, -0.3173456546024602, -0.3173429939602287, -0.31734089488693396, -0.31733926026226716], \"xaxis\": \"x2\", \"y\": [0.0, 0.05654761074620635, 0.10893378804666826, 0.15576674239243876, 0.19617100656924708, 0.23001622762941615, 0.25779752325589206, 0.28031952743733446, 0.298444605731176, 0.312966237870432, 0.32456622691352277, 0.33381167338445483, 0.3411667693712719, 0.3470083087263204, 0.3516406539341158, 0.3553088687209171, 0.35820972901345516, 0.3605008960176747, 0.36230840264771436, 0.3637328085848013, 0.364854192437121, 0.36573619015638587, 0.36642931809935264, 0.3669735841613463, 0.3674006478589418, 0.36773551693828394, 0.36799793694538896, 0.3682034538821892, 0.36836433120600515, 0.3684902006144745, 0.36858862927790315, 0.36866557165899777, 0.3687256871409768, 0.3687726485484641, 0.3688093220756966, 0.3688379498148002, 0.36886027603953764, 0.36887770542867765, 0.36889130033904133, 0.368901899331075, 0.36891016362860296, 0.36891660044147007, 0.3689216174866076, 0.36892552364913217, 0.36892856820522735, 0.36893094069992644, 0.3689327906087841, 0.3689342399525199, 0.36893536580804265, 0.3689362338075601], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"5c479130-a35c-4acd-ac9b-dfd8e83bff2e\"}, {\"line\": {\"color\": \"rgb(34, 119, 158)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_x\", \"x\": [0.0, 0.12244897959183673, 0.24489795918367346, 0.36734693877551017, 0.4897959183673469, 0.6122448979591837, 0.7346938775510203, 0.8571428571428571, 0.9795918367346939, 1.1020408163265305, 1.2244897959183674, 1.346938775510204, 1.4693877551020407, 1.5918367346938775, 1.7142857142857142, 1.836734693877551, 1.9591836734693877, 2.0816326530612246, 2.204081632653061, 2.326530612244898, 2.4489795918367347, 2.571428571428571, 2.693877551020408, 2.816326530612245, 2.9387755102040813, 3.061224489795918, 3.183673469387755, 3.306122448979592, 3.4285714285714284, 3.5510204081632653, 3.673469387755102, 3.7959183673469385, 3.9183673469387754, 4.040816326530612, 4.163265306122449, 4.285714285714286, 4.408163265306122, 4.530612244897959, 4.653061224489796, 4.775510204081632, 4.8979591836734695, 5.020408163265306, 5.142857142857142, 5.26530612244898, 5.387755102040816, 5.5102040816326525, 5.63265306122449, 5.755102040816326, 5.877551020408163, 6.0], \"y\": [2.0, 1.5898879908691512, 0.09116942641742595, -1.2892733499556064, -1.7165722278067121, -1.6573383762683993, -1.2653104863867892, -0.3448977664688868, 0.5682569166303237, 1.26649086372092, 1.4530781713391205, 1.4245766699976528, 1.344648982234877, 1.0068703140648665, 0.4442478277248246, -0.12666749748682002, -0.6554371459430208, -1.057511259522562, -1.2181668044692728, -1.224144014204532, -1.2023429492275, -1.159771559853107, -1.0177113207432569, -0.7362904289965733, -0.3866537610747874, -0.03693757942190929, 0.29735850533321767, 0.6073011803455556, 0.8548867068620034, 0.9959188043497199, 1.0396474121880794, 1.0363370876827478, 1.0232723007911548, 1.0075545529869299, 0.9714057912086662, 0.8856605224634109, 0.7357719346072075, 0.5386279402870494, 0.3240740556179248, 0.11077027099574432, -0.09604216601885016, -0.29569168161482734, -0.4837677418636904, -0.6478248104004547, -0.7711576939697778, -0.8449592922377955, -0.8761546988691105, -0.8812318472485765, -0.8754213849225162, -0.8674041731287144], \"type\": \"scatter\", \"uid\": \"7f4be7c1-4a4c-43a5-bcef-c3a63eec6a92\"}, {\"line\": {\"color\": \"rgb(201, 79, 14)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_y\", \"x\": [0.0, 0.12244897959183673, 0.24489795918367346, 0.36734693877551017, 0.4897959183673469, 0.6122448979591837, 0.7346938775510203, 0.8571428571428571, 0.9795918367346939, 1.1020408163265305, 1.2244897959183674, 1.346938775510204, 1.4693877551020407, 1.5918367346938775, 1.7142857142857142, 1.836734693877551, 1.9591836734693877, 2.0816326530612246, 2.204081632653061, 2.326530612244898, 2.4489795918367347, 2.571428571428571, 2.693877551020408, 2.816326530612245, 2.9387755102040813, 3.061224489795918, 3.183673469387755, 3.306122448979592, 3.4285714285714284, 3.5510204081632653, 3.673469387755102, 3.7959183673469385, 3.9183673469387754, 4.040816326530612, 4.163265306122449, 4.285714285714286, 4.408163265306122, 4.530612244897959, 4.653061224489796, 4.775510204081632, 4.8979591836734695, 5.020408163265306, 5.142857142857142, 5.26530612244898, 5.387755102040816, 5.5102040816326525, 5.63265306122449, 5.755102040816326, 5.877551020408163, 6.0], \"y\": [0.0, 1.6347817545429182, 1.844674383806059, 1.6378754737516776, 0.5998667122634451, -0.5858387521501, -1.4411780782906123, -1.5740080287701013, -1.5219753216299392, -1.246228749011831, -0.563979027922285, 0.17013422399462957, 0.8339999042616996, 1.2538388566248573, 1.3366957083936806, 1.3122265055435145, 1.2637416972353306, 1.071145121769598, 0.6805697949460937, 0.22651994818854593, -0.2110414557078644, -0.618010917261637, -0.9395152275568366, -1.1009851626928493, -1.13164900965241, -1.1180436278539654, -1.0998703474931357, -1.0584917104036367, -0.9444137589948657, -0.735567231443964, -0.46969440862493866, -0.19377068428574815, 0.07376774145461143, 0.3305289903480158, 0.5687452211048643, 0.7647389541573506, 0.8916711271367963, 0.9476595221233626, 0.9579168432316844, 0.950365550632471, 0.940043438875845, 0.9277165512939757, 0.9027774758134423, 0.8485228199192166, 0.7525208133424066, 0.6170523010719102, 0.45738255489858415, 0.28973124380035203, 0.12341746107635407, -0.038649916191955624], \"type\": \"scatter\", \"uid\": \"c578e072-98be-42aa-9a4a-f10b918499d3\"}, {\"line\": {\"color\": \"rgb(173, 7, 7)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_xy\", \"x\": [2.0, 1.5898879908691512, 0.09116942641742595, -1.2892733499556064, -1.7165722278067121, -1.6573383762683993, -1.2653104863867892, -0.3448977664688868, 0.5682569166303237, 1.26649086372092, 1.4530781713391205, 1.4245766699976528, 1.344648982234877, 1.0068703140648665, 0.4442478277248246, -0.12666749748682002, -0.6554371459430208, -1.057511259522562, -1.2181668044692728, -1.224144014204532, -1.2023429492275, -1.159771559853107, -1.0177113207432569, -0.7362904289965733, -0.3866537610747874, -0.03693757942190929, 0.29735850533321767, 0.6073011803455556, 0.8548867068620034, 0.9959188043497199, 1.0396474121880794, 1.0363370876827478, 1.0232723007911548, 1.0075545529869299, 0.9714057912086662, 0.8856605224634109, 0.7357719346072075, 0.5386279402870494, 0.3240740556179248, 0.11077027099574432, -0.09604216601885016, -0.29569168161482734, -0.4837677418636904, -0.6478248104004547, -0.7711576939697778, -0.8449592922377955, -0.8761546988691105, -0.8812318472485765, -0.8754213849225162, -0.8674041731287144], \"xaxis\": \"x2\", \"y\": [0.0, 1.6347817545429182, 1.844674383806059, 1.6378754737516776, 0.5998667122634451, -0.5858387521501, -1.4411780782906123, -1.5740080287701013, -1.5219753216299392, -1.246228749011831, -0.563979027922285, 0.17013422399462957, 0.8339999042616996, 1.2538388566248573, 1.3366957083936806, 1.3122265055435145, 1.2637416972353306, 1.071145121769598, 0.6805697949460937, 0.22651994818854593, -0.2110414557078644, -0.618010917261637, -0.9395152275568366, -1.1009851626928493, -1.13164900965241, -1.1180436278539654, -1.0998703474931357, -1.0584917104036367, -0.9444137589948657, -0.735567231443964, -0.46969440862493866, -0.19377068428574815, 0.07376774145461143, 0.3305289903480158, 0.5687452211048643, 0.7647389541573506, 0.8916711271367963, 0.9476595221233626, 0.9579168432316844, 0.950365550632471, 0.940043438875845, 0.9277165512939757, 0.9027774758134423, 0.8485228199192166, 0.7525208133424066, 0.6170523010719102, 0.45738255489858415, 0.28973124380035203, 0.12341746107635407, -0.038649916191955624], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"f0d8e031-eb66-41fc-afb3-756087438c94\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 1}, \"name\": \"current_x\", \"x\": [0.0, 0.12244897959183673, 0.24489795918367346, 0.36734693877551017, 0.4897959183673469, 0.6122448979591837, 0.7346938775510203, 0.8571428571428571, 0.9795918367346939, 1.1020408163265305, 1.2244897959183674, 1.346938775510204, 1.4693877551020407, 1.5918367346938775, 1.7142857142857142, 1.836734693877551, 1.9591836734693877, 2.0816326530612246, 2.204081632653061, 2.326530612244898, 2.4489795918367347, 2.571428571428571, 2.693877551020408, 2.816326530612245, 2.9387755102040813, 3.061224489795918, 3.183673469387755, 3.306122448979592, 3.4285714285714284, 3.5510204081632653, 3.673469387755102, 3.7959183673469385, 3.9183673469387754, 4.040816326530612, 4.163265306122449, 4.285714285714286, 4.408163265306122, 4.530612244897959, 4.653061224489796, 4.775510204081632, 4.8979591836734695, 5.020408163265306, 5.142857142857142, 5.26530612244898, 5.387755102040816, 5.5102040816326525, 5.63265306122449, 5.755102040816326, 5.877551020408163, 6.0], \"y\": [2.0, 1.436581963587984, 0.9696813817069801, 0.6037645827844602, 0.3297921045100084, 0.13105453462244457, -0.010232517338564335, -0.10937556649944526, -0.1782864612071781, -0.2257873393913745, -0.2582518050705372, -0.28022479902923064, -0.294922593030897, -0.3046088680265031, -0.31086953985409943, -0.31481027332807776, -0.31719782406609276, -0.3185605691636698, -0.31925991957613686, -0.3195408060355251, -0.3195673718246092, -0.31944804806327165, -0.31925312753621327, -0.3190270368476306, -0.31879686378820005, -0.3185782234979382, -0.31837928286496364, -0.3182034729367215, -0.31805137419076124, -0.3179218825359452, -0.317813006150125, -0.3177223756088433, -0.3176475526541967, -0.31758619192908394, -0.31753616311436, -0.3174955696739524, -0.3174627753689424, -0.31743638194744755, -0.3174152058966373, -0.31739826231258284, -0.3173847388545805, -0.31737397182441696, -0.3173654159623826, -0.31735863389935026, -0.3173532602614974, -0.3173490124570506, -0.3173456546024602, -0.3173429939602287, -0.31734089488693396, -0.31733926026226716], \"type\": \"scatter\", \"uid\": \"9516cbb1-c2d2-4fbc-863f-1d64a9752b66\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 1}, \"name\": \"current_y\", \"x\": [0.0, 0.12244897959183673, 0.24489795918367346, 0.36734693877551017, 0.4897959183673469, 0.6122448979591837, 0.7346938775510203, 0.8571428571428571, 0.9795918367346939, 1.1020408163265305, 1.2244897959183674, 1.346938775510204, 1.4693877551020407, 1.5918367346938775, 1.7142857142857142, 1.836734693877551, 1.9591836734693877, 2.0816326530612246, 2.204081632653061, 2.326530612244898, 2.4489795918367347, 2.571428571428571, 2.693877551020408, 2.816326530612245, 2.9387755102040813, 3.061224489795918, 3.183673469387755, 3.306122448979592, 3.4285714285714284, 3.5510204081632653, 3.673469387755102, 3.7959183673469385, 3.9183673469387754, 4.040816326530612, 4.163265306122449, 4.285714285714286, 4.408163265306122, 4.530612244897959, 4.653061224489796, 4.775510204081632, 4.8979591836734695, 5.020408163265306, 5.142857142857142, 5.26530612244898, 5.387755102040816, 5.5102040816326525, 5.63265306122449, 5.755102040816326, 5.877551020408163, 6.0], \"y\": [0.0, 0.05654761074620635, 0.10893378804666826, 0.15576674239243876, 0.19617100656924708, 0.23001622762941615, 0.25779752325589206, 0.28031952743733446, 0.298444605731176, 0.312966237870432, 0.32456622691352277, 0.33381167338445483, 0.3411667693712719, 0.3470083087263204, 0.3516406539341158, 0.3553088687209171, 0.35820972901345516, 0.3605008960176747, 0.36230840264771436, 0.3637328085848013, 0.364854192437121, 0.36573619015638587, 0.36642931809935264, 0.3669735841613463, 0.3674006478589418, 0.36773551693828394, 0.36799793694538896, 0.3682034538821892, 0.36836433120600515, 0.3684902006144745, 0.36858862927790315, 0.36866557165899777, 0.3687256871409768, 0.3687726485484641, 0.3688093220756966, 0.3688379498148002, 0.36886027603953764, 0.36887770542867765, 0.36889130033904133, 0.368901899331075, 0.36891016362860296, 0.36891660044147007, 0.3689216174866076, 0.36892552364913217, 0.36892856820522735, 0.36893094069992644, 0.3689327906087841, 0.3689342399525199, 0.36893536580804265, 0.3689362338075601], \"type\": \"scatter\", \"uid\": \"2e28f5ee-22cb-47b2-826c-03a8c21444ed\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 2}, \"name\": \"current_xy\", \"x\": [2.0, 1.436581963587984, 0.9696813817069801, 0.6037645827844602, 0.3297921045100084, 0.13105453462244457, -0.010232517338564335, -0.10937556649944526, -0.1782864612071781, -0.2257873393913745, -0.2582518050705372, -0.28022479902923064, -0.294922593030897, -0.3046088680265031, -0.31086953985409943, -0.31481027332807776, -0.31719782406609276, -0.3185605691636698, -0.31925991957613686, -0.3195408060355251, -0.3195673718246092, -0.31944804806327165, -0.31925312753621327, -0.3190270368476306, -0.31879686378820005, -0.3185782234979382, -0.31837928286496364, -0.3182034729367215, -0.31805137419076124, -0.3179218825359452, -0.317813006150125, -0.3177223756088433, -0.3176475526541967, -0.31758619192908394, -0.31753616311436, -0.3174955696739524, -0.3174627753689424, -0.31743638194744755, -0.3174152058966373, -0.31739826231258284, -0.3173847388545805, -0.31737397182441696, -0.3173654159623826, -0.31735863389935026, -0.3173532602614974, -0.3173490124570506, -0.3173456546024602, -0.3173429939602287, -0.31734089488693396, -0.31733926026226716], \"xaxis\": \"x2\", \"y\": [0.0, 0.05654761074620635, 0.10893378804666826, 0.15576674239243876, 0.19617100656924708, 0.23001622762941615, 0.25779752325589206, 0.28031952743733446, 0.298444605731176, 0.312966237870432, 0.32456622691352277, 0.33381167338445483, 0.3411667693712719, 0.3470083087263204, 0.3516406539341158, 0.3553088687209171, 0.35820972901345516, 0.3605008960176747, 0.36230840264771436, 0.3637328085848013, 0.364854192437121, 0.36573619015638587, 0.36642931809935264, 0.3669735841613463, 0.3674006478589418, 0.36773551693828394, 0.36799793694538896, 0.3682034538821892, 0.36836433120600515, 0.3684902006144745, 0.36858862927790315, 0.36866557165899777, 0.3687256871409768, 0.3687726485484641, 0.3688093220756966, 0.3688379498148002, 0.36886027603953764, 0.36887770542867765, 0.36889130033904133, 0.368901899331075, 0.36891016362860296, 0.36891660044147007, 0.3689216174866076, 0.36892552364913217, 0.36892856820522735, 0.36893094069992644, 0.3689327906087841, 0.3689342399525199, 0.36893536580804265, 0.3689362338075601], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"79e694c7-7b30-42b4-a01f-f41412e7b34c\"}], {\"height\": 680, \"title\": {\"text\": \"Step 5<br>Best Loss 97.33419046026196<br>Current Loss 97.33419046026196\"}, \"width\": 1024, \"xaxis\": {\"range\": [-0.25, 6.25]}, \"yaxis\": {\"domain\": [0.55, 1.0], \"range\": [-2.5, 2.5]}, \"xaxis2\": {\"range\": [-2.5, 2.5]}, \"yaxis2\": {\"domain\": [0, 0.45], \"range\": [-2.5, 2.5]}}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"8ed2c56a-90b9-4ee6-8664-c5cde61b20e7\" style=\"height: 680px; width: 1024px;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"8ed2c56a-90b9-4ee6-8664-c5cde61b20e7\", [{\"line\": {\"color\": \"rgb(34, 119, 158)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_x\", \"x\": [0.0, 0.12244897959183673, 0.24489795918367346, 0.36734693877551017, 0.4897959183673469, 0.6122448979591837, 0.7346938775510203, 0.8571428571428571, 0.9795918367346939, 1.1020408163265305, 1.2244897959183674, 1.346938775510204, 1.4693877551020407, 1.5918367346938775, 1.7142857142857142, 1.836734693877551, 1.9591836734693877, 2.0816326530612246, 2.204081632653061, 2.326530612244898, 2.4489795918367347, 2.571428571428571, 2.693877551020408, 2.816326530612245, 2.9387755102040813, 3.061224489795918, 3.183673469387755, 3.306122448979592, 3.4285714285714284, 3.5510204081632653, 3.673469387755102, 3.7959183673469385, 3.9183673469387754, 4.040816326530612, 4.163265306122449, 4.285714285714286, 4.408163265306122, 4.530612244897959, 4.653061224489796, 4.775510204081632, 4.8979591836734695, 5.020408163265306, 5.142857142857142, 5.26530612244898, 5.387755102040816, 5.5102040816326525, 5.63265306122449, 5.755102040816326, 5.877551020408163, 6.0], \"y\": [2.0, 1.436581963587984, 0.9696813817069801, 0.6037645827844602, 0.3297921045100084, 0.13105453462244457, -0.010232517338564335, -0.10937556649944526, -0.1782864612071781, -0.2257873393913745, -0.2582518050705372, -0.28022479902923064, -0.294922593030897, -0.3046088680265031, -0.31086953985409943, -0.31481027332807776, -0.31719782406609276, -0.3185605691636698, -0.31925991957613686, -0.3195408060355251, -0.3195673718246092, -0.31944804806327165, -0.31925312753621327, -0.3190270368476306, -0.31879686378820005, -0.3185782234979382, -0.31837928286496364, -0.3182034729367215, -0.31805137419076124, -0.3179218825359452, -0.317813006150125, -0.3177223756088433, -0.3176475526541967, -0.31758619192908394, -0.31753616311436, -0.3174955696739524, -0.3174627753689424, -0.31743638194744755, -0.3174152058966373, -0.31739826231258284, -0.3173847388545805, -0.31737397182441696, -0.3173654159623826, -0.31735863389935026, -0.3173532602614974, -0.3173490124570506, -0.3173456546024602, -0.3173429939602287, -0.31734089488693396, -0.31733926026226716], \"type\": \"scatter\", \"uid\": \"5b80efc4-89a2-499e-af1f-3c80959d6d75\"}, {\"line\": {\"color\": \"rgb(201, 79, 14)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_y\", \"x\": [0.0, 0.12244897959183673, 0.24489795918367346, 0.36734693877551017, 0.4897959183673469, 0.6122448979591837, 0.7346938775510203, 0.8571428571428571, 0.9795918367346939, 1.1020408163265305, 1.2244897959183674, 1.346938775510204, 1.4693877551020407, 1.5918367346938775, 1.7142857142857142, 1.836734693877551, 1.9591836734693877, 2.0816326530612246, 2.204081632653061, 2.326530612244898, 2.4489795918367347, 2.571428571428571, 2.693877551020408, 2.816326530612245, 2.9387755102040813, 3.061224489795918, 3.183673469387755, 3.306122448979592, 3.4285714285714284, 3.5510204081632653, 3.673469387755102, 3.7959183673469385, 3.9183673469387754, 4.040816326530612, 4.163265306122449, 4.285714285714286, 4.408163265306122, 4.530612244897959, 4.653061224489796, 4.775510204081632, 4.8979591836734695, 5.020408163265306, 5.142857142857142, 5.26530612244898, 5.387755102040816, 5.5102040816326525, 5.63265306122449, 5.755102040816326, 5.877551020408163, 6.0], \"y\": [0.0, 0.05654761074620635, 0.10893378804666826, 0.15576674239243876, 0.19617100656924708, 0.23001622762941615, 0.25779752325589206, 0.28031952743733446, 0.298444605731176, 0.312966237870432, 0.32456622691352277, 0.33381167338445483, 0.3411667693712719, 0.3470083087263204, 0.3516406539341158, 0.3553088687209171, 0.35820972901345516, 0.3605008960176747, 0.36230840264771436, 0.3637328085848013, 0.364854192437121, 0.36573619015638587, 0.36642931809935264, 0.3669735841613463, 0.3674006478589418, 0.36773551693828394, 0.36799793694538896, 0.3682034538821892, 0.36836433120600515, 0.3684902006144745, 0.36858862927790315, 0.36866557165899777, 0.3687256871409768, 0.3687726485484641, 0.3688093220756966, 0.3688379498148002, 0.36886027603953764, 0.36887770542867765, 0.36889130033904133, 0.368901899331075, 0.36891016362860296, 0.36891660044147007, 0.3689216174866076, 0.36892552364913217, 0.36892856820522735, 0.36893094069992644, 0.3689327906087841, 0.3689342399525199, 0.36893536580804265, 0.3689362338075601], \"type\": \"scatter\", \"uid\": \"748da336-c1ee-4018-931e-7fab0127a954\"}, {\"line\": {\"color\": \"rgb(8, 173, 99)\", \"width\": 1}, \"mode\": \"lines+markers\", \"name\": \"best_xy\", \"x\": [2.0, 1.436581963587984, 0.9696813817069801, 0.6037645827844602, 0.3297921045100084, 0.13105453462244457, -0.010232517338564335, -0.10937556649944526, -0.1782864612071781, -0.2257873393913745, -0.2582518050705372, -0.28022479902923064, -0.294922593030897, -0.3046088680265031, -0.31086953985409943, -0.31481027332807776, -0.31719782406609276, -0.3185605691636698, -0.31925991957613686, -0.3195408060355251, -0.3195673718246092, -0.31944804806327165, -0.31925312753621327, -0.3190270368476306, -0.31879686378820005, -0.3185782234979382, -0.31837928286496364, -0.3182034729367215, -0.31805137419076124, -0.3179218825359452, -0.317813006150125, -0.3177223756088433, -0.3176475526541967, -0.31758619192908394, -0.31753616311436, -0.3174955696739524, -0.3174627753689424, -0.31743638194744755, -0.3174152058966373, -0.31739826231258284, -0.3173847388545805, -0.31737397182441696, -0.3173654159623826, -0.31735863389935026, -0.3173532602614974, -0.3173490124570506, -0.3173456546024602, -0.3173429939602287, -0.31734089488693396, -0.31733926026226716], \"xaxis\": \"x2\", \"y\": [0.0, 0.05654761074620635, 0.10893378804666826, 0.15576674239243876, 0.19617100656924708, 0.23001622762941615, 0.25779752325589206, 0.28031952743733446, 0.298444605731176, 0.312966237870432, 0.32456622691352277, 0.33381167338445483, 0.3411667693712719, 0.3470083087263204, 0.3516406539341158, 0.3553088687209171, 0.35820972901345516, 0.3605008960176747, 0.36230840264771436, 0.3637328085848013, 0.364854192437121, 0.36573619015638587, 0.36642931809935264, 0.3669735841613463, 0.3674006478589418, 0.36773551693828394, 0.36799793694538896, 0.3682034538821892, 0.36836433120600515, 0.3684902006144745, 0.36858862927790315, 0.36866557165899777, 0.3687256871409768, 0.3687726485484641, 0.3688093220756966, 0.3688379498148002, 0.36886027603953764, 0.36887770542867765, 0.36889130033904133, 0.368901899331075, 0.36891016362860296, 0.36891660044147007, 0.3689216174866076, 0.36892552364913217, 0.36892856820522735, 0.36893094069992644, 0.3689327906087841, 0.3689342399525199, 0.36893536580804265, 0.3689362338075601], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"5c479130-a35c-4acd-ac9b-dfd8e83bff2e\"}, {\"line\": {\"color\": \"rgb(34, 119, 158)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_x\", \"x\": [0.0, 0.12244897959183673, 0.24489795918367346, 0.36734693877551017, 0.4897959183673469, 0.6122448979591837, 0.7346938775510203, 0.8571428571428571, 0.9795918367346939, 1.1020408163265305, 1.2244897959183674, 1.346938775510204, 1.4693877551020407, 1.5918367346938775, 1.7142857142857142, 1.836734693877551, 1.9591836734693877, 2.0816326530612246, 2.204081632653061, 2.326530612244898, 2.4489795918367347, 2.571428571428571, 2.693877551020408, 2.816326530612245, 2.9387755102040813, 3.061224489795918, 3.183673469387755, 3.306122448979592, 3.4285714285714284, 3.5510204081632653, 3.673469387755102, 3.7959183673469385, 3.9183673469387754, 4.040816326530612, 4.163265306122449, 4.285714285714286, 4.408163265306122, 4.530612244897959, 4.653061224489796, 4.775510204081632, 4.8979591836734695, 5.020408163265306, 5.142857142857142, 5.26530612244898, 5.387755102040816, 5.5102040816326525, 5.63265306122449, 5.755102040816326, 5.877551020408163, 6.0], \"y\": [2.0, 1.5898879908691512, 0.09116942641742595, -1.2892733499556064, -1.7165722278067121, -1.6573383762683993, -1.2653104863867892, -0.3448977664688868, 0.5682569166303237, 1.26649086372092, 1.4530781713391205, 1.4245766699976528, 1.344648982234877, 1.0068703140648665, 0.4442478277248246, -0.12666749748682002, -0.6554371459430208, -1.057511259522562, -1.2181668044692728, -1.224144014204532, -1.2023429492275, -1.159771559853107, -1.0177113207432569, -0.7362904289965733, -0.3866537610747874, -0.03693757942190929, 0.29735850533321767, 0.6073011803455556, 0.8548867068620034, 0.9959188043497199, 1.0396474121880794, 1.0363370876827478, 1.0232723007911548, 1.0075545529869299, 0.9714057912086662, 0.8856605224634109, 0.7357719346072075, 0.5386279402870494, 0.3240740556179248, 0.11077027099574432, -0.09604216601885016, -0.29569168161482734, -0.4837677418636904, -0.6478248104004547, -0.7711576939697778, -0.8449592922377955, -0.8761546988691105, -0.8812318472485765, -0.8754213849225162, -0.8674041731287144], \"type\": \"scatter\", \"uid\": \"7f4be7c1-4a4c-43a5-bcef-c3a63eec6a92\"}, {\"line\": {\"color\": \"rgb(201, 79, 14)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_y\", \"x\": [0.0, 0.12244897959183673, 0.24489795918367346, 0.36734693877551017, 0.4897959183673469, 0.6122448979591837, 0.7346938775510203, 0.8571428571428571, 0.9795918367346939, 1.1020408163265305, 1.2244897959183674, 1.346938775510204, 1.4693877551020407, 1.5918367346938775, 1.7142857142857142, 1.836734693877551, 1.9591836734693877, 2.0816326530612246, 2.204081632653061, 2.326530612244898, 2.4489795918367347, 2.571428571428571, 2.693877551020408, 2.816326530612245, 2.9387755102040813, 3.061224489795918, 3.183673469387755, 3.306122448979592, 3.4285714285714284, 3.5510204081632653, 3.673469387755102, 3.7959183673469385, 3.9183673469387754, 4.040816326530612, 4.163265306122449, 4.285714285714286, 4.408163265306122, 4.530612244897959, 4.653061224489796, 4.775510204081632, 4.8979591836734695, 5.020408163265306, 5.142857142857142, 5.26530612244898, 5.387755102040816, 5.5102040816326525, 5.63265306122449, 5.755102040816326, 5.877551020408163, 6.0], \"y\": [0.0, 1.6347817545429182, 1.844674383806059, 1.6378754737516776, 0.5998667122634451, -0.5858387521501, -1.4411780782906123, -1.5740080287701013, -1.5219753216299392, -1.246228749011831, -0.563979027922285, 0.17013422399462957, 0.8339999042616996, 1.2538388566248573, 1.3366957083936806, 1.3122265055435145, 1.2637416972353306, 1.071145121769598, 0.6805697949460937, 0.22651994818854593, -0.2110414557078644, -0.618010917261637, -0.9395152275568366, -1.1009851626928493, -1.13164900965241, -1.1180436278539654, -1.0998703474931357, -1.0584917104036367, -0.9444137589948657, -0.735567231443964, -0.46969440862493866, -0.19377068428574815, 0.07376774145461143, 0.3305289903480158, 0.5687452211048643, 0.7647389541573506, 0.8916711271367963, 0.9476595221233626, 0.9579168432316844, 0.950365550632471, 0.940043438875845, 0.9277165512939757, 0.9027774758134423, 0.8485228199192166, 0.7525208133424066, 0.6170523010719102, 0.45738255489858415, 0.28973124380035203, 0.12341746107635407, -0.038649916191955624], \"type\": \"scatter\", \"uid\": \"c578e072-98be-42aa-9a4a-f10b918499d3\"}, {\"line\": {\"color\": \"rgb(173, 7, 7)\", \"dash\": \"dot\", \"width\": 2}, \"name\": \"groundtruth_xy\", \"x\": [2.0, 1.5898879908691512, 0.09116942641742595, -1.2892733499556064, -1.7165722278067121, -1.6573383762683993, -1.2653104863867892, -0.3448977664688868, 0.5682569166303237, 1.26649086372092, 1.4530781713391205, 1.4245766699976528, 1.344648982234877, 1.0068703140648665, 0.4442478277248246, -0.12666749748682002, -0.6554371459430208, -1.057511259522562, -1.2181668044692728, -1.224144014204532, -1.2023429492275, -1.159771559853107, -1.0177113207432569, -0.7362904289965733, -0.3866537610747874, -0.03693757942190929, 0.29735850533321767, 0.6073011803455556, 0.8548867068620034, 0.9959188043497199, 1.0396474121880794, 1.0363370876827478, 1.0232723007911548, 1.0075545529869299, 0.9714057912086662, 0.8856605224634109, 0.7357719346072075, 0.5386279402870494, 0.3240740556179248, 0.11077027099574432, -0.09604216601885016, -0.29569168161482734, -0.4837677418636904, -0.6478248104004547, -0.7711576939697778, -0.8449592922377955, -0.8761546988691105, -0.8812318472485765, -0.8754213849225162, -0.8674041731287144], \"xaxis\": \"x2\", \"y\": [0.0, 1.6347817545429182, 1.844674383806059, 1.6378754737516776, 0.5998667122634451, -0.5858387521501, -1.4411780782906123, -1.5740080287701013, -1.5219753216299392, -1.246228749011831, -0.563979027922285, 0.17013422399462957, 0.8339999042616996, 1.2538388566248573, 1.3366957083936806, 1.3122265055435145, 1.2637416972353306, 1.071145121769598, 0.6805697949460937, 0.22651994818854593, -0.2110414557078644, -0.618010917261637, -0.9395152275568366, -1.1009851626928493, -1.13164900965241, -1.1180436278539654, -1.0998703474931357, -1.0584917104036367, -0.9444137589948657, -0.735567231443964, -0.46969440862493866, -0.19377068428574815, 0.07376774145461143, 0.3305289903480158, 0.5687452211048643, 0.7647389541573506, 0.8916711271367963, 0.9476595221233626, 0.9579168432316844, 0.950365550632471, 0.940043438875845, 0.9277165512939757, 0.9027774758134423, 0.8485228199192166, 0.7525208133424066, 0.6170523010719102, 0.45738255489858415, 0.28973124380035203, 0.12341746107635407, -0.038649916191955624], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"f0d8e031-eb66-41fc-afb3-756087438c94\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 1}, \"name\": \"current_x\", \"x\": [0.0, 0.12244897959183673, 0.24489795918367346, 0.36734693877551017, 0.4897959183673469, 0.6122448979591837, 0.7346938775510203, 0.8571428571428571, 0.9795918367346939, 1.1020408163265305, 1.2244897959183674, 1.346938775510204, 1.4693877551020407, 1.5918367346938775, 1.7142857142857142, 1.836734693877551, 1.9591836734693877, 2.0816326530612246, 2.204081632653061, 2.326530612244898, 2.4489795918367347, 2.571428571428571, 2.693877551020408, 2.816326530612245, 2.9387755102040813, 3.061224489795918, 3.183673469387755, 3.306122448979592, 3.4285714285714284, 3.5510204081632653, 3.673469387755102, 3.7959183673469385, 3.9183673469387754, 4.040816326530612, 4.163265306122449, 4.285714285714286, 4.408163265306122, 4.530612244897959, 4.653061224489796, 4.775510204081632, 4.8979591836734695, 5.020408163265306, 5.142857142857142, 5.26530612244898, 5.387755102040816, 5.5102040816326525, 5.63265306122449, 5.755102040816326, 5.877551020408163, 6.0], \"y\": [2.0, 1.436581963587984, 0.9696813817069801, 0.6037645827844602, 0.3297921045100084, 0.13105453462244457, -0.010232517338564335, -0.10937556649944526, -0.1782864612071781, -0.2257873393913745, -0.2582518050705372, -0.28022479902923064, -0.294922593030897, -0.3046088680265031, -0.31086953985409943, -0.31481027332807776, -0.31719782406609276, -0.3185605691636698, -0.31925991957613686, -0.3195408060355251, -0.3195673718246092, -0.31944804806327165, -0.31925312753621327, -0.3190270368476306, -0.31879686378820005, -0.3185782234979382, -0.31837928286496364, -0.3182034729367215, -0.31805137419076124, -0.3179218825359452, -0.317813006150125, -0.3177223756088433, -0.3176475526541967, -0.31758619192908394, -0.31753616311436, -0.3174955696739524, -0.3174627753689424, -0.31743638194744755, -0.3174152058966373, -0.31739826231258284, -0.3173847388545805, -0.31737397182441696, -0.3173654159623826, -0.31735863389935026, -0.3173532602614974, -0.3173490124570506, -0.3173456546024602, -0.3173429939602287, -0.31734089488693396, -0.31733926026226716], \"type\": \"scatter\", \"uid\": \"9516cbb1-c2d2-4fbc-863f-1d64a9752b66\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 1}, \"name\": \"current_y\", \"x\": [0.0, 0.12244897959183673, 0.24489795918367346, 0.36734693877551017, 0.4897959183673469, 0.6122448979591837, 0.7346938775510203, 0.8571428571428571, 0.9795918367346939, 1.1020408163265305, 1.2244897959183674, 1.346938775510204, 1.4693877551020407, 1.5918367346938775, 1.7142857142857142, 1.836734693877551, 1.9591836734693877, 2.0816326530612246, 2.204081632653061, 2.326530612244898, 2.4489795918367347, 2.571428571428571, 2.693877551020408, 2.816326530612245, 2.9387755102040813, 3.061224489795918, 3.183673469387755, 3.306122448979592, 3.4285714285714284, 3.5510204081632653, 3.673469387755102, 3.7959183673469385, 3.9183673469387754, 4.040816326530612, 4.163265306122449, 4.285714285714286, 4.408163265306122, 4.530612244897959, 4.653061224489796, 4.775510204081632, 4.8979591836734695, 5.020408163265306, 5.142857142857142, 5.26530612244898, 5.387755102040816, 5.5102040816326525, 5.63265306122449, 5.755102040816326, 5.877551020408163, 6.0], \"y\": [0.0, 0.05654761074620635, 0.10893378804666826, 0.15576674239243876, 0.19617100656924708, 0.23001622762941615, 0.25779752325589206, 0.28031952743733446, 0.298444605731176, 0.312966237870432, 0.32456622691352277, 0.33381167338445483, 0.3411667693712719, 0.3470083087263204, 0.3516406539341158, 0.3553088687209171, 0.35820972901345516, 0.3605008960176747, 0.36230840264771436, 0.3637328085848013, 0.364854192437121, 0.36573619015638587, 0.36642931809935264, 0.3669735841613463, 0.3674006478589418, 0.36773551693828394, 0.36799793694538896, 0.3682034538821892, 0.36836433120600515, 0.3684902006144745, 0.36858862927790315, 0.36866557165899777, 0.3687256871409768, 0.3687726485484641, 0.3688093220756966, 0.3688379498148002, 0.36886027603953764, 0.36887770542867765, 0.36889130033904133, 0.368901899331075, 0.36891016362860296, 0.36891660044147007, 0.3689216174866076, 0.36892552364913217, 0.36892856820522735, 0.36893094069992644, 0.3689327906087841, 0.3689342399525199, 0.36893536580804265, 0.3689362338075601], \"type\": \"scatter\", \"uid\": \"2e28f5ee-22cb-47b2-826c-03a8c21444ed\"}, {\"line\": {\"color\": \"rgb(24, 24, 24)\", \"dash\": \"dash\", \"width\": 2}, \"name\": \"current_xy\", \"x\": [2.0, 1.436581963587984, 0.9696813817069801, 0.6037645827844602, 0.3297921045100084, 0.13105453462244457, -0.010232517338564335, -0.10937556649944526, -0.1782864612071781, -0.2257873393913745, -0.2582518050705372, -0.28022479902923064, -0.294922593030897, -0.3046088680265031, -0.31086953985409943, -0.31481027332807776, -0.31719782406609276, -0.3185605691636698, -0.31925991957613686, -0.3195408060355251, -0.3195673718246092, -0.31944804806327165, -0.31925312753621327, -0.3190270368476306, -0.31879686378820005, -0.3185782234979382, -0.31837928286496364, -0.3182034729367215, -0.31805137419076124, -0.3179218825359452, -0.317813006150125, -0.3177223756088433, -0.3176475526541967, -0.31758619192908394, -0.31753616311436, -0.3174955696739524, -0.3174627753689424, -0.31743638194744755, -0.3174152058966373, -0.31739826231258284, -0.3173847388545805, -0.31737397182441696, -0.3173654159623826, -0.31735863389935026, -0.3173532602614974, -0.3173490124570506, -0.3173456546024602, -0.3173429939602287, -0.31734089488693396, -0.31733926026226716], \"xaxis\": \"x2\", \"y\": [0.0, 0.05654761074620635, 0.10893378804666826, 0.15576674239243876, 0.19617100656924708, 0.23001622762941615, 0.25779752325589206, 0.28031952743733446, 0.298444605731176, 0.312966237870432, 0.32456622691352277, 0.33381167338445483, 0.3411667693712719, 0.3470083087263204, 0.3516406539341158, 0.3553088687209171, 0.35820972901345516, 0.3605008960176747, 0.36230840264771436, 0.3637328085848013, 0.364854192437121, 0.36573619015638587, 0.36642931809935264, 0.3669735841613463, 0.3674006478589418, 0.36773551693828394, 0.36799793694538896, 0.3682034538821892, 0.36836433120600515, 0.3684902006144745, 0.36858862927790315, 0.36866557165899777, 0.3687256871409768, 0.3687726485484641, 0.3688093220756966, 0.3688379498148002, 0.36886027603953764, 0.36887770542867765, 0.36889130033904133, 0.368901899331075, 0.36891016362860296, 0.36891660044147007, 0.3689216174866076, 0.36892552364913217, 0.36892856820522735, 0.36893094069992644, 0.3689327906087841, 0.3689342399525199, 0.36893536580804265, 0.3689362338075601], \"yaxis\": \"y2\", \"type\": \"scatter\", \"uid\": \"79e694c7-7b30-42b4-a01f-f41412e7b34c\"}], {\"height\": 680, \"title\": {\"text\": \"Step 5<br>Best Loss 97.33419046026196<br>Current Loss 97.33419046026196\"}, \"width\": 1024, \"xaxis\": {\"range\": [-0.25, 6.25]}, \"yaxis\": {\"domain\": [0.55, 1.0], \"range\": [-2.5, 2.5]}, \"xaxis2\": {\"range\": [-2.5, 2.5]}, \"yaxis2\": {\"domain\": [0, 0.45], \"range\": [-2.5, 2.5]}}, {\"showLink\": false, \"linkText\": \"Export to plot.ly\", \"plotlyServerURL\": \"https://plot.ly\"})});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "optimize(predict_fn2, ode_scipy_solver2, loss_L2, ground_truth2, z0, ts,\n",
    "         args=None, variables=model2.variables, epochs=5,\n",
    "         x_axis_min=-0.25, x_axis_max=6.25,\n",
    "         y_axis_min=-2.5, y_axis_max=2.5,\n",
    "         x2_axis_min=-2.5, x2_axis_max=2.5,\n",
    "         y2_axis_min=-2.5, y2_axis_max=2.5,\n",
    "         output_dir=\"ode_dense_damped_oscillator_2d\", start=0)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Post Work'em\n",
    "\n",
    "This is not a post mortem as the study is not finished yet and still on-going.\n",
    "\n",
    "After reaching this step, I'm truly happy as it works and sounds promising about the paper concepts which are reproducible... I don't pretend it's perfect yet and I can say that trainings are not very fast and tend to differ between runs. Sometimes, it seems to converge at first and then loop without converging further. Yet, it's already quite cool to me :)\n",
    "\n",
    "But it has been painful... Ok, I'm not yet an expert in TF or Python and I'm still learning both deeper concepts and libraries. But I can say I've been developing for tens of years in complex backend production environment and distributed systems with Native, Java and Scala/FP. From this experience, current state of maturity of TF and AI Python ecosystem, in general, is far from those in terms of robustness, consistency and determinism. Ok, you can have fun with a few pre-built samples but as soon as you go out of the paved road, it becomes hairy and not very deterministic or consistent... Everything takes a lot of time and effort to reach acceptable state... Specially when dealing with custom gradients which tend to crash with no error at all so it's hard to know which part of your graph fails..\n",
    "\n",
    "I'm (in) eager(-mode) to discover incoming TF 2.x. which should make TF a bit more practical (more like Pytorch) and consistent in terms API... I'm also looking at TF-Swift, not that I like Swift but because a multi-stage compiled auto-diff remains quite appealing to me compared to Pythonic weakness...\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Next steps...\n",
    "\n",
    "_This notebook is the first one of a few incoming studies I'll publish when I find a few minutes as I'm also implementing Time-series models and Continuous Normalizing Flows as explained in the paper..._\n",
    "\n",
    "I want to test more networks for derivative or time distribution and see how it can speed trainings...\n",
    "\n",
    "I want to make a quick Pytorch implementation and compare to TF 1.x.\n",
    "\n",
    "I want to see how they check full-GPU ODE Solvers in Julia or Pytorch...\n",
    "\n",
    "I want to play with depth distribution as a learnable parameter..."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}