02 deeplearning using flux - 01 introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Binder](https://mybinder.org/badge_logo.svg)](https://notebooks.gesis.org/binder/v2/gh/jolin-io/fall-in-love-with-julia/main?filepath=02%20deeplearning%20using%20flux%20-%2001%20introduction.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![JuliaLogo](https://julialang.org/assets/infra/logo.svg)\n",
    "# Welcome to the deep learning tutorial to Julia\n",
    "\n",
    "Julia shines when it comes to letting packages interact seamlessly with one another. One such example is julia's deep learning capabilities.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question to you:** What are the key ingredients to do deep-learning?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Flux.jl\n",
    "\n",
    "In Julia there are two actively maintained deep-learning packages: [Knet.jl](https://github.com/denizyuret/Knet.jl) and [Flux.jl](https://github.com/FluxML/Flux.jl). Both are written in 100% julia and hence interact seamlessly with everything else in Julia.\n",
    "\n",
    "Flux.jl is considered the easier package for starters as it is more similar to Keras, while Knet.jl stays more low-level in its API.\n",
    "Even more important, Flux.jl is focussing on interoperability with other Julia packages, enabling NeuralDifferentialEquations and other advances in [scientific machine learning](https://sciml.ai/).\n",
    "\n",
    "The most up-to-date blog-post I could find on comparing Knet.jl, Flux.jl and Tensorflow [is this one](https://estadistika.github.io//julia/python/packages/knet/flux/tensorflow/machine-learning/deep-learning/2019/06/20/Deep-Learning-Exploring-High-Level-APIs-of-Knet.jl-and-Flux.jl-in-comparison-to-Tensorflow-Keras.html). It is nice and simple comparison with real code, I can recommend it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "using Flux  # takes about a minute when run the first time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let me start with citing Flux.ml\n",
    "\n",
    "**Flux: The Julia Machine Learning Library**\n",
    "\n",
    "Flux is a library for machine learning. It comes \"batteries-included\" with many useful tools built in, but also lets you use the full power of the Julia language where you need it. We follow a few key principles:\n",
    "\n",
    "* **Doing the obvious thing.** Flux has relatively few explicit APIs for features like regularisation or embeddings. Instead, writing down the mathematical form will work – and be fast.\n",
    "* **You could have written Flux.** All of it, from LSTMs to GPU kernels, is straightforward Julia code. When in doubt, it’s well worth looking at the source. If you need something different, you can easily roll your own.\n",
    "* **Play nicely with others.** Flux works well with Julia libraries from data frames and images to differential equation solvers, so you can easily build complex data processing pipelines that integrate Flux models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Flux.jl Basics\n",
    "I want to go with you through the introductory example of the [Flux.jl documentation](https://fluxml.ai/Flux.jl/stable/models/basics/) which is very good and highly educational."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Taking Gradients\n",
    "Flux's core feature is taking gradients of Julia code. The gradient function takes another Julia function f and a set of arguments, and returns the gradient with respect to each argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f(x) = 3x^2 + 2x + 1;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df/dx = 6x + 2\n",
    "f'(2)  # after `using Flux` you have access to automatic differentiation of arbitrary functions (actually this is given by Zygote.jl which is a subpackage of the Flux eco system)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# d²f/dx² = 6\n",
    "f''(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# d³f/dx³ = 0\n",
    "f'''(2)  # takes quite long to compile, but works"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is fancy syntax for the underlying core function `gradient`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df(x) = gradient(f, x)[1]; # df/dx = 6x + 2\n",
    "df(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d2f(x) = gradient(df, x)[1]; # d²f/dx² = 6\n",
    "d2f(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may ask how far does this go? Can everything be autodifferentiated? Actually almost everything, including arbitrary controlflows, recursions, loops, and even mutable datastructures. See https://fluxml.ai/Zygote.jl/latest/#Taking-Gradients-1 for details.\n",
    "\n",
    "When a function has many parameters, we can get gradients of each one at the same time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f(x, y) = sum((x .- y).^2);\n",
    "\n",
    "@show gradient(f, 2, 2)\n",
    "@show gradient(f, 1, 0)\n",
    "gradient(f, [2, 1], [2, 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But machine learning models can have hundreds of parameters! To handle this, Flux lets you work with collections of parameters, via params. You can get the gradient of all parameters used in a program without explicitly passing them in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = [2, 1];\n",
    "y = [2, 0];\n",
    "gs = gradient(params(x, y)) do\n",
    "    f(x, y)\n",
    "end\n",
    "\n",
    "@show gs[x]\n",
    "@show gs[y];"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, gradient takes a zero-argument function; no arguments are necessary because the params tell it what to differentiate.\n",
    "\n",
    "This will come in really handy when dealing with big, complicated models. For now, though, let's start with something simple."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple Models\n",
    "Consider a simple linear regression, which tries to predict an output array y from an input x."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "W = rand(2, 5)\n",
    "b = rand(2)\n",
    "\n",
    "predict(x) = W*x .+ b\n",
    "\n",
    "function loss(x, y)\n",
    "  ŷ = predict(x)\n",
    "  sum((y .- ŷ).^2)\n",
    "end\n",
    "\n",
    "x, y = rand(5), rand(2) # Dummy data\n",
    "loss(x, y) # ~ 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To improve the prediction we can take the gradients of W and b with respect to the loss and perform gradient descent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gs = gradient(() -> loss(x, y), params(W, b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have gradients, we can pull them out and update W to train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "W̄ = gs[W]\n",
    "W .-= 0.1 .* W̄\n",
    "loss(x, y) # ~ 2.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The loss has decreased a little, meaning that our prediction x is closer to the target y. If we have some data we can already try [training the model](https://fluxml.ai/Flux.jl/stable/training/training/).\n",
    "\n",
    "All deep learning in Flux, however complex, is a simple generalisation of this example. Of course, models can look very different – they might have millions of parameters or complex control flow. Let's see how Flux handles more complex models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building Layers\n",
    "It's common to create more complex models than the linear regression above. For example, we might want to have two linear layers with a nonlinearity like sigmoid (σ) in between them. In the above style we could write this as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "using Flux\n",
    "\n",
    "W1 = rand(3, 5)\n",
    "b1 = rand(3)\n",
    "layer1(x) = W1 * x .+ b1\n",
    "\n",
    "W2 = rand(2, 3)\n",
    "b2 = rand(2)\n",
    "layer2(x) = W2 * x .+ b2\n",
    "\n",
    "model(x) = layer2(σ.(layer1(x)))  # TODO run `?σ` to see what this is, try run `methods(σ)` to see where it comes from\n",
    "\n",
    "model(rand(5)) # => 2-element vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This works but is fairly unwieldy, with a lot of repetition – especially as we add more layers. One way to factor this out is to create a function that returns linear layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "function linear(in, out)\n",
    "  W = randn(out, in)\n",
    "  b = randn(out)\n",
    "  x -> W * x .+ b\n",
    "end\n",
    "\n",
    "linear1 = linear(5, 3) # we can access linear1.W etc\n",
    "linear2 = linear(3, 2)\n",
    "\n",
    "model(x) = linear2(σ.(linear1(x)))\n",
    "\n",
    "model(rand(5)) # => 2-element vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another (equivalent) way is to create a struct that explicitly represents the affine layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "struct Affine\n",
    "  W\n",
    "  b\n",
    "end\n",
    "\n",
    "Affine(in::Integer, out::Integer) =\n",
    "  Affine(randn(out, in), randn(out))\n",
    "\n",
    "# Overload call, so the object can be used as a function\n",
    "(m::Affine)(x) = m.W * x .+ m.b\n",
    "\n",
    "a = Affine(10, 5)\n",
    "\n",
    "a(rand(10)) # => 5-element vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! You just built the Dense layer that comes with Flux. Flux has many interesting layers available, but they're all things you could have built yourself very easily.\n",
    "\n",
    "(There is one small difference with Dense – for convenience it also takes an activation function, like ``Dense(10, 5, σ)``.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stacking It Up\n",
    "It's pretty common to write models that look something like:\n",
    "```julia\n",
    "layer1 = Dense(10, 5, σ)\n",
    "# ...\n",
    "model(x) = layer3(layer2(layer1(x)))\n",
    "```\n",
    "For long chains, it might be a bit more intuitive to have a list of layers, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "layers = [Dense(10, 5, σ), Dense(5, 2), softmax]\n",
    "model(x) = foldl((x, m) -> m(x), layers, init = x)\n",
    "\n",
    "model(rand(10)) # => 2-element vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Handily, this is also provided for in Flux:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model2 = Chain(\n",
    "  Dense(10, 5, σ),\n",
    "  Dense(5, 2),\n",
    "  softmax)\n",
    "\n",
    "model2(rand(10)) # => 2-element vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This quickly starts to look like a high-level deep learning library; yet you can see how it falls out of simple abstractions, and we lose none of the power of Julia code.\n",
    "\n",
    "A nice property of this approach is that because \"models\" are just functions (possibly with trainable parameters), you can also see this as simple function composition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = Dense(5, 2) ∘ Dense(10, 5, σ)\n",
    "\n",
    "m(rand(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likewise, Chain will happily work with any Julia function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = Chain(x -> x^2, x -> x+1)\n",
    "\n",
    "m(5) # => 26"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### To be continued...\n",
    "For further details you can continue at the official [Flux documentation](https://fluxml.ai/Flux.jl/stable/models/basics/#Layer-helpers-1).\n",
    "\n",
    "Flux.jl also comes with a [model zoo](https://github.com/FluxML/model-zoo) where you can find a bunch of ready examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scientific Machine Learning\n",
    "\n",
    "At the end I want to show you some cool new stuff you can do with Flux.jl which you cannot do easily with other deep learning frameworks.\n",
    "\n",
    "### Differentiable Control Problems, Physical Systems, Chemistry, Biology ... include arbitrary domain knowledge\n",
    "\n",
    "[This blog-post about differentiable control problems in Julia](https://fluxml.ai/2019/03/05/dp-vs-rl.html) really flashed me. It demonstrates that you can learn through arbitrary dynamical systems with Flux.jl.\n",
    "\n",
    "One such dynamical system is a trebuchet:\n",
    "\n",
    "![trebuchet-visualization](https://fluxml.ai/assets/2019-03-05-dp-vs-rl/trebuchet-basic.gif)\n",
    "\n",
    "And you can straightforward incorporate it as part of your machine learning model:\n",
    "![architecture-diagram-how-to-learn-through-dynamical-system](https://fluxml.ai/assets/2019-03-05-dp-vs-rl/trebuchet-flow.png)\n",
    "\n",
    "The same logic applies to other fields like Physics where you have differential equations describing your system. Similar in Chemistry or Biology. Actually it is not bound to anything specific, any julia code will do, and hence **you can include really arbitrary domain specific knowledge into your machine learning model**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SciML\n",
    "\n",
    "This idea is so far reaching that a whole new community is already born around it. It named itself scientific machine learning, short SciML https://sciml.ai/.\n",
    "\n",
    "Here is a very good summary about their goal and their current reach: https://sciml.ai/2020/03/29/SciML.html."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Thanks for participating ;-)\n",
    "\n",
    "In case of any questions feel free to reach me at stephan.sahm@gmx.de\n",
    "\n",
    "If you are curious for more or want to do a Julia project, just tell me. I am always glad about new enthusiasts.\n",
    "\n",
    "Believe me, it's the future of applied-math, including data-science.\n",
    "![fans](https://images.unsplash.com/photo-1490078615078-3e40c20db36c?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=2125&q=80)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.6.1",
   "language": "julia",
   "name": "julia-1.6"
  },
  "language_info": {
   "name": "julia",
   "version": ""
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}