index.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fraud POC\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "This is a prototype/proof-of-concept of a ML solution for Fraud Detection for e-commerce,\n",
    "completely built in Python and backed by Dask to parallelize data processing and model training\n",
    "and hopeit.engine to \"productionize\" training pipeline and prediction service as microservices."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Author: Leo Smerling \n",
    "\n",
    "LinkedIn: https://www.linkedin.com/in/leosmerling/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Dask](https://dask.org/) is a distributed processing engine for Python that can be used to process high data loads in distributed environments, such a Dask cluster. It has APIs built on top of popular Numpy and Pandas libraries.\n",
    "\n",
    "[hopeit.engine](https://github.com/hopeit-git/hopeit.engine): is an (upcoming) open-source library that I am contributing to, that enables to quickly develop microservices in Python. hopeit.engine is built on top of aiohttp to provide API endpoints and async processing, and also provides distributed processing of streaming events using [Redis Streams](https://redis.io/topics/streams-intro). Streaming data, authorization, logging, metrics and tracking/tracing are added to your microservice out of the box.\n",
    "\n",
    "To enable development, testing and working with the data in an unified environment I use [nbdev](https://github.com/fastai/nbdev). nbdev allows to explore data, create the services and test using Jupyter Notebooks. hopeit.engine and Dask plays well also with Juypyter notebooks, so the whole pipeline and prediction service can be developed and tested from Jupyter.\n",
    "\n",
    "This repo shows and end-to-end example of a Fraud Detection system consisting of:\n",
    "- Data preprocessing and partitioning (using Dasks Dataframes API)\n",
    "- Feature calculation (using Dasks Dataframes API)\n",
    "- Preparing data for model training (using Dasks Dataframes API)\n",
    "- Training a model (distributed Dask XGBoost)\n",
    "- Preparing data to serve predictions (using Dask + Redis)\n",
    "- Prepare and run a microservice to orchestrate and monitor the data + training pipeline (using hopeit.engine)\n",
    "- Prepare and run a microservice to predict fraud on new orders (using hopeit.engine) \n",
    "\n",
    "**DISCLAIMER**: The objective of this project is to quickly show an example on how Data + Feature Extraction +\n",
    "Model Trainig + Prediction can be developed and prepared for production. The data used for this example\n",
    "is randomly generated orders, and neither the features selected and model parameteres were optimized\n",
    "given the nature of data used. The intention is to give an overview of the tools and the approach to quickstart a project that could evolve into a mature state by improving each one of its pieces.\n",
    "\n",
    "\n",
    "### Getting started\n",
    "\n",
    "(Feel free to report issues if you find the procedure below not working, I've tested only in a Linux environment)\n",
    "\n",
    "* I recommend to install [Anaconda](https://docs.anaconda.com/anaconda/install/) (virtualenv can be used also -- not tested --)\n",
    "\n",
    "\n",
    "* Create a conda environment, activate and install jupyterlab, nbdev and dask\n",
    "```\n",
    "conda create -n fraud-poc python=3.7\n",
    "conda activate fraud-poc\n",
    "conda install jupyterlab\n",
    "conda install -c conda-forge dask graphviz python-graphviz \n",
    "pip install nbdev\n",
    "nbdev_install_git_hooks\n",
    "```\n",
    "\n",
    "\n",
    "* Install hopeit.engine from provided library (preview version, do not use in production):\n",
    "```\n",
    "cd install\n",
    "source install-hopeit-engine.sh\n",
    "cd ..\n",
    "```\n",
    "\n",
    "\n",
    "* Finally install this project and dependencies in development mode\n",
    "```\n",
    "pip install -e .\n",
    "```\n",
    "\n",
    "\n",
    "* In order to run the microservices (optional) Redis is required. You can run redis for development, from the provided docker configuration:\n",
    "```\n",
    "cd docker\n",
    "pip install docker-compose\n",
    "docker-compose up -d redis\n",
    "cd ..\n",
    "```\n",
    "\n",
    "* You can start a Dask cluster using docker or locally:\n",
    "\n",
    "Docker:\n",
    "```\n",
    "docker-compose up dask-scheduler\n",
    "docker-compose up dask-worker\n",
    "```\n",
    "\n",
    "Locally:\n",
    "```\n",
    "dask-scheduler\n",
    "dask-worker tcp://localhost:8786 --memory-limit 2GB\n",
    "```\n",
    "\n",
    "* Create a folder to store data\n",
    "```\n",
    "mkdir data\n",
    "```\n",
    "(location can be changed from config files)\n",
    "\n",
    "\n",
    "* To open, visualize and edit notebooks run from the root folder of the project\n",
    "```\n",
    "jupyter lab\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overview:\n",
    "\n",
    "* Notebooks prefixed from 00* to 09* are created for each component/event of the pipeline and prediction service. Check each notebook for a brief description of what they do:\n",
    "    \n",
    "* Cells marked with #export, will generate a python file inside fraud_poc/ folder that can be executed by hopeit.engine\n",
    "        \n",
    "* To generate/update the code, run `nbdev_build_lib` (no need to do it if you haven't change the code, modules are already generated in the repo)\n",
    "        \n",
    "* Usually the last cells of the notebooks, are a test case that can be run locally and invoke the generated file, gather data and do some checks/analysis. I saved the notebooks with the outputs so you can visualize some examples without needing to install anything.        \n",
    "        \n",
    "* Inside config/ folder there are configuration files to run two microservices:\n",
    "\n",
    "    * `training_pipeline.json` and `openapi-training.json` describe the service to run data preparation and training pipeline using hopeit.engine\n",
    "    \n",
    "    * `fraud-service.json` and `openapi-service.json` configure a service to perform real-time predictions on new orders based on the training model and aggregated data\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training Pipeline\n",
    "\n",
    "As a result of configuration in `config/training-config.json` plus implemented Python modules generated from Notebooks 00* to 07, the following training pipeline is implemented, where event (green) is notified by the previous step using streams (blueish):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "from hopeit.testing.apps import config\n",
    "from fraud_poc.diagrams import draw_graph\n",
    "\n",
    "training_config = config('config/training-pipeline.json')\n",
    "pipeline_diagram = draw_graph(training_config, show_streams=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.42.3 (20191010.1750)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"396pt\" height=\"737pt\"\n",
       " viewBox=\"0.00 0.00 396.40 736.60\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 732.6)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-732.6 392.4,-732.6 392.4,4 -4,4\"/>\n",
       "<!-- submit_training_pipeline.POST -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>submit_training_pipeline.POST</title>\n",
       "<ellipse fill=\"#aaccaa\" stroke=\"black\" cx=\"111.65\" cy=\"-726.8\" rx=\"1.8\" ry=\"1.8\"/>\n",
       "</g>\n",
       "<!-- submit_training_pipeline -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>submit_training_pipeline</title>\n",
       "<polygon fill=\"#aaccaa\" stroke=\"black\" points=\"189.9,-689 33.39,-689 33.39,-653 189.9,-653 189.9,-689\"/>\n",
       "<text text-anchor=\"middle\" x=\"111.65\" y=\"-666.8\" font-family=\"Times,serif\" font-size=\"14.00\">submit_training_pipeline</text>\n",
       "</g>\n",
       "<!-- submit_training_pipeline.POST&#45;&gt;submit_training_pipeline -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>submit_training_pipeline.POST&#45;&gt;submit_training_pipeline</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M111.65,-724.87C111.65,-721.64 111.65,-710.55 111.65,-699.35\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"115.15,-699.26 111.65,-689.26 108.15,-699.26 115.15,-699.26\"/>\n",
       "</g>\n",
       "<!-- fraud_poc.training.submit_training_pipeline -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>fraud_poc.training.submit_training_pipeline</title>\n",
       "<path fill=\"#aaaaff\" stroke=\"black\" d=\"M98.69,-580.5C98.69,-580.5 292.6,-580.5 292.6,-580.5 298.6,-580.5 304.6,-586.5 304.6,-592.5 304.6,-592.5 304.6,-604.5 304.6,-604.5 304.6,-610.5 298.6,-616.5 292.6,-616.5 292.6,-616.5 98.69,-616.5 98.69,-616.5 92.69,-616.5 86.69,-610.5 86.69,-604.5 86.69,-604.5 86.69,-592.5 86.69,-592.5 86.69,-586.5 92.69,-580.5 98.69,-580.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"195.65\" y=\"-595.74\" font-family=\"Times,serif\" font-size=\"10.00\">→ fraud_poc.training.submit_training_pipeline →</text>\n",
       "</g>\n",
       "<!-- data.preprocess -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>data.preprocess</title>\n",
       "<polygon fill=\"#aaccaa\" stroke=\"black\" points=\"246.83,-544 144.47,-544 144.47,-508 246.83,-508 246.83,-544\"/>\n",
       "<text text-anchor=\"middle\" x=\"195.65\" y=\"-521.8\" font-family=\"Times,serif\" font-size=\"14.00\">data.preprocess</text>\n",
       "</g>\n",
       "<!-- fraud_poc.training.submit_training_pipeline&#45;&gt;data.preprocess -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>fraud_poc.training.submit_training_pipeline&#45;&gt;data.preprocess</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M195.65,-580.43C195.65,-572.67 195.65,-563.28 195.65,-554.56\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"199.15,-554.4 195.65,-544.4 192.15,-554.4 199.15,-554.4\"/>\n",
       "</g>\n",
       "<!-- submit_training_pipeline&#45;&gt;fraud_poc.training.submit_training_pipeline -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>submit_training_pipeline&#45;&gt;fraud_poc.training.submit_training_pipeline</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M131.98,-652.93C142.63,-643.99 155.84,-632.91 167.47,-623.15\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"169.8,-625.76 175.21,-616.65 165.3,-620.4 169.8,-625.76\"/>\n",
       "</g>\n",
       "<!-- data.make&#45;sample&#45;data.POST -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>data.make&#45;sample&#45;data.POST</title>\n",
       "<ellipse fill=\"#aaccaa\" stroke=\"black\" cx=\"280.65\" cy=\"-726.8\" rx=\"1.8\" ry=\"1.8\"/>\n",
       "</g>\n",
       "<!-- data.make&#45;sample&#45;data -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>data.make&#45;sample&#45;data</title>\n",
       "<polygon fill=\"#aaccaa\" stroke=\"black\" points=\"353.58,-689 207.71,-689 207.71,-653 353.58,-653 353.58,-689\"/>\n",
       "<text text-anchor=\"middle\" x=\"280.65\" y=\"-666.8\" font-family=\"Times,serif\" font-size=\"14.00\">data.make&#45;sample&#45;data</text>\n",
       "</g>\n",
       "<!-- data.make&#45;sample&#45;data.POST&#45;&gt;data.make&#45;sample&#45;data -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>data.make&#45;sample&#45;data.POST&#45;&gt;data.make&#45;sample&#45;data</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M280.65,-724.87C280.65,-721.64 280.65,-710.55 280.65,-699.35\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"284.15,-699.26 280.65,-689.26 277.15,-699.26 284.15,-699.26\"/>\n",
       "</g>\n",
       "<!-- data.make&#45;sample&#45;data&#45;&gt;fraud_poc.training.submit_training_pipeline -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>data.make&#45;sample&#45;data&#45;&gt;fraud_poc.training.submit_training_pipeline</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M260.07,-652.93C249.29,-643.99 235.92,-632.91 224.16,-623.15\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"226.26,-620.34 216.33,-616.65 221.79,-625.73 226.26,-620.34\"/>\n",
       "</g>\n",
       "<!-- fraud_poc.training.data.preprocess -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>fraud_poc.training.data.preprocess</title>\n",
       "<path fill=\"#aaaaff\" stroke=\"black\" d=\"M118,-435.5C118,-435.5 273.29,-435.5 273.29,-435.5 279.29,-435.5 285.29,-441.5 285.29,-447.5 285.29,-447.5 285.29,-459.5 285.29,-459.5 285.29,-465.5 279.29,-471.5 273.29,-471.5 273.29,-471.5 118,-471.5 118,-471.5 112,-471.5 106,-465.5 106,-459.5 106,-459.5 106,-447.5 106,-447.5 106,-441.5 112,-435.5 118,-435.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"195.65\" y=\"-450.74\" font-family=\"Times,serif\" font-size=\"10.00\">→ fraud_poc.training.data.preprocess →</text>\n",
       "</g>\n",
       "<!-- data.feature&#45;calc -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>data.feature&#45;calc</title>\n",
       "<polygon fill=\"#aaccaa\" stroke=\"black\" points=\"250.01,-399 141.28,-399 141.28,-363 250.01,-363 250.01,-399\"/>\n",
       "<text text-anchor=\"middle\" x=\"195.65\" y=\"-376.8\" font-family=\"Times,serif\" font-size=\"14.00\">data.feature&#45;calc</text>\n",
       "</g>\n",
       "<!-- fraud_poc.training.data.preprocess&#45;&gt;data.feature&#45;calc -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>fraud_poc.training.data.preprocess&#45;&gt;data.feature&#45;calc</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M195.65,-435.43C195.65,-427.67 195.65,-418.28 195.65,-409.56\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"199.15,-409.4 195.65,-399.4 192.15,-409.4 199.15,-409.4\"/>\n",
       "</g>\n",
       "<!-- data.preprocess&#45;&gt;fraud_poc.training.data.preprocess -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>data.preprocess&#45;&gt;fraud_poc.training.data.preprocess</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M195.65,-507.93C195.65,-500.17 195.65,-490.78 195.65,-482.06\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"199.15,-481.9 195.65,-471.9 192.15,-481.9 199.15,-481.9\"/>\n",
       "</g>\n",
       "<!-- fraud_poc.training.data.feature_calc -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>fraud_poc.training.data.feature_calc</title>\n",
       "<path fill=\"#aaaaff\" stroke=\"black\" d=\"M114.96,-290.5C114.96,-290.5 276.34,-290.5 276.34,-290.5 282.34,-290.5 288.34,-296.5 288.34,-302.5 288.34,-302.5 288.34,-314.5 288.34,-314.5 288.34,-320.5 282.34,-326.5 276.34,-326.5 276.34,-326.5 114.96,-326.5 114.96,-326.5 108.96,-326.5 102.96,-320.5 102.96,-314.5 102.96,-314.5 102.96,-302.5 102.96,-302.5 102.96,-296.5 108.96,-290.5 114.96,-290.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"195.65\" y=\"-305.74\" font-family=\"Times,serif\" font-size=\"10.00\">→ fraud_poc.training.data.feature_calc →</text>\n",
       "</g>\n",
       "<!-- data.training&#45;data -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>data.training&#45;data</title>\n",
       "<polygon fill=\"#aaccaa\" stroke=\"black\" points=\"189.99,-254 75.3,-254 75.3,-218 189.99,-218 189.99,-254\"/>\n",
       "<text text-anchor=\"middle\" x=\"132.65\" y=\"-231.8\" font-family=\"Times,serif\" font-size=\"14.00\">data.training&#45;data</text>\n",
       "</g>\n",
       "<!-- fraud_poc.training.data.feature_calc&#45;&gt;data.training&#45;data -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>fraud_poc.training.data.feature_calc&#45;&gt;data.training&#45;data</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M180.39,-290.43C172.77,-281.9 163.4,-271.41 154.98,-261.99\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"157.47,-259.52 148.19,-254.4 152.25,-264.19 157.47,-259.52\"/>\n",
       "</g>\n",
       "<!-- data.prepare&#45;db -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>data.prepare&#45;db</title>\n",
       "<polygon fill=\"#aaccaa\" stroke=\"black\" points=\"330.59,-254 226.7,-254 226.7,-218 330.59,-218 330.59,-254\"/>\n",
       "<text text-anchor=\"middle\" x=\"278.65\" y=\"-231.8\" font-family=\"Times,serif\" font-size=\"14.00\">data.prepare&#45;db</text>\n",
       "</g>\n",
       "<!-- fraud_poc.training.data.feature_calc&#45;&gt;data.prepare&#45;db -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>fraud_poc.training.data.feature_calc&#45;&gt;data.prepare&#45;db</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M215.74,-290.43C226.26,-281.49 239.32,-270.41 250.8,-260.65\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"253.1,-263.29 258.45,-254.15 248.56,-257.96 253.1,-263.29\"/>\n",
       "</g>\n",
       "<!-- data.feature&#45;calc&#45;&gt;fraud_poc.training.data.feature_calc -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>data.feature&#45;calc&#45;&gt;fraud_poc.training.data.feature_calc</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M195.65,-362.93C195.65,-355.17 195.65,-345.78 195.65,-337.06\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"199.15,-336.9 195.65,-326.9 192.15,-336.9 199.15,-336.9\"/>\n",
       "</g>\n",
       "<!-- fraud_poc.training.data.training_data -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>fraud_poc.training.data.training_data</title>\n",
       "<path fill=\"#aaaaff\" stroke=\"black\" d=\"M12,-145.5C12,-145.5 177.29,-145.5 177.29,-145.5 183.29,-145.5 189.29,-151.5 189.29,-157.5 189.29,-157.5 189.29,-169.5 189.29,-169.5 189.29,-175.5 183.29,-181.5 177.29,-181.5 177.29,-181.5 12,-181.5 12,-181.5 6,-181.5 0,-175.5 0,-169.5 0,-169.5 0,-157.5 0,-157.5 0,-151.5 6,-145.5 12,-145.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"94.65\" y=\"-160.74\" font-family=\"Times,serif\" font-size=\"10.00\">→ fraud_poc.training.data.training_data →</text>\n",
       "</g>\n",
       "<!-- model.train -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>model.train</title>\n",
       "<polygon fill=\"#aaccaa\" stroke=\"black\" points=\"134.79,-109 54.5,-109 54.5,-73 134.79,-73 134.79,-109\"/>\n",
       "<text text-anchor=\"middle\" x=\"94.65\" y=\"-86.8\" font-family=\"Times,serif\" font-size=\"14.00\">model.train</text>\n",
       "</g>\n",
       "<!-- fraud_poc.training.data.training_data&#45;&gt;model.train -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>fraud_poc.training.data.training_data&#45;&gt;model.train</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M94.65,-145.43C94.65,-137.67 94.65,-128.28 94.65,-119.56\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"98.15,-119.4 94.65,-109.4 91.15,-119.4 98.15,-119.4\"/>\n",
       "</g>\n",
       "<!-- data.training&#45;data&#45;&gt;fraud_poc.training.data.training_data -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>data.training&#45;data&#45;&gt;fraud_poc.training.data.training_data</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M123.45,-217.93C119.08,-209.83 113.76,-199.96 108.88,-190.91\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"111.85,-189.04 104.02,-181.9 105.69,-192.36 111.85,-189.04\"/>\n",
       "</g>\n",
       "<!-- fraud_poc.training.data.prepare_db -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>fraud_poc.training.data.prepare_db</title>\n",
       "<path fill=\"#aaaaff\" stroke=\"black\" d=\"M218.89,-145.5C218.89,-145.5 376.4,-145.5 376.4,-145.5 382.4,-145.5 388.4,-151.5 388.4,-157.5 388.4,-157.5 388.4,-169.5 388.4,-169.5 388.4,-175.5 382.4,-181.5 376.4,-181.5 376.4,-181.5 218.89,-181.5 218.89,-181.5 212.89,-181.5 206.89,-175.5 206.89,-169.5 206.89,-169.5 206.89,-157.5 206.89,-157.5 206.89,-151.5 212.89,-145.5 218.89,-145.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"297.65\" y=\"-160.74\" font-family=\"Times,serif\" font-size=\"10.00\">→ fraud_poc.training.data.prepare_db →</text>\n",
       "</g>\n",
       "<!-- data.prepare&#45;db&#45;&gt;fraud_poc.training.data.prepare_db -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>data.prepare&#45;db&#45;&gt;fraud_poc.training.data.prepare_db</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M283.24,-217.93C285.36,-210.08 287.92,-200.58 290.3,-191.77\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"293.73,-192.46 292.96,-181.9 286.97,-190.64 293.73,-192.46\"/>\n",
       "</g>\n",
       "<!-- fraud_poc.training.model.train -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>fraud_poc.training.model.train</title>\n",
       "<path fill=\"#aaaaff\" stroke=\"black\" d=\"M25.05,-0.5C25.05,-0.5 164.24,-0.5 164.24,-0.5 170.24,-0.5 176.24,-6.5 176.24,-12.5 176.24,-12.5 176.24,-24.5 176.24,-24.5 176.24,-30.5 170.24,-36.5 164.24,-36.5 164.24,-36.5 25.05,-36.5 25.05,-36.5 19.05,-36.5 13.05,-30.5 13.05,-24.5 13.05,-24.5 13.05,-12.5 13.05,-12.5 13.05,-6.5 19.05,-0.5 25.05,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"94.65\" y=\"-15.74\" font-family=\"Times,serif\" font-size=\"10.00\">→ fraud_poc.training.model.train →</text>\n",
       "</g>\n",
       "<!-- model.train&#45;&gt;fraud_poc.training.model.train -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>model.train&#45;&gt;fraud_poc.training.model.train</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M94.65,-72.93C94.65,-65.17 94.65,-55.78 94.65,-47.06\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"98.15,-46.9 94.65,-36.9 91.15,-46.9 98.15,-46.9\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x1053bc6d0>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_diagram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two entry points:\n",
    "\n",
    "* make_sample_data: endpoint to create sample data and trigger the pipeline steps.\n",
    "* submit_training_pipleine: endpoint to trigger pipeline steps from already existing data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Inference Service\n",
    "\n",
    "As configured in `fraud-service.json` with events implementes in notebooks 08* to 09*, there is a resulting service with two endpoints:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "from hopeit.testing.apps import config\n",
    "from fraud_poc.diagrams import draw_graph\n",
    "\n",
    "service_config = config('config/fraud-service.json')\n",
    "service_diagram = draw_graph(service_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.42.3 (20191010.1750)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"208pt\" height=\"84pt\"\n",
       " viewBox=\"0.00 0.00 208.04 83.60\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 79.6)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-79.6 204.04,-79.6 204.04,4 -4,4\"/>\n",
       "<!-- live.predict.POST -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>live.predict.POST</title>\n",
       "<ellipse fill=\"#aaccaa\" stroke=\"black\" cx=\"39.68\" cy=\"-73.8\" rx=\"1.8\" ry=\"1.8\"/>\n",
       "</g>\n",
       "<!-- live.predict -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>live.predict</title>\n",
       "<polygon fill=\"#aaccaa\" stroke=\"black\" points=\"79.54,-36 -0.18,-36 -0.18,0 79.54,0 79.54,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"39.68\" y=\"-13.8\" font-family=\"Times,serif\" font-size=\"14.00\">live.predict</text>\n",
       "</g>\n",
       "<!-- live.predict.POST&#45;&gt;live.predict -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>live.predict.POST&#45;&gt;live.predict</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M39.68,-71.87C39.68,-68.64 39.68,-57.55 39.68,-46.35\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"43.18,-46.26 39.68,-36.26 36.18,-46.26 43.18,-46.26\"/>\n",
       "</g>\n",
       "<!-- test.find_orders.GET -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>test.find_orders.GET</title>\n",
       "<ellipse fill=\"#aaccaa\" stroke=\"black\" cx=\"148.68\" cy=\"-73.8\" rx=\"1.8\" ry=\"1.8\"/>\n",
       "</g>\n",
       "<!-- test.find_orders -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>test.find_orders</title>\n",
       "<polygon fill=\"#aaccaa\" stroke=\"black\" points=\"199.9,-36 97.47,-36 97.47,0 199.9,0 199.9,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"148.68\" y=\"-13.8\" font-family=\"Times,serif\" font-size=\"14.00\">test.find_orders</text>\n",
       "</g>\n",
       "<!-- test.find_orders.GET&#45;&gt;test.find_orders -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>test.find_orders.GET&#45;&gt;test.find_orders</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M148.68,-71.87C148.68,-68.64 148.68,-57.55 148.68,-46.35\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"152.18,-46.26 148.68,-36.26 145.18,-46.26 152.18,-46.26\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x1067616a0>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "service_diagram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* live.predict: endpoint required to enter order information and return predictions and calculated feature values.\n",
    "* test.find_orders: it's a helper endpoint to find random generated orders to be used in this example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data processing and training pipeline\n",
    "\n",
    "* To run training pipeline service:\n",
    "\n",
    "```\n",
    "hopeit_server run --config-files=config/server.json,config/training-pipeline.json --api-file=config/openapi-training.json --start-streams --port=8020\n",
    "```\n",
    "        \n",
    "You should see a couple endpoints in http://localhost:8020/api/docs\n",
    "\n",
    "* The first endpoint \"Data: Make Sample Data\" will run the whole data+training pipeline end to end if you click in `Try It`\n",
    "![](docs/img/api01.png)\n",
    "\n",
    "1) **create-sample-data**: will create random orders in parquet format into folder `./data/raw/`. This dataset is partitioned by time periods, i.e. 1 file per 30-day batch in this example. Once this step is finished the end of the job will be notified using hopeit.engine streams funcionallity and the next job will take place once the event is consumed.\n",
    "\n",
    "2) **preprocess**: reads data generated in previous step and creates new parquet files partitioned by customer_id and email, so aggregations on those two dimensions can be performed more efficiently later. Again, once the job is finished, the next step will be notified. This generated files also can be use for data analysis and feature discovering using Jupyter and Dask.\n",
    "\n",
    "3) **feature-calc**: calculates aggregations on customer_ids and emails (i.e. accumulates most recent emails, ip_addrs, counts, order_amounts, etc) and stores a new data set of orders enriched with this extra information.\n",
    "\n",
    "4) **training-data**: prepares data for training: obtain labels for the orders (in this POC `is_fraud` label field is just assigned using a combination of calculations with some randomness) and creates a more balanced dataset subsampling non-fraud cases, creates a validation set using more recent non-fraud and fraud labeled transactions. Next step is notified when data is ready. The dataset is shuffle randomly into N partitions (10 in the example) so training can be performed from each partition using fairly-balanced datasets.\n",
    "\n",
    "5) **train-model**: trains an XGBoost model on sampled data using Dask distributed implementation. Validates model precision and recall using validation dataset and if validation passes a configured treshold, model is saved to be used in prediction service.\n",
    "\n",
    "6) **prepare-db**: stores most recent customer_id and email features calculated in step 3) into a Redis database that can be used for real-time prediction service. (Notice that this data should be continuously updated on new orders but this is not provided in this POC)\n",
    "\n",
    "Since data generation could be tedious, there is a second endpoint that allows to run just from step 02, assuming\n",
    "you already have raw data:\n",
    "![](docs/img/api02.png)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fraud prediction service\n",
    "\n",
    "To run the live prediction service:\n",
    "\n",
    "```\n",
    "hopeit_server run --config-files=config/server.json,config/fraud-service.json --api-file=config/openapi-service.json --start-streams --port=8021\n",
    "```\n",
    "\n",
    "You can try the endpoints using in http://localhost:8021/api/docs\n",
    "\n",
    "* First extract some valid customer_id and email using:\n",
    "```\n",
    "curl -X GET \"http://localhost:8021/api/fraud-poc/0x0x1-service/test/find-orders?prefix=*&num_items=10\" \\\n",
    " -H \"Accept: application/json\" \n",
    "```\n",
    "This POC only can predict fraud for known customer_id and email in the generated data.\n",
    "\n",
    "Using a customer_id and email, pass a new order to the service using the Live: Predict Endpoint:\n",
    "![](docs/img/api03.png)\n",
    "\n",
    "And check the results, all calcualted features plus an is_fraud field is returned:\n",
    "```\n",
    "{\n",
    "  \"order_id\": \"ce4798f5-6127-4d6e-bf1d-dda810eab26b\",\n",
    "  \"order_date\": \"2020-07-07T06:33:18+00:00\",\n",
    "  \"customer_id\": \"271d8c5e-e4e3-4377-a3e3-673ccf153664\",\n",
    "  \"ip_addr\": \"f95e9c978b7f88dde5b9eb39417070251603db2d\",\n",
    "  \"order_amount\": 100.7097195892065,\n",
    "  \"email\": \"7545576ffe1b7c1d9d8d2e82d0191fa057df695f\",\n",
    "  \"customer_id_by_email\": [\n",
    "    \"271d8c5e-e4e3-4377-a3e3-673ccf153664\"\n",
    "  ],\n",
    "  \"num_customer_id_by_email\": 1,\n",
    "  \"last_customer_id_by_email\": \"271d8c5e-e4e3-4377-a3e3-673ccf153664\",\n",
    "  \"same_customer_id_by_email\": 1,\n",
    "  \"known_customer_id_by_email\": 1,\n",
    "  \"order_amount_mean_by_email\": 468.79164250074143,\n",
    "  \"order_amount_std_by_email\": 317.0635415216074,\n",
    "  \"order_amount_min_by_email\": 68.2940660160266,\n",
    "  \"order_amount_max_by_email\": 916.7097195892065,\n",
    "  \"order_amount_sum_by_email\": 4687.916425007415,\n",
    "  \"order_amount_by_email\": [\n",
    "    769.0840886685221,\n",
    "    68.2940660160266,\n",
    "    164.22372869469348,\n",
    "    198.35357128773578,\n",
    "    454.66931470215576,\n",
    "    100.7097195892065,\n",
    "    779.1408217338134,\n",
    "    916.7097195892065,\n",
    "    854.4217419999278,\n",
    "    382.3096527261267\n",
    "  ],\n",
    "  \"key\": \"271d8c5e-e4e3-4377-a3e3-673ccf153664\",\n",
    "  \"email_by_customer_id\": [\n",
    "    \"7545576ffe1b7c1d9d8d2e82d0191fa057df695f\"\n",
    "  ],\n",
    "  \"ip_addr_by_customer_id\": [\n",
    "    \"f95e9c978b7f88dde5b9eb39417070251603db2d\",\n",
    "    \"788e574cf1934b34e9510ce897d8a593ab9dbcc9\",\n",
    "    \"d02eae79264a401d76e853c41bdb781484443db2\"\n",
    "  ],\n",
    "  \"num_email_by_customer_id\": 1,\n",
    "  \"num_ip_addr_by_customer_id\": 3,\n",
    "  \"last_email_by_customer_id\": \"7545576ffe1b7c1d9d8d2e82d0191fa057df695f\",\n",
    "  \"last_ip_addr_by_customer_id\": \"f95e9c978b7f88dde5b9eb39417070251603db2d\",\n",
    "  \"same_email_by_customer_id\": 1,\n",
    "  \"same_ip_addr_by_customer_id\": 1,\n",
    "  \"known_email_by_customer_id\": 1,\n",
    "  \"known_ip_addr_by_customer_id\": 1,\n",
    "  \"order_amount_mean_by_customer_id\": 468.79164250074143,\n",
    "  \"order_amount_std_by_customer_id\": 317.0635415216074,\n",
    "  \"order_amount_min_by_customer_id\": 68.2940660160266,\n",
    "  \"order_amount_max_by_customer_id\": 916.7097195892065,\n",
    "  \"order_amount_sum_by_customer_id\": 4687.916425007415,\n",
    "  \"order_amount_by_customer_id\": [\n",
    "    769.0840886685221,\n",
    "    68.2940660160266,\n",
    "    164.22372869469348,\n",
    "    198.35357128773578,\n",
    "    454.66931470215576,\n",
    "    100.7097195892065,\n",
    "    779.1408217338134,\n",
    "    916.7097195892065,\n",
    "    854.4217419999278,\n",
    "    382.3096527261267\n",
    "  ],\n",
    "  \"location_lat\": 0,\n",
    "  \"location_long\": 0,\n",
    "  \"is_fraud\": 0.5424039363861084\n",
    "}\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So that's it, please feel free to submit feedback and suggestions! Please contact me in case you want to improve pieces like dataset generation, model tuning, etc.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I Hope you enjoyed it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}