notebooks/assignments/Week4Notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "d54e336e-4354-4b7c-8d05-0492855b81ba",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "# Scope of Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "ccc2775f-ba77-49e0-9ced-ff4bcbf3efdf",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "The goal of this notebook is to showcase how you can use, in your own environment, a pre-trained model along with some profile data extracted from the Adobe Experience Platform to generate propensity scores and ingest those back to enrich the Unified Profile.\n",
    "\n",
    "![Workflow](../media/CMLE-Notebooks-Week4-Workflow.png)\n",
    "\n",
    "We'll go through several steps:\n",
    "- **Reading the featurized data** from the Data Landing Zone\n",
    "- Generating the **scores**\n",
    "- Creating a **target dataset**\n",
    "- Creating a **dataflow** to deliver data in the right format to that dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "83c69965-e562-493c-b90c-df08b1ad0008",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "ac6b4958-397f-4aad-9d81-afb1d45ea0b2",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "This notebook requires some configuration data to properly authenticate to your Adobe Experience Platform instance. You should be able to find all the values required above by following the Setup section of the **README**.\n",
    "\n",
    "The next cell will be looking for your configuration file under your **ADOBE_HOME** path to fetch the values used throughout this notebook. See more details in the Setup section of the **README** to understand how to create your configuration file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "978eccff-5e2b-4dd3-9bc7-844e055d80b3",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from configparser import ConfigParser\n",
    "\n",
    "if \"ADOBE_HOME\" not in os.environ:\n",
    "    raise Exception(\"ADOBE_HOME environment variable needs to be set.\")\n",
    "\n",
    "config = ConfigParser()\n",
    "config_path = os.path.join(os.environ[\"ADOBE_HOME\"], \"conf\", \"config.ini\")\n",
    "\n",
    "if not os.path.exists(config_path):\n",
    "    raise Exception(f\"Looking for configuration under {config_path} but config not found, please verify path\")\n",
    "\n",
    "config.read(config_path)\n",
    "\n",
    "ims_org_id = config.get(\"Platform\", \"ims_org_id\")\n",
    "sandbox_name = config.get(\"Platform\", \"sandbox_name\")\n",
    "environment = config.get(\"Platform\", \"environment\")\n",
    "client_id = config.get(\"Authentication\", \"client_id\")\n",
    "client_secret = config.get(\"Authentication\", \"client_secret\")\n",
    "scopes = config.get(\"Authentication\", \"scopes\")\n",
    "dataset_id = config.get(\"Platform\", \"dataset_id\")\n",
    "featurized_dataset_id = config.get(\"Platform\", \"featurized_dataset_id\")\n",
    "export_path = config.get(\"Cloud\", \"export_path\")\n",
    "import_path = config.get(\"Cloud\", \"import_path\")\n",
    "data_format = config.get(\"Cloud\", \"data_format\")\n",
    "compression_type = config.get(\"Cloud\", \"compression_type\")\n",
    "model_name = config.get(\"Cloud\", \"model_name\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "180a214a-3a8a-4e87-94e6-c459112c88ef",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Some utility functions that will be used throughout this notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "54c6b121-d3aa-461e-9445-72b7e07efb0c",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "def get_ui_link(tenant_id, resource_type, resource_id):\n",
    "    if environment == \"prod\":\n",
    "        prefix = f\"https://experience.adobe.com\"\n",
    "    else:\n",
    "        prefix = f\"https://experience-{environment}.adobe.com\"\n",
    "    return f\"{prefix}/#/@{tenant_id}/sname:{sandbox_name}/platform/{resource_type}/{resource_id}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "df081db7-0179-422b-8cdf-41278a8ba8bb",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Username: cmenguy@adobe.com\n",
      "Unique ID: cmenguy_adobe_com\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "username = dbutils.notebook.entry_point.getDbutils().notebook().getContext().userName().get()\n",
    "unique_id = s = re.sub(\"[^0-9a-zA-Z]+\", \"_\", username)\n",
    "\n",
    "print(f\"Username: {username}\")\n",
    "print(f\"Unique ID: {unique_id}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "8cf3303b-e086-4638-9aa2-7d62bdca24bd",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Before we run anything, make sure to install the following required libraries for this notebook. They are all publicly available libraries and the latest version should work fine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "44dd3429-850b-4679-82e3-dfea9688993b",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting mlflow\r\n",
      "  Downloading mlflow-2.2.2-py3-none-any.whl (17.6 MB)\r\n",
      "\u001b[?25l\r",
      "\u001b[K     |                                | 10 kB 38.3 MB/s eta 0:00:01\r",
      "\u001b[K     |                                | 20 kB 10.6 MB/s eta 0:00:02\r",
      "\u001b[K     |                                | 30 kB 15.0 MB/s eta 0:00:02\r",
      "\u001b[K     |                                | 40 kB 10.1 MB/s eta 0:00:02\r",
      "\u001b[K     |                                | 51 kB 8.7 MB/s eta 0:00:03\r",
      "\u001b[K     |▏                               | 61 kB 10.3 MB/s eta 0:00:02\r",
      "\u001b[K     |▏                               | 71 kB 10.7 MB/s eta 0:00:02\r",
      "\u001b[K     |▏                               | 81 kB 12.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▏                               | 92 kB 10.4 MB/s eta 0:00:02\r",
      "\u001b[K     |▏                               | 102 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▏                               | 112 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▎                               | 122 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▎                               | 133 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▎                               | 143 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▎                               | 153 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▎                               | 163 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▎                               | 174 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▍                               | 184 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▍                               | 194 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▍                               | 204 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▍                               | 215 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▍                               | 225 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▍                               | 235 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▌                               | 245 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▌                               | 256 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▌                               | 266 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▌                               | 276 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▌                               | 286 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▌                               | 296 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▋                               | 307 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▋                               | 317 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▋                               | 327 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▋                               | 337 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▋                               | 348 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▋                               | 358 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▊                               | 368 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▊                               | 378 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▊                               | 389 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▊                               | 399 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▊                               | 409 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▊                               | 419 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▉                               | 430 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▉                               | 440 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▉                               | 450 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▉                               | 460 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▉                               | 471 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |▉                               | 481 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█                               | 491 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█                               | 501 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█                               | 512 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█                               | 522 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█                               | 532 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█                               | 542 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█                               | 552 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█                               | 563 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█                               | 573 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█                               | 583 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█                               | 593 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█                               | 604 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▏                              | 614 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▏                              | 624 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▏                              | 634 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▏                              | 645 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▏                              | 655 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▏                              | 665 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▎                              | 675 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▎                              | 686 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▎                              | 696 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▎                              | 706 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▎                              | 716 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▎                              | 727 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▍                              | 737 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▍                              | 747 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▍                              | 757 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▍                              | 768 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▍                              | 778 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▍                              | 788 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▌                              | 798 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▌                              | 808 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▌                              | 819 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▌                              | 829 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▌                              | 839 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▌                              | 849 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▋                              | 860 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▋                              | 870 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▋                              | 880 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▋                              | 890 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▋                              | 901 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▋                              | 911 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▊                              | 921 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▊                              | 931 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▊                              | 942 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▊                              | 952 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▊                              | 962 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▊                              | 972 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▉                              | 983 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▉                              | 993 kB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▉                              | 1.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▉                              | 1.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▉                              | 1.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█▉                              | 1.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██                              | 1.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██                              | 1.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██                              | 1.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██                              | 1.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██                              | 1.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██                              | 1.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██                              | 1.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██                              | 1.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██                              | 1.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██                              | 1.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██                              | 1.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██                              | 1.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▏                             | 1.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▏                             | 1.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▏                             | 1.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▏                             | 1.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▏                             | 1.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▏                             | 1.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▎                             | 1.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▎                             | 1.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▎                             | 1.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▎                             | 1.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▎                             | 1.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▎                             | 1.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▍                             | 1.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▍                             | 1.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▍                             | 1.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▍                             | 1.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▍                             | 1.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▍                             | 1.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▌                             | 1.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▌                             | 1.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▌                             | 1.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▌                             | 1.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▌                             | 1.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▌                             | 1.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▋                             | 1.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▋                             | 1.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▋                             | 1.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▋                             | 1.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▋                             | 1.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▊                             | 1.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▊                             | 1.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▊                             | 1.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▊                             | 1.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▊                             | 1.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▊                             | 1.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▉                             | 1.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▉                             | 1.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▉                             | 1.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▉                             | 1.6 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▉                             | 1.6 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██▉                             | 1.6 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███                             | 1.6 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███                             | 1.6 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███                             | 1.6 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███                             | 1.6 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███                             | 1.6 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███                             | 1.6 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███                             | 1.6 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███                             | 1.7 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███                             | 1.7 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███                             | 1.7 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███                             | 1.7 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███                             | 1.7 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▏                            | 1.7 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▏                            | 1.7 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▏                            | 1.7 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▏                            | 1.7 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▏                            | 1.8 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▏                            | 1.8 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▎                            | 1.8 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▎                            | 1.8 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▎                            | 1.8 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▎                            | 1.8 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▎                            | 1.8 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▎                            | 1.8 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▍                            | 1.8 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▍                            | 1.8 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▍                            | 1.9 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▍                            | 1.9 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▍                            | 1.9 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▍                            | 1.9 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▌                            | 1.9 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▌                            | 1.9 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▌                            | 1.9 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▌                            | 1.9 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▌                            | 1.9 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▌                            | 1.9 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▋                            | 2.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▋                            | 2.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▋                            | 2.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▋                            | 2.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▋                            | 2.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▋                            | 2.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▊                            | 2.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▊                            | 2.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▊                            | 2.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▊                            | 2.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▊                            | 2.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▊                            | 2.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▉                            | 2.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▉                            | 2.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▉                            | 2.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▉                            | 2.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▉                            | 2.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |███▉                            | 2.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████                            | 2.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████                            | 2.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████                            | 2.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████                            | 2.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████                            | 2.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████                            | 2.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████                            | 2.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████                            | 2.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████                            | 2.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████                            | 2.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████                            | 2.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████                            | 2.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▏                           | 2.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▏                           | 2.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▏                           | 2.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▏                           | 2.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▏                           | 2.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▏                           | 2.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▎                           | 2.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▎                           | 2.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▎                           | 2.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▎                           | 2.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▎                           | 2.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▎                           | 2.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▍                           | 2.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▍                           | 2.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▍                           | 2.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▍                           | 2.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▍                           | 2.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▍                           | 2.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▌                           | 2.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▌                           | 2.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▌                           | 2.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▌                           | 2.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▌                           | 2.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▌                           | 2.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▋                           | 2.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▋                           | 2.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▋                           | 2.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▋                           | 2.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▋                           | 2.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▋                           | 2.6 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▊                           | 2.6 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▊                           | 2.6 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▊                           | 2.6 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▊                           | 2.6 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▊                           | 2.6 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▊                           | 2.6 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▉                           | 2.6 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▉                           | 2.6 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▉                           | 2.7 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▉                           | 2.7 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▉                           | 2.7 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |████▉                           | 2.7 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████                           | 2.7 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████                           | 2.7 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████                           | 2.7 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████                           | 2.7 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████                           | 2.7 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████                           | 2.7 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████                           | 2.8 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████                           | 2.8 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████                           | 2.8 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████                           | 2.8 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████                           | 2.8 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████                           | 2.8 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▏                          | 2.8 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▏                          | 2.8 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▏                          | 2.8 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▏                          | 2.8 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▏                          | 2.9 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▏                          | 2.9 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▎                          | 2.9 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▎                          | 2.9 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▎                          | 2.9 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▎                          | 2.9 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▎                          | 2.9 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▍                          | 2.9 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▍                          | 2.9 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▍                          | 2.9 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▍                          | 3.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▍                          | 3.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▍                          | 3.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▌                          | 3.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▌                          | 3.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▌                          | 3.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▌                          | 3.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▌                          | 3.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▌                          | 3.0 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▋                          | 3.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▋                          | 3.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▋                          | 3.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▋                          | 3.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▋                          | 3.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▋                          | 3.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▊                          | 3.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▊                          | 3.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▊                          | 3.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▊                          | 3.1 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▊                          | 3.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▊                          | 3.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▉                          | 3.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▉                          | 3.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▉                          | 3.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▉                          | 3.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▉                          | 3.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |█████▉                          | 3.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████                          | 3.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████                          | 3.2 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████                          | 3.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████                          | 3.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████                          | 3.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████                          | 3.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████                          | 3.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████                          | 3.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████                          | 3.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████                          | 3.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████                          | 3.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████                          | 3.3 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████▏                         | 3.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████▏                         | 3.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████▏                         | 3.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████▏                         | 3.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████▏                         | 3.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████▏                         | 3.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████▎                         | 3.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████▎                         | 3.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████▎                         | 3.4 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████▎                         | 3.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████▎                         | 3.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████▎                         | 3.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████▍                         | 3.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████▍                         | 3.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████▍                         | 3.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████▍                         | 3.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████▍                         | 3.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████▍                         | 3.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████▌                         | 3.5 MB 11.0 MB/s eta 0:00:02\r",
      "\u001b[K     |██████▌ \n",
      "\n",
      "*** WARNING: max output size exceeded, skipping output. ***\n",
      "\n",
      "██████████████████▋            | 1.7 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████▊            | 1.7 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████▉            | 1.7 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████            | 1.7 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████            | 1.7 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████▏           | 1.8 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████▎           | 1.8 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████▍           | 1.8 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████▌           | 1.8 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████▋           | 1.8 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████▊           | 1.8 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████▉           | 1.8 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████           | 1.8 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████           | 1.8 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████▏          | 1.8 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████▍          | 1.9 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████▌          | 1.9 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████▋          | 1.9 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████▊          | 1.9 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████▉          | 1.9 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████          | 1.9 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████          | 1.9 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████▏         | 1.9 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████▎         | 1.9 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████▍         | 1.9 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████▌         | 2.0 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████▋         | 2.0 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████▊         | 2.0 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████▉         | 2.0 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████         | 2.0 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████         | 2.0 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████▏        | 2.0 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████▍        | 2.0 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████▌        | 2.0 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████▋        | 2.0 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████▊        | 2.1 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████▉        | 2.1 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████        | 2.1 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████        | 2.1 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████▏       | 2.1 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████▎       | 2.1 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████▍       | 2.1 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████▌       | 2.1 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████▋       | 2.1 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████▊       | 2.2 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████▉       | 2.2 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████       | 2.2 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████       | 2.2 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▎      | 2.2 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▍      | 2.2 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▌      | 2.2 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▋      | 2.2 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▊      | 2.2 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▉      | 2.2 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████      | 2.3 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████      | 2.3 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▏     | 2.3 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▎     | 2.3 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▍     | 2.3 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▌     | 2.3 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▋     | 2.3 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▊     | 2.3 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▉     | 2.3 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████     | 2.3 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████     | 2.4 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▎    | 2.4 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▍    | 2.4 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▌    | 2.4 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▋    | 2.4 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▊    | 2.4 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▉    | 2.4 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████    | 2.4 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████    | 2.4 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▏   | 2.4 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▎   | 2.5 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▍   | 2.5 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▌   | 2.5 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▋   | 2.5 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▊   | 2.5 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▉   | 2.5 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████   | 2.5 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▏  | 2.5 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▎  | 2.5 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▍  | 2.5 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▌  | 2.6 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▋  | 2.6 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▊  | 2.6 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▉  | 2.6 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████  | 2.6 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████  | 2.6 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▏ | 2.6 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▎ | 2.6 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▍ | 2.6 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▌ | 2.7 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▋ | 2.7 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▊ | 2.7 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▉ | 2.7 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████ | 2.7 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▏| 2.7 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▎| 2.7 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▍| 2.7 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▌| 2.7 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▋| 2.7 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▊| 2.8 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▉| 2.8 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████████| 2.8 MB 86.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████████| 2.8 MB 86.7 MB/s \r\n",
      "\u001b[?25hRequirement already satisfied: packaging<24 in /databricks/python3/lib/python3.9/site-packages (from mlflow) (21.3)\r\n",
      "Requirement already satisfied: sqlparse<1,>=0.4.0 in /databricks/python3/lib/python3.9/site-packages (from mlflow) (0.4.2)\r\n",
      "Requirement already satisfied: databricks-cli<1,>=0.8.7 in /databricks/python3/lib/python3.9/site-packages (from mlflow) (0.17.4)\r\n",
      "Requirement already satisfied: click<9,>=7.0 in /databricks/python3/lib/python3.9/site-packages (from mlflow) (8.0.4)\r\n",
      "Requirement already satisfied: Mako in /databricks/python3/lib/python3.9/site-packages (from alembic<2->mlflow) (1.2.0)\r\n",
      "Requirement already satisfied: typing-extensions>=4 in /databricks/python3/lib/python3.9/site-packages (from alembic<2->mlflow) (4.1.1)\r\n",
      "Requirement already satisfied: tabulate>=0.7.7 in /databricks/python3/lib/python3.9/site-packages (from databricks-cli<1,>=0.8.7->mlflow) (0.8.9)\r\n",
      "Requirement already satisfied: six>=1.10.0 in /databricks/python3/lib/python3.9/site-packages (from databricks-cli<1,>=0.8.7->mlflow) (1.16.0)\r\n",
      "Requirement already satisfied: oauthlib>=3.1.0 in /databricks/python3/lib/python3.9/site-packages (from databricks-cli<1,>=0.8.7->mlflow) (3.2.0)\r\n",
      "Requirement already satisfied: pyjwt>=1.7.0 in /databricks/python3/lib/python3.9/site-packages (from databricks-cli<1,>=0.8.7->mlflow) (2.6.0)\r\n",
      "Requirement already satisfied: urllib3>=1.26.0 in /databricks/python3/lib/python3.9/site-packages (from docker<7,>=4.0.0->mlflow) (1.26.9)\r\n",
      "Requirement already satisfied: websocket-client>=0.32.0 in /databricks/python3/lib/python3.9/site-packages (from docker<7,>=4.0.0->mlflow) (0.58.0)\r\n",
      "Requirement already satisfied: itsdangerous>=0.24 in /databricks/python3/lib/python3.9/site-packages (from Flask<3->mlflow) (2.0.1)\r\n",
      "Requirement already satisfied: Werkzeug>=0.15 in /databricks/python3/lib/python3.9/site-packages (from Flask<3->mlflow) (2.0.3)\r\n",
      "Requirement already satisfied: gitdb<5,>=4.0.1 in /databricks/python3/lib/python3.9/site-packages (from gitpython<4,>=2.1.0->mlflow) (4.0.10)\r\n",
      "Requirement already satisfied: smmap<6,>=3.0.1 in /databricks/python3/lib/python3.9/site-packages (from gitdb<5,>=4.0.1->gitpython<4,>=2.1.0->mlflow) (5.0.0)\r\n",
      "Requirement already satisfied: setuptools>=3.0 in /databricks/python3/lib/python3.9/site-packages (from gunicorn<21->mlflow) (61.2.0)\r\n",
      "Requirement already satisfied: zipp>=0.5 in /databricks/python3/lib/python3.9/site-packages (from importlib-metadata!=4.7.0,<7,>=3.7.0->mlflow) (3.7.0)\r\n",
      "Requirement already satisfied: MarkupSafe>=0.23 in /databricks/python3/lib/python3.9/site-packages (from Jinja2<4,>=2.11->mlflow) (2.0.1)\r\n",
      "Requirement already satisfied: python-dateutil>=2.7 in /databricks/python3/lib/python3.9/site-packages (from matplotlib<4->mlflow) (2.8.2)\r\n",
      "Requirement already satisfied: fonttools>=4.22.0 in /databricks/python3/lib/python3.9/site-packages (from matplotlib<4->mlflow) (4.25.0)\r\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /databricks/python3/lib/python3.9/site-packages (from matplotlib<4->mlflow) (1.3.2)\r\n",
      "Requirement already satisfied: pyparsing>=2.2.1 in /databricks/python3/lib/python3.9/site-packages (from matplotlib<4->mlflow) (3.0.4)\r\n",
      "Requirement already satisfied: cycler>=0.10 in /databricks/python3/lib/python3.9/site-packages (from matplotlib<4->mlflow) (0.11.0)\r\n",
      "Requirement already satisfied: pillow>=6.2.0 in /databricks/python3/lib/python3.9/site-packages (from matplotlib<4->mlflow) (9.0.1)\r\n",
      "Requirement already satisfied: idna<4,>=2.5 in /databricks/python3/lib/python3.9/site-packages (from requests<3,>=2.17.3->mlflow) (3.3)\r\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /databricks/python3/lib/python3.9/site-packages (from requests<3,>=2.17.3->mlflow) (2.0.4)\r\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /databricks/python3/lib/python3.9/site-packages (from requests<3,>=2.17.3->mlflow) (2021.10.8)\r\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /databricks/python3/lib/python3.9/site-packages (from scikit-learn<2->mlflow) (2.2.0)\r\n",
      "Requirement already satisfied: joblib>=0.11 in /databricks/python3/lib/python3.9/site-packages (from scikit-learn<2->mlflow) (1.1.0)\r\n",
      "Requirement already satisfied: numba in /databricks/python3/lib/python3.9/site-packages (from shap<1,>=0.40->mlflow) (0.55.1)\r\n",
      "Requirement already satisfied: tqdm>4.25.0 in /databricks/python3/lib/python3.9/site-packages (from shap<1,>=0.40->mlflow) (4.64.0)\r\n",
      "Requirement already satisfied: slicer==0.0.7 in /databricks/python3/lib/python3.9/site-packages (from shap<1,>=0.40->mlflow) (0.0.7)\r\n",
      "Collecting typing-extensions>=4\r\n",
      "  Downloading typing_extensions-4.5.0-py3-none-any.whl (27 kB)\r\n",
      "Collecting greenlet!=0.4.17\r\n",
      "  Downloading greenlet-2.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (610 kB)\r\n",
      "\u001b[?25l\r",
      "\u001b[K     |▌                               | 10 kB 37.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█                               | 20 kB 44.8 MB/s eta 0:00:01\r",
      "\u001b[K     |█▋                              | 30 kB 55.8 MB/s eta 0:00:01\r",
      "\u001b[K     |██▏                             | 40 kB 62.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██▊                             | 51 kB 67.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███▏                            | 61 kB 74.8 MB/s eta 0:00:01\r",
      "\u001b[K     |███▊                            | 71 kB 79.2 MB/s eta 0:00:01\r",
      "\u001b[K     |████▎                           | 81 kB 84.4 MB/s eta 0:00:01\r",
      "\u001b[K     |████▉                           | 92 kB 87.9 MB/s eta 0:00:01\r",
      "\u001b[K     |█████▍                          | 102 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████                          | 112 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████▍                         | 122 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████                         | 133 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████▌                        | 143 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████                        | 153 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████▋                       | 163 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████▏                      | 174 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████▋                      | 184 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████▏                     | 194 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████▊                     | 204 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████▎                    | 215 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████▉                    | 225 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████▍                   | 235 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████▉                   | 245 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████▍                  | 256 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████                  | 266 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████▌                 | 276 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████                 | 286 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████▋                | 296 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████                | 307 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████▋               | 317 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████▏              | 327 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████▊              | 337 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████▎             | 348 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████▊             | 358 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████▎            | 368 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████▉            | 378 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████▍           | 389 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████           | 399 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████▌          | 409 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████          | 419 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████▌         | 430 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████         | 440 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████▋        | 450 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████▏       | 460 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████▊       | 471 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▏      | 481 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████▊      | 491 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▎     | 501 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████▉     | 512 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████▍    | 522 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████    | 532 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████▍   | 542 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████   | 552 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |█████████████████████████████▌  | 563 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████  | 573 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |██████████████████████████████▋ | 583 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▏| 593 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |███████████████████████████████▋| 604 kB 90.7 MB/s eta 0:00:01\r",
      "\u001b[K     |████████████████████████████████| 610 kB 90.7 MB/s \r\n",
      "\u001b[?25hRequirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in /databricks/python3/lib/python3.9/site-packages (from numba->shap<1,>=0.40->mlflow) (0.38.0)\r\n",
      "Installing collected packages: typing-extensions, greenlet, sqlalchemy, querystring-parser, docker, alembic, mlflow\r\n",
      "  Attempting uninstall: typing-extensions\r\n",
      "    Found existing installation: typing-extensions 4.1.1\r\n",
      "    Not uninstalling typing-extensions at /databricks/python3/lib/python3.9/site-packages, outside environment /local_disk0/.ephemeral_nfs/envs/pythonEnv-ea947958-e611-4e90-942a-530ed26066bd\r\n",
      "    Can't uninstall 'typing-extensions'. No files were found to uninstall.\r\n",
      "Successfully installed alembic-1.10.2 docker-6.0.1 greenlet-2.0.2 mlflow-2.2.2 querystring-parser-1.2.4 sqlalchemy-2.0.8 typing-extensions-4.5.0\r\n",
      "\u001b[33mWARNING: You are using pip version 21.2.4; however, version 23.0.1 is available.\r\n",
      "You should consider upgrading via the '/local_disk0/.ephemeral_nfs/envs/pythonEnv-ea947958-e611-4e90-942a-530ed26066bd/bin/python -m pip install --upgrade pip' command.\u001b[0m\r\n",
      "Collecting aepp\r\n",
      "  Using cached aepp-0.2.9-py3-none-any.whl (120 kB)\r\n",
      "Collecting pathlib2\r\n",
      "  Using cached pathlib2-2.3.7.post1-py2.py3-none-any.whl (18 kB)\r\n",
      "Collecting pathlib\r\n",
      "  Using cached pathlib-1.0.1-py3-none-any.whl (14 kB)\r\n",
      "Requirement already satisfied: PyJWT in /databricks/python3/lib/python3.9/site-packages (from aepp) (2.6.0)\r\n",
      "Requirement already satisfied: pandas in /databricks/python3/lib/python3.9/site-packages (from aepp) (1.4.2)\r\n",
      "Requirement already satisfied: requests in /databricks/python3/lib/python3.9/site-packages (from aepp) (2.27.1)\r\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in /databricks/python3/lib/python3.9/site-packages (from pandas->aepp) (2.8.2)\r\n",
      "Requirement already satisfied: pytz>=2020.1 in /databricks/python3/lib/python3.9/site-packages (from pandas->aepp) (2021.3)\r\n",
      "Requirement already satisfied: numpy>=1.18.5 in /databricks/python3/lib/python3.9/site-packages (from pandas->aepp) (1.21.5)\r\n",
      "Requirement already satisfied: six>=1.5 in /databricks/python3/lib/python3.9/site-packages (from python-dateutil>=2.8.1->pandas->aepp) (1.16.0)\r\n",
      "Requirement already satisfied: cryptography>=3.4.0 in /databricks/python3/lib/python3.9/site-packages (from PyJWT->aepp) (3.4.8)\r\n",
      "Requirement already satisfied: cffi>=1.12 in /databricks/python3/lib/python3.9/site-packages (from cryptography>=3.4.0->PyJWT->aepp) (1.15.0)\r\n",
      "Requirement already satisfied: pycparser in /databricks/python3/lib/python3.9/site-packages (from cffi>=1.12->cryptography>=3.4.0->PyJWT->aepp) (2.21)\r\n",
      "Requirement already satisfied: idna<4,>=2.5 in /databricks/python3/lib/python3.9/site-packages (from requests->aepp) (3.3)\r\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /databricks/python3/lib/python3.9/site-packages (from requests->aepp) (2.0.4)\r\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /databricks/python3/lib/python3.9/site-packages (from requests->aepp) (1.26.9)\r\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /databricks/python3/lib/python3.9/site-packages (from requests->aepp) (2021.10.8)\r\n",
      "Installing collected packages: pathlib2, pathlib, aepp\r\n",
      "Successfully installed aepp-0.2.9 pathlib-1.0.1 pathlib2-2.3.7.post1\r\n",
      "\u001b[33mWARNING: You are using pip version 21.2.4; however, version 23.0.1 is available.\r\n",
      "You should consider upgrading via the '/local_disk0/.ephemeral_nfs/envs/pythonEnv-ea947958-e611-4e90-942a-530ed26066bd/bin/python -m pip install --upgrade pip' command.\u001b[0m\r\n",
      "Collecting adlfs\r\n",
      "  Using cached adlfs-2023.1.0-py3-none-any.whl (25 kB)\r\n",
      "Requirement already satisfied: fsspec>=2021.10.1 in /databricks/python3/lib/python3.9/site-packages (from adlfs) (2022.2.0)\r\n",
      "Collecting azure-datalake-store<0.1,>=0.0.46\r\n",
      "  Using cached azure_datalake_store-0.0.52-py2.py3-none-any.whl (61 kB)\r\n",
      "Collecting azure-storage-blob>=12.12.0\r\n",
      "  Using cached azure_storage_blob-12.15.0-py3-none-any.whl (387 kB)\r\n",
      "Requirement already satisfied: azure-core<2.0.0,>=1.23.1 in /databricks/python3/lib/python3.9/site-packages (from adlfs) (1.26.1)\r\n",
      "Collecting aiohttp>=3.7.0\r\n",
      "  Using cached aiohttp-3.8.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)\r\n",
      "Collecting azure-identity\r\n",
      "  Using cached azure_identity-1.12.0-py3-none-any.whl (135 kB)\r\n",
      "Collecting frozenlist>=1.1.1\r\n",
      "  Using cached frozenlist-1.3.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (158 kB)\r\n",
      "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /databricks/python3/lib/python3.9/site-packages (from aiohttp>=3.7.0->adlfs) (2.0.4)\r\n",
      "Collecting yarl<2.0,>=1.0\r\n",
      "  Using cached yarl-1.8.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (264 kB)\r\n",
      "Collecting async-timeout<5.0,>=4.0.0a3\r\n",
      "  Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB)\r\n",
      "Collecting aiosignal>=1.1.2\r\n",
      "  Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)\r\n",
      "Collecting multidict<7.0,>=4.5\r\n",
      "  Using cached multidict-6.0.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB)\r\n",
      "Requirement already satisfied: attrs>=17.3.0 in /databricks/python3/lib/python3.9/site-packages (from aiohttp>=3.7.0->adlfs) (21.4.0)\r\n",
      "Requirement already satisfied: six>=1.11.0 in /databricks/python3/lib/python3.9/site-packages (from azure-core<2.0.0,>=1.23.1->adlfs) (1.16.0)\r\n",
      "Requirement already satisfied: requests>=2.18.4 in /databricks/python3/lib/python3.9/site-packages (from azure-core<2.0.0,>=1.23.1->adlfs) (2.27.1)\r\n",
      "Requirement already satisfied: typing-extensions>=4.0.1 in /local_disk0/.ephemeral_nfs/envs/pythonEnv-ea947958-e611-4e90-942a-530ed26066bd/lib/python3.9/site-packages (from azure-core<2.0.0,>=1.23.1->adlfs) (4.5.0)\r\n",
      "Collecting adal>=0.4.2\r\n",
      "  Using cached adal-1.2.7-py2.py3-none-any.whl (55 kB)\r\n",
      "Requirement already satisfied: cffi in /databricks/python3/lib/python3.9/site-packages (from azure-datalake-store<0.1,>=0.0.46->adlfs) (1.15.0)\r\n",
      "Requirement already satisfied: python-dateutil<3,>=2.1.0 in /databricks/python3/lib/python3.9/site-packages (from adal>=0.4.2->azure-datalake-store<0.1,>=0.0.46->adlfs) (2.8.2)\r\n",
      "Requirement already satisfied: cryptography>=1.1.0 in /databricks/python3/lib/python3.9/site-packages (from adal>=0.4.2->azure-datalake-store<0.1,>=0.0.46->adlfs) (3.4.8)\r\n",
      "Requirement already satisfied: PyJWT<3,>=1.0.0 in /databricks/python3/lib/python3.9/site-packages (from adal>=0.4.2->azure-datalake-store<0.1,>=0.0.46->adlfs) (2.6.0)\r\n",
      "Requirement already satisfied: isodate>=0.6.1 in /databricks/python3/lib/python3.9/site-packages (from azure-storage-blob>=12.12.0->adlfs) (0.6.1)\r\n",
      "Requirement already satisfied: pycparser in /databricks/python3/lib/python3.9/site-packages (from cffi->azure-datalake-store<0.1,>=0.0.46->adlfs) (2.21)\r\n",
      "Requirement already satisfied: idna<4,>=2.5 in /databricks/python3/lib/python3.9/site-packages (from requests>=2.18.4->azure-core<2.0.0,>=1.23.1->adlfs) (3.3)\r\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /databricks/python3/lib/python3.9/site-packages (from requests>=2.18.4->azure-core<2.0.0,>=1.23.1->adlfs) (1.26.9)\r\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /databricks/python3/lib/python3.9/site-packages (from requests>=2.18.4->azure-core<2.0.0,>=1.23.1->adlfs) (2021.10.8)\r\n",
      "Collecting msal-extensions<2.0.0,>=0.3.0\r\n",
      "  Using cached msal_extensions-1.0.0-py2.py3-none-any.whl (19 kB)\r\n",
      "Collecting msal<2.0.0,>=1.12.0\r\n",
      "  Using cached msal-1.21.0-py2.py3-none-any.whl (89 kB)\r\n",
      "Collecting portalocker<3,>=1.0\r\n",
      "  Using cached portalocker-2.7.0-py2.py3-none-any.whl (15 kB)\r\n",
      "Installing collected packages: portalocker, multidict, msal, frozenlist, yarl, msal-extensions, async-timeout, aiosignal, adal, azure-storage-blob, azure-identity, azure-datalake-store, aiohttp, adlfs\r\n",
      "Successfully installed adal-1.2.7 adlfs-2023.1.0 aiohttp-3.8.4 aiosignal-1.3.1 async-timeout-4.0.2 azure-datalake-store-0.0.52 azure-identity-1.12.0 azure-storage-blob-12.15.0 frozenlist-1.3.3 msal-1.21.0 msal-extensions-1.0.0 multidict-6.0.4 portalocker-2.7.0 yarl-1.8.2\r\n",
      "\u001b[33mWARNING: You are using pip version 21.2.4; however, version 23.0.1 is available.\r\n",
      "You should consider upgrading via the '/local_disk0/.ephemeral_nfs/envs/pythonEnv-ea947958-e611-4e90-942a-530ed26066bd/bin/python -m pip install --upgrade pip' command.\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!pip install mlflow\n",
    "!pip install aepp\n",
    "!pip install adlfs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "18d805c9-3923-46b1-bd0e-e1979a4496e2",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "We'll be using the [aepp Python library](https://github.com/pitchmuc/aepp) here to interact with AEP APIs and create a schema and dataset suitable for adding our synthetic data further down the line. This library simply provides a programmatic interface around the REST APIs, but all these steps could be completed similarly using the raw APIs directly or even in the UI. For more information on the underlying APIs please see [the API reference guide](https://developer.adobe.com/experience-platform-apis/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "3ce72370-0266-460b-b00c-9a2d98d626f9",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Before any calls can take place, we need to configure the library and setup authentication credentials. For this you'll need the following piece of information. For information about how you can get these, please refer to the `Setup` section of the **Readme**:\n",
    "- Client ID\n",
    "- Client secret"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "ebba43c4-a1cc-4e49-8ba1-162a6b24ebba",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "The private key needs to be accessible on disk from this notebook. We recommend uploading it to DBFS and refering to it with the `/dbfs` prefix. This can be achieved by clicking in the Databricks notebook interface on `File > Upload data to DBFS` and then selecting the **private.key** file you downloaded during the setup, click `Next` and then you should have the option to copy the path. Make sure it starts with `/dbfs/FileStore` - for example if you uploaded your private key into `/FileStore/shared_upload/your_username` then the final path should be `/dbfs/FileStore/shared_uploads/your_username/private.key`. Make sure this value is properly set in the `private_key_path` variable of your configuration file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "5fd8f161-e8bd-4ecf-b00c-2b6ed8c10e2f",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "import aepp\n",
    "\n",
    "aepp.configure(\n",
    "  org_id=ims_org_id,\n",
    "  secret=client_secret,\n",
    "  scopes=scopes,\n",
    "  client_id=client_id,\n",
    "  environment=environment,\n",
    "  sandbox=sandbox_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "c09c6abb-b5fd-4fed-99cb-49e101f72827",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "# 1. Generating Propensity Scores Using the Trained Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "a6b6b184-f1e8-4174-b3ef-4eba7a520840",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "## 1.1 Reading the Featurized Data from the Data Landing Zone"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "663c6af0-a7f2-482b-ab3f-679d04f9cce9",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "In the second weekly assignment we had written our featurized data into the Data Landing Zone, and then on the third assignment we just read a sampled portion of it for training our model. At that point we want to score all of the profiles, so we need to read everything."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "d5877a0e-10f6-432f-aa5a-506e69a20390",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "The featurized data exported into the Data Landing Zone is under the format **cmle/egress/$DATASETID/exportTime=$EXPORTTIME**. We know the dataset ID which is in your config under `featurized_dataset_id` so we're just missing the export time so we know what to read. To get that we can simply list files in the DLZ and find what the value is. The first step is to retrieve the credentials for the DLZ related to the destination container:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "3f027217-03a3-4240-ad9b-0fac534ed49f",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "import aepp\n",
    " \n",
    "from aepp import connector\n",
    " \n",
    "connector = connector.AdobeRequest(\n",
    "  config_object=aepp.config.config_object,\n",
    "  header=aepp.config.header,\n",
    "  loggingEnabled=False,\n",
    "  logger=None,\n",
    ")\n",
    "\n",
    "endpoint = aepp.config.endpoints[\"global\"] + \"/data/foundation/connectors/landingzone/credentials\"\n",
    " \n",
    "dlz_credentials = connector.getData(endpoint=endpoint, params={\n",
    "  \"type\": \"dlz_destination\"\n",
    "})\n",
    "dlz_container = dlz_credentials[\"containerName\"]\n",
    "dlz_sas_token = dlz_credentials[\"SASToken\"]\n",
    "dlz_storage_account = dlz_credentials[\"storageAccountName\"]\n",
    "dlz_sas_uri = dlz_credentials[\"SASUri\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "7b3d28c1-feb9-4bc7-9f7a-75ac705538f7",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Now we use some Python libraries to authenticate and issue listing commands so we can get the paths and extract the time from it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "c0dd0f48-0d70-48d1-a298-030110c9a730",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using featurized data export time of 20230401140556\n"
     ]
    }
   ],
   "source": [
    "from adlfs import AzureBlobFileSystem\n",
    "from fsspec import AbstractFileSystem\n",
    "\n",
    "fs = AzureBlobFileSystem(\n",
    "  account_name=dlz_storage_account,\n",
    "  sas_token=dlz_sas_token\n",
    ")\n",
    "\n",
    "def get_export_time(fs: AbstractFileSystem, container_name: str, base_path: str, dataset_id: str):\n",
    "  featurized_data_base_path = f\"{container_name}/{base_path}/{dataset_id}\"\n",
    "  featurized_data_export_paths = fs.ls(featurized_data_base_path)\n",
    "  \n",
    "  if len(featurized_data_export_paths) == 0:\n",
    "    raise Exception(f\"Found no exports for featurized data from dataset ID {dataset_id} under path {featurized_data_base_path}\")\n",
    "  elif len(featurized_data_export_paths) > 1:\n",
    "    print(f\"Found {len(featurized_data_export_paths)} exports from dataset dataset ID {dataset_id} under path {featurized_data_base_path}, using most recent one\")\n",
    "  \n",
    "  featurized_data_export_path = featurized_data_export_paths[-1]\n",
    "  featurized_data_export_time = featurized_data_export_path.strip().split(\"/\")[-1].split(\"=\")[-1]\n",
    "  return featurized_data_export_time\n",
    "\n",
    "export_time = get_export_time(fs, dlz_container, export_path, featurized_dataset_id)\n",
    "print(f\"Using featurized data export time of {export_time}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "a8ef6afe-1d93-4a3e-947d-497ca04c6092",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "At that point we're ready to read this data. We're using Spark since it could be pretty large as we're not doing any sampling. Spark needs the following properties to be able to authenticate using SAS:\n",
    "- `fs.azure.account.auth.type.$ACCOUNT.dfs.core.windows.net` should be set to `SAS`.\n",
    "- `fs.azure.sas.token.provider.type.$ACCOUNT.dfs.core.windows.net` should be set to `org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider`.\n",
    "- `fs.azure.sas.fixed.token.$ACCOUNT.dfs.core.windows.net` should be set to the SAS token retrieved earlier.\n",
    "\n",
    "Let's put that in practice and create a Spark dataframe containing the entire featurized data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "a027af84-b920-4f32-a8da-438cb46fc1ba",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- userId: string (nullable = true)\n",
      " |-- eventType: string (nullable = true)\n",
      " |-- timestamp: timestamp (nullable = true)\n",
      " |-- subscriptionOccurred: long (nullable = true)\n",
      " |-- emailsReceived: long (nullable = true)\n",
      " |-- emailsOpened: long (nullable = true)\n",
      " |-- emailsClicked: long (nullable = true)\n",
      " |-- productsViewed: long (nullable = true)\n",
      " |-- propositionInteracts: long (nullable = true)\n",
      " |-- propositionDismissed: long (nullable = true)\n",
      " |-- webLinkClicks: long (nullable = true)\n",
      " |-- minutes_since_emailSent: integer (nullable = true)\n",
      " |-- minutes_since_emailOpened: integer (nullable = true)\n",
      " |-- minutes_since_emailClick: integer (nullable = true)\n",
      " |-- minutes_since_productView: integer (nullable = true)\n",
      " |-- minutes_since_propositionInteract: integer (nullable = true)\n",
      " |-- minutes_since_propositionDismiss: integer (nullable = true)\n",
      " |-- minutes_since_linkClick: integer (nullable = true)\n",
      " |-- random_row_number_for_user: integer (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "spark.conf.set(f\"fs.azure.account.auth.type.{dlz_storage_account}.dfs.core.windows.net\", \"SAS\")\n",
    "spark.conf.set(f\"fs.azure.sas.token.provider.type.{dlz_storage_account}.dfs.core.windows.net\", \"org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider\")\n",
    "spark.conf.set(f\"fs.azure.sas.fixed.token.{dlz_storage_account}.dfs.core.windows.net\", dlz_sas_token)\n",
    "\n",
    "protocol = \"abfss\"\n",
    "input_path = f\"{protocol}://{dlz_container}@{dlz_storage_account}.dfs.core.windows.net/{export_path}/{featurized_dataset_id}/exportTime={export_time}/\"\n",
    "\n",
    "df = spark.read.parquet(input_path)\n",
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "3a8f66b0-eb3e-44f0-b1e7-62084d98408a",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "We can verify it matches what we had written out in the second weekly assignment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "bbe92849-00f0-43c4-a8a5-5a365961365e",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out[9]: 89921"
     ]
    }
   ],
   "source": [
    "df.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "d66b787c-5184-422d-9aab-1d3fddc54cbb",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "And also do a sanity check on the data to make sure it looks good:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "18799e70-9596-47ad-9399-aff7d36edd02",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+--------------------+--------------------+--------------+------------+-------------+--------------+--------------------+--------------------+-------------+-----------------------+-------------------------+------------------------+-------------------------+---------------------------------+--------------------------------+-----------------------+--------------------------+\n",
      "|              userId|           eventType|           timestamp|subscriptionOccurred|emailsReceived|emailsOpened|emailsClicked|productsViewed|propositionInteracts|propositionDismissed|webLinkClicks|minutes_since_emailSent|minutes_since_emailOpened|minutes_since_emailClick|minutes_since_productView|minutes_since_propositionInteract|minutes_since_propositionDismiss|minutes_since_linkClick|random_row_number_for_user|\n",
      "+--------------------+--------------------+--------------------+--------------------+--------------+------------+-------------+--------------+--------------------+--------------------+-------------+-----------------------+-------------------------+------------------------+-------------------------+---------------------------------+--------------------------------+-----------------------+--------------------------+\n",
      "|11016499788150990...|web.webpagedetail...|2023-03-13 12:35:...|                   0|             3|           2|            1|             0|                   0|                   0|            0|                   1966|                        9|                       2|                     null|                             null|                            null|                   null|                         1|\n",
      "|87713101502605469...|directMarketing.e...|2023-03-13 12:35:...|                   0|             7|           2|            0|             1|                   0|                   0|            0|                      0|                    45587|                    null|                    61427|                             null|                            null|                   null|                         1|\n",
      "|60978314658731272...|web.webpagedetail...|2023-03-13 12:36:...|                   0|             3|           2|            0|             0|                   0|                   0|            0|                  17079|                    27335|                    null|                     null|                             null|                            null|                   null|                         1|\n",
      "|17558197385005669...|directMarketing.e...|2023-03-13 12:40:...|                   0|             7|           0|            0|             1|                   0|                   0|            0|                      0|                     null|                    null|                    48310|                             null|                            null|                   null|                         1|\n",
      "|01260911688018982...|advertising.impre...|2023-03-13 12:41:...|                   0|             6|           0|            0|             0|                   0|                   0|            0|                  18543|                     null|                    null|                     null|                             null|                            null|                   null|                         1|\n",
      "|73960490737088770...|web.webpagedetail...|2023-03-13 12:43:...|                   0|             3|           0|            0|             0|                   0|                   0|            0|                  65060|                     null|                    null|                     null|                             null|                            null|                   null|                         1|\n",
      "|07155275535911112...|directMarketing.e...|2023-03-13 12:44:...|                   0|             3|           2|            0|             0|                   0|                   0|            0|                    494|                        0|                    null|                     null|                             null|                            null|                   null|                         1|\n",
      "|24074127159040175...|directMarketing.e...|2023-03-13 12:44:...|                   0|             5|           1|            1|             0|                   0|                   0|            0|                      0|                    87646|                   87596|                     null|                             null|                            null|                   null|                         1|\n",
      "|19274473770753176...|directMarketing.e...|2023-03-13 12:46:...|                   0|             6|           1|            1|             0|                   0|                   0|            0|                      0|                    76684|                   76659|                     null|                             null|                            null|                   null|                         1|\n",
      "|59304643265641088...|directMarketing.e...|2023-03-13 12:46:...|                   0|             4|           0|            0|             0|                   0|                   0|            0|                      0|                     null|                    null|                     null|                             null|                            null|                   null|                         1|\n",
      "|56098422002231905...|web.webinteractio...|2023-03-13 12:47:...|                   0|             5|           0|            0|             0|                   0|                   0|            1|                   7047|                     null|                    null|                     null|                             null|                            null|                      0|                         1|\n",
      "|39458942610781556...|web.webpagedetail...|2023-03-13 12:47:...|                   0|             5|           1|            0|             0|                   0|                   0|            0|                     99|                    28508|                    null|                     null|                             null|                            null|                   null|                         1|\n",
      "|68651848422281677...|advertising.impre...|2023-03-13 12:48:...|                   0|             1|           0|            0|             0|                   0|                   0|            0|                  47567|                     null|                    null|                     null|                             null|                            null|                   null|                         1|\n",
      "|12884474211843442...|   web.formFilledOut|2023-03-13 12:49:...|                   1|             4|           1|            0|             0|                   0|                   0|            1|                   5273|                    62123|                    null|                     null|                             null|                            null|                      1|                         9|\n",
      "|24863386561915550...|advertising.impre...|2023-03-13 12:50:...|                   0|             5|           1|            0|             0|                   0|                   0|            3|                  12293|                    22161|                    null|                     null|                             null|                            null|                    217|                         1|\n",
      "|19898711844809940...|directMarketing.e...|2023-03-13 12:53:...|                   0|             2|           2|            0|             0|                   0|                   0|            0|                   2532|                        0|                    null|                     null|                             null|                            null|                   null|                         1|\n",
      "|39423033032148946...|directMarketing.e...|2023-03-13 12:55:...|                   0|             3|           1|            0|             0|                   0|                   0|            0|                      0|                      139|                    null|                     null|                             null|                            null|                   null|                         1|\n",
      "|78840901414561282...|directMarketing.e...|2023-03-13 12:56:...|                   0|             3|           1|            0|             1|                   0|                   0|            0|                      0|                    41200|                    null|                    55405|                             null|                            null|                   null|                         1|\n",
      "|27226125779868933...|directMarketing.e...|2023-03-13 12:56:...|                   0|             4|           0|            0|             0|                   0|                   0|            0|                      0|                     null|                    null|                     null|                             null|                            null|                   null|                         1|\n",
      "|44153509104259612...|directMarketing.e...|2023-03-13 12:56:...|                   0|             1|           0|            0|             1|                   0|                   0|            0|                      0|                     null|                    null|                    43203|                             null|                            null|                   null|                         1|\n",
      "+--------------------+--------------------+--------------------+--------------------+--------------+------------+-------------+--------------+--------------------+--------------------+-------------+-----------------------+-------------------------+------------------------+-------------------------+---------------------------------+--------------------------------+-----------------------+--------------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "6f7e844c-d044-4b51-a3db-7c65065288ee",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+--------------------+--------------------+--------------+------------+-------------+--------------+--------------------+--------------------+-------------+-----------------------+-------------------------+------------------------+-------------------------+---------------------------------+--------------------------------+-----------------------+--------------------------+\n",
      "|              userId|           eventType|           timestamp|subscriptionOccurred|emailsReceived|emailsOpened|emailsClicked|productsViewed|propositionInteracts|propositionDismissed|webLinkClicks|minutes_since_emailSent|minutes_since_emailOpened|minutes_since_emailClick|minutes_since_productView|minutes_since_propositionInteract|minutes_since_propositionDismiss|minutes_since_linkClick|random_row_number_for_user|\n",
      "+--------------------+--------------------+--------------------+--------------------+--------------+------------+-------------+--------------+--------------------+--------------------+-------------+-----------------------+-------------------------+------------------------+-------------------------+---------------------------------+--------------------------------+-----------------------+--------------------------+\n",
      "|11016499788150990...|web.webpagedetail...|2023-03-13 12:35:...|                   0|             3|           2|            1|             0|                   0|                   0|            0|                   1966|                        9|                       2|                        0|                                0|                               0|                      0|                         1|\n",
      "|87713101502605469...|directMarketing.e...|2023-03-13 12:35:...|                   0|             7|           2|            0|             1|                   0|                   0|            0|                      0|                    45587|                       0|                    61427|                                0|                               0|                      0|                         1|\n",
      "|60978314658731272...|web.webpagedetail...|2023-03-13 12:36:...|                   0|             3|           2|            0|             0|                   0|                   0|            0|                  17079|                    27335|                       0|                        0|                                0|                               0|                      0|                         1|\n",
      "|17558197385005669...|directMarketing.e...|2023-03-13 12:40:...|                   0|             7|           0|            0|             1|                   0|                   0|            0|                      0|                        0|                       0|                    48310|                                0|                               0|                      0|                         1|\n",
      "|01260911688018982...|advertising.impre...|2023-03-13 12:41:...|                   0|             6|           0|            0|             0|                   0|                   0|            0|                  18543|                        0|                       0|                        0|                                0|                               0|                      0|                         1|\n",
      "|73960490737088770...|web.webpagedetail...|2023-03-13 12:43:...|                   0|             3|           0|            0|             0|                   0|                   0|            0|                  65060|                        0|                       0|                        0|                                0|                               0|                      0|                         1|\n",
      "|07155275535911112...|directMarketing.e...|2023-03-13 12:44:...|                   0|             3|           2|            0|             0|                   0|                   0|            0|                    494|                        0|                       0|                        0|                                0|                               0|                      0|                         1|\n",
      "|24074127159040175...|directMarketing.e...|2023-03-13 12:44:...|                   0|             5|           1|            1|             0|                   0|                   0|            0|                      0|                    87646|                   87596|                        0|                                0|                               0|                      0|                         1|\n",
      "|19274473770753176...|directMarketing.e...|2023-03-13 12:46:...|                   0|             6|           1|            1|             0|                   0|                   0|            0|                      0|                    76684|                   76659|                        0|                                0|                               0|                      0|                         1|\n",
      "|59304643265641088...|directMarketing.e...|2023-03-13 12:46:...|                   0|             4|           0|            0|             0|                   0|                   0|            0|                      0|                        0|                       0|                        0|                                0|                               0|                      0|                         1|\n",
      "|56098422002231905...|web.webinteractio...|2023-03-13 12:47:...|                   0|             5|           0|            0|             0|                   0|                   0|            1|                   7047|                        0|                       0|                        0|                                0|                               0|                      0|                         1|\n",
      "|39458942610781556...|web.webpagedetail...|2023-03-13 12:47:...|                   0|             5|           1|            0|             0|                   0|                   0|            0|                     99|                    28508|                       0|                        0|                                0|                               0|                      0|                         1|\n",
      "|68651848422281677...|advertising.impre...|2023-03-13 12:48:...|                   0|             1|           0|            0|             0|                   0|                   0|            0|                  47567|                        0|                       0|                        0|                                0|                               0|                      0|                         1|\n",
      "|12884474211843442...|   web.formFilledOut|2023-03-13 12:49:...|                   1|             4|           1|            0|             0|                   0|                   0|            1|                   5273|                    62123|                       0|                        0|                                0|                               0|                      1|                         9|\n",
      "|24863386561915550...|advertising.impre...|2023-03-13 12:50:...|                   0|             5|           1|            0|             0|                   0|                   0|            3|                  12293|                    22161|                       0|                        0|                                0|                               0|                    217|                         1|\n",
      "|19898711844809940...|directMarketing.e...|2023-03-13 12:53:...|                   0|             2|           2|            0|             0|                   0|                   0|            0|                   2532|                        0|                       0|                        0|                                0|                               0|                      0|                         1|\n",
      "|39423033032148946...|directMarketing.e...|2023-03-13 12:55:...|                   0|             3|           1|            0|             0|                   0|                   0|            0|                      0|                      139|                       0|                        0|                                0|                               0|                      0|                         1|\n",
      "|78840901414561282...|directMarketing.e...|2023-03-13 12:56:...|                   0|             3|           1|            0|             1|                   0|                   0|            0|                      0|                    41200|                       0|                    55405|                                0|                               0|                      0|                         1|\n",
      "|27226125779868933...|directMarketing.e...|2023-03-13 12:56:...|                   0|             4|           0|            0|             0|                   0|                   0|            0|                      0|                        0|                       0|                        0|                                0|                               0|                      0|                         1|\n",
      "|44153509104259612...|directMarketing.e...|2023-03-13 12:56:...|                   0|             1|           0|            0|             1|                   0|                   0|            0|                      0|                        0|                       0|                    43203|                                0|                               0|                      0|                         1|\n",
      "+--------------------+--------------------+--------------------+--------------------+--------------+------------+-------------+--------------+--------------------+--------------------+-------------+-----------------------+-------------------------+------------------------+-------------------------+---------------------------------+--------------------------------+-----------------------+--------------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df = df.fillna(0)\n",
    "df.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "6a5c9a39-606a-4e56-b1be-f5c535d4bbfb",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "## 1.2 Scoring the Profiles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "6f70c947-de11-4b2b-863e-996f976af79c",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "For scoring we need 2 things:\n",
    "1. The **data** to score.\n",
    "2. The **trained model** that will be used to do the scoring.\n",
    "\n",
    "We just created a dataframe containing the first one, and in the previous weekly assignment we created a production model that can operate on this data, so let's fetch this model from our model hub and turn it into a Spark UDF so it can interact with our data easily:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "7c0c8834-8095-4e72-90bb-deee87b45830",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023/04/05 17:28:22 WARNING mlflow.pyfunc: Calling `spark_udf()` with `env_manager=\"local\"` does not recreate the same environment that was used during training, which may lead to errors or inaccurate predictions. We recommend specifying `env_manager=\"conda\"`, which automatically recreates the environment that was used to train the model and performs inference in the recreated environment.\n",
      "2023/04/05 17:28:23 INFO mlflow.models.flavor_backend_registry: Selected backend for flavor 'python_function'\n"
     ]
    }
   ],
   "source": [
    "import mlflow.pyfunc\n",
    "\n",
    "model_udf = mlflow.pyfunc.spark_udf(spark, f\"models:/{model_name}/production\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "44e48326-4d4f-43dc-8ab5-d77a5bbaff36",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Now we're ready to apply our trained model on top of the entire dataset. For that, we need to give this UDF its inputs - which in our case are all the columns that the model needs to operate on. We can get that easily as the Spark dataframe contains metadata about its columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "727f3932-65ea-4a49-9d97-48183d4a0f14",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- userId: string (nullable = true)\n",
      " |-- eventType: string (nullable = true)\n",
      " |-- timestamp: timestamp (nullable = true)\n",
      " |-- subscriptionOccurred: long (nullable = true)\n",
      " |-- emailsReceived: long (nullable = true)\n",
      " |-- emailsOpened: long (nullable = true)\n",
      " |-- emailsClicked: long (nullable = true)\n",
      " |-- productsViewed: long (nullable = true)\n",
      " |-- propositionInteracts: long (nullable = true)\n",
      " |-- propositionDismissed: long (nullable = true)\n",
      " |-- webLinkClicks: long (nullable = true)\n",
      " |-- minutes_since_emailSent: integer (nullable = true)\n",
      " |-- minutes_since_emailOpened: integer (nullable = true)\n",
      " |-- minutes_since_emailClick: integer (nullable = true)\n",
      " |-- minutes_since_productView: integer (nullable = true)\n",
      " |-- minutes_since_propositionInteract: integer (nullable = true)\n",
      " |-- minutes_since_propositionDismiss: integer (nullable = true)\n",
      " |-- minutes_since_linkClick: integer (nullable = true)\n",
      " |-- random_row_number_for_user: integer (nullable = true)\n",
      " |-- prediction: double (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql.functions import struct\n",
    " \n",
    "# Apply the model to the new data\n",
    "udf_inputs = struct(*(df.columns))\n",
    " \n",
    "df_scored = df.withColumn(\n",
    "  \"prediction\",\n",
    "  model_udf(udf_inputs)\n",
    ")\n",
    "df_scored.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "73a1897c-37db-4090-ab4a-373d0a54c671",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "If we look at the data we should see a new column called `prediction` which corresponds to the score generated by the model for this particular profile based on all the features computed earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "3687ef6e-45db-4dcc-b6c4-4e58573e097a",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+--------------------+--------------------+--------------+------------+-------------+--------------+--------------------+--------------------+-------------+-----------------------+-------------------------+------------------------+-------------------------+---------------------------------+--------------------------------+-----------------------+--------------------------+--------------------+\n",
      "|              userId|           eventType|           timestamp|subscriptionOccurred|emailsReceived|emailsOpened|emailsClicked|productsViewed|propositionInteracts|propositionDismissed|webLinkClicks|minutes_since_emailSent|minutes_since_emailOpened|minutes_since_emailClick|minutes_since_productView|minutes_since_propositionInteract|minutes_since_propositionDismiss|minutes_since_linkClick|random_row_number_for_user|          prediction|\n",
      "+--------------------+--------------------+--------------------+--------------------+--------------+------------+-------------+--------------+--------------------+--------------------+-------------+-----------------------+-------------------------+------------------------+-------------------------+---------------------------------+--------------------------------+-----------------------+--------------------------+--------------------+\n",
      "|11016499788150990...|web.webpagedetail...|2023-03-13 12:35:...|                   0|             3|           2|            1|             0|                   0|                   0|            0|                   1966|                        9|                       2|                        0|                                0|                               0|                      0|                         1|  0.7488606278736331|\n",
      "|87713101502605469...|directMarketing.e...|2023-03-13 12:35:...|                   0|             7|           2|            0|             1|                   0|                   0|            0|                      0|                    45587|                       0|                    61427|                                0|                               0|                      0|                         1| 0.05426953828223503|\n",
      "|60978314658731272...|web.webpagedetail...|2023-03-13 12:36:...|                   0|             3|           2|            0|             0|                   0|                   0|            0|                  17079|                    27335|                       0|                        0|                                0|                               0|                      0|                         1|  0.2675003625191588|\n",
      "|17558197385005669...|directMarketing.e...|2023-03-13 12:40:...|                   0|             7|           0|            0|             1|                   0|                   0|            0|                      0|                        0|                       0|                    48310|                                0|                               0|                      0|                         1|0.027538360646405295|\n",
      "|01260911688018982...|advertising.impre...|2023-03-13 12:41:...|                   0|             6|           0|            0|             0|                   0|                   0|            0|                  18543|                        0|                       0|                        0|                                0|                               0|                      0|                         1| 0.15652258952798903|\n",
      "|73960490737088770...|web.webpagedetail...|2023-03-13 12:43:...|                   0|             3|           0|            0|             0|                   0|                   0|            0|                  65060|                        0|                       0|                        0|                                0|                               0|                      0|                         1| 0.08045210856584475|\n",
      "|07155275535911112...|directMarketing.e...|2023-03-13 12:44:...|                   0|             3|           2|            0|             0|                   0|                   0|            0|                    494|                        0|                       0|                        0|                                0|                               0|                      0|                         1|6.521356121291099E-4|\n",
      "|24074127159040175...|directMarketing.e...|2023-03-13 12:44:...|                   0|             5|           1|            1|             0|                   0|                   0|            0|                      0|                    87646|                   87596|                        0|                                0|                               0|                      0|                         1|                 0.0|\n",
      "|19274473770753176...|directMarketing.e...|2023-03-13 12:46:...|                   0|             6|           1|            1|             0|                   0|                   0|            0|                      0|                    76684|                   76659|                        0|                                0|                               0|                      0|                         1|                 0.0|\n",
      "|59304643265641088...|directMarketing.e...|2023-03-13 12:46:...|                   0|             4|           0|            0|             0|                   0|                   0|            0|                      0|                        0|                       0|                        0|                                0|                               0|                      0|                         1|0.002396242128429...|\n",
      "|56098422002231905...|web.webinteractio...|2023-03-13 12:47:...|                   0|             5|           0|            0|             0|                   0|                   0|            1|                   7047|                        0|                       0|                        0|                                0|                               0|                      0|                         1| 0.18235710702975552|\n",
      "|39458942610781556...|web.webpagedetail...|2023-03-13 12:47:...|                   0|             5|           1|            0|             0|                   0|                   0|            0|                     99|                    28508|                       0|                        0|                                0|                               0|                      0|                         1|   0.176432800926713|\n",
      "|68651848422281677...|advertising.impre...|2023-03-13 12:48:...|                   0|             1|           0|            0|             0|                   0|                   0|            0|                  47567|                        0|                       0|                        0|                                0|                               0|                      0|                         1| 0.05165010331047478|\n",
      "|12884474211843442...|   web.formFilledOut|2023-03-13 12:49:...|                   1|             4|           1|            0|             0|                   0|                   0|            1|                   5273|                    62123|                       0|                        0|                                0|                               0|                      1|                         9|   0.601910577952276|\n",
      "|24863386561915550...|advertising.impre...|2023-03-13 12:50:...|                   0|             5|           1|            0|             0|                   0|                   0|            3|                  12293|                    22161|                       0|                        0|                                0|                               0|                    217|                         1|  0.4759538474862529|\n",
      "|19898711844809940...|directMarketing.e...|2023-03-13 12:53:...|                   0|             2|           2|            0|             0|                   0|                   0|            0|                   2532|                        0|                       0|                        0|                                0|                               0|                      0|                         1|0.004020075366077656|\n",
      "|39423033032148946...|directMarketing.e...|2023-03-13 12:55:...|                   0|             3|           1|            0|             0|                   0|                   0|            0|                      0|                      139|                       0|                        0|                                0|                               0|                      0|                         1|0.001505529710294848|\n",
      "|78840901414561282...|directMarketing.e...|2023-03-13 12:56:...|                   0|             3|           1|            0|             1|                   0|                   0|            0|                      0|                    41200|                       0|                    55405|                                0|                               0|                      0|                         1| 0.06383670726116872|\n",
      "|27226125779868933...|directMarketing.e...|2023-03-13 12:56:...|                   0|             4|           0|            0|             0|                   0|                   0|            0|                      0|                        0|                       0|                        0|                                0|                               0|                      0|                         1|0.002396242128429...|\n",
      "|44153509104259612...|directMarketing.e...|2023-03-13 12:56:...|                   0|             1|           0|            0|             1|                   0|                   0|            0|                      0|                        0|                       0|                    43203|                                0|                               0|                      0|                         1| 0.03231089580938638|\n",
      "+--------------------+--------------------+--------------------+--------------------+--------------+------------+-------------+--------------+--------------------+--------------------+-------------+-----------------------+-------------------------+------------------------+-------------------------+---------------------------------+--------------------------------+-----------------------+--------------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df_scored.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "2f29f1f7-e74f-4d53-9d0f-3a40d94610f7",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "When you think about bringing the scored profiles back into the Adobe Experience Platform, we don't need to bring back all the features. In fact, we only really need 2 columns:\n",
    "- The user ID, so we know in the Unified Profile to which profile this row corresponds.\n",
    "- The score for this user ID."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "0c52fd86-9e48-4a4d-b9db-33fb75ccc213",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- userId: string (nullable = true)\n",
      " |-- prediction: double (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql.functions import udf, col, lit, create_map, array, struct, current_timestamp\n",
    "\n",
    "from itertools import chain\n",
    "\n",
    "df_to_ingest = df_scored.select(\n",
    "  \"userId\",\n",
    "  \"prediction\"\n",
    ").cache()\n",
    "df_to_ingest.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "c0fb950e-1960-4e60-9d60-ade5681050a9",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out[20]: 89921"
     ]
    }
   ],
   "source": [
    "df_to_ingest.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "f84f9bfb-c9f4-45b7-a186-d9b2ad71fadb",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+\n",
      "|              userId|          prediction|\n",
      "+--------------------+--------------------+\n",
      "|11016499788150990...|  0.7488606278736331|\n",
      "|87713101502605469...| 0.05426953828223503|\n",
      "|60978314658731272...|  0.2675003625191588|\n",
      "|17558197385005669...|0.027538360646405295|\n",
      "|01260911688018982...| 0.15652258952798903|\n",
      "|73960490737088770...| 0.08045210856584475|\n",
      "|07155275535911112...|6.521356121291099E-4|\n",
      "|24074127159040175...|                 0.0|\n",
      "|19274473770753176...|                 0.0|\n",
      "|59304643265641088...|0.002396242128429...|\n",
      "|56098422002231905...| 0.18235710702975552|\n",
      "|39458942610781556...|   0.176432800926713|\n",
      "|68651848422281677...| 0.05165010331047478|\n",
      "|12884474211843442...|   0.601910577952276|\n",
      "|24863386561915550...|  0.4759538474862529|\n",
      "|19898711844809940...|0.004020075366077656|\n",
      "|39423033032148946...|0.001505529710294848|\n",
      "|78840901414561282...| 0.06383670726116872|\n",
      "|27226125779868933...|0.002396242128429...|\n",
      "|44153509104259612...| 0.03231089580938638|\n",
      "+--------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df_to_ingest.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "c3a3a7f9-35fa-4135-8dcb-7c7c36b9ea4c",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "At that point we have the scored profiles and exactly what we need to bring back into Adobe Experience Platform. But we're not quite ready to write the results yet, there's a bit of setup that needs to happen first:\n",
    "- We need to create and configure a destination **dataset** in Adobe Experience Platform where our data will end up.\n",
    "- We need to setup a **data flow** that will be able to take this data, convert it into an XDM format, and deliver it to this dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "e9ab9303-0fa3-412f-9d21-480e73e1d6b8",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "# 2. Bringing the Scores back into Unified Profile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "ea5c13b9-f14b-488b-bd4f-d8b2450ff81c",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "## 2.1 Create ingestion schema and dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "621b31e9-44a3-4586-b3c7-68b0d28fd38d",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "The first step is to define where this propensity data we are creating as the output of our model should end up in the Unified Profile. We need to create a few entities for that:\n",
    "- A **fieldgroup** that will define the XDM for where propensity scores should be stored.\n",
    "- A **schema** based on that field group that will tie it back to the concept of profile.\n",
    "- A **dataset** based on that schema that will hold the data.\n",
    "\n",
    "As for the structure itself it's pretty simple, we just need 2 fields:\n",
    "- The **propensity** itself as a decimal number.\n",
    "- The **user ID** to which this propensity score relates.\n",
    "\n",
    "Let's put that in practice and create the field group. Note that because we are creating custom fields here, they need to be nested under the tenant ID corresponding to your organization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "4965fb32-b9c1-492f-a31d-d45aedbf89d4",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out[22]: 'aamds'"
     ]
    }
   ],
   "source": [
    "from aepp import schema\n",
    "\n",
    "schema_conn = schema.Schema()\n",
    "\n",
    "tenant_id = schema_conn.getTenantId()\n",
    "tenant_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "97f582f8-4d6b-41e8-b0e2-4cd22f28a121",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out[23]: 'https://ns.adobe.com/aamds/mixins/a1ddf130a5a91546734d651fcc1923513e4849591c1bc63d'"
     ]
    }
   ],
   "source": [
    "fieldgroup_res = schema_conn.createFieldGroup({\n",
    "  \t\"type\": \"object\",\n",
    "\t\"title\": f\"[CMLE][Week4] Fieldgroup for user propensity (created by {username})\",\n",
    "\t\"description\": \"This mixin is used to define a propensity score that can be assigned to a given profile.\",\n",
    "\t\"allOf\": [{\n",
    "\t\t\"$ref\": \"#/definitions/customFields\"\n",
    "\t}],\n",
    "\t\"meta:containerId\": \"tenant\",\n",
    "\t\"meta:resourceType\": \"mixins\",\n",
    "\t\"meta:xdmType\": \"object\",\n",
    "\t\"definitions\": {\n",
    "      \"customFields\": {\n",
    "        \"type\": \"object\",\n",
    "        \"properties\": {\n",
    "          f\"_{tenant_id}\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "              \"propensity\": {\n",
    "                \"title\": \"Propensity\",\n",
    "                \"description\": \"This refers to the propensity of a user towards an outcome.\",\n",
    "                \"type\": \"number\"\n",
    "              },\n",
    "              \"userid\": {\n",
    "                \"title\": \"User ID\",\n",
    "                \"description\": \"This refers to the user having a propensity towards an outcome.\",\n",
    "                \"type\": \"string\"\n",
    "              }\n",
    "            }\n",
    "          }\n",
    "        }\n",
    "      }\n",
    "\t},\n",
    "\t\"meta:intendedToExtend\": [\"https://ns.adobe.com/xdm/context/profile\"]\n",
    "})\n",
    "\n",
    "fieldgroup_id = fieldgroup_res[\"$id\"]\n",
    "fieldgroup_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "900d6fb7-12c4-4405-832c-3c1009161bc9",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "From this field group ID we can add it to a brand new schema that will be marked for profiles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "abcc2406-0b94-4f99-8fae-ca84ec136dc1",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Schema ID: https://ns.adobe.com/aamds/schemas/fdc01e8953e31a43d6b1e949a7870fe7e0a37131400d567c\n",
      "Schema Alt ID: _aamds.schemas.fdc01e8953e31a43d6b1e949a7870fe7e0a37131400d567c\n"
     ]
    }
   ],
   "source": [
    "schema_res = schema_conn.createProfileSchema(\n",
    "  name=f\"[CMLE][Week4] Schema for user propensity ingestion (created by {username})\",\n",
    "  mixinIds=[\n",
    "    fieldgroup_id\n",
    "  ],\n",
    "  description=\"Schema generated by CMLE for user propensity score ingestion\",\n",
    ")\n",
    "\n",
    "schema_id = schema_res[\"$id\"]\n",
    "schema_alt_id = schema_res[\"meta:altId\"]\n",
    "\n",
    "print(f\"Schema ID: {schema_id}\")\n",
    "print(f\"Schema Alt ID: {schema_alt_id}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "f5a0c413-14c2-474f-a194-eee7221a51a8",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Because we eventually intend for these scores to end up in the Unified Profile, we need to specify which field of the schema corresponds to an identity so it can resolve the corresponding profile. In our case, the `userid` field is an ECID and we mark it as such."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "4e90e5ba-b5c0-41ac-af73-bbc9feea3794",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out[25]: {'@id': '8f6039aa5369bf18d9f0bb357e14c8d2d6c16741475b81d9',\n",
      " '@type': 'xdm:descriptorIdentity',\n",
      " 'xdm:sourceSchema': 'https://ns.adobe.com/aamds/schemas/fdc01e8953e31a43d6b1e949a7870fe7e0a37131400d567c',\n",
      " 'xdm:sourceVersion': 1,\n",
      " 'xdm:sourceProperty': '/_aamds/userid',\n",
      " 'imsOrg': '36045A905CA640040A495E73@AdobeOrg',\n",
      " 'version': '1',\n",
      " 'xdm:namespace': 'ECID',\n",
      " 'xdm:property': 'xdm:id',\n",
      " 'xdm:isPrimary': True,\n",
      " 'meta:containerId': 'f0388270-ec6b-4f0a-b882-70ec6b8f0a30',\n",
      " 'meta:sandboxId': 'f0388270-ec6b-4f0a-b882-70ec6b8f0a30',\n",
      " 'meta:sandboxType': 'development'}"
     ]
    }
   ],
   "source": [
    "identity_type = \"ECID\"\n",
    "descriptor_res = schema_conn.createDescriptor(\n",
    "  descriptorObj = {\n",
    "    \"@type\": \"xdm:descriptorIdentity\",\n",
    "    \"xdm:sourceSchema\": schema_id,\n",
    "    \"xdm:sourceVersion\": 1,\n",
    "    \"xdm:sourceProperty\": f\"/_{tenant_id}/userid\",\n",
    "    \"xdm:namespace\": identity_type,\n",
    "    \"xdm:property\": \"xdm:id\",\n",
    "    \"xdm:isPrimary\": True\n",
    "  }\n",
    ")\n",
    "descriptor_res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "d860b8cb-9f8b-429f-b673-582bb8d03a73",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "And of course that schema needs to be enabled for Unified Profile consumption, so it can be added to the profile union schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "15a029df-bba6-4c8a-b721-9ba50b115cd0",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out[26]: {'$id': 'https://ns.adobe.com/aamds/schemas/fdc01e8953e31a43d6b1e949a7870fe7e0a37131400d567c',\n",
      " 'meta:altId': '_aamds.schemas.fdc01e8953e31a43d6b1e949a7870fe7e0a37131400d567c',\n",
      " 'meta:resourceType': 'schemas',\n",
      " 'version': '1.1',\n",
      " 'title': '[CMLE][Week4] Schema for user propensity ingestion (created by cmenguy@adobe.com)',\n",
      " 'type': 'object',\n",
      " 'description': 'Schema generated by CMLE for user propensity score ingestion',\n",
      " 'allOf': [{'$ref': 'https://ns.adobe.com/xdm/context/profile',\n",
      "   'type': 'object',\n",
      "   'meta:xdmType': 'object'},\n",
      "  {'$ref': 'https://ns.adobe.com/aamds/mixins/a1ddf130a5a91546734d651fcc1923513e4849591c1bc63d',\n",
      "   'type': 'object',\n",
      "   'meta:xdmType': 'object'}],\n",
      " 'refs': ['https://ns.adobe.com/xdm/context/profile',\n",
      "  'https://ns.adobe.com/aamds/mixins/a1ddf130a5a91546734d651fcc1923513e4849591c1bc63d'],\n",
      " 'imsOrg': '36045A905CA640040A495E73@AdobeOrg',\n",
      " 'additionalInfo': {'numberOfIdentities': 1,\n",
      "  'numberOfRelationShips': 0,\n",
      "  'classTitle': 'XDM Individual Profile',\n",
      "  'behavior': 'Record',\n",
      "  'hasPrimaryIdentity': 'true',\n",
      "  'primaryIdentityNamespace': 'ECID',\n",
      "  'hasRelationShip': False,\n",
      "  'profileEnabled': True},\n",
      " 'meta:extensible': False,\n",
      " 'meta:abstract': False,\n",
      " 'meta:extends': ['https://ns.adobe.com/xdm/common/auditable',\n",
      "  'https://ns.adobe.com/xdm/data/record',\n",
      "  'https://ns.adobe.com/xdm/context/profile',\n",
      "  'https://ns.adobe.com/aamds/mixins/a1ddf130a5a91546734d651fcc1923513e4849591c1bc63d'],\n",
      " 'meta:xdmType': 'object',\n",
      " 'meta:registryMetadata': {'repo:createdDate': 1680715782797,\n",
      "  'repo:lastModifiedDate': 1680715790308,\n",
      "  'xdm:createdClientId': 'a123472de32743d597ae10556b72977f',\n",
      "  'xdm:lastModifiedClientId': 'a123472de32743d597ae10556b72977f',\n",
      "  'xdm:createdUserId': '5FC626B363D4274A0A495E0D@techacct.adobe.com',\n",
      "  'xdm:lastModifiedUserId': '5FC626B363D4274A0A495E0D@techacct.adobe.com',\n",
      "  'eTag': '477e23267417953931c535ab9ebbf616ee9ec39afee36ad8534bd1115d81ec8e',\n",
      "  'meta:globalLibVersion': '1.40.3'},\n",
      " 'meta:class': 'https://ns.adobe.com/xdm/context/profile',\n",
      " 'meta:containerId': 'f0388270-ec6b-4f0a-b882-70ec6b8f0a30',\n",
      " 'meta:sandboxId': 'f0388270-ec6b-4f0a-b882-70ec6b8f0a30',\n",
      " 'meta:sandboxType': 'development',\n",
      " 'meta:tenantNamespace': '_aamds',\n",
      " 'meta:immutableTags': ['union'],\n",
      " 'meta:descriptorStatus': {'result': []}}"
     ]
    }
   ],
   "source": [
    "enable_res = schema_conn.enableSchemaForRealTime(schema_alt_id)\n",
    "enable_res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "6374ad2a-7df6-4d11-be98-87ee8747a033",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "At that point we're ready to create the dataset that will hold our propensity scores. This dataset is based on our schema we just created and nothing more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "782f1db5-964c-4359-8308-4241c0199d54",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out[27]: '642db013d8fc0e1c07f46b12'"
     ]
    }
   ],
   "source": [
    "from aepp import catalog\n",
    "\n",
    "cat_conn = catalog.Catalog()\n",
    "\n",
    "ingestion_dataset_res = cat_conn.createDataSets(\n",
    "  name=f\"[CMLE][Week4] Dataset for user propensity ingestion (created by {username})\",\n",
    "  schemaId=schema_id\n",
    ")\n",
    "\n",
    "ingestion_dataset_id = ingestion_dataset_res[0].split(\"/\")[-1]\n",
    "ingestion_dataset_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "6c7157e8-794c-486a-9b24-ff75e167c643",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "And similarly that dataset needs to be enabled for Unified Profile consumption, so that any batch of data written to this dataset is automatically picked up and processed to insert into the individual profiles and create new fragments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "ed5d7223-fbe7-43af-9983-8caaed7c6fe6",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out[28]: ['@/dataSets/642db013d8fc0e1c07f46b12']"
     ]
    }
   ],
   "source": [
    "# TODO: this is currently failing due to invalid content type, need to fix in aepp, see https://github.com/pitchmuc/aepp/issues/15\n",
    "# for now just enable in the UI...\n",
    "cat_conn.enableDatasetProfile(ingestion_dataset_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "109165ae-6d68-454c-938e-0acfbb697f36",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "You should be able to see your dataset in the UI at the link below, and it should match the new schema created as shown in the following screenshot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "2ad9529e-cb3d-4782-8e1b-b996d7baf1a2",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset ID 642db013d8fc0e1c07f46b12 available under https://experience.adobe.com/#/@aamds/sname:cmle/platform/dataset/browse/642db013d8fc0e1c07f46b12\n"
     ]
    }
   ],
   "source": [
    "ingestion_dataset_link = get_ui_link(tenant_id, \"dataset/browse\", ingestion_dataset_id)\n",
    "print(f\"Dataset ID {ingestion_dataset_id} available under {ingestion_dataset_link}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "506adcc5-4f11-4a22-b574-1f2781140b86",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "![Dataset](../media/CMLE-Notebooks-Week4-ScoringDataset.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "2d2df930-a20a-4d28-93b5-cec402b560d3",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "## 2.2 Setup ingestion data flow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "43094e80-b29d-4e0c-b79e-3dd8abd729a4",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Now that all the dataset and schema setup is completed, we're ready to define our Data Flow. The Data Flow defines the contract between the source and destination dataset.\n",
    "\n",
    "For the purposes of this notebook we will be using the [Data Landing Zone (DLZ)](https://experienceleague.adobe.com/docs/experience-platform/sources/api-tutorials/create/cloud-storage/data-landing-zone.html?lang=en) as the source filesystem under which the scoring results will be written. Every Adobe Experience Platform has a DLZ already setup as an [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs) container. We'll be using that as a delivery mechanism for the featurized data, but this step can be customized to delivery this data to any cloud storage filesystem.\n",
    "\n",
    "To setup the delivery pipeline, we'll be using the [Flow Service for Destinations](https://developer.adobe.com/experience-platform-apis/references/destinations/) which will be responsible for picking up the featurized data and dump it into the DLZ. There's a few steps involved:\n",
    "- Creating a **source connection**.\n",
    "- Creating a **target connection**.\n",
    "- Creating a **transformation**.\n",
    "- Creating a **data flow**.\n",
    "\n",
    "Note that, although we already got DLZ credentials earlier in this notebook, there were for a different container where all the destination data is written (`dlz-destination`), but here we want to get credentials for a different container corresponding to your user drop zone (`dlz-user-container`).\n",
    "\n",
    "For that, again we use `aepp` to abstract all the APIs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "d0ca9582-141f-4a60-8c4f-db85777c3252",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dlz-user-container\n"
     ]
    }
   ],
   "source": [
    "from aepp import flowservice\n",
    "\n",
    "flow_conn = flowservice.FlowService()\n",
    "\n",
    "dlz_credentials = flow_conn.getLandingZoneCredential()\n",
    "dlz_container = dlz_credentials[\"containerName\"]\n",
    "dlz_sas_token = dlz_credentials[\"SASToken\"]\n",
    "dlz_storage_account = dlz_credentials[\"storageAccountName\"]\n",
    "dlz_sas_uri = dlz_credentials[\"SASUri\"]\n",
    "print(dlz_container)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "339f1b21-6518-428c-a8a4-d38768b4d348",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "The **source connection** is responsible for connecting to your cloud storage account (in our case here, the Data Landing Zone) so that the resulting Data Flow will know from where data needs to be picked up.\n",
    "\n",
    "For reference, here is a list of all the connection specs available for the most popular cloud storage accounts (these IDs are global across every single customer account and sandbox):\n",
    "\n",
    "| Cloud Storage Type    | Connection Spec ID                   | Connection Spec Name\n",
    "|-----------------------|--------------------------------------|----------------------\n",
    "| Amazon S3             | ecadc60c-7455-4d87-84dc-2a0e293d997b | amazon-s3\n",
    "| Azure Blob Storage    | d771e9c1-4f26-40dc-8617-ce58c4b53702 | google-adwords\n",
    "| Azure Data Lake       | b3ba5556-48be-44b7-8b85-ff2b69b46dc4 | adls-gen2\n",
    "| Data Landing Zone     | 26f526f2-58f4-4712-961d-e41bf1ccc0e8 | landing-zone\n",
    "| Google Cloud Storage  | 32e8f412-cdf7-464c-9885-78184cb113fd | google-cloud\n",
    "| SFTP                  | b7bf2577-4520-42c9-bae9-cad01560f7bc | sftp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "a3b934a9-6ea4-4512-acd2-1ef11f7019b0",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out[53]: '9acfd387-6abc-4fe2-b72a-00ec85c23107'"
     ]
    }
   ],
   "source": [
    "connection_spec_id = \"26f526f2-58f4-4712-961d-e41bf1ccc0e8\"\n",
    "source_res = flow_conn.createSourceConnection({\n",
    "  \"name\": \"[CMLE][Week4] Data Landing Zone source connection for propensity scores\",\n",
    "  \"data\": {\n",
    "      \"format\": \"delimited\"\n",
    "  },\n",
    "  \"params\": {\n",
    "    \"path\": f\"{dlz_container}/{import_path}\",\n",
    "    \"type\": \"folder\",\n",
    "    \"recursive\": True\n",
    "  },\n",
    "  \"connectionSpec\": {\n",
    "      \"id\": connection_spec_id,\n",
    "      \"version\": \"1.0\"\n",
    "  }\n",
    "})\n",
    "\n",
    "source_connection_id = source_res[\"id\"]\n",
    "source_connection_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "92b70e88-71b9-42da-8299-aae8837be628",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "The **target connection** is responsible for connecting to your Adobe Experience Platform dataset so that the resulting Data Flow will know where the data needs to be written. Because we already created our ingestion dataset in the previous section, we can simply tie it to that dataset ID and the corresponding schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "0f0f0532-beb9-46aa-9908-7fff63d88bfa",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out[54]: '7dfe5e37-58c4-41ac-b652-31edbcb46ecb'"
     ]
    }
   ],
   "source": [
    "target_res = flow_conn.createTargetConnectionDataLake(\n",
    "  name=\"[CMLE][Week4] User Propensity Target Connection\",\n",
    "  datasetId=ingestion_dataset_id,\n",
    "  schemaId=schema_id\n",
    ")\n",
    "\n",
    "target_connection_id = target_res[\"id\"]\n",
    "target_connection_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "6c7270e8-1ea9-4a07-bab4-91ff6c1b6212",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "We're still missing one step. If you look back to the previous cells, this is what we have as the schema of our scored dataframe:\n",
    "- `userId`\n",
    "- `prediction`\n",
    "\n",
    "And this is what we have as the schema of our ingestion dataset:\n",
    "- `_$TENANTID.userid`\n",
    "- `_$TENANTID.propensity`\n",
    "\n",
    "Although it may look obvious to us, we still need to let the platform know which fields maps to what. This can be achieved using the [Data Prep service](https://experienceleague.adobe.com/docs/experience-platform/data-prep/home.html) which allows you to specify a set of **transformations** to map one field to another. In our case the transformation is pretty simple, we just need to match the schemas without making any changes, but you can do a lot more extensive transformations using this service if needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "c03d77f0-078f-4f34-9dd7-b99a5f301601",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "from aepp import dataprep\n",
    "\n",
    "dataprep_conn = dataprep.DataPrep()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "0859f72c-dcc5-4933-92aa-a3ac21b5677b",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out[56]: '802b8d47ac864c45a4b42910ec301587'"
     ]
    }
   ],
   "source": [
    "mapping_res = dataprep_conn.createMappingSet(\n",
    "  schemaId=schema_id,\n",
    "  validate=True,\n",
    "  mappingList=[\n",
    "    {\n",
    "      \"sourceType\": \"ATTRIBUTE\",\n",
    "      \"source\": \"prediction\",\n",
    "      \"destination\": f\"_{tenant_id}.propensity\"\n",
    "    },\n",
    "    {\n",
    "      \"sourceType\": \"ATTRIBUTE\",\n",
    "      \"source\": \"userId\",\n",
    "      \"destination\": f\"_{tenant_id}.userid\"\n",
    "    }\n",
    "  ]\n",
    ")\n",
    "\n",
    "mapping_id = mapping_res[\"id\"]\n",
    "mapping_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "ea168844-d4f3-46e3-befd-781f65c4b574",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "At that point we have everything we need to create a **Data Flow**. A data flow is the \"recipe\" that describes where the data comes from and where it should end up. We can also specify how often checks happen to find new data, but it cannot be lower than 15 minutes currently for platform stability reasons. A data flow is tied to a flow spec ID which contains the instructions for transfering data in an optimized way between a source and destination.\n",
    "\n",
    "For reference, here is a list of all the flow specs available for the most popular cloud storage accounts (these IDs are global across every single customer account and sandbox):\n",
    "\n",
    "| Cloud Storage Type    | Flow Spec ID                         | Flow Spec Name\n",
    "|-----------------------|--------------------------------------|------------------\n",
    "| Amazon S3             | 9753525b-82c7-4dce-8a9b-5ccfce2b9876 | CloudStorageToAEP\n",
    "| Azure Blob Storage    | 14518937-270c-4525-bdec-c2ba7cce3860 | CRMToAEP\n",
    "| Azure Data Lake       | 9753525b-82c7-4dce-8a9b-5ccfce2b9876 | CloudStorageToAEP\n",
    "| Data Landing Zone     | 9753525b-82c7-4dce-8a9b-5ccfce2b9876 | CloudStorageToAEP\n",
    "| Google Cloud Storage  | 9753525b-82c7-4dce-8a9b-5ccfce2b9876 | CloudStorageToAEP\n",
    "| SFTP                  | 9753525b-82c7-4dce-8a9b-5ccfce2b9876 | CloudStorageToAEP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "0ce4d087-2095-4044-aa7d-2f6c5c1b51fb",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out[57]: '9753525b-82c7-4dce-8a9b-5ccfce2b9876'"
     ]
    }
   ],
   "source": [
    "flow_spec = flow_conn.getFlowSpecs(\"name==CloudStorageToAEP\")\n",
    "flow_spec_id = flow_spec[0][\"id\"]\n",
    "flow_spec_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "efd9dac5-314f-46f4-aee2-54a19db895ca",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out[58]: '1090c6fc-2044-499b-bda7-acf3109b97b6'"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "# TODO: cleanup in aepp, first param should not be required\n",
    "flow_res = flow_conn.createFlow(flow_spec_id, obj={\n",
    "  \"name\": f\"[CMLE][Week4] DLZ to AEP for user propensity (created by {username})\",\n",
    "  \"flowSpec\": {\n",
    "      \"id\": flow_spec_id,\n",
    "      \"version\": \"1.0\"\n",
    "  },\n",
    "  \"sourceConnectionIds\": [\n",
    "      source_connection_id\n",
    "  ],\n",
    "  \"targetConnectionIds\": [\n",
    "      target_connection_id\n",
    "  ],\n",
    "  \"transformations\": [\n",
    "      {\n",
    "          \"name\": \"Mapping\",\n",
    "          \"params\": {\n",
    "              \"mappingId\": mapping_id,\n",
    "              \"mappingVersion\": 0\n",
    "          }\n",
    "      }\n",
    "  ],\n",
    "  \"scheduleParams\": {\n",
    "      \"startTime\": str(int(time.time())),\n",
    "      \"frequency\": \"minute\",\n",
    "      \"interval\": \"15\"\n",
    "  }\n",
    "})\n",
    "dataflow_id = flow_res[\"id\"]\n",
    "dataflow_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the name of the transformation has to be set to `Mapping` or the job will fail."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "8fc21e3c-3c60-4692-8de3-40c33d1020ca",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "You should be able to see your Data Flow in the UI at the link below, and you may see some executions depending on when you check since it runs on a schedule and will still show the run even if there was no data to process, as shown in the screenshot below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "44e869f1-01d7-4d25-b421-25c0e1a1380c",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data Flow created as ID 1090c6fc-2044-499b-bda7-acf3109b97b6 available under https://experience.adobe.com/#/@aamds/sname:cmle/platform/source/dataflows/1090c6fc-2044-499b-bda7-acf3109b97b6\n"
     ]
    }
   ],
   "source": [
    "dataflow_link = get_ui_link(tenant_id, \"source/dataflows\", dataflow_id)\n",
    "print(f\"Data Flow created as ID {dataflow_id} available under {dataflow_link}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "a355015f-6cf4-4bbe-a638-6dc1fd4a148d",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "![Source Dataflow](../media/CMLE-Notebooks-Week4-Dataflow.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "497720c7-5521-48e7-84c5-7b21b2aa8ad8",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Note: If you would like to switch to a different cloud storage, you need to update the `flow_spec_id` variable above to the matching value in the table mentioned earlier in this section. You can refer to the name from the table above to find out the ID."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "773f6f7f-5c5d-4e9c-bab3-28e0e40ada5b",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "## 2.3 Ingest the scored users into the Unified Profile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "7f3e1476-959a-47d8-bf92-fe7fbf7c6098",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "At that point we have successfully setup a Data Flow that is listening on any files being written to the DLZ under our import path, and will automatically convert it to XDM and deliver it to our dataset where they will be picked up for ingestion into the Unified Profile. All that is left to do is to actually deliver the data. For that, we refer to our Spark dataframe computed earlier `df_to_ingest` and we need to write it to the DLZ under the afore-mentioned folder.\n",
    "\n",
    "Before we can do that we need to update the credentials, because we'll be writing to a different container in the DLZ (`dlz-user-container` instead of `dlz-destination`) so the credentials are different. This should not cause an issue with the lazy computation of our dataframe because we used `.cache()` to cache it so it should already be in memory right now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "2da7607a-eafb-4287-96f7-becc73ce78d9",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "spark.conf.set(f\"fs.azure.account.auth.type.{dlz_storage_account}.dfs.core.windows.net\", \"SAS\")\n",
    "spark.conf.set(f\"fs.azure.sas.token.provider.type.{dlz_storage_account}.dfs.core.windows.net\", \"org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider\")\n",
    "spark.conf.set(f\"fs.azure.sas.fixed.token.{dlz_storage_account}.dfs.core.windows.net\", dlz_sas_token)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "17698fa2-65c3-41bc-bffb-7bf221981f21",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Now let's determine the full path where we need to write. We use a similar convention as when the data as been egressed, which is `cmle/ingress/$DATASETID/exportTime=$EXPORTTIME`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "5044cb13-86f5-4830-8c6b-1be8090208a8",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Out[39]: 'abfss://dlz-user-container@sndbxdtlndv43vd4kug0dnjn.dfs.core.windows.net/cmle/ingress/642db013d8fc0e1c07f46b12/exportTime=20230405173234/'"
     ]
    }
   ],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "protocol = \"abfss\"\n",
    "scoring_export_time = datetime.utcnow().strftime('%Y%m%d%H%M%S')\n",
    "output_path = f\"{protocol}://{dlz_container}@{dlz_storage_account}.dfs.core.windows.net/{import_path}/{ingestion_dataset_id}/exportTime={scoring_export_time}/\"\n",
    "output_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "00eeec8d-739e-4342-94ff-1ab6ae7926b8",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Now we just need to write the datafrae to this output path. Note that because we chose `delimited` format in our Data Flow setup, we're just going to write the resulting files as CSV format, and include the header so the transformation knows which field is which column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "f7a82868-cacc-4020-89de-09c8109d2ee4",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "df_to_ingest \\\n",
    "  .write \\\n",
    "  .option(\"header\", True) \\\n",
    "  .format(\"csv\") \\\n",
    "  .save(output_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "639a2e67-d3c0-4d0e-a3c2-6a28f3a61f02",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Because the Data Flow is executed asynchronously every 15 minutes, it may take a few minutes before the data is ingested in the dataset. We can check the status of the runs below until we can see the run has successfully completed to check some summary statistics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "67e7facf-d0a9-4329-86ed-2a16af1f729b",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Run ID fdf6cf83-3ba4-44b5-894a-1554759ae799 completed with: duration=91.82 secs; size=3.213498115539551 MB; num_rows=89921; num_files=1\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "# TODO: handle that more gracefully in aepp\n",
    "finished = False\n",
    "while not finished:\n",
    "  try:\n",
    "      runs = flow_conn.getRuns(prop=f\"flowId=={dataflow_id}\")\n",
    "      for run in runs:\n",
    "          run_id = run[\"id\"]\n",
    "          run_started_at = run[\"metrics\"][\"durationSummary\"][\"startedAtUTC\"]\n",
    "          run_ended_at = run[\"metrics\"][\"durationSummary\"][\"completedAtUTC\"]\n",
    "          run_duration_secs = (run_ended_at - run_started_at) / 1000\n",
    "          run_size_mb = run[\"metrics\"][\"sizeSummary\"][\"outputBytes\"] / 1024. / 1024.\n",
    "          run_num_rows = run[\"metrics\"][\"recordSummary\"][\"outputRecordCount\"]\n",
    "          run_num_files = run[\"metrics\"][\"fileSummary\"][\"outputFileCount\"]\n",
    "          print(f\"Run ID {run_id} completed with: duration={run_duration_secs} secs; size={run_size_mb} MB; num_rows={run_num_rows}; num_files={run_num_files}\")\n",
    "      finished = True\n",
    "  except Exception as e:\n",
    "      print(f\"No runs completed yet for flow {dataflow_id}\")\n",
    "      time.sleep(60)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "64e45b1d-40dc-484f-9c0a-b81106e07e28",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Once this is done, you should be able to go back in your dataset at the same link as before and see a batch created successfully in it. You should also notice for that batch that the records ingested will also show up under **Existing Profile Fragments** which means they have been ingested in the Unified Profile successfully."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "d38c2b98-47d1-4211-8cee-6009cbcda197",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "![Ingestion](../media/CMLE-Notebooks-Week4-Ingestion.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "8f64d990-e799-47f2-ad80-8120b727ca77",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "## 2.4 Storing the scoring dataset ID in the configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "048215d6-22b9-4260-9e1a-eea7d3968f74",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Now that we got everything working, we just need to save the `ingestion_dataset_id` variable in the original configuration file, so we can refer to it in the following weekly assignment. To do that, execute the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "af89ab3a-a166-4753-b6ab-121c27e7ba85",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "config.set(\"Platform\", \"scoring_dataset_id\", ingestion_dataset_id)\n",
    "\n",
    "with open(config_path, \"w\") as configfile:\n",
    "    config.write(configfile)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "fe24871e-2bec-4a36-85f5-29688bfb313e",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "application/vnd.databricks.v1+notebook": {
   "dashboards": [],
   "language": "python",
   "notebookMetadata": {
    "pythonIndentUnit": 2,
    "widgetLayout": [
     {
      "breakBefore": false,
      "name": "pqsHost",
      "width": 204
     },
     {
      "breakBefore": false,
      "name": "imsOrgId",
      "width": 204
     },
     {
      "breakBefore": false,
      "name": "sandboxName",
      "width": 204
     },
     {
      "breakBefore": false,
      "name": "sandboxId",
      "width": 204
     },
     {
      "breakBefore": false,
      "name": "clientId",
      "width": 204
     },
     {
      "breakBefore": false,
      "name": "clientSecret",
      "width": 204
     },
     {
      "breakBefore": false,
      "name": "privateKeyPath",
      "width": 204
     },
     {
      "breakBefore": false,
      "name": "techAccountId",
      "width": 204
     },
     {
      "breakBefore": false,
      "name": "pqsPort",
      "width": 204
     },
     {
      "breakBefore": false,
      "name": "pqsDB",
      "width": 204
     },
     {
      "breakBefore": false,
      "name": "pqsUser",
      "width": 204
     },
     {
      "breakBefore": false,
      "name": "pqsPass",
      "width": 204
     },
     {
      "breakBefore": false,
      "name": "datasetId",
      "width": 204
     }
    ]
   },
   "notebookName": "Week4Notebook",
   "notebookOrigID": 3680348664249761,
   "widgets": {}
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}