course1/week3/C1_W3_Assignment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train a model with Amazon SageMaker Autopilot\n",
    "\n",
    "### Introduction\n",
    "\n",
    "In this lab, you will use Amazon Sagemaker Autopilot to train a BERT-based natural language processing (NLP) model. The model will analyze customer feedback and classify the messages into positive (1), neutral (0) and negative (-1) sentiment.\n",
    "\n",
    "### Table of Contents\n",
    "\n",
    "- [1. Review transformed dataset](#c1w3-1.)\n",
    "- [2. Configure the Autopilot job](#c1w3-2.)\n",
    "  - [2.1. Upload data to S3 bucket](#c1w3-2.1.)\n",
    "  - [2.2. S3 output for generated assets](#c1w3-2.2.)\n",
    "  - [2.3. Configure the Autopilot job](#c1w3-2.3.)\n",
    "    - [Exercise 1](#c1w3-ex-1)\n",
    "- [3. Launch the Autopilot job](#c1w3-3.)\n",
    "    - [Exercise 2](#c1w3-ex-2)\n",
    "- [4. Track Autopilot job progress](#c1w3-4.)\n",
    "  - [4.1. Autopilot job description](#c1w3-4.1.)\n",
    "  - [4.2. Autopilot job status](#c1w3-4.2.)\n",
    "  - [4.3. Review the SageMaker processing jobs](#c1w3-4.3.)\n",
    "  - [4.4. Wait for the data analysis step to finish](#c1w3-4.4.)\n",
    "  - [4.5. View generated notebooks](#c1w3-4.5.)\n",
    "    - [Exercise 3](#c1w3-ex-3)\n",
    "    - [Exercise 4](#c1w3-ex-4)\n",
    "- [5. Feature engineering](#c1w3-5.)\n",
    "    - [Exercise 5](#c1w3-ex-5)\n",
    "- [6. Model training and tuning](#c1w3-6.)\n",
    "  - [6.1. Wait for training and tuning](#c1w3-6.1.)\n",
    "    - [Exercise 6](#c1w3-ex-6)\n",
    "  - [6.2. Compare model candidates](#c1w3-6.2.)\n",
    "    - [Exercise 7](#c1w3-ex-7)\n",
    "  - [6.3. Review best candidate](#c1w3-6.3.)\n",
    "    - [Exercise 8](#c1w3-ex-8)\n",
    "- [7. Review all output in S3 bucket](#c1w3-7.)\n",
    "- [8. Deploy and test best candidate model](#c1w3-8.)\n",
    "  - [8.1. Deploy best candidate model](#c1w3-8.1.)\n",
    "  - [8.2. Test the model](#c1w3-8.2.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Amazon SageMaker Autopilot automatically trains and tunes the best machine learning models for classification or regression, based on your data while allowing to maintain full control and visibility.\n",
    "\n",
    "SageMaker Autopilot will inspect the raw dataset, apply feature processors, pick the best set of algorithms, train and tune multiple models, and then rank the models based on performance - all with just a few clicks.  Autopilot transparently generates a set of Python scripts and notebooks for a complete end-to-end pipeline including data analysis, candidate generation, feature engineering, and model training/tuning.\n",
    "\n",
    "SageMaker Autopilot job consists of the following high-level steps: \n",
    "* _Data analysis_ where the data is summarized and analyzed to determine which feature engineering techniques, hyper-parameters, and models to explore.\n",
    "* _Feature engineering_ where the data is scrubbed, balanced, combined, and split into train and validation.\n",
    "* _Model training and tuning_ where the top performing features, hyper-parameters, and models are selected and trained.\n",
    "\n",
    "<img src=\"images/autopilot-transparent.png\" width=\"100%\">\n",
    "\n",
    "These re-usable scripts and notebooks give us full visibility into how the model candidates were created.  Since Autopilot integrates natively with SageMaker Studio, we can visually explore the different models generated by SageMaker Autopilot.\n",
    "\n",
    "SageMaker Autopilot can be used by people without machine learning experience to automatically train a model from a dataset.  Additionally, experienced developers can use Autopilot to train a baseline model from which they can iterate and manually improve.\n",
    "\n",
    "Autopilot is available through the SageMaker Studio UI and AWS Python SDK.  In this notebook, you will use the AWS Python SDK to train a series of text-classification models and deploy the model with the highest accuracy.\n",
    "\n",
    "For more details on Autopilot, have a look at this [**Amazon Science Publication**](https://www.amazon.science/publications/amazon-sagemaker-autopilot-a-white-box-automl-solution-at-scale).\n",
    "\n",
    "### Use case: analyze customer sentiment\n",
    "\n",
    "Customer feedback appears across many channels including social media and partner websites.  As a company, you want to capture this valuable product feedback to spot negative trends and improve the situation, if needed. Here you will train a model to classify the feedback messages into positive (1), neutral (0) and negative (-1) sentiment.\n",
    "\n",
    "First, let's install and import required modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/secretstorage/dhcrypto.py:16: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead\n",
      "  from cryptography.utils import int_from_bytes\n",
      "/opt/conda/lib/python3.7/site-packages/secretstorage/util.py:25: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead\n",
      "  from cryptography.utils import int_from_bytes\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# please ignore warning messages during the installation\n",
    "!pip install --disable-pip-version-check -q sagemaker==2.35.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import botocore\n",
    "import time\n",
    "import json\n",
    "\n",
    "config = botocore.config.Config(user_agent_extra='dlai-pds/c1/w3')\n",
    "\n",
    "# low-level service client of the boto3 session\n",
    "sm = boto3.client(service_name='sagemaker', \n",
    "                  config=config)\n",
    "\n",
    "sm_runtime = boto3.client('sagemaker-runtime',\n",
    "                          config=config)\n",
    "\n",
    "sess = sagemaker.Session(sagemaker_client=sm,\n",
    "                         sagemaker_runtime_client=sm_runtime)\n",
    "\n",
    "bucket = sess.default_bucket()\n",
    "role = sagemaker.get_execution_role()\n",
    "region = sess.boto_region_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format='retina'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-1.'></a>\n",
    "# 1. Review transformed dataset\n",
    "\n",
    "Let's transform the dataset into a format that Autopilot recognizes.  Specifically, a comma-separated file of `label,features` as shown here:\n",
    "```\n",
    "sentiment,review_body\n",
    "-1,\"this is bad\"\n",
    "0,\"this is ok\"\n",
    "1,\"this is great\"\n",
    "...\n",
    "```\n",
    "\n",
    "Sentiment is one of three classes: negative (-1), neutral (0), or positive (1).  Autopilot requires that the target variable, `sentiment` is first and the set of features, just `review_body` in this case, come next.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "download: s3://dlai-practical-data-science/data/balanced/womens_clothing_ecommerce_reviews_balanced.csv to ./womens_clothing_ecommerce_reviews_balanced.csv\n"
     ]
    }
   ],
   "source": [
    "!aws s3 cp 's3://dlai-practical-data-science/data/balanced/womens_clothing_ecommerce_reviews_balanced.csv' ./"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentiment</th>\n",
       "      <th>review_body</th>\n",
       "      <th>product_category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-1</td>\n",
       "      <td>This suit did nothing for me. the top has zero...</td>\n",
       "      <td>Swim</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1</td>\n",
       "      <td>Like other reviewers  i saw this dress on the ...</td>\n",
       "      <td>Dresses</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-1</td>\n",
       "      <td>I wish i had read the reviews before purchasin...</td>\n",
       "      <td>Knits</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-1</td>\n",
       "      <td>I ordered these pants in my usual size (xl) an...</td>\n",
       "      <td>Legwear</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-1</td>\n",
       "      <td>I noticed this top on one of the sales associa...</td>\n",
       "      <td>Knits</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sentiment                                        review_body  \\\n",
       "0         -1  This suit did nothing for me. the top has zero...   \n",
       "1         -1  Like other reviewers  i saw this dress on the ...   \n",
       "2         -1  I wish i had read the reviews before purchasin...   \n",
       "3         -1  I ordered these pants in my usual size (xl) an...   \n",
       "4         -1  I noticed this top on one of the sales associa...   \n",
       "\n",
       "  product_category  \n",
       "0             Swim  \n",
       "1          Dresses  \n",
       "2            Knits  \n",
       "3          Legwear  \n",
       "4            Knits  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = './womens_clothing_ecommerce_reviews_balanced.csv'\n",
    "\n",
    "df = pd.read_csv(path, delimiter=',')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_autopilot = './womens_clothing_ecommerce_reviews_balanced_for_autopilot.csv'\n",
    "\n",
    "df[['sentiment', 'review_body']].to_csv(path_autopilot, \n",
    "                                        sep=',', \n",
    "                                        index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-2.'></a>\n",
    "# 2. Configure the Autopilot job"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-2.1.'></a>\n",
    "### 2.1. Upload data to S3 bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'s3://sagemaker-us-east-1-118421937497/autopilot/data/womens_clothing_ecommerce_reviews_balanced_for_autopilot.csv'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "autopilot_train_s3_uri = sess.upload_data(bucket=bucket, key_prefix='autopilot/data', path=path_autopilot)\n",
    "autopilot_train_s3_uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the existence of the dataset in this S3 bucket folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-09-12 12:18:04    2253749 womens_clothing_ecommerce_reviews_balanced_for_autopilot.csv\n"
     ]
    }
   ],
   "source": [
    "!aws s3 ls $autopilot_train_s3_uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-2.2.'></a>\n",
    "### 2.2. S3 output for generated assets\n",
    "Set the S3 output path for the Autopilot outputs. This includes Jupyter notebooks (analysis), Python scripts (feature engineering), and trained models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sagemaker-us-east-1-118421937497/autopilot\n"
     ]
    }
   ],
   "source": [
    "model_output_s3_uri = 's3://{}/autopilot'.format(bucket)\n",
    "\n",
    "print(model_output_s3_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-2.3.'></a>\n",
    "### 2.3. Configure the Autopilot job"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the Autopilot job name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "timestamp = int(time.time())\n",
    "\n",
    "auto_ml_job_name = 'automl-dm-{}'.format(timestamp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When configuring our Autopilot job, you need to specify the maximum number of candidates, `max_candidates`, to explore as well as the input/output S3 locations and target column to predict. In this case, you want to predict `sentiment` from the review text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-ex-1'></a>\n",
    "### Exercise 1\n",
    "\n",
    "Configure the Autopilot job.\n",
    "\n",
    "**Instructions**: Create an instance of the `sagemaker.automl.automl.AutoML` estimator class passing the required configuration parameters. Target attribute for predictions here is `sentiment`.\n",
    "\n",
    "```python\n",
    "automl = sagemaker.automl.automl.AutoML(\n",
    "    target_attribute_name='...', # the name of the target attribute for predictions\n",
    "    base_job_name=..., # Autopilot job name\n",
    "    output_path=..., # output data path\n",
    "    max_candidates=..., # maximum number of candidates\n",
    "    sagemaker_session=sess,\n",
    "    role=role,\n",
    "    max_runtime_per_training_job_in_seconds=1200,\n",
    "    total_job_runtime_in_seconds=7200\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_candidates = 3\n",
    "\n",
    "automl = sagemaker.automl.automl.AutoML(\n",
    "    ### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    target_attribute_name='sentiment', # Replace None\n",
    "    base_job_name=auto_ml_job_name, # Replace None\n",
    "    output_path=model_output_s3_uri, # Replace None\n",
    "    ### END SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    max_candidates=max_candidates,\n",
    "    sagemaker_session=sess,\n",
    "    role=role,\n",
    "    max_runtime_per_training_job_in_seconds=1200,\n",
    "    total_job_runtime_in_seconds=7200\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-3.'></a>\n",
    "# 3. Launch the Autopilot job"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-ex-2'></a>\n",
    "### Exercise 2\n",
    "\n",
    "Launch the Autopilot job.\n",
    "\n",
    "**Instructions**: Call `fit` function of the configured estimator passing the S3 bucket input data path and the Autopilot job name.\n",
    "\n",
    "```python\n",
    "automl.fit(\n",
    "    ..., # input data path\n",
    "    job_name=auto_ml_job_name, # Autopilot job name\n",
    "    wait=False, \n",
    "    logs=False\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl.fit(\n",
    "    ### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    autopilot_train_s3_uri, # Replace None\n",
    "    ### END SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    job_name=auto_ml_job_name, \n",
    "    wait=False, \n",
    "    logs=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-4.'></a>\n",
    "# 4. Track Autopilot job progress\n",
    "\n",
    "Once the Autopilot job has been launched, you can track the job progress directly from the notebook using the SDK capabilities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-4.1.'></a>\n",
    "### 4.1. Autopilot job description\n",
    "\n",
    "Function `describe_auto_ml_job` of the Amazon SageMaker service returns the information about the AutoML job in dictionary format. You can review the response syntax and response elements in the [**documentation**](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJob.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "job_description_response = automl.describe_auto_ml_job(job_name=auto_ml_job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-4.2.'></a>\n",
    "### 4.2. Autopilot job status\n",
    "\n",
    "To track the job progress you can use two response elements: `AutoMLJobStatus` and `AutoMLJobSecondaryStatus`, which correspond to the primary (Completed | InProgress | Failed | Stopped | Stopping) and secondary (AnalyzingData | FeatureEngineering | ModelTuning etc.) job states respectively. To see if the AutoML job has started, you can check the existence of the `AutoMLJobStatus` and `AutoMLJobSecondaryStatus` elements in the job description response.\n",
    "\n",
    "In this notebook, you will use the following scheme to track the job progress:\n",
    "\n",
    "```python\n",
    "# check if the job is still at certain stage\n",
    "while [check 'AutoMLJobStatus' and 'AutoMLJobSecondaryStatus'] in job_description_response:\n",
    "    # update the job description response\n",
    "    job_description_response = automl.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "    # print the message the Autopilot job is in the stage ...\n",
    "    print([message])\n",
    "    # git a time step to check the status again\n",
    "    sleep(15)\n",
    "print(\"Autopilot job complete...\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[OK] AutoML job started.\n"
     ]
    }
   ],
   "source": [
    "while 'AutoMLJobStatus' not in job_description_response.keys() and 'AutoMLJobSecondaryStatus' not in job_description_response.keys():\n",
    "    job_description_response = automl.describe_auto_ml_job(job_name=auto_ml_job_name)\n",
    "    print('[INFO] Autopilot job has not yet started. Please wait. ')\n",
    "    # function `json.dumps` encodes JSON string for printing.\n",
    "    print(json.dumps(job_description_response, indent=4, sort_keys=True, default=str))\n",
    "    print('[INFO] Waiting for Autopilot job to start...')\n",
    "    sleep(15)\n",
    "\n",
    "print('[OK] AutoML job started.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-4.3.'></a>\n",
    "### 4.3. Review the SageMaker processing jobs\n",
    "\n",
    "The Autopilot creates required SageMaker processing jobs during the run:\n",
    "\n",
    "* First processing job (data splitter) checks the data sanity, performs stratified shuffling and splits the data into training and validation. \n",
    "* Second processing job (candidate generator) first streams through the data to compute statistics for the dataset. Then, uses these statistics to identify the problem type, and possible types of every column-predictor: numeric, categorical, natural language, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/processing-jobs/\">processing jobs</a></b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "display(HTML('<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/processing-jobs/\">processing jobs</a></b>'.format(region)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can review the updates on that page during the run of the Autopilot job."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-4.4.'></a>\n",
    "### 4.4. Wait for the data analysis step to finish\n",
    "\n",
    "Here you will use the same scheme as above to check the completion of the data analysis step. This step can be identified with the (primary) job status value `InProgress` and secondary job status values `Starting` and then `AnalyzingData`.\n",
    "### _This cell will take approximately 10 minutes to run._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "InProgress Starting\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress AnalyzingData\n",
      "InProgress FeatureEngineering\n",
      "[OK] Data analysis phase completed.\n",
      "\n",
      "{\n",
      "    \"AutoMLJobArn\": \"arn:aws:sagemaker:us-east-1:118421937497:automl-job/automl-dm-1631449084\",\n",
      "    \"AutoMLJobArtifacts\": {\n",
      "        \"CandidateDefinitionNotebookLocation\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/sagemaker-automl-candidates/automl-dm-1631449084-pr-1-60810fda05ca49e5b46872ae7f80056ccfe25/notebooks/SageMakerAutopilotCandidateDefinitionNotebook.ipynb\",\n",
      "        \"DataExplorationNotebookLocation\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/sagemaker-automl-candidates/automl-dm-1631449084-pr-1-60810fda05ca49e5b46872ae7f80056ccfe25/notebooks/SageMakerAutopilotDataExplorationNotebook.ipynb\"\n",
      "    },\n",
      "    \"AutoMLJobConfig\": {\n",
      "        \"CompletionCriteria\": {\n",
      "            \"MaxAutoMLJobRuntimeInSeconds\": 7200,\n",
      "            \"MaxCandidates\": 3,\n",
      "            \"MaxRuntimePerTrainingJobInSeconds\": 1200\n",
      "        },\n",
      "        \"SecurityConfig\": {\n",
      "            \"EnableInterContainerTrafficEncryption\": false\n",
      "        }\n",
      "    },\n",
      "    \"AutoMLJobName\": \"automl-dm-1631449084\",\n",
      "    \"AutoMLJobSecondaryStatus\": \"FeatureEngineering\",\n",
      "    \"AutoMLJobStatus\": \"InProgress\",\n",
      "    \"CreationTime\": \"2021-09-12 12:18:05.084000+00:00\",\n",
      "    \"GenerateCandidateDefinitionsOnly\": false,\n",
      "    \"InputDataConfig\": [\n",
      "        {\n",
      "            \"DataSource\": {\n",
      "                \"S3DataSource\": {\n",
      "                    \"S3DataType\": \"S3Prefix\",\n",
      "                    \"S3Uri\": \"s3://sagemaker-us-east-1-118421937497/autopilot/data/womens_clothing_ecommerce_reviews_balanced_for_autopilot.csv\"\n",
      "                }\n",
      "            },\n",
      "            \"TargetAttributeName\": \"sentiment\"\n",
      "        }\n",
      "    ],\n",
      "    \"LastModifiedTime\": \"2021-09-12 12:27:05.557000+00:00\",\n",
      "    \"OutputDataConfig\": {\n",
      "        \"S3OutputPath\": \"s3://sagemaker-us-east-1-118421937497/autopilot\"\n",
      "    },\n",
      "    \"ResolvedAttributes\": {\n",
      "        \"AutoMLJobObjective\": {\n",
      "            \"MetricName\": \"Accuracy\"\n",
      "        },\n",
      "        \"CompletionCriteria\": {\n",
      "            \"MaxAutoMLJobRuntimeInSeconds\": 7200,\n",
      "            \"MaxCandidates\": 3,\n",
      "            \"MaxRuntimePerTrainingJobInSeconds\": 1200\n",
      "        },\n",
      "        \"ProblemType\": \"MulticlassClassification\"\n",
      "    },\n",
      "    \"ResponseMetadata\": {\n",
      "        \"HTTPHeaders\": {\n",
      "            \"content-length\": \"1721\",\n",
      "            \"content-type\": \"application/x-amz-json-1.1\",\n",
      "            \"date\": \"Sun, 12 Sep 2021 12:27:09 GMT\",\n",
      "            \"x-amzn-requestid\": \"5a8b3024-3c5d-4670-b641-5872016d4763\"\n",
      "        },\n",
      "        \"HTTPStatusCode\": 200,\n",
      "        \"RequestId\": \"5a8b3024-3c5d-4670-b641-5872016d4763\",\n",
      "        \"RetryAttempts\": 0\n",
      "    },\n",
      "    \"RoleArn\": \"arn:aws:iam::118421937497:role/c21581a406814l993653t1w1184-SageMakerExecutionRole-1FRZBBBH9OZN0\"\n",
      "}\n",
      "CPU times: user 429 ms, sys: 31.3 ms, total: 461 ms\n",
      "Wall time: 9min 18s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "job_status = job_description_response['AutoMLJobStatus']\n",
    "job_sec_status = job_description_response['AutoMLJobSecondaryStatus']\n",
    "\n",
    "if job_status not in ('Stopped', 'Failed'):\n",
    "    while job_status in ('InProgress') and job_sec_status in ('Starting', 'AnalyzingData'):\n",
    "        job_description_response = automl.describe_auto_ml_job(job_name=auto_ml_job_name)\n",
    "        job_status = job_description_response['AutoMLJobStatus']\n",
    "        job_sec_status = job_description_response['AutoMLJobSecondaryStatus']\n",
    "        print(job_status, job_sec_status)\n",
    "        time.sleep(15)\n",
    "    print('[OK] Data analysis phase completed.\\n')\n",
    "    \n",
    "print(json.dumps(job_description_response, indent=4, sort_keys=True, default=str))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait for Autopilot to finish generating the notebooks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-4.5.'></a>\n",
    "### 4.5. View generated notebooks\n",
    "Once data analysis is complete, SageMaker AutoPilot generates two notebooks: \n",
    "* Data exploration\n",
    "* Candidate definition\n",
    "\n",
    "Notebooks are included in the AutoML job artifacts generated during the run. Before checking the existence of the notebooks, you can check if the artifacts have been generated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-ex-3'></a>\n",
    "### Exercise 3\n",
    "\n",
    "Check if the Autopilot job artifacts have been generated.\n",
    "\n",
    "**Instructions**: Use status check scheme described above. The generation of artifacts can be identified by existence of `AutoMLJobArtifacts` element in the keys of the job description response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[OK] AutoMLJobArtifacts generated.\n"
     ]
    }
   ],
   "source": [
    "### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "# get the information about the running Autopilot job\n",
    "job_description_response = automl.describe_auto_ml_job(job_name=auto_ml_job_name) # Replace None\n",
    "\n",
    "# keep in the while loop until the Autopilot job artifacts will be generated\n",
    "while 'AutoMLJobArtifacts' not in job_description_response.keys(): # Replace all None\n",
    "    # update the information about the running Autopilot job\n",
    "    job_description_response = automl.describe_auto_ml_job(job_name=auto_ml_job_name) # Replace None\n",
    "    ### END SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    print('[INFO] Autopilot job has not yet generated the artifacts. Please wait. ')\n",
    "    print(json.dumps(job_description_response, indent=4, sort_keys=True, default=str))\n",
    "    print('[INFO] Waiting for AutoMLJobArtifacts...')\n",
    "    time.sleep(15)\n",
    "\n",
    "print('[OK] AutoMLJobArtifacts generated.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait for Autopilot to make the notebooks available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-ex-4'></a>\n",
    "### Exercise 4\n",
    "\n",
    "Check if the notebooks have been created.\n",
    "\n",
    "**Instructions**: Use status check scheme described above. Notebooks creation can be identified by existence of `DataExplorationNotebookLocation` element in the keys of the `job_description_response['AutoMLJobArtifacts']` dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[OK] DataExplorationNotebookLocation found.\n"
     ]
    }
   ],
   "source": [
    "### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "# get the information about the running Autopilot job\n",
    "job_description_response = automl.describe_auto_ml_job(job_name=auto_ml_job_name) # Replace None\n",
    "\n",
    "# keep in the while loop until the notebooks will be created\n",
    "while 'DataExplorationNotebookLocation' not in job_description_response['AutoMLJobArtifacts'].keys(): # Replace all None\n",
    "    # update the information about the running Autopilot job\n",
    "    job_description_response = automl.describe_auto_ml_job(job_name=auto_ml_job_name) # Replace None\n",
    "    ### END SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    print('[INFO] Autopilot job has not yet generated the notebooks. Please wait. ')\n",
    "    print(json.dumps(job_description_response, indent=4, sort_keys=True, default=str))\n",
    "    print('[INFO] Waiting for DataExplorationNotebookLocation...')\n",
    "    time.sleep(15)\n",
    "\n",
    "print('[OK] DataExplorationNotebookLocation found.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Review the generated resources in S3 directly. Following the link, you can find the notebooks in the folder `notebooks` and download them by clicking on object `Actions`/`Object actions` -> `Download as`/`Download`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b>Review <a target=\"blank\" href=\"https://s3.console.aws.amazon.com/s3/buckets/sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/sagemaker-automl-candidates/automl-dm-1631449084-pr-1-60810fda05ca49e5b46872ae7f80056ccfe25/\">generated notebooks</a> in S3 bucket</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "generated_resources = job_description_response['AutoMLJobArtifacts']['DataExplorationNotebookLocation']\n",
    "download_path = generated_resources.rsplit('/notebooks/SageMakerAutopilotDataExplorationNotebook.ipynb')[0]\n",
    "job_id = download_path.rsplit('/', 1)[-1]\n",
    "\n",
    "if not job_id: \n",
    "    print('No AutoMLJobArtifacts found.')\n",
    "else: \n",
    "    display(HTML('<b>Review <a target=\"blank\" href=\"https://s3.console.aws.amazon.com/s3/buckets/{}/autopilot/{}/sagemaker-automl-candidates/{}/\">generated notebooks</a> in S3 bucket</b>'.format(bucket, auto_ml_job_name, job_id)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-5.'></a>\n",
    "# 5. Feature engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-ex-5'></a>\n",
    "### Exercise 5\n",
    "\n",
    "Check the completion of the feature engineering step.\n",
    "\n",
    "**Instructions**: Use status check scheme described above. Feature engineering step can be identified with the (primary) job status value `InProgress` and secondary job status value `FeatureEngineering`.\n",
    "\n",
    "### _This cell will take approximately 10 minutes to run._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "InProgress\n",
      "FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress FeatureEngineering\n",
      "InProgress ModelTuning\n",
      "[OK] Feature engineering phase completed.\n",
      "\n",
      "{\n",
      "    \"AutoMLJobArn\": \"arn:aws:sagemaker:us-east-1:118421937497:automl-job/automl-dm-1631449084\",\n",
      "    \"AutoMLJobArtifacts\": {\n",
      "        \"CandidateDefinitionNotebookLocation\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/sagemaker-automl-candidates/automl-dm-1631449084-pr-1-60810fda05ca49e5b46872ae7f80056ccfe25/notebooks/SageMakerAutopilotCandidateDefinitionNotebook.ipynb\",\n",
      "        \"DataExplorationNotebookLocation\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/sagemaker-automl-candidates/automl-dm-1631449084-pr-1-60810fda05ca49e5b46872ae7f80056ccfe25/notebooks/SageMakerAutopilotDataExplorationNotebook.ipynb\"\n",
      "    },\n",
      "    \"AutoMLJobConfig\": {\n",
      "        \"CompletionCriteria\": {\n",
      "            \"MaxAutoMLJobRuntimeInSeconds\": 7200,\n",
      "            \"MaxCandidates\": 3,\n",
      "            \"MaxRuntimePerTrainingJobInSeconds\": 1200\n",
      "        },\n",
      "        \"SecurityConfig\": {\n",
      "            \"EnableInterContainerTrafficEncryption\": false\n",
      "        }\n",
      "    },\n",
      "    \"AutoMLJobName\": \"automl-dm-1631449084\",\n",
      "    \"AutoMLJobSecondaryStatus\": \"ModelTuning\",\n",
      "    \"AutoMLJobStatus\": \"InProgress\",\n",
      "    \"CreationTime\": \"2021-09-12 12:18:05.084000+00:00\",\n",
      "    \"GenerateCandidateDefinitionsOnly\": false,\n",
      "    \"InputDataConfig\": [\n",
      "        {\n",
      "            \"DataSource\": {\n",
      "                \"S3DataSource\": {\n",
      "                    \"S3DataType\": \"S3Prefix\",\n",
      "                    \"S3Uri\": \"s3://sagemaker-us-east-1-118421937497/autopilot/data/womens_clothing_ecommerce_reviews_balanced_for_autopilot.csv\"\n",
      "                }\n",
      "            },\n",
      "            \"TargetAttributeName\": \"sentiment\"\n",
      "        }\n",
      "    ],\n",
      "    \"LastModifiedTime\": \"2021-09-12 12:36:33.820000+00:00\",\n",
      "    \"OutputDataConfig\": {\n",
      "        \"S3OutputPath\": \"s3://sagemaker-us-east-1-118421937497/autopilot\"\n",
      "    },\n",
      "    \"ResolvedAttributes\": {\n",
      "        \"AutoMLJobObjective\": {\n",
      "            \"MetricName\": \"Accuracy\"\n",
      "        },\n",
      "        \"CompletionCriteria\": {\n",
      "            \"MaxAutoMLJobRuntimeInSeconds\": 7200,\n",
      "            \"MaxCandidates\": 3,\n",
      "            \"MaxRuntimePerTrainingJobInSeconds\": 1200\n",
      "        },\n",
      "        \"ProblemType\": \"MulticlassClassification\"\n",
      "    },\n",
      "    \"ResponseMetadata\": {\n",
      "        \"HTTPHeaders\": {\n",
      "            \"content-length\": \"1713\",\n",
      "            \"content-type\": \"application/x-amz-json-1.1\",\n",
      "            \"date\": \"Sun, 12 Sep 2021 12:36:34 GMT\",\n",
      "            \"x-amzn-requestid\": \"a8abd57a-e0b3-4017-bd9a-0ec51efc5686\"\n",
      "        },\n",
      "        \"HTTPStatusCode\": 200,\n",
      "        \"RequestId\": \"a8abd57a-e0b3-4017-bd9a-0ec51efc5686\",\n",
      "        \"RetryAttempts\": 0\n",
      "    },\n",
      "    \"RoleArn\": \"arn:aws:iam::118421937497:role/c21581a406814l993653t1w1184-SageMakerExecutionRole-1FRZBBBH9OZN0\"\n",
      "}\n",
      "CPU times: user 403 ms, sys: 30.3 ms, total: 434 ms\n",
      "Wall time: 9min 15s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "job_description_response = automl.describe_auto_ml_job(job_name=auto_ml_job_name)\n",
    "job_status = job_description_response['AutoMLJobStatus']\n",
    "job_sec_status = job_description_response['AutoMLJobSecondaryStatus']\n",
    "print(job_status)\n",
    "print(job_sec_status)\n",
    "if job_status not in ('Stopped', 'Failed'):\n",
    "    ### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    while job_status in ('InProgress') and job_sec_status in ('FeatureEngineering'): # Replace all None\n",
    "    ### END SOLUTION - DO NOT delete this comment for grading purposes\n",
    "        job_description_response = automl.describe_auto_ml_job(job_name=auto_ml_job_name)\n",
    "        job_status = job_description_response['AutoMLJobStatus']\n",
    "        job_sec_status = job_description_response['AutoMLJobSecondaryStatus']\n",
    "        print(job_status, job_sec_status)\n",
    "        time.sleep(5)\n",
    "    print('[OK] Feature engineering phase completed.\\n')\n",
    "    \n",
    "print(json.dumps(job_description_response, indent=4, sort_keys=True, default=str))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-6.'></a>\n",
    "# 6. Model training and tuning\n",
    "When you launched the Autopilot job, you requested that 3 model candidates are generated and compared. Therefore, you should see three (3) SageMaker training jobs below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/hyper-tuning-jobs/\">hyper-parameter tuning jobs</a></b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "display(HTML('<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/hyper-tuning-jobs/\">hyper-parameter tuning jobs</a></b>'.format(region)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-6.1.'></a>\n",
    "### 6.1. Wait for training and tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-ex-6'></a>\n",
    "### Exercise 6\n",
    "\n",
    "Check the completion of the model tuning step.\n",
    "\n",
    "**Instructions**: Use status check scheme described above. Model tuning step can be identified with the (primary) job status value `InProgress` and secondary job status value `ModelTuning`.\n",
    "\n",
    "### _This cell will take approximately 5-10 minutes to run._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "InProgress\n",
      "ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress ModelTuning\n",
      "InProgress GeneratingExplainabilityReport\n",
      "[OK] Model tuning phase completed.\n",
      "\n",
      "{\n",
      "    \"AutoMLJobArn\": \"arn:aws:sagemaker:us-east-1:118421937497:automl-job/automl-dm-1631449084\",\n",
      "    \"AutoMLJobArtifacts\": {\n",
      "        \"CandidateDefinitionNotebookLocation\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/sagemaker-automl-candidates/automl-dm-1631449084-pr-1-60810fda05ca49e5b46872ae7f80056ccfe25/notebooks/SageMakerAutopilotCandidateDefinitionNotebook.ipynb\",\n",
      "        \"DataExplorationNotebookLocation\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/sagemaker-automl-candidates/automl-dm-1631449084-pr-1-60810fda05ca49e5b46872ae7f80056ccfe25/notebooks/SageMakerAutopilotDataExplorationNotebook.ipynb\"\n",
      "    },\n",
      "    \"AutoMLJobConfig\": {\n",
      "        \"CompletionCriteria\": {\n",
      "            \"MaxAutoMLJobRuntimeInSeconds\": 7200,\n",
      "            \"MaxCandidates\": 3,\n",
      "            \"MaxRuntimePerTrainingJobInSeconds\": 1200\n",
      "        },\n",
      "        \"SecurityConfig\": {\n",
      "            \"EnableInterContainerTrafficEncryption\": false\n",
      "        }\n",
      "    },\n",
      "    \"AutoMLJobName\": \"automl-dm-1631449084\",\n",
      "    \"AutoMLJobSecondaryStatus\": \"GeneratingExplainabilityReport\",\n",
      "    \"AutoMLJobStatus\": \"InProgress\",\n",
      "    \"BestCandidate\": {\n",
      "        \"CandidateName\": \"automl-dm-1631449084uVUArkOHWnPD-001-eb259203\",\n",
      "        \"CandidateProperties\": {\n",
      "            \"CandidateMetrics\": [\n",
      "                {\n",
      "                    \"MetricName\": \"Accuracy\",\n",
      "                    \"Set\": \"Validation\",\n",
      "                    \"Value\": 0.596340000629425\n",
      "                },\n",
      "                {\n",
      "                    \"MetricName\": \"F1macro\",\n",
      "                    \"Set\": \"Validation\",\n",
      "                    \"Value\": 0.5948699712753296\n",
      "                }\n",
      "            ]\n",
      "        },\n",
      "        \"CandidateStatus\": \"Completed\",\n",
      "        \"CandidateSteps\": [\n",
      "            {\n",
      "                \"CandidateStepArn\": \"arn:aws:sagemaker:us-east-1:118421937497:processing-job/automl-dm-1631449084-db-1-8e2c5d2b4f33496ca0c2b88b057c2c2afc1f2\",\n",
      "                \"CandidateStepName\": \"automl-dm-1631449084-db-1-8e2c5d2b4f33496ca0c2b88b057c2c2afc1f2\",\n",
      "                \"CandidateStepType\": \"AWS::SageMaker::ProcessingJob\"\n",
      "            },\n",
      "            {\n",
      "                \"CandidateStepArn\": \"arn:aws:sagemaker:us-east-1:118421937497:training-job/automl-dm-1631449084-dpp0-1-463056e6a4fa496bb36edad681360736942\",\n",
      "                \"CandidateStepName\": \"automl-dm-1631449084-dpp0-1-463056e6a4fa496bb36edad681360736942\",\n",
      "                \"CandidateStepType\": \"AWS::SageMaker::TrainingJob\"\n",
      "            },\n",
      "            {\n",
      "                \"CandidateStepArn\": \"arn:aws:sagemaker:us-east-1:118421937497:transform-job/automl-dm-1631449084-dpp0-rpb-1-ed5c4f47891746f7b74089486c2c50b\",\n",
      "                \"CandidateStepName\": \"automl-dm-1631449084-dpp0-rpb-1-ed5c4f47891746f7b74089486c2c50b\",\n",
      "                \"CandidateStepType\": \"AWS::SageMaker::TransformJob\"\n",
      "            },\n",
      "            {\n",
      "                \"CandidateStepArn\": \"arn:aws:sagemaker:us-east-1:118421937497:training-job/automl-dm-1631449084uvuarkohwnpd-001-eb259203\",\n",
      "                \"CandidateStepName\": \"automl-dm-1631449084uVUArkOHWnPD-001-eb259203\",\n",
      "                \"CandidateStepType\": \"AWS::SageMaker::TrainingJob\"\n",
      "            }\n",
      "        ],\n",
      "        \"CreationTime\": \"2021-09-12 12:39:03+00:00\",\n",
      "        \"EndTime\": \"2021-09-12 12:40:18+00:00\",\n",
      "        \"FinalAutoMLJobObjectiveMetric\": {\n",
      "            \"MetricName\": \"validation:accuracy\",\n",
      "            \"Value\": 0.596340000629425\n",
      "        },\n",
      "        \"InferenceContainers\": [\n",
      "            {\n",
      "                \"Environment\": {\n",
      "                    \"AUTOML_SPARSE_ENCODE_RECORDIO_PROTOBUF\": \"1\",\n",
      "                    \"AUTOML_TRANSFORM_MODE\": \"feature-transform\",\n",
      "                    \"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT\": \"application/x-recordio-protobuf\",\n",
      "                    \"SAGEMAKER_PROGRAM\": \"sagemaker_serve\",\n",
      "                    \"SAGEMAKER_SUBMIT_DIRECTORY\": \"/opt/ml/model/code\"\n",
      "                },\n",
      "                \"Image\": \"683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-sklearn-automl:2.2.1-1-cpu-py3\",\n",
      "                \"ModelDataUrl\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/data-processor-models/automl-dm-1631449084-dpp0-1-463056e6a4fa496bb36edad681360736942/output/model.tar.gz\"\n",
      "            },\n",
      "            {\n",
      "                \"Environment\": {\n",
      "                    \"MAX_CONTENT_LENGTH\": \"20971520\",\n",
      "                    \"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT\": \"text/csv\",\n",
      "                    \"SAGEMAKER_INFERENCE_OUTPUT\": \"predicted_label\",\n",
      "                    \"SAGEMAKER_INFERENCE_SUPPORTED\": \"predicted_label,probability,probabilities\"\n",
      "                },\n",
      "                \"Image\": \"683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-xgboost:1.2-2-cpu-py3\",\n",
      "                \"ModelDataUrl\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/tuning/automl-dm--dpp0-xgb/automl-dm-1631449084uVUArkOHWnPD-001-eb259203/output/model.tar.gz\"\n",
      "            },\n",
      "            {\n",
      "                \"Environment\": {\n",
      "                    \"AUTOML_TRANSFORM_MODE\": \"inverse-label-transform\",\n",
      "                    \"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT\": \"text/csv\",\n",
      "                    \"SAGEMAKER_INFERENCE_INPUT\": \"predicted_label\",\n",
      "                    \"SAGEMAKER_INFERENCE_OUTPUT\": \"predicted_label\",\n",
      "                    \"SAGEMAKER_INFERENCE_SUPPORTED\": \"predicted_label,probability,labels,probabilities\",\n",
      "                    \"SAGEMAKER_PROGRAM\": \"sagemaker_serve\",\n",
      "                    \"SAGEMAKER_SUBMIT_DIRECTORY\": \"/opt/ml/model/code\"\n",
      "                },\n",
      "                \"Image\": \"683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-sklearn-automl:2.2.1-1-cpu-py3\",\n",
      "                \"ModelDataUrl\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/data-processor-models/automl-dm-1631449084-dpp0-1-463056e6a4fa496bb36edad681360736942/output/model.tar.gz\"\n",
      "            }\n",
      "        ],\n",
      "        \"LastModifiedTime\": \"2021-09-12 12:40:45.737000+00:00\",\n",
      "        \"ObjectiveStatus\": \"Succeeded\"\n",
      "    },\n",
      "    \"CreationTime\": \"2021-09-12 12:18:05.084000+00:00\",\n",
      "    \"GenerateCandidateDefinitionsOnly\": false,\n",
      "    \"InputDataConfig\": [\n",
      "        {\n",
      "            \"DataSource\": {\n",
      "                \"S3DataSource\": {\n",
      "                    \"S3DataType\": \"S3Prefix\",\n",
      "                    \"S3Uri\": \"s3://sagemaker-us-east-1-118421937497/autopilot/data/womens_clothing_ecommerce_reviews_balanced_for_autopilot.csv\"\n",
      "                }\n",
      "            },\n",
      "            \"TargetAttributeName\": \"sentiment\"\n",
      "        }\n",
      "    ],\n",
      "    \"LastModifiedTime\": \"2021-09-12 12:40:47.204000+00:00\",\n",
      "    \"OutputDataConfig\": {\n",
      "        \"S3OutputPath\": \"s3://sagemaker-us-east-1-118421937497/autopilot\"\n",
      "    },\n",
      "    \"ResolvedAttributes\": {\n",
      "        \"AutoMLJobObjective\": {\n",
      "            \"MetricName\": \"Accuracy\"\n",
      "        },\n",
      "        \"CompletionCriteria\": {\n",
      "            \"MaxAutoMLJobRuntimeInSeconds\": 7200,\n",
      "            \"MaxCandidates\": 3,\n",
      "            \"MaxRuntimePerTrainingJobInSeconds\": 1200\n",
      "        },\n",
      "        \"ProblemType\": \"MulticlassClassification\"\n",
      "    },\n",
      "    \"ResponseMetadata\": {\n",
      "        \"HTTPHeaders\": {\n",
      "            \"content-length\": \"5082\",\n",
      "            \"content-type\": \"application/x-amz-json-1.1\",\n",
      "            \"date\": \"Sun, 12 Sep 2021 12:40:48 GMT\",\n",
      "            \"x-amzn-requestid\": \"c3c0593b-7dd7-46c8-9e18-edf32689dd26\"\n",
      "        },\n",
      "        \"HTTPStatusCode\": 200,\n",
      "        \"RequestId\": \"c3c0593b-7dd7-46c8-9e18-edf32689dd26\",\n",
      "        \"RetryAttempts\": 0\n",
      "    },\n",
      "    \"RoleArn\": \"arn:aws:iam::118421937497:role/c21581a406814l993653t1w1184-SageMakerExecutionRole-1FRZBBBH9OZN0\"\n",
      "}\n",
      "CPU times: user 188 ms, sys: 14.9 ms, total: 203 ms\n",
      "Wall time: 4min 12s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "job_description_response = automl.describe_auto_ml_job(job_name=auto_ml_job_name)\n",
    "job_status = job_description_response['AutoMLJobStatus']\n",
    "job_sec_status = job_description_response['AutoMLJobSecondaryStatus']\n",
    "print(job_status)\n",
    "print(job_sec_status)\n",
    "if job_status not in ('Stopped', 'Failed'):\n",
    "    ### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    while job_status in ('InProgress') and job_sec_status in ('ModelTuning'): # Replace all None\n",
    "    ### END SOLUTION - DO NOT delete this comment for grading purposes\n",
    "        job_description_response = automl.describe_auto_ml_job(job_name=auto_ml_job_name)\n",
    "        job_status = job_description_response['AutoMLJobStatus']\n",
    "        job_sec_status = job_description_response['AutoMLJobSecondaryStatus']\n",
    "        print(job_status, job_sec_status)\n",
    "        time.sleep(5)\n",
    "    print('[OK] Model tuning phase completed.\\n')\n",
    "    \n",
    "print(json.dumps(job_description_response, indent=4, sort_keys=True, default=str))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Please wait until ^^ Autopilot ^^ completes above_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, you can check the completion of the Autopilot job looking for the `Completed` job status."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'AutoMLJobArn': 'arn:aws:sagemaker:us-east-1:118421937497:automl-job/automl-dm-1631449084',\n",
      " 'AutoMLJobArtifacts': {'CandidateDefinitionNotebookLocation': 's3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/sagemaker-automl-candidates/automl-dm-1631449084-pr-1-60810fda05ca49e5b46872ae7f80056ccfe25/notebooks/SageMakerAutopilotCandidateDefinitionNotebook.ipynb',\n",
      "                        'DataExplorationNotebookLocation': 's3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/sagemaker-automl-candidates/automl-dm-1631449084-pr-1-60810fda05ca49e5b46872ae7f80056ccfe25/notebooks/SageMakerAutopilotDataExplorationNotebook.ipynb'},\n",
      " 'AutoMLJobConfig': {'CompletionCriteria': {'MaxAutoMLJobRuntimeInSeconds': 7200,\n",
      "                                            'MaxCandidates': 3,\n",
      "                                            'MaxRuntimePerTrainingJobInSeconds': 1200},\n",
      "                     'SecurityConfig': {'EnableInterContainerTrafficEncryption': False}},\n",
      " 'AutoMLJobName': 'automl-dm-1631449084',\n",
      " 'AutoMLJobSecondaryStatus': 'GeneratingExplainabilityReport',\n",
      " 'AutoMLJobStatus': 'InProgress',\n",
      " 'BestCandidate': {'CandidateName': 'automl-dm-1631449084uVUArkOHWnPD-001-eb259203',\n",
      "                   'CandidateProperties': {'CandidateMetrics': [{'MetricName': 'Accuracy',\n",
      "                                                                 'Set': 'Validation',\n",
      "                                                                 'Value': 0.596340000629425},\n",
      "                                                                {'MetricName': 'F1macro',\n",
      "                                                                 'Set': 'Validation',\n",
      "                                                                 'Value': 0.5948699712753296}]},\n",
      "                   'CandidateStatus': 'Completed',\n",
      "                   'CandidateSteps': [{'CandidateStepArn': 'arn:aws:sagemaker:us-east-1:118421937497:processing-job/automl-dm-1631449084-db-1-8e2c5d2b4f33496ca0c2b88b057c2c2afc1f2',\n",
      "                                       'CandidateStepName': 'automl-dm-1631449084-db-1-8e2c5d2b4f33496ca0c2b88b057c2c2afc1f2',\n",
      "                                       'CandidateStepType': 'AWS::SageMaker::ProcessingJob'},\n",
      "                                      {'CandidateStepArn': 'arn:aws:sagemaker:us-east-1:118421937497:training-job/automl-dm-1631449084-dpp0-1-463056e6a4fa496bb36edad681360736942',\n",
      "                                       'CandidateStepName': 'automl-dm-1631449084-dpp0-1-463056e6a4fa496bb36edad681360736942',\n",
      "                                       'CandidateStepType': 'AWS::SageMaker::TrainingJob'},\n",
      "                                      {'CandidateStepArn': 'arn:aws:sagemaker:us-east-1:118421937497:transform-job/automl-dm-1631449084-dpp0-rpb-1-ed5c4f47891746f7b74089486c2c50b',\n",
      "                                       'CandidateStepName': 'automl-dm-1631449084-dpp0-rpb-1-ed5c4f47891746f7b74089486c2c50b',\n",
      "                                       'CandidateStepType': 'AWS::SageMaker::TransformJob'},\n",
      "                                      {'CandidateStepArn': 'arn:aws:sagemaker:us-east-1:118421937497:training-job/automl-dm-1631449084uvuarkohwnpd-001-eb259203',\n",
      "                                       'CandidateStepName': 'automl-dm-1631449084uVUArkOHWnPD-001-eb259203',\n",
      "                                       'CandidateStepType': 'AWS::SageMaker::TrainingJob'}],\n",
      "                   'CreationTime': datetime.datetime(2021, 9, 12, 12, 39, 3, tzinfo=tzlocal()),\n",
      "                   'EndTime': datetime.datetime(2021, 9, 12, 12, 40, 18, tzinfo=tzlocal()),\n",
      "                   'FinalAutoMLJobObjectiveMetric': {'MetricName': 'validation:accuracy',\n",
      "                                                     'Value': 0.596340000629425},\n",
      "                   'InferenceContainers': [{'Environment': {'AUTOML_SPARSE_ENCODE_RECORDIO_PROTOBUF': '1',\n",
      "                                                            'AUTOML_TRANSFORM_MODE': 'feature-transform',\n",
      "                                                            'SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT': 'application/x-recordio-protobuf',\n",
      "                                                            'SAGEMAKER_PROGRAM': 'sagemaker_serve',\n",
      "                                                            'SAGEMAKER_SUBMIT_DIRECTORY': '/opt/ml/model/code'},\n",
      "                                            'Image': '683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-sklearn-automl:2.2.1-1-cpu-py3',\n",
      "                                            'ModelDataUrl': 's3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/data-processor-models/automl-dm-1631449084-dpp0-1-463056e6a4fa496bb36edad681360736942/output/model.tar.gz'},\n",
      "                                           {'Environment': {'MAX_CONTENT_LENGTH': '20971520',\n",
      "                                                            'SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT': 'text/csv',\n",
      "                                                            'SAGEMAKER_INFERENCE_OUTPUT': 'predicted_label',\n",
      "                                                            'SAGEMAKER_INFERENCE_SUPPORTED': 'predicted_label,probability,probabilities'},\n",
      "                                            'Image': '683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-xgboost:1.2-2-cpu-py3',\n",
      "                                            'ModelDataUrl': 's3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/tuning/automl-dm--dpp0-xgb/automl-dm-1631449084uVUArkOHWnPD-001-eb259203/output/model.tar.gz'},\n",
      "                                           {'Environment': {'AUTOML_TRANSFORM_MODE': 'inverse-label-transform',\n",
      "                                                            'SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT': 'text/csv',\n",
      "                                                            'SAGEMAKER_INFERENCE_INPUT': 'predicted_label',\n",
      "                                                            'SAGEMAKER_INFERENCE_OUTPUT': 'predicted_label',\n",
      "                                                            'SAGEMAKER_INFERENCE_SUPPORTED': 'predicted_label,probability,labels,probabilities',\n",
      "                                                            'SAGEMAKER_PROGRAM': 'sagemaker_serve',\n",
      "                                                            'SAGEMAKER_SUBMIT_DIRECTORY': '/opt/ml/model/code'},\n",
      "                                            'Image': '683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-sklearn-automl:2.2.1-1-cpu-py3',\n",
      "                                            'ModelDataUrl': 's3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/data-processor-models/automl-dm-1631449084-dpp0-1-463056e6a4fa496bb36edad681360736942/output/model.tar.gz'}],\n",
      "                   'LastModifiedTime': datetime.datetime(2021, 9, 12, 12, 40, 45, 737000, tzinfo=tzlocal()),\n",
      "                   'ObjectiveStatus': 'Succeeded'},\n",
      " 'CreationTime': datetime.datetime(2021, 9, 12, 12, 18, 5, 84000, tzinfo=tzlocal()),\n",
      " 'GenerateCandidateDefinitionsOnly': False,\n",
      " 'InputDataConfig': [{'DataSource': {'S3DataSource': {'S3DataType': 'S3Prefix',\n",
      "                                                      'S3Uri': 's3://sagemaker-us-east-1-118421937497/autopilot/data/womens_clothing_ecommerce_reviews_balanced_for_autopilot.csv'}},\n",
      "                      'TargetAttributeName': 'sentiment'}],\n",
      " 'LastModifiedTime': datetime.datetime(2021, 9, 12, 12, 40, 49, 812000, tzinfo=tzlocal()),\n",
      " 'OutputDataConfig': {'S3OutputPath': 's3://sagemaker-us-east-1-118421937497/autopilot'},\n",
      " 'ResolvedAttributes': {'AutoMLJobObjective': {'MetricName': 'Accuracy'},\n",
      "                        'CompletionCriteria': {'MaxAutoMLJobRuntimeInSeconds': 7200,\n",
      "                                               'MaxCandidates': 3,\n",
      "                                               'MaxRuntimePerTrainingJobInSeconds': 1200},\n",
      "                        'ProblemType': 'MulticlassClassification'},\n",
      " 'ResponseMetadata': {'HTTPHeaders': {'content-length': '5082',\n",
      "                                      'content-type': 'application/x-amz-json-1.1',\n",
      "                                      'date': 'Sun, 12 Sep 2021 12:40:53 GMT',\n",
      "                                      'x-amzn-requestid': '38253a25-62a6-4e0f-b52d-4ef3e66421df'},\n",
      "                      'HTTPStatusCode': 200,\n",
      "                      'RequestId': '38253a25-62a6-4e0f-b52d-4ef3e66421df',\n",
      "                      'RetryAttempts': 0},\n",
      " 'RoleArn': 'arn:aws:iam::118421937497:role/c21581a406814l993653t1w1184-SageMakerExecutionRole-1FRZBBBH9OZN0'}\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  InProgress\n",
      "Secondary job status:  GeneratingExplainabilityReport\n",
      "Job status:  Completed\n",
      "Secondary job status:  Completed\n",
      "[OK] Autopilot job completed.\n",
      "\n",
      "CPU times: user 564 ms, sys: 34 ms, total: 598 ms\n",
      "Wall time: 7min 34s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "from pprint import pprint\n",
    "\n",
    "job_description_response = automl.describe_auto_ml_job(job_name=auto_ml_job_name)\n",
    "pprint(job_description_response)\n",
    "job_status = job_description_response['AutoMLJobStatus']\n",
    "job_sec_status = job_description_response['AutoMLJobSecondaryStatus']\n",
    "print('Job status:  {}'.format(job_status))\n",
    "print('Secondary job status:  {}'.format(job_sec_status))\n",
    "if job_status not in ('Stopped', 'Failed'):\n",
    "    while job_status not in ('Completed'):\n",
    "        job_description_response = automl.describe_auto_ml_job(job_name=auto_ml_job_name)\n",
    "        job_status = job_description_response['AutoMLJobStatus']\n",
    "        job_sec_status = job_description_response['AutoMLJobSecondaryStatus']\n",
    "        print('Job status:  {}'.format(job_status))\n",
    "        print('Secondary job status:  {}'.format(job_sec_status))        \n",
    "        time.sleep(10)\n",
    "    print('[OK] Autopilot job completed.\\n')\n",
    "else:\n",
    "    print('Job status: {}'.format(job_status))\n",
    "    print('Secondary job status: {}'.format(job_status))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before moving to the next section make sure the status above indicates `Autopilot job completed`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-6.2.'></a>\n",
    "### 6.2. Compare model candidates\n",
    "Once model tuning is complete, you can view all the candidates (pipeline evaluations with different hyperparameter combinations) that were explored by AutoML and sort them by their final performance metric."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-ex-7'></a>\n",
    "### Exercise 7\n",
    "\n",
    "List candidates generated by Autopilot sorted by accuracy from highest to lowest.\n",
    "\n",
    "**Instructions**: Use `list_candidates` function passing the Autopilot job name `auto_ml_job_name` with the accuracy field `FinalObjectiveMetricValue`. It returns the list of candidates with the information about them.\n",
    "\n",
    "```python\n",
    "candidates = automl.list_candidates(\n",
    "    job_name=..., # Autopilot job name\n",
    "    sort_by='...' # accuracy field name\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "candidates = automl.list_candidates(\n",
    "    ### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    job_name=auto_ml_job_name, # Replace None\n",
    "    sort_by='FinalObjectiveMetricValue' # Replace None\n",
    "    ### END SOLUTION - DO NOT delete this comment for grading purposes\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can review the response syntax and response elements of the function `list_candidates` in the [**documentation**](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_AutoMLCandidate.html). Now let's put the candidate existence check into the loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[OK] Candidates generated.\n"
     ]
    }
   ],
   "source": [
    "while candidates == []:\n",
    "    candidates = automl.list_candidates(job_name=auto_ml_job_name)\n",
    "    print('[INFO] Autopilot job is generating the candidates. Please wait.')\n",
    "    time.sleep(10)\n",
    "\n",
    "print('[OK] Candidates generated.') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The information about each of the candidates is in the dictionary with the following keys:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['CandidateName', 'FinalAutoMLJobObjectiveMetric', 'ObjectiveStatus', 'CandidateSteps', 'CandidateStatus', 'InferenceContainers', 'CreationTime', 'EndTime', 'LastModifiedTime', 'CandidateProperties'])\n"
     ]
    }
   ],
   "source": [
    "print(candidates[0].keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`CandidateName` contains the candidate name and the `FinalAutoMLJobObjectiveMetric` element contains the metric information which can be used to identify the best candidate later. Let's check that they were generated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[OK] CandidateName generated.\n"
     ]
    }
   ],
   "source": [
    "while 'CandidateName' not in candidates[0]:\n",
    "    candidates = automl.list_candidates(job_name=auto_ml_job_name)\n",
    "    print('[INFO] Autopilot job is generating CandidateName. Please wait. ')\n",
    "    sleep(10)\n",
    "\n",
    "print('[OK] CandidateName generated.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[OK] FinalAutoMLJobObjectiveMetric generated.\n"
     ]
    }
   ],
   "source": [
    "while 'FinalAutoMLJobObjectiveMetric' not in candidates[0]:\n",
    "    candidates = automl.list_candidates(job_name=auto_ml_job_name)\n",
    "    print('[INFO] Autopilot job is generating FinalAutoMLJobObjectiveMetric. Please wait. ')\n",
    "    sleep(10)\n",
    "\n",
    "print('[OK] FinalAutoMLJobObjectiveMetric generated.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "    {\n",
      "        \"CandidateName\": \"automl-dm-1631449084uVUArkOHWnPD-001-eb259203\",\n",
      "        \"CandidateProperties\": {\n",
      "            \"CandidateArtifactLocations\": {\n",
      "                \"Explainability\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/documentation/explainability/output\"\n",
      "            },\n",
      "            \"CandidateMetrics\": [\n",
      "                {\n",
      "                    \"MetricName\": \"Accuracy\",\n",
      "                    \"Set\": \"Validation\",\n",
      "                    \"Value\": 0.596340000629425\n",
      "                },\n",
      "                {\n",
      "                    \"MetricName\": \"F1macro\",\n",
      "                    \"Set\": \"Validation\",\n",
      "                    \"Value\": 0.5948699712753296\n",
      "                }\n",
      "            ]\n",
      "        },\n",
      "        \"CandidateStatus\": \"Completed\",\n",
      "        \"CandidateSteps\": [\n",
      "            {\n",
      "                \"CandidateStepArn\": \"arn:aws:sagemaker:us-east-1:118421937497:processing-job/automl-dm-1631449084-db-1-8e2c5d2b4f33496ca0c2b88b057c2c2afc1f2\",\n",
      "                \"CandidateStepName\": \"automl-dm-1631449084-db-1-8e2c5d2b4f33496ca0c2b88b057c2c2afc1f2\",\n",
      "                \"CandidateStepType\": \"AWS::SageMaker::ProcessingJob\"\n",
      "            },\n",
      "            {\n",
      "                \"CandidateStepArn\": \"arn:aws:sagemaker:us-east-1:118421937497:training-job/automl-dm-1631449084-dpp0-1-463056e6a4fa496bb36edad681360736942\",\n",
      "                \"CandidateStepName\": \"automl-dm-1631449084-dpp0-1-463056e6a4fa496bb36edad681360736942\",\n",
      "                \"CandidateStepType\": \"AWS::SageMaker::TrainingJob\"\n",
      "            },\n",
      "            {\n",
      "                \"CandidateStepArn\": \"arn:aws:sagemaker:us-east-1:118421937497:transform-job/automl-dm-1631449084-dpp0-rpb-1-ed5c4f47891746f7b74089486c2c50b\",\n",
      "                \"CandidateStepName\": \"automl-dm-1631449084-dpp0-rpb-1-ed5c4f47891746f7b74089486c2c50b\",\n",
      "                \"CandidateStepType\": \"AWS::SageMaker::TransformJob\"\n",
      "            },\n",
      "            {\n",
      "                \"CandidateStepArn\": \"arn:aws:sagemaker:us-east-1:118421937497:training-job/automl-dm-1631449084uvuarkohwnpd-001-eb259203\",\n",
      "                \"CandidateStepName\": \"automl-dm-1631449084uVUArkOHWnPD-001-eb259203\",\n",
      "                \"CandidateStepType\": \"AWS::SageMaker::TrainingJob\"\n",
      "            }\n",
      "        ],\n",
      "        \"CreationTime\": \"2021-09-12 12:39:03+00:00\",\n",
      "        \"EndTime\": \"2021-09-12 12:40:18+00:00\",\n",
      "        \"FinalAutoMLJobObjectiveMetric\": {\n",
      "            \"MetricName\": \"validation:accuracy\",\n",
      "            \"Value\": 0.596340000629425\n",
      "        },\n",
      "        \"InferenceContainers\": [\n",
      "            {\n",
      "                \"Environment\": {\n",
      "                    \"AUTOML_SPARSE_ENCODE_RECORDIO_PROTOBUF\": \"1\",\n",
      "                    \"AUTOML_TRANSFORM_MODE\": \"feature-transform\",\n",
      "                    \"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT\": \"application/x-recordio-protobuf\",\n",
      "                    \"SAGEMAKER_PROGRAM\": \"sagemaker_serve\",\n",
      "                    \"SAGEMAKER_SUBMIT_DIRECTORY\": \"/opt/ml/model/code\"\n",
      "                },\n",
      "                \"Image\": \"683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-sklearn-automl:2.2.1-1-cpu-py3\",\n",
      "                \"ModelDataUrl\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/data-processor-models/automl-dm-1631449084-dpp0-1-463056e6a4fa496bb36edad681360736942/output/model.tar.gz\"\n",
      "            },\n",
      "            {\n",
      "                \"Environment\": {\n",
      "                    \"MAX_CONTENT_LENGTH\": \"20971520\",\n",
      "                    \"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT\": \"text/csv\",\n",
      "                    \"SAGEMAKER_INFERENCE_OUTPUT\": \"predicted_label\",\n",
      "                    \"SAGEMAKER_INFERENCE_SUPPORTED\": \"predicted_label,probability,probabilities\"\n",
      "                },\n",
      "                \"Image\": \"683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-xgboost:1.2-2-cpu-py3\",\n",
      "                \"ModelDataUrl\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/tuning/automl-dm--dpp0-xgb/automl-dm-1631449084uVUArkOHWnPD-001-eb259203/output/model.tar.gz\"\n",
      "            },\n",
      "            {\n",
      "                \"Environment\": {\n",
      "                    \"AUTOML_TRANSFORM_MODE\": \"inverse-label-transform\",\n",
      "                    \"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT\": \"text/csv\",\n",
      "                    \"SAGEMAKER_INFERENCE_INPUT\": \"predicted_label\",\n",
      "                    \"SAGEMAKER_INFERENCE_OUTPUT\": \"predicted_label\",\n",
      "                    \"SAGEMAKER_INFERENCE_SUPPORTED\": \"predicted_label,probability,labels,probabilities\",\n",
      "                    \"SAGEMAKER_PROGRAM\": \"sagemaker_serve\",\n",
      "                    \"SAGEMAKER_SUBMIT_DIRECTORY\": \"/opt/ml/model/code\"\n",
      "                },\n",
      "                \"Image\": \"683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-sklearn-automl:2.2.1-1-cpu-py3\",\n",
      "                \"ModelDataUrl\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/data-processor-models/automl-dm-1631449084-dpp0-1-463056e6a4fa496bb36edad681360736942/output/model.tar.gz\"\n",
      "            }\n",
      "        ],\n",
      "        \"LastModifiedTime\": \"2021-09-12 12:40:45.737000+00:00\",\n",
      "        \"ObjectiveStatus\": \"Succeeded\"\n",
      "    },\n",
      "    {\n",
      "        \"CandidateName\": \"automl-dm-1631449084uVUArkOHWnPD-003-07853f6c\",\n",
      "        \"CandidateProperties\": {\n",
      "            \"CandidateMetrics\": [\n",
      "                {\n",
      "                    \"MetricName\": \"Accuracy\",\n",
      "                    \"Set\": \"Validation\",\n",
      "                    \"Value\": 0.5938100218772888\n",
      "                },\n",
      "                {\n",
      "                    \"MetricName\": \"F1macro\",\n",
      "                    \"Set\": \"Validation\",\n",
      "                    \"Value\": 0.5918700098991394\n",
      "                }\n",
      "            ]\n",
      "        },\n",
      "        \"CandidateStatus\": \"Completed\",\n",
      "        \"CandidateSteps\": [\n",
      "            {\n",
      "                \"CandidateStepArn\": \"arn:aws:sagemaker:us-east-1:118421937497:processing-job/automl-dm-1631449084-db-1-8e2c5d2b4f33496ca0c2b88b057c2c2afc1f2\",\n",
      "                \"CandidateStepName\": \"automl-dm-1631449084-db-1-8e2c5d2b4f33496ca0c2b88b057c2c2afc1f2\",\n",
      "                \"CandidateStepType\": \"AWS::SageMaker::ProcessingJob\"\n",
      "            },\n",
      "            {\n",
      "                \"CandidateStepArn\": \"arn:aws:sagemaker:us-east-1:118421937497:training-job/automl-dm-1631449084-dpp2-1-c3bd7e163f83404db5169e09305adb44e64\",\n",
      "                \"CandidateStepName\": \"automl-dm-1631449084-dpp2-1-c3bd7e163f83404db5169e09305adb44e64\",\n",
      "                \"CandidateStepType\": \"AWS::SageMaker::TrainingJob\"\n",
      "            },\n",
      "            {\n",
      "                \"CandidateStepArn\": \"arn:aws:sagemaker:us-east-1:118421937497:transform-job/automl-dm-1631449084-dpp2-rpb-1-3275b5cc6dae4f03aa954476eb1325a\",\n",
      "                \"CandidateStepName\": \"automl-dm-1631449084-dpp2-rpb-1-3275b5cc6dae4f03aa954476eb1325a\",\n",
      "                \"CandidateStepType\": \"AWS::SageMaker::TransformJob\"\n",
      "            },\n",
      "            {\n",
      "                \"CandidateStepArn\": \"arn:aws:sagemaker:us-east-1:118421937497:training-job/automl-dm-1631449084uvuarkohwnpd-003-07853f6c\",\n",
      "                \"CandidateStepName\": \"automl-dm-1631449084uVUArkOHWnPD-003-07853f6c\",\n",
      "                \"CandidateStepType\": \"AWS::SageMaker::TrainingJob\"\n",
      "            }\n",
      "        ],\n",
      "        \"CreationTime\": \"2021-09-12 12:39:13+00:00\",\n",
      "        \"EndTime\": \"2021-09-12 12:40:21+00:00\",\n",
      "        \"FinalAutoMLJobObjectiveMetric\": {\n",
      "            \"MetricName\": \"validation:accuracy\",\n",
      "            \"Value\": 0.5938100218772888\n",
      "        },\n",
      "        \"InferenceContainers\": [\n",
      "            {\n",
      "                \"Environment\": {\n",
      "                    \"AUTOML_SPARSE_ENCODE_RECORDIO_PROTOBUF\": \"1\",\n",
      "                    \"AUTOML_TRANSFORM_MODE\": \"feature-transform\",\n",
      "                    \"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT\": \"application/x-recordio-protobuf\",\n",
      "                    \"SAGEMAKER_PROGRAM\": \"sagemaker_serve\",\n",
      "                    \"SAGEMAKER_SUBMIT_DIRECTORY\": \"/opt/ml/model/code\"\n",
      "                },\n",
      "                \"Image\": \"683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-sklearn-automl:2.2.1-1-cpu-py3\",\n",
      "                \"ModelDataUrl\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/data-processor-models/automl-dm-1631449084-dpp2-1-c3bd7e163f83404db5169e09305adb44e64/output/model.tar.gz\"\n",
      "            },\n",
      "            {\n",
      "                \"Environment\": {\n",
      "                    \"MAX_CONTENT_LENGTH\": \"20971520\",\n",
      "                    \"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT\": \"text/csv\",\n",
      "                    \"SAGEMAKER_INFERENCE_OUTPUT\": \"predicted_label\",\n",
      "                    \"SAGEMAKER_INFERENCE_SUPPORTED\": \"predicted_label,probability,probabilities\"\n",
      "                },\n",
      "                \"Image\": \"683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-xgboost:1.2-2-cpu-py3\",\n",
      "                \"ModelDataUrl\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/tuning/automl-dm--dpp2-xgb/automl-dm-1631449084uVUArkOHWnPD-003-07853f6c/output/model.tar.gz\"\n",
      "            },\n",
      "            {\n",
      "                \"Environment\": {\n",
      "                    \"AUTOML_TRANSFORM_MODE\": \"inverse-label-transform\",\n",
      "                    \"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT\": \"text/csv\",\n",
      "                    \"SAGEMAKER_INFERENCE_INPUT\": \"predicted_label\",\n",
      "                    \"SAGEMAKER_INFERENCE_OUTPUT\": \"predicted_label\",\n",
      "                    \"SAGEMAKER_INFERENCE_SUPPORTED\": \"predicted_label,probability,labels,probabilities\",\n",
      "                    \"SAGEMAKER_PROGRAM\": \"sagemaker_serve\",\n",
      "                    \"SAGEMAKER_SUBMIT_DIRECTORY\": \"/opt/ml/model/code\"\n",
      "                },\n",
      "                \"Image\": \"683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-sklearn-automl:2.2.1-1-cpu-py3\",\n",
      "                \"ModelDataUrl\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/data-processor-models/automl-dm-1631449084-dpp2-1-c3bd7e163f83404db5169e09305adb44e64/output/model.tar.gz\"\n",
      "            }\n",
      "        ],\n",
      "        \"LastModifiedTime\": \"2021-09-12 12:40:45.653000+00:00\",\n",
      "        \"ObjectiveStatus\": \"Succeeded\"\n",
      "    },\n",
      "    {\n",
      "        \"CandidateName\": \"automl-dm-1631449084uVUArkOHWnPD-002-9dddc734\",\n",
      "        \"CandidateProperties\": {\n",
      "            \"CandidateMetrics\": [\n",
      "                {\n",
      "                    \"MetricName\": \"Accuracy\",\n",
      "                    \"Set\": \"Validation\",\n",
      "                    \"Value\": 0.428600013256073\n",
      "                },\n",
      "                {\n",
      "                    \"MetricName\": \"F1macro\",\n",
      "                    \"Set\": \"Validation\",\n",
      "                    \"Value\": 0.4290100038051605\n",
      "                }\n",
      "            ]\n",
      "        },\n",
      "        \"CandidateStatus\": \"Completed\",\n",
      "        \"CandidateSteps\": [\n",
      "            {\n",
      "                \"CandidateStepArn\": \"arn:aws:sagemaker:us-east-1:118421937497:processing-job/automl-dm-1631449084-db-1-8e2c5d2b4f33496ca0c2b88b057c2c2afc1f2\",\n",
      "                \"CandidateStepName\": \"automl-dm-1631449084-db-1-8e2c5d2b4f33496ca0c2b88b057c2c2afc1f2\",\n",
      "                \"CandidateStepType\": \"AWS::SageMaker::ProcessingJob\"\n",
      "            },\n",
      "            {\n",
      "                \"CandidateStepArn\": \"arn:aws:sagemaker:us-east-1:118421937497:training-job/automl-dm-1631449084-dpp1-1-f01a6f5da7fa4be1bda1abf2215cfac0cf3\",\n",
      "                \"CandidateStepName\": \"automl-dm-1631449084-dpp1-1-f01a6f5da7fa4be1bda1abf2215cfac0cf3\",\n",
      "                \"CandidateStepType\": \"AWS::SageMaker::TrainingJob\"\n",
      "            },\n",
      "            {\n",
      "                \"CandidateStepArn\": \"arn:aws:sagemaker:us-east-1:118421937497:transform-job/automl-dm-1631449084-dpp1-csv-1-6bc173285d60439f87b873dc87153a6\",\n",
      "                \"CandidateStepName\": \"automl-dm-1631449084-dpp1-csv-1-6bc173285d60439f87b873dc87153a6\",\n",
      "                \"CandidateStepType\": \"AWS::SageMaker::TransformJob\"\n",
      "            },\n",
      "            {\n",
      "                \"CandidateStepArn\": \"arn:aws:sagemaker:us-east-1:118421937497:training-job/automl-dm-1631449084uvuarkohwnpd-002-9dddc734\",\n",
      "                \"CandidateStepName\": \"automl-dm-1631449084uVUArkOHWnPD-002-9dddc734\",\n",
      "                \"CandidateStepType\": \"AWS::SageMaker::TrainingJob\"\n",
      "            }\n",
      "        ],\n",
      "        \"CreationTime\": \"2021-09-12 12:39:12+00:00\",\n",
      "        \"EndTime\": \"2021-09-12 12:40:24+00:00\",\n",
      "        \"FinalAutoMLJobObjectiveMetric\": {\n",
      "            \"MetricName\": \"validation:accuracy\",\n",
      "            \"Value\": 0.428600013256073\n",
      "        },\n",
      "        \"InferenceContainers\": [\n",
      "            {\n",
      "                \"Environment\": {\n",
      "                    \"AUTOML_TRANSFORM_MODE\": \"feature-transform\",\n",
      "                    \"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT\": \"application/x-recordio-protobuf\",\n",
      "                    \"SAGEMAKER_PROGRAM\": \"sagemaker_serve\",\n",
      "                    \"SAGEMAKER_SUBMIT_DIRECTORY\": \"/opt/ml/model/code\"\n",
      "                },\n",
      "                \"Image\": \"683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-sklearn-automl:2.2.1-1-cpu-py3\",\n",
      "                \"ModelDataUrl\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/data-processor-models/automl-dm-1631449084-dpp1-1-f01a6f5da7fa4be1bda1abf2215cfac0cf3/output/model.tar.gz\"\n",
      "            },\n",
      "            {\n",
      "                \"Environment\": {\n",
      "                    \"MAX_CONTENT_LENGTH\": \"20971520\",\n",
      "                    \"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT\": \"text/csv\",\n",
      "                    \"SAGEMAKER_INFERENCE_OUTPUT\": \"predicted_label\",\n",
      "                    \"SAGEMAKER_INFERENCE_SUPPORTED\": \"predicted_label,probability,probabilities\"\n",
      "                },\n",
      "                \"Image\": \"683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-xgboost:1.2-2-cpu-py3\",\n",
      "                \"ModelDataUrl\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/tuning/automl-dm--dpp1-xgb/automl-dm-1631449084uVUArkOHWnPD-002-9dddc734/output/model.tar.gz\"\n",
      "            },\n",
      "            {\n",
      "                \"Environment\": {\n",
      "                    \"AUTOML_TRANSFORM_MODE\": \"inverse-label-transform\",\n",
      "                    \"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT\": \"text/csv\",\n",
      "                    \"SAGEMAKER_INFERENCE_INPUT\": \"predicted_label\",\n",
      "                    \"SAGEMAKER_INFERENCE_OUTPUT\": \"predicted_label\",\n",
      "                    \"SAGEMAKER_INFERENCE_SUPPORTED\": \"predicted_label,probability,labels,probabilities\",\n",
      "                    \"SAGEMAKER_PROGRAM\": \"sagemaker_serve\",\n",
      "                    \"SAGEMAKER_SUBMIT_DIRECTORY\": \"/opt/ml/model/code\"\n",
      "                },\n",
      "                \"Image\": \"683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-sklearn-automl:2.2.1-1-cpu-py3\",\n",
      "                \"ModelDataUrl\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/data-processor-models/automl-dm-1631449084-dpp1-1-f01a6f5da7fa4be1bda1abf2215cfac0cf3/output/model.tar.gz\"\n",
      "            }\n",
      "        ],\n",
      "        \"LastModifiedTime\": \"2021-09-12 12:40:45.653000+00:00\",\n",
      "        \"ObjectiveStatus\": \"Succeeded\"\n",
      "    }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(candidates, indent=4, sort_keys=True, default=str))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can print the names of the candidates with their metric values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "metric validation:accuracy\n",
      "0  automl-dm-1631449084uVUArkOHWnPD-001-eb259203  0.596340000629425\n",
      "1  automl-dm-1631449084uVUArkOHWnPD-003-07853f6c  0.5938100218772888\n",
      "2  automl-dm-1631449084uVUArkOHWnPD-002-9dddc734  0.428600013256073\n"
     ]
    }
   ],
   "source": [
    "print(\"metric \" + str(candidates[0]['FinalAutoMLJobObjectiveMetric']['MetricName']))\n",
    "\n",
    "for index, candidate in enumerate(candidates):\n",
    "    print(str(index) + \"  \" \n",
    "        + candidate['CandidateName'] + \"  \" \n",
    "        + str(candidate['FinalAutoMLJobObjectiveMetric']['Value']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-6.3.'></a>\n",
    "### 6.3. Review best candidate\n",
    "\n",
    "Now that you have successfully completed the Autopilot job on the dataset and visualized the trials, you can get the information about the best candidate model and review it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-ex-8'></a>\n",
    "### Exercise 8\n",
    "\n",
    "Get the information about the generated best candidate job. \n",
    "\n",
    "**Instructions**: Use `best_candidate` function passing the Autopilot job name. This function will give an error if candidates have not been generated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"CandidateName\": \"automl-dm-1631449084uVUArkOHWnPD-001-eb259203\",\n",
      "    \"CandidateProperties\": {\n",
      "        \"CandidateArtifactLocations\": {\n",
      "            \"Explainability\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/documentation/explainability/output\"\n",
      "        },\n",
      "        \"CandidateMetrics\": [\n",
      "            {\n",
      "                \"MetricName\": \"Accuracy\",\n",
      "                \"Set\": \"Validation\",\n",
      "                \"Value\": 0.596340000629425\n",
      "            },\n",
      "            {\n",
      "                \"MetricName\": \"F1macro\",\n",
      "                \"Set\": \"Validation\",\n",
      "                \"Value\": 0.5948699712753296\n",
      "            }\n",
      "        ]\n",
      "    },\n",
      "    \"CandidateStatus\": \"Completed\",\n",
      "    \"CandidateSteps\": [\n",
      "        {\n",
      "            \"CandidateStepArn\": \"arn:aws:sagemaker:us-east-1:118421937497:processing-job/automl-dm-1631449084-db-1-8e2c5d2b4f33496ca0c2b88b057c2c2afc1f2\",\n",
      "            \"CandidateStepName\": \"automl-dm-1631449084-db-1-8e2c5d2b4f33496ca0c2b88b057c2c2afc1f2\",\n",
      "            \"CandidateStepType\": \"AWS::SageMaker::ProcessingJob\"\n",
      "        },\n",
      "        {\n",
      "            \"CandidateStepArn\": \"arn:aws:sagemaker:us-east-1:118421937497:training-job/automl-dm-1631449084-dpp0-1-463056e6a4fa496bb36edad681360736942\",\n",
      "            \"CandidateStepName\": \"automl-dm-1631449084-dpp0-1-463056e6a4fa496bb36edad681360736942\",\n",
      "            \"CandidateStepType\": \"AWS::SageMaker::TrainingJob\"\n",
      "        },\n",
      "        {\n",
      "            \"CandidateStepArn\": \"arn:aws:sagemaker:us-east-1:118421937497:transform-job/automl-dm-1631449084-dpp0-rpb-1-ed5c4f47891746f7b74089486c2c50b\",\n",
      "            \"CandidateStepName\": \"automl-dm-1631449084-dpp0-rpb-1-ed5c4f47891746f7b74089486c2c50b\",\n",
      "            \"CandidateStepType\": \"AWS::SageMaker::TransformJob\"\n",
      "        },\n",
      "        {\n",
      "            \"CandidateStepArn\": \"arn:aws:sagemaker:us-east-1:118421937497:training-job/automl-dm-1631449084uvuarkohwnpd-001-eb259203\",\n",
      "            \"CandidateStepName\": \"automl-dm-1631449084uVUArkOHWnPD-001-eb259203\",\n",
      "            \"CandidateStepType\": \"AWS::SageMaker::TrainingJob\"\n",
      "        }\n",
      "    ],\n",
      "    \"CreationTime\": \"2021-09-12 12:39:03+00:00\",\n",
      "    \"EndTime\": \"2021-09-12 12:40:18+00:00\",\n",
      "    \"FinalAutoMLJobObjectiveMetric\": {\n",
      "        \"MetricName\": \"validation:accuracy\",\n",
      "        \"Value\": 0.596340000629425\n",
      "    },\n",
      "    \"InferenceContainers\": [\n",
      "        {\n",
      "            \"Environment\": {\n",
      "                \"AUTOML_SPARSE_ENCODE_RECORDIO_PROTOBUF\": \"1\",\n",
      "                \"AUTOML_TRANSFORM_MODE\": \"feature-transform\",\n",
      "                \"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT\": \"application/x-recordio-protobuf\",\n",
      "                \"SAGEMAKER_PROGRAM\": \"sagemaker_serve\",\n",
      "                \"SAGEMAKER_SUBMIT_DIRECTORY\": \"/opt/ml/model/code\"\n",
      "            },\n",
      "            \"Image\": \"683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-sklearn-automl:2.2.1-1-cpu-py3\",\n",
      "            \"ModelDataUrl\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/data-processor-models/automl-dm-1631449084-dpp0-1-463056e6a4fa496bb36edad681360736942/output/model.tar.gz\"\n",
      "        },\n",
      "        {\n",
      "            \"Environment\": {\n",
      "                \"MAX_CONTENT_LENGTH\": \"20971520\",\n",
      "                \"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT\": \"text/csv\",\n",
      "                \"SAGEMAKER_INFERENCE_OUTPUT\": \"predicted_label\",\n",
      "                \"SAGEMAKER_INFERENCE_SUPPORTED\": \"predicted_label,probability,probabilities\"\n",
      "            },\n",
      "            \"Image\": \"683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-xgboost:1.2-2-cpu-py3\",\n",
      "            \"ModelDataUrl\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/tuning/automl-dm--dpp0-xgb/automl-dm-1631449084uVUArkOHWnPD-001-eb259203/output/model.tar.gz\"\n",
      "        },\n",
      "        {\n",
      "            \"Environment\": {\n",
      "                \"AUTOML_TRANSFORM_MODE\": \"inverse-label-transform\",\n",
      "                \"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT\": \"text/csv\",\n",
      "                \"SAGEMAKER_INFERENCE_INPUT\": \"predicted_label\",\n",
      "                \"SAGEMAKER_INFERENCE_OUTPUT\": \"predicted_label\",\n",
      "                \"SAGEMAKER_INFERENCE_SUPPORTED\": \"predicted_label,probability,labels,probabilities\",\n",
      "                \"SAGEMAKER_PROGRAM\": \"sagemaker_serve\",\n",
      "                \"SAGEMAKER_SUBMIT_DIRECTORY\": \"/opt/ml/model/code\"\n",
      "            },\n",
      "            \"Image\": \"683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-sklearn-automl:2.2.1-1-cpu-py3\",\n",
      "            \"ModelDataUrl\": \"s3://sagemaker-us-east-1-118421937497/autopilot/automl-dm-1631449084/data-processor-models/automl-dm-1631449084-dpp0-1-463056e6a4fa496bb36edad681360736942/output/model.tar.gz\"\n",
      "        }\n",
      "    ],\n",
      "    \"LastModifiedTime\": \"2021-09-12 12:40:45.737000+00:00\",\n",
      "    \"ObjectiveStatus\": \"Succeeded\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "candidates = automl.list_candidates(job_name=auto_ml_job_name)\n",
    "\n",
    "if candidates != []:\n",
    "    best_candidate = automl.best_candidate(\n",
    "        ### BEGIN SOLUTION - DO NOT delete this comment for grading purposes\n",
    "        job_name=auto_ml_job_name # Replace None\n",
    "        ### END SOLUTION - DO NOT delete this comment for grading purposes\n",
    "    )\n",
    "    print(json.dumps(best_candidate, indent=4, sort_keys=True, default=str))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the existence of the candidate name for the best candidate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[OK] BestCandidate CandidateName generated.\n"
     ]
    }
   ],
   "source": [
    "while 'CandidateName' not in best_candidate:\n",
    "    best_candidate = automl.best_candidate(job_name=auto_ml_job_name)\n",
    "    print('[INFO] Autopilot Job is generating BestCandidate CandidateName. Please wait. ')\n",
    "    print(json.dumps(best_candidate, indent=4, sort_keys=True, default=str))\n",
    "    sleep(10)\n",
    "\n",
    "print('[OK] BestCandidate CandidateName generated.')  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the existence of the metric value for the best candidate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[OK] BestCandidate FinalAutoMLJobObjectiveMetric generated.\n"
     ]
    }
   ],
   "source": [
    "while 'FinalAutoMLJobObjectiveMetric' not in best_candidate:\n",
    "    best_candidate = automl.best_candidate(job_name=auto_ml_job_name)\n",
    "    print('[INFO] Autopilot Job is generating BestCandidate FinalAutoMLJobObjectiveMetric. Please wait. ')\n",
    "    print(json.dumps(best_candidate, indent=4, sort_keys=True, default=str))\n",
    "    sleep(10)\n",
    "\n",
    "print('[OK] BestCandidate FinalAutoMLJobObjectiveMetric generated.')  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the information about the best candidate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Candidate name: automl-dm-1631449084uVUArkOHWnPD-001-eb259203\n",
      "Metric name: validation:accuracy\n",
      "Metric value: 0.596340000629425\n"
     ]
    }
   ],
   "source": [
    "best_candidate_identifier = best_candidate['CandidateName']\n",
    "print(\"Candidate name: \" + best_candidate_identifier)\n",
    "print(\"Metric name: \" + best_candidate['FinalAutoMLJobObjectiveMetric']['MetricName'])\n",
    "print(\"Metric value: \" + str(best_candidate['FinalAutoMLJobObjectiveMetric']['Value']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-7.'></a>\n",
    "# 7. Review all output in S3 bucket\n",
    "\n",
    "You will see the artifacts generated by Autopilot including the following:\n",
    "```\n",
    "data-processor-models/        # \"models\" learned to transform raw data into features \n",
    "documentation/                # explainability and other documentation about your model\n",
    "preprocessed-data/            # data for train and validation\n",
    "sagemaker-automl-candidates/  # candidate models which autopilot compares\n",
    "transformed-data/             # candidate-specific data for train and validation\n",
    "tuning/                       # candidate-specific tuning results\n",
    "validations/                  # validation results\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b>Review all <a target=\"blank\" href=\"https://s3.console.aws.amazon.com/s3/buckets/sagemaker-us-east-1-118421937497?region=us-east-1&prefix=autopilot/automl-dm-1631449084/\">output in S3</a></b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "display(\n",
    "    HTML(\n",
    "        '<b>Review all <a target=\"blank\" href=\"https://s3.console.aws.amazon.com/s3/buckets/{}?region={}&prefix=autopilot/{}/\">output in S3</a></b>'.format(\n",
    "            bucket, region, auto_ml_job_name\n",
    "        )\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-8.'></a>\n",
    "# 8. Deploy and test best candidate model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-8.1.'></a>\n",
    "### 8.1. Deploy best candidate model\n",
    "\n",
    "While batch transformations are supported, you will deploy our model as a REST Endpoint in this example.\n",
    "\n",
    "First, you need to customize the inference response. The inference containers generated by SageMaker Autopilot allow you to select the response content for predictions. By default the inference containers are configured to generate the `predicted_label`. But you can add `probability` into the list of inference response keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "inference_response_keys = ['predicted_label', 'probability']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you will create a SageMaker endpoint from the best candidate generated by Autopilot. Wait for SageMaker to deploy the endpoint.\n",
    "\n",
    "### _This cell will take approximately 5-10 minutes to run._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------!\n",
      "Endpoint name:  sagemaker-sklearn-automl-2021-09-12-12-48-28-953\n"
     ]
    }
   ],
   "source": [
    "autopilot_model = automl.deploy(\n",
    "    initial_instance_count=1,\n",
    "    instance_type='ml.m5.large',\n",
    "    candidate=best_candidate,\n",
    "    inference_response_keys=inference_response_keys,\n",
    "    predictor_cls=sagemaker.predictor.Predictor,\n",
    "    serializer=sagemaker.serializers.JSONSerializer(),\n",
    "    deserializer=sagemaker.deserializers.JSONDeserializer()\n",
    ")\n",
    "\n",
    "print('\\nEndpoint name:  {}'.format(autopilot_model.endpoint_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Please wait until the ^^ endpoint ^^ is deployed._ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Review the SageMaker endpoint in the AWS console."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/endpoints/sagemaker-sklearn-automl-2021-09-12-12-48-28-953\">SageMaker REST endpoint</a></b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "display(HTML('<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/endpoints/{}\">SageMaker REST endpoint</a></b>'.format(region, autopilot_model.endpoint_name)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='c1w3-8.2.'></a>\n",
    "### 8.2. Test the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Invoke a few predictions for the actual reviews using the deployed endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review:  This product is great!  Predicated class: 1\n",
      "Review:  OK but not great.  Predicated class: 1\n",
      "Review:  This is not the right product.  Predicated class: -1\n",
      "(-1 = Negative, 0=Neutral, 1=Positive)\n"
     ]
    }
   ],
   "source": [
    "#sm_runtime = boto3.client('sagemaker-runtime')\n",
    "\n",
    "review_list = ['This product is great!',\n",
    "               'OK, but not great.',\n",
    "               'This is not the right product.']\n",
    "\n",
    "for review in review_list:\n",
    "    \n",
    "    # remove commas from the review since we're passing the inputs as a CSV\n",
    "    review = review.replace(\",\", \"\")\n",
    "\n",
    "    response = sm_runtime.invoke_endpoint(\n",
    "        EndpointName=autopilot_model.endpoint_name, # endpoint name\n",
    "        ContentType='text/csv', # type of input data\n",
    "        Accept='text/csv', # type of the inference in the response\n",
    "        Body=review # review text\n",
    "        )\n",
    "\n",
    "    response_body=response['Body'].read().decode('utf-8').strip().split(',')\n",
    "\n",
    "    print('Review: ', review, ' Predicated class: {}'.format(response_body[0]))\n",
    "\n",
    "print(\"(-1 = Negative, 0=Neutral, 1=Positive)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You used Amazon SageMaker Autopilot to automatically find the best model, hyper-parameters, and feature-engineering scripts for our dataset. Autopilot uses a uniquely-transparent approach to AutoML by generating re-usable Python scripts and notebooks.\n",
    "\n",
    "Upload the notebook into S3 bucket for grading purposes.\n",
    "\n",
    "**Note:** you may need to click on \"Save\" button before the upload."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "upload: ./C1_W3_Assignment.ipynb to s3://sagemaker-us-east-1-118421937497/C1_W3_Assignment_Learner.ipynb\n"
     ]
    }
   ],
   "source": [
    "!aws s3 cp ./C1_W3_Assignment.ipynb s3://$bucket/C1_W3_Assignment_Learner.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please go to the main lab window and click on `Submit` button (see the `Finish the lab` section of the instructions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}