autopilot/autopilot_california_housing.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Housing Price Prediction with Amazon SageMaker Autopilot\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/autopilot|autopilot_california_housing.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_**Using Autopilot to Predict House Prices in California**_\n",
    "\n",
    "\n",
    "Kernel `Python 3 (Data Science)` works well with this notebook. You will have the best experience running this within SageMaker Studio.\n",
    "\n",
    "---\n",
    "\n",
    "## Contents\n",
    "\n",
    "1. [Introduction](#Introduction)\n",
    "1. [Setup](#Setup)\n",
    "1. [Prepare Training Data](#Data)\n",
    "1. [Train](#Settingup)\n",
    "1. [Autopilot Results](#Results)\n",
    "1. [Evaluate Using Test Data](#Evaluate)\n",
    "1. [Cleanup](#Cleanup)\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "## Introduction\n",
    "\n",
    "Amazon SageMaker Autopilot is an automated machine learning (commonly referred to as AutoML) solution for tabular datasets. You can use SageMaker Autopilot in different ways: on autopilot (without any human input) or with human guidance, without code through SageMaker Studio or scripted using the AWS SDKs. This notebook will use the AWS SDKs to simply create and deploy a machine learning model without doing any feature engineering manually. We will also explore the auto-generated feature importance report.\n",
    "\n",
    "Predicting housing prices is a classic linear regression problem in ML. The data pertains to the houses found in a given California district (as a group) and include summary stats about them based on census data from 1990. The dataset variables are easily understandable, and the columns are as follows:\n",
    "\n",
    "* ```longitude```\n",
    "* ```latitude```\n",
    "* ```housingMedianAge```\n",
    "* ```totalRooms```\n",
    "* ```totalBedrooms```\n",
    "* ```population```\n",
    "* ```households```\n",
    "* ```medianIncome```\n",
    "* ```medianHouseValue``` (target)\n",
    "\n",
    "What we're going to try to predict is the median house value for a district. We will let Autopilot perform feature engineering, model selection, model tuning, and give us the best candidate model ready to use for inferences.\n",
    "\n",
    "---\n",
    "## Setup\n",
    "\n",
    "_This notebook was created and tested on a ml.t3.medium notebook instance._\n",
    "\n",
    "Let's start by specifying:\n",
    "\n",
    "- The S3 bucket and prefix to use for training and model data.  This should be within the same region as the Notebook Instance, training, and hosting. The following code will use SageMaker's default S3 bucket (and create one if it doesn't exist).\n",
    "- The IAM role ARN used to give training and hosting access to your data. See the documentation for how to create these. The following code will use the SageMaker execution role."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "isConfigCell": true,
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import boto3\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "region = boto3.Session().region_name\n",
    "\n",
    "session = sagemaker.Session()\n",
    "\n",
    "# You can modify the following to use a bucket of your choosing\n",
    "bucket = session.default_bucket()\n",
    "prefix = \"sagemaker/DEMO-autopilot-housing\"\n",
    "\n",
    "role = get_execution_role()\n",
    "\n",
    "# This is the client we will use to interact with SageMaker Autopilot\n",
    "sm = boto3.Session().client(service_name=\"sagemaker\", region_name=region)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll import the Python libraries we'll need for the remainder of the exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import io\n",
    "from urllib.parse import urlparse\n",
    "import time"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Prepare Training Data<a name=\"Data\"></a>\n",
    "\n",
    "We will use the publicly available California housing dataset. The target variable is the median house value for California districts.\n",
    "\n",
    "This dataset was obtained from the StatLib repository (http://lib.stat.cmu.edu/datasets/) and derived from the 1990 U.S. census, using one row per census block group. A block group is the smallest geographical unit for which the U.S. Census Bureau publishes sample data (a block group typically has a population of 600 to 3,000 people)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3 = boto3.client(\"s3\")\n",
    "s3.download_file(\n",
    "    f\"sagemaker-example-files-prod-{region}\",\n",
    "    \"datasets/tabular/california_housing/cal_housing.tgz\",\n",
    "    \"cal_housing.tgz\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -zxf cal_housing.tgz -o"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "columns = [\n",
    "    \"longitude\",\n",
    "    \"latitude\",\n",
    "    \"housingMedianAge\",\n",
    "    \"totalRooms\",\n",
    "    \"totalBedrooms\",\n",
    "    \"population\",\n",
    "    \"households\",\n",
    "    \"medianIncome\",\n",
    "    \"medianHouseValue\",\n",
    "]\n",
    "\n",
    "target = \"medianHouseValue\"\n",
    "\n",
    "cal_housing_df = pd.read_csv(\"CaliforniaHousing/cal_housing.data\", names=columns, header=None)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before starting an Autopilot job, it is good practice to inspect the data to ensure there are no obvious issues with it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option(\"display.max_columns\", 500)\n",
    "cal_housing_df"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the various columns have values in different ranges. For example, values under the target column ```medianHouseValue``` are orders of magnitude higher than other columns. This difference in scale often causes issues when training an ML model, which is why it's a common feature engineering practice to normalize or standardize values (depending on the nature of the numeric distribution and presence of outliers).\n",
    "\n",
    "However, because Autopilot handles feature engineering automatically (among other things), we're not going to make any analysis or transformations ourselves.\n",
    "\n",
    "To also illustrate how Autopilot handles data issues, such as missing values, let's introduce some random empty values in our dataset for the housing median age column.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Take a small, random sample of the dataset\n",
    "cal_housing_df_sample = cal_housing_df.sample(frac=0.01, random_state=100)\n",
    "\n",
    "cal_housing_df[\"housingMedianAge\"].loc[cal_housing_df_sample.index] = (\n",
    "    cal_housing_df[\"housingMedianAge\"].loc[cal_housing_df_sample.index].apply(lambda row: np.nan)\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now our dataset should contain empty values for ~1% of the rows under the ```housingMedianAge``` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "isna_sum = cal_housing_df[\"housingMedianAge\"].isna().sum()\n",
    "print(\"{0} values removed from housingMedianAge column\".format(isna_sum))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split data for training and testing (80/20 split)\n",
    "\n",
    "train_data = cal_housing_df.sample(frac=0.8, random_state=200)\n",
    "\n",
    "test_data = cal_housing_df.drop(train_data.index)\n",
    "\n",
    "test_data_no_target = test_data.drop(columns=[target])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Upload the dataset to S3\n",
    "\n",
    "train_file = \"train_data.csv\"\n",
    "train_data.to_csv(train_file, index=False, header=True)\n",
    "train_data_s3_path = session.upload_data(path=train_file, key_prefix=prefix + \"/train\")\n",
    "print(\"Train data uploaded to: \" + train_data_s3_path)\n",
    "\n",
    "test_file = \"test_data_no_target.csv\"\n",
    "test_data_no_target.to_csv(test_file, index=False, header=False)\n",
    "test_data_s3_path = session.upload_data(path=test_file, key_prefix=prefix + \"/test\")\n",
    "print(\"Test data uploaded to: \" + test_data_s3_path)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Setting up the SageMaker Autopilot Job<a name=\"Settingup\"></a>\n",
    "\n",
    "After uploading the dataset to Amazon S3, you can invoke Autopilot to find the best ML pipeline to train a model on this dataset. \n",
    "\n",
    "The required inputs for invoking an Autopilot job are:\n",
    "* Amazon S3 location for input dataset and for all output artifacts\n",
    "* Name of the target column in the dataset for predictions \n",
    "* An IAM role\n",
    "\n",
    "Currently, Autopilot supports only tabular datasets in CSV format. Either all files should have a header row, or the first file of the dataset, when sorted in alphabetical/lexical order by name, is expected to have a header row.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_data_config = [\n",
    "    {\n",
    "        \"DataSource\": {\n",
    "            \"S3DataSource\": {\n",
    "                \"S3DataType\": \"S3Prefix\",\n",
    "                \"S3Uri\": \"s3://{}/{}/train\".format(bucket, prefix),\n",
    "            }\n",
    "        },\n",
    "        \"TargetAttributeName\": target,\n",
    "    }\n",
    "]\n",
    "\n",
    "job_config = {\"CompletionCriteria\": {\"MaxCandidates\": 10}}\n",
    "\n",
    "\n",
    "output_data_config = {\"S3OutputPath\": \"s3://{}/{}/output\".format(bucket, prefix)}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also specify the type of problem you want to solve with your dataset (`Regression, MulticlassClassification, BinaryClassification`). In case you are not sure, SageMaker Autopilot will infer the problem type based on statistics of the target column (the column you want to predict). \n",
    "\n",
    "Because the target attribute, ```medianHouseValue```, is a continuous numeric variable, Autopilot will infer the problem type as regression. \n",
    "\n",
    "You have the option to limit the running time of a SageMaker Autopilot job by providing either the maximum number of pipeline evaluations or candidates (one pipeline evaluation is called a `Candidate` because it generates a candidate model) or providing the total time allocated for the overall Autopilot job. Under default settings, this job may take several hours to run. This varies between runs because of the nature of the exploratory process Autopilot uses to find optimal training parameters.\n",
    "\n",
    "For this demo, we limit the number of candidates to 10 so that the job finishes in under 1 hour.\n",
    "\n",
    "Finally, you also have the option to deploy the winning model to a SageMaker endpoint automatically upon completion. In this case, we will not deploy the endpoint. We'll run a batch prediction job later instead to evaluate our model.\n",
    "\n",
    "For guidance on how to configure the job parameters, check out the SDK documentation."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Launching the SageMaker Autopilot Job<a name=\"Launching\"></a>\n",
    "\n",
    "You can now launch the Autopilot job by calling the `create_auto_ml_job` API. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import gmtime, strftime, sleep\n",
    "\n",
    "timestamp_suffix = strftime(\"%Y%m%d-%H-%M\", gmtime())\n",
    "\n",
    "auto_ml_job_name = \"automl-housing-\" + timestamp_suffix\n",
    "print(\"AutoMLJobName: \" + auto_ml_job_name)\n",
    "\n",
    "sm.create_auto_ml_job(\n",
    "    AutoMLJobName=auto_ml_job_name,\n",
    "    InputDataConfig=input_data_config,\n",
    "    OutputDataConfig=output_data_config,\n",
    "    AutoMLJobConfig=job_config,\n",
    "    # Uncomment to automatically deploy an endpoint\n",
    "    # ModelDeployConfig={\n",
    "    #'AutoGenerateEndpointName': True,\n",
    "    #'EndpointName': 'autopilot-DEMO-housing-' + timestamp_suffix\n",
    "    # },\n",
    "    RoleArn=role,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "The Autopilot job will now be performing the following steps:\n",
    "\n",
    "* Data Analysis\n",
    "* Feature Engineering\n",
    "* Model selection\n",
    "* Model tuning (hyperparameter optimization)\n",
    "* Model feature importance (SageMaker Clarify)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tracking SageMaker Autopilot job progress"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"JobStatus - Secondary Status\")\n",
    "print(\"------------------------------\")\n",
    "\n",
    "\n",
    "describe_response = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "print(describe_response[\"AutoMLJobStatus\"] + \" - \" + describe_response[\"AutoMLJobSecondaryStatus\"])\n",
    "job_run_status = describe_response[\"AutoMLJobStatus\"]\n",
    "\n",
    "while job_run_status not in (\"Failed\", \"Completed\", \"Stopped\"):\n",
    "    describe_response = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "    job_run_status = describe_response[\"AutoMLJobStatus\"]\n",
    "\n",
    "    print(\n",
    "        describe_response[\"AutoMLJobStatus\"] + \" - \" + describe_response[\"AutoMLJobSecondaryStatus\"]\n",
    "    )\n",
    "    sleep(60)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true
   },
   "source": [
    "---\n",
    "## Results\n",
    "\n",
    "Now you can use the ```describe_auto_ml_job``` API to look up the best candidate selected by the SageMaker Autopilot job. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_candidate = sm.describe_auto_ml_job(AutoMLJobName=auto_ml_job_name)[\"BestCandidate\"]\n",
    "best_candidate_name = best_candidate[\"CandidateName\"]\n",
    "\n",
    "print(\"\\n\")\n",
    "print(\"CandidateName: \" + best_candidate_name)\n",
    "print(\n",
    "    \"FinalAutoMLJobObjectiveMetricName: \"\n",
    "    + best_candidate[\"FinalAutoMLJobObjectiveMetric\"][\"MetricName\"]\n",
    ")\n",
    "print(\n",
    "    \"FinalAutoMLJobObjectiveMetricValue: \"\n",
    "    + str(best_candidate[\"FinalAutoMLJobObjectiveMetric\"][\"Value\"])\n",
    ")\n",
    "print(\"\\nBest candidate details:: \" + str(best_candidate))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are curious to explore the performance of other algorithms that Autopilot explored, you can enumerate them via ```list_candidates_for_auto_ml_job``` API call"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sm_dict = sm.list_candidates_for_auto_ml_job(AutoMLJobName=auto_ml_job_name)\n",
    "for item in sm_dict[\"Candidates\"]:\n",
    "    print(item[\"CandidateName\"], item[\"FinalAutoMLJobObjectiveMetric\"])\n",
    "    print(item[\"InferenceContainers\"][1][\"Image\"], \"\\n\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "Autopilot automatically generates two executable Jupyter Notebooks:  \n",
    "\n",
    "- ```SageMakerAutopilotDataExplorationNotebook.ipynb```\n",
    "- ```SageMakerAutopilotCandidateDefinitionNotebook.ipynb```\n",
    "\n",
    "These notebooks are stored in S3. Let us download them onto our SageMaker Notebook instance, so we can explore them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "candidate_nbk = describe_response[\"AutoMLJobArtifacts\"][\"CandidateDefinitionNotebookLocation\"]\n",
    "data_explore_nbk = describe_response[\"AutoMLJobArtifacts\"][\"DataExplorationNotebookLocation\"]\n",
    "\n",
    "\n",
    "def split_s3_path(s3_path):\n",
    "    path_parts = s3_path.replace(\"s3://\", \"\").split(\"/\")\n",
    "    bucket = path_parts.pop(0)\n",
    "    key = \"/\".join(path_parts)\n",
    "    return bucket, key\n",
    "\n",
    "\n",
    "s3_bucket, candidate_nbk_key = split_s3_path(candidate_nbk)\n",
    "_, data_explore_nbk_key = split_s3_path(data_explore_nbk)\n",
    "\n",
    "print(s3_bucket, candidate_nbk_key, data_explore_nbk_key)\n",
    "\n",
    "session.download_data(path=\"./\", bucket=s3_bucket, key_prefix=candidate_nbk_key)\n",
    "\n",
    "session.download_data(path=\"./\", bucket=s3_bucket, key_prefix=data_explore_nbk_key)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Exploration and Candidate Generation Notebooks"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Take some time to inspect the Data Exploration notebook. Check the ```Column Analysis and Descriptive Statistics``` section to see the analysis carried out by Autopilot.**"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Now, take some time to inspect the Candidate Generation notebook. Check the ```Generated Candidates``` section to see the different algorithms and data transformation strategies used by Autopilot.**"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "Autopilot also automatically generates a feature importance report.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "explainability_prefix = best_candidate[\"CandidateProperties\"][\"CandidateArtifactLocations\"][\n",
    "    \"Explainability\"\n",
    "]\n",
    "\n",
    "s3_bucket, explainability_dir = split_s3_path(explainability_prefix)\n",
    "\n",
    "session.download_data(path=\"./\", bucket=s3_bucket, key_prefix=explainability_dir)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The preceding code will download a directory to our local environment. In that directory (the prefix is the autopilot job name, the suffix is automatically generated), you should see the SageMaker Clarify artifacts. SageMaker Clarify provides greater visibility into training data and models to identify and limit bias and explain predictions. The following code will open the feature importance report:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import IFrame\n",
    "\n",
    "# Fetch the auto-generated directory name for the SageMaker Clarify artifacts\n",
    "dir_name = (\n",
    "    session.list_s3_files(bucket=s3_bucket, key_prefix=explainability_dir)[0]\n",
    "    .replace(explainability_dir, \"\")\n",
    "    .split(\"/\")[1]\n",
    ")\n",
    "\n",
    "# Display HTML report\n",
    "IFrame(src=f\"{dir_name}/report.html\", width=700, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your results may vary. But you're likely to see latitude and longitude (i.e., location) on top, along with population size and median income, which are stronger predictors of housing prices than the other features in the dataset."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In SageMaker Studio, you can also navigate to SageMaker resources tab, click on Experiments and trials, and find your Autopilot experiment. You can double-click on the experiment name to list all trials, and from there you can double-click on a specific trial to see its details, including charts and metrics."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Evaluate Model Using Test Dataset<a name=\"Evaluate\"></a>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To evaluate the model on previously unseen data, we will test it against the test dataset we prepared earlier. For that, we don't necessarily need to deploy the model to an endpoint, we can simply run a batch transform job to get predictions for our unlabeled test dataset."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up Transform Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import AutoML\n",
    "\n",
    "automl = AutoML.attach(auto_ml_job_name=auto_ml_job_name)\n",
    "\n",
    "s3_transform_output_path = \"s3://{}/{}/inference-results/\".format(s3_bucket, prefix)\n",
    "\n",
    "model_name = \"{0}-model\".format(best_candidate_name)\n",
    "\n",
    "model = automl.create_model(\n",
    "    name=model_name,\n",
    "    candidate=best_candidate,\n",
    ")\n",
    "\n",
    "output_path = s3_transform_output_path + best_candidate_name + \"/\"\n",
    "\n",
    "transformer = model.transformer(\n",
    "    instance_count=1,\n",
    "    instance_type=\"ml.m5.xlarge\",\n",
    "    assemble_with=\"Line\",\n",
    "    strategy=\"SingleRecord\",\n",
    "    output_path=output_path,\n",
    "    env={\"SAGEMAKER_MODEL_SERVER_TIMEOUT\": \"100\", \"SAGEMAKER_MODEL_SERVER_WORKERS\": \"1\"},\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Launch Transform Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "transformer.transform(\n",
    "    data=test_data_s3_path,\n",
    "    split_type=\"Line\",\n",
    "    content_type=\"text/csv\",\n",
    "    wait=False,\n",
    "    model_client_config={\"InvocationsTimeoutInSeconds\": 80, \"InvocationsMaxRetries\": 1},\n",
    ")\n",
    "\n",
    "print(\"Starting transform job {}\".format(transformer._current_job_name))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Track Transform Job Status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Wait for jobs to finish\n",
    "pending_complete = True\n",
    "job_name = transformer._current_job_name\n",
    "\n",
    "while pending_complete:\n",
    "    pending_complete = False\n",
    "\n",
    "    description = sm.describe_transform_job(TransformJobName=job_name)\n",
    "    if description[\"TransformJobStatus\"] not in [\"Failed\", \"Completed\"]:\n",
    "        pending_complete = True\n",
    "\n",
    "    print(\"{} transform job is running.\".format(job_name))\n",
    "    time.sleep(60)\n",
    "\n",
    "print(\"\\nCompleted.\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate the Inference Results"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The transform job will have now generated a CSV file with inference results for the test dataset. We will use those results and compare them with the real test labels to see how the model performs compared to real data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_csv_from_s3(s3uri, file_name):\n",
    "    parsed_url = urlparse(s3uri)\n",
    "    bucket_name = parsed_url.netloc\n",
    "    prefix = parsed_url.path[1:].strip(\"/\")\n",
    "    s3 = boto3.resource(\"s3\")\n",
    "    obj = s3.Object(bucket_name, \"{}/{}\".format(prefix, file_name))\n",
    "    return obj.get()[\"Body\"].read().decode(\"utf-8\")\n",
    "\n",
    "\n",
    "job_status = sm.describe_transform_job(TransformJobName=job_name)[\"TransformJobStatus\"]\n",
    "\n",
    "if job_status == \"Completed\":\n",
    "    pred_csv = get_csv_from_s3(transformer.output_path, \"{}.out\".format(test_file))\n",
    "    predictions = pd.read_csv(io.StringIO(pred_csv), header=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "from math import sqrt\n",
    "\n",
    "labels_df = test_data[target]\n",
    "mse = mean_squared_error(labels_df, predictions)\n",
    "rmse = sqrt(mse)\n",
    "\n",
    "print(\"MSE: {0}\\nRMSE: {1}\".format(mse, rmse))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Cleanup\n",
    "\n",
    "The Autopilot job creates many underlying artifacts such as dataset splits, preprocessing scripts, or preprocessed data, etc. This code, when un-commented, deletes them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3 = boto3.resource(\"s3\")\n",
    "s3_bucket = s3.Bucket(bucket)\n",
    "\n",
    "print(s3_bucket)\n",
    "job_outputs_prefix = \"{}/output/{}\".format(prefix, auto_ml_job_name)\n",
    "print(job_outputs_prefix)\n",
    "\n",
    "# Delete S3 objects\n",
    "s3_bucket.objects.filter(Prefix=job_outputs_prefix).delete()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then delete all the experiment and model resources created by the Autopilot experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cleanup_experiment_resources(experiment_name):\n",
    "    trials = sm.list_trials(ExperimentName=experiment_name)[\"TrialSummaries\"]\n",
    "    print(\"TrialNames:\")\n",
    "    for trial in trials:\n",
    "        trial_name = trial[\"TrialName\"]\n",
    "        print(f\"\\n{trial_name}\")\n",
    "\n",
    "        components_in_trial = sm.list_trial_components(TrialName=trial_name)\n",
    "        print(\"\\tTrialComponentNames:\")\n",
    "        for component in components_in_trial[\"TrialComponentSummaries\"]:\n",
    "            component_name = component[\"TrialComponentName\"]\n",
    "            print(f\"\\t{component_name}\")\n",
    "            sm.disassociate_trial_component(TrialComponentName=component_name, TrialName=trial_name)\n",
    "            try:\n",
    "                # comment out to keep trial components\n",
    "                sm.delete_trial_component(TrialComponentName=component_name)\n",
    "            except:\n",
    "                # component is associated with another trial\n",
    "                continue\n",
    "            # to prevent throttling\n",
    "            time.sleep(5)\n",
    "        sm.delete_trial(TrialName=trial_name)\n",
    "    sm.delete_experiment(ExperimentName=experiment_name)\n",
    "    print(f\"\\nExperiment {experiment_name} deleted\")\n",
    "\n",
    "\n",
    "def cleanup_autopilot_models(autopilot_job_name):\n",
    "    print(\"{0}:\\n\".format(autopilot_job_name))\n",
    "    response = sm.list_models(NameContains=autopilot_job_name)\n",
    "\n",
    "    for model in response[\"Models\"]:\n",
    "        model_name = model[\"ModelName\"]\n",
    "        print(f\"\\t{model_name}\")\n",
    "        sm.delete_model(ModelName=model_name)\n",
    "        # to prevent throttling\n",
    "        time.sleep(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleanup_experiment_resources(\"{0}-aws-auto-ml-job\".format(auto_ml_job_name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cleanup_autopilot_models(auto_ml_job_name)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, the following code, when uncommented, will delete the local files used in this demo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil\n",
    "import glob\n",
    "import os\n",
    "\n",
    "\n",
    "def delete_local_files():\n",
    "    base_path = \"\"\n",
    "    dir_list = glob.iglob(os.path.join(base_path, \"{0}*\".format(auto_ml_job_name)))\n",
    "\n",
    "    for path in dir_list:\n",
    "        if os.path.isdir(path):\n",
    "            shutil.rmtree(path)\n",
    "\n",
    "    if os.path.exists(\"CaliforniaHousing\"):\n",
    "        shutil.rmtree(\"CaliforniaHousing\")\n",
    "\n",
    "    if os.path.exists(\"cal_housing.tgz\"):\n",
    "        os.remove(\"cal_housing.tgz\")\n",
    "\n",
    "    if os.path.exists(\"SageMakerAutopilotCandidateDefinitionNotebook.ipynb\"):\n",
    "        os.remove(\"SageMakerAutopilotCandidateDefinitionNotebook.ipynb\")\n",
    "\n",
    "    if os.path.exists(\"SageMakerAutopilotDataExplorationNotebook.ipynb\"):\n",
    "        os.remove(\"SageMakerAutopilotDataExplorationNotebook.ipynb\")\n",
    "\n",
    "    if os.path.exists(\"test_data_no_target.csv\"):\n",
    "        os.remove(\"test_data_no_target.csv\")\n",
    "\n",
    "    if os.path.exists(\"test_data.csv\"):\n",
    "        os.remove(\"test_data.csv\")\n",
    "\n",
    "    if os.path.exists(\"train_data.csv\"):\n",
    "        os.remove(\"train_data.csv\")\n",
    "\n",
    "\n",
    "## UNCOMMENT TO CLEAN UP LOCAL FILES\n",
    "# delete_local_files()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note: If you enabled automatic endpoint creation, you will need to delete the endpoint manually.**"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/autopilot|autopilot_california_housing.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/autopilot|autopilot_california_housing.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/autopilot|autopilot_california_housing.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/autopilot|autopilot_california_housing.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/autopilot|autopilot_california_housing.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/autopilot|autopilot_california_housing.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/autopilot|autopilot_california_housing.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/autopilot|autopilot_california_housing.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/autopilot|autopilot_california_housing.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/autopilot|autopilot_california_housing.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/autopilot|autopilot_california_housing.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/autopilot|autopilot_california_housing.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/autopilot|autopilot_california_housing.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/autopilot|autopilot_california_housing.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/autopilot|autopilot_california_housing.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}