vertex_pipelines_simple.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pknVo1kM2wI2"
   },
   "source": [
    "##### Copyright 2021 The TensorFlow Authors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "SoFqANDE222Y"
   },
   "outputs": [],
   "source": [
    "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "# https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6x1ypzczQCwy"
   },
   "source": [
    "# Simple TFX Pipeline for Vertex Pipelines\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_VuwrlnvQJ5k"
   },
   "source": [
    "This notebook-based tutorial will create a simple TFX pipeline and run it using\n",
    "Google Cloud Vertex Pipelines.  This notebook is based on the TFX pipeline\n",
    "built in\n",
    "[Simple TFX Pipeline Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple).\n",
    "\n",
    "Google Cloud Vertex Pipelines helps you to automate, monitor, and govern\n",
    "your ML systems by orchestrating your ML workflow in a serverless manner. You\n",
    "can define your ML pipelines using Python with TFX, and then execute your\n",
    "pipelines on Google Cloud. See\n",
    "[Vertex Pipelines introduction](https://cloud.google.com/vertex-ai/docs/pipelines/introduction)\n",
    "to learn more about Vertex Pipelines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fwZ0aXisoBFW"
   },
   "source": [
    "## Setup\n",
    "### Install python packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WC9W_S-bONgl"
   },
   "source": [
    "We will install required Python packages including TFX and KFP to author ML\n",
    "pipelines and submit jobs to Vertex Pipelines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "iyQtljP-qPHY"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pip in /opt/conda/lib/python3.7/site-packages (22.2.1)\n",
      "Requirement already satisfied: tfx[kfp]<2 in /opt/conda/lib/python3.7/site-packages (1.9.0)\n",
      "Requirement already satisfied: tensorflow-hub<0.13,>=0.9.0 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (0.12.0)\n",
      "Requirement already satisfied: typing-extensions<5,>=3.10.0.2 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (4.0.1)\n",
      "Requirement already satisfied: ml-metadata<1.10.0,>=1.9.0 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.9.0)\n",
      "Requirement already satisfied: google-apitools<1,>=0.5 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (0.5.31)\n",
      "Requirement already satisfied: google-cloud-aiplatform<2,>=1.6.2 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.8.1)\n",
      "Requirement already satisfied: numpy<2,>=1.16 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.21.6)\n",
      "Requirement already satisfied: jinja2<4,>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (2.11.3)\n",
      "Requirement already satisfied: tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (2.9.1)\n",
      "Requirement already satisfied: keras-tuner<2,>=1.0.4 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.1.0)\n",
      "Requirement already satisfied: absl-py<2.0.0,>=0.9 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.0.0)\n",
      "Requirement already satisfied: pyyaml<6,>=3.12 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (5.4.1)\n",
      "Requirement already satisfied: attrs<21,>=19.3.0 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (20.3.0)\n",
      "Requirement already satisfied: pyarrow<6,>=1 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (5.0.0)\n",
      "Requirement already satisfied: apache-beam[gcp]<3,>=2.38 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (2.40.0)\n",
      "Requirement already satisfied: docker<5,>=4.1 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (4.4.4)\n",
      "Requirement already satisfied: tensorflow-serving-api!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<3,>=1.15 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (2.9.1)\n",
      "Requirement already satisfied: kubernetes<13,>=10.0.1 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (12.0.1)\n",
      "Requirement already satisfied: ml-pipelines-sdk==1.9.0 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.9.0)\n",
      "Requirement already satisfied: tensorflow-transform<1.10.0,>=1.9.0 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.9.0)\n",
      "Requirement already satisfied: grpcio<2,>=1.28.1 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.43.0)\n",
      "Requirement already satisfied: portpicker<2,>=1.3.1 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.5.2)\n",
      "Requirement already satisfied: packaging<21,>=20 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (20.9)\n",
      "Requirement already satisfied: tensorflow-data-validation<1.10.0,>=1.9.0 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.9.0)\n",
      "Requirement already satisfied: tfx-bsl<1.10.0,>=1.9.0 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.9.0)\n",
      "Requirement already satisfied: click<8,>=7 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (7.1.2)\n",
      "Requirement already satisfied: google-api-python-client<2,>=1.8 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.12.11)\n",
      "Requirement already satisfied: google-api-core<2 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.32.0)\n",
      "Requirement already satisfied: google-cloud-bigquery<3,>=2.26.0 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (2.31.0)\n",
      "Requirement already satisfied: protobuf<4,>=3.13 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (3.19.1)\n",
      "Requirement already satisfied: tensorflow-model-analysis<0.41,>=0.40.0 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (0.40.0)\n",
      "Requirement already satisfied: kfp-pipeline-spec<0.2,>=0.1.10 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (0.1.16)\n",
      "Requirement already satisfied: kfp<2,>=1.8.5 in /opt/conda/lib/python3.7/site-packages (from tfx[kfp]<2) (1.8.13)\n",
      "Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from absl-py<2.0.0,>=0.9->tfx[kfp]<2) (1.16.0)\n",
      "Requirement already satisfied: dill<0.3.2,>=0.3.1.1 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (0.3.1.1)\n",
      "Requirement already satisfied: cloudpickle<3,>=2.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (2.1.0)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.24.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (2.26.0)\n",
      "Requirement already satisfied: python-dateutil<3,>=2.8.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (2.8.2)\n",
      "Requirement already satisfied: fastavro<2,>=0.23.6 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (1.4.7)\n",
      "Requirement already satisfied: pymongo<4.0.0,>=3.8.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (3.12.3)\n",
      "Requirement already satisfied: crcmod<2.0,>=1.7 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (1.7)\n",
      "Requirement already satisfied: pytz>=2018.3 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (2021.3)\n",
      "Requirement already satisfied: proto-plus<2,>=1.7.1 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (1.19.8)\n",
      "Requirement already satisfied: httplib2<0.21.0,>=0.8 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (0.20.2)\n",
      "Requirement already satisfied: orjson<4.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (3.6.5)\n",
      "Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (2.6.0)\n",
      "Requirement already satisfied: pydot<2,>=1.2.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (1.4.2)\n",
      "Requirement already satisfied: google-cloud-spanner<2,>=1.13.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (1.19.3)\n",
      "Requirement already satisfied: google-auth-httplib2<0.2.0,>=0.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (0.1.0)\n",
      "Requirement already satisfied: google-cloud-core<2,>=0.28.1 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (1.7.3)\n",
      "Requirement already satisfied: google-cloud-recommendations-ai<=0.2.0,>=0.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (0.2.0)\n",
      "Requirement already satisfied: google-cloud-dlp<4,>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (3.8.0)\n",
      "Requirement already satisfied: cachetools<5,>=3.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (4.2.4)\n",
      "Requirement already satisfied: google-auth<3,>=1.18.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (1.35.0)\n",
      "Requirement already satisfied: google-cloud-pubsub<3,>=2.1.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (2.13.4)\n",
      "Requirement already satisfied: grpcio-gcp<1,>=0.2.2 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (0.2.2)\n",
      "Requirement already satisfied: google-cloud-vision<2,>=0.38.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (1.0.2)\n",
      "Requirement already satisfied: google-cloud-videointelligence<2,>=1.8.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (1.16.3)\n",
      "Requirement already satisfied: google-cloud-bigquery-storage>=2.6.3 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (2.10.1)\n",
      "Requirement already satisfied: google-cloud-language<2,>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (1.3.2)\n",
      "Requirement already satisfied: google-cloud-bigtable<2,>=0.31.1 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (1.7.2)\n",
      "Requirement already satisfied: google-cloud-pubsublite<2,>=1.2.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (1.4.2)\n",
      "Requirement already satisfied: google-cloud-datastore<2,>=1.8.0 in /opt/conda/lib/python3.7/site-packages (from apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (1.15.5)\n",
      "Requirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker<5,>=4.1->tfx[kfp]<2) (1.2.3)\n",
      "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core<2->tfx[kfp]<2) (1.56.4)\n",
      "Requirement already satisfied: setuptools>=40.3.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core<2->tfx[kfp]<2) (59.6.0)\n",
      "Requirement already satisfied: uritemplate<4dev,>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.8->tfx[kfp]<2) (3.0.1)\n",
      "Requirement already satisfied: oauth2client>=1.4.12 in /opt/conda/lib/python3.7/site-packages (from google-apitools<1,>=0.5->tfx[kfp]<2) (4.1.3)\n",
      "Requirement already satisfied: fasteners>=0.14 in /opt/conda/lib/python3.7/site-packages (from google-apitools<1,>=0.5->tfx[kfp]<2) (0.16.3)\n",
      "Requirement already satisfied: google-cloud-storage<2.0.0dev,>=1.32.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform<2,>=1.6.2->tfx[kfp]<2) (1.43.0)\n",
      "Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3,>=2.26.0->tfx[kfp]<2) (2.1.0)\n",
      "Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/lib/python3.7/site-packages (from jinja2<4,>=2.7.3->tfx[kfp]<2) (1.1.1)\n",
      "Requirement already satisfied: tensorboard in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx[kfp]<2) (2.9.1)\n",
      "Requirement already satisfied: kt-legacy in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx[kfp]<2) (1.0.4)\n",
      "Requirement already satisfied: ipython in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx[kfp]<2) (7.30.1)\n",
      "Requirement already satisfied: scipy in /opt/conda/lib/python3.7/site-packages (from keras-tuner<2,>=1.0.4->tfx[kfp]<2) (1.7.3)\n",
      "Requirement already satisfied: docstring-parser<1,>=0.7.3 in /opt/conda/lib/python3.7/site-packages (from kfp<2,>=1.8.5->tfx[kfp]<2) (0.14.1)\n",
      "Requirement already satisfied: tabulate<1,>=0.8.6 in /opt/conda/lib/python3.7/site-packages (from kfp<2,>=1.8.5->tfx[kfp]<2) (0.8.10)\n",
      "Requirement already satisfied: Deprecated<2,>=1.2.7 in /opt/conda/lib/python3.7/site-packages (from kfp<2,>=1.8.5->tfx[kfp]<2) (1.2.13)\n",
      "Requirement already satisfied: kfp-server-api<2.0.0,>=1.1.2 in /opt/conda/lib/python3.7/site-packages (from kfp<2,>=1.8.5->tfx[kfp]<2) (1.8.3)\n",
      "Requirement already satisfied: pydantic<2,>=1.8.2 in /opt/conda/lib/python3.7/site-packages (from kfp<2,>=1.8.5->tfx[kfp]<2) (1.8.2)\n",
      "Requirement already satisfied: strip-hints<1,>=0.1.8 in /opt/conda/lib/python3.7/site-packages (from kfp<2,>=1.8.5->tfx[kfp]<2) (0.1.10)\n",
      "Requirement already satisfied: jsonschema<4,>=3.0.1 in /opt/conda/lib/python3.7/site-packages (from kfp<2,>=1.8.5->tfx[kfp]<2) (3.2.0)\n",
      "Requirement already satisfied: typer<1.0,>=0.3.2 in /opt/conda/lib/python3.7/site-packages (from kfp<2,>=1.8.5->tfx[kfp]<2) (0.6.1)\n",
      "Requirement already satisfied: requests-toolbelt<1,>=0.8.0 in /opt/conda/lib/python3.7/site-packages (from kfp<2,>=1.8.5->tfx[kfp]<2) (0.9.1)\n",
      "Requirement already satisfied: fire<1,>=0.3.1 in /opt/conda/lib/python3.7/site-packages (from kfp<2,>=1.8.5->tfx[kfp]<2) (0.4.0)\n",
      "Requirement already satisfied: certifi>=14.05.14 in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=10.0.1->tfx[kfp]<2) (2021.10.8)\n",
      "Requirement already satisfied: requests-oauthlib in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=10.0.1->tfx[kfp]<2) (1.3.0)\n",
      "Requirement already satisfied: urllib3>=1.24.2 in /opt/conda/lib/python3.7/site-packages (from kubernetes<13,>=10.0.1->tfx[kfp]<2) (1.26.7)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging<21,>=20->tfx[kfp]<2) (3.0.6)\n",
      "Requirement already satisfied: psutil in /opt/conda/lib/python3.7/site-packages (from portpicker<2,>=1.3.1->tfx[kfp]<2) (5.8.0)\n",
      "Requirement already satisfied: tensorflow-estimator<2.10.0,>=2.9.0rc0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5->tfx[kfp]<2) (2.9.0)\n",
      "Requirement already satisfied: libclang>=13.0.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5->tfx[kfp]<2) (14.0.1)\n",
      "Requirement already satisfied: google-pasta>=0.1.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5->tfx[kfp]<2) (0.2.0)\n",
      "Requirement already satisfied: keras-preprocessing>=1.1.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5->tfx[kfp]<2) (1.1.2)\n",
      "Requirement already satisfied: gast<=0.4.0,>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5->tfx[kfp]<2) (0.4.0)\n",
      "Requirement already satisfied: opt-einsum>=2.3.2 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5->tfx[kfp]<2) (3.3.0)\n",
      "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5->tfx[kfp]<2) (0.23.1)\n",
      "Requirement already satisfied: flatbuffers<2,>=1.12 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5->tfx[kfp]<2) (1.12)\n",
      "Requirement already satisfied: h5py>=2.9.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5->tfx[kfp]<2) (3.6.0)\n",
      "Requirement already satisfied: termcolor>=1.1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5->tfx[kfp]<2) (1.1.0)\n",
      "Requirement already satisfied: astunparse>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5->tfx[kfp]<2) (1.6.3)\n",
      "Requirement already satisfied: wrapt>=1.11.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5->tfx[kfp]<2) (1.13.3)\n",
      "Requirement already satisfied: keras<2.10.0,>=2.9.0rc0 in /opt/conda/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5->tfx[kfp]<2) (2.9.0)\n",
      "Requirement already satisfied: joblib<0.15,>=0.12 in /opt/conda/lib/python3.7/site-packages (from tensorflow-data-validation<1.10.0,>=1.9.0->tfx[kfp]<2) (0.14.1)\n",
      "Requirement already satisfied: pandas<2,>=1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow-data-validation<1.10.0,>=1.9.0->tfx[kfp]<2) (1.3.5)\n",
      "Requirement already satisfied: tensorflow-metadata<1.10,>=1.9.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow-data-validation<1.10.0,>=1.9.0->tfx[kfp]<2) (1.9.0)\n",
      "Requirement already satisfied: pyfarmhash<0.4,>=0.2 in /opt/conda/lib/python3.7/site-packages (from tensorflow-data-validation<1.10.0,>=1.9.0->tfx[kfp]<2) (0.3.2)\n",
      "Requirement already satisfied: ipywidgets<8,>=7 in /opt/conda/lib/python3.7/site-packages (from tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (7.6.5)\n",
      "Requirement already satisfied: wheel<1.0,>=0.23.0 in /opt/conda/lib/python3.7/site-packages (from astunparse>=1.6.0->tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5->tfx[kfp]<2) (0.37.0)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<3,>=1.18.0->apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (4.8)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<3,>=1.18.0->apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (0.2.7)\n",
      "Requirement already satisfied: libcst>=0.2.5 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery-storage>=2.6.3->apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (0.3.23)\n",
      "Requirement already satisfied: grpc-google-iam-v1<0.13dev,>=0.12.3 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigtable<2,>=0.31.1->apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (0.12.4)\n",
      "Requirement already satisfied: grpcio-status>=1.16.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-pubsub<3,>=2.1.0->apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (1.43.0)\n",
      "Requirement already satisfied: overrides<7.0.0,>=6.0.1 in /opt/conda/lib/python3.7/site-packages (from google-cloud-pubsublite<2,>=1.2.0->apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (6.1.0)\n",
      "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.7/site-packages (from google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3,>=2.26.0->tfx[kfp]<2) (1.1.2)\n",
      "Requirement already satisfied: cached-property in /opt/conda/lib/python3.7/site-packages (from h5py>=2.9.0->tensorflow!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<2.10,>=1.15.5->tfx[kfp]<2) (1.5.2)\n",
      "Requirement already satisfied: docopt in /opt/conda/lib/python3.7/site-packages (from hdfs<3.0.0,>=2.1.0->apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (0.6.2)\n",
      "Requirement already satisfied: backcall in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.2.0)\n",
      "Requirement already satisfied: pexpect>4.3 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (4.8.0)\n",
      "Requirement already satisfied: pickleshare in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.7.5)\n",
      "Requirement already satisfied: traitlets>=4.2 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (5.1.1)\n",
      "Requirement already satisfied: decorator in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (5.1.0)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (3.0.24)\n",
      "Requirement already satisfied: jedi>=0.16 in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.18.1)\n",
      "Requirement already satisfied: pygments in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (2.10.0)\n",
      "Requirement already satisfied: matplotlib-inline in /opt/conda/lib/python3.7/site-packages (from ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.1.3)\n",
      "Requirement already satisfied: ipykernel>=4.5.1 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (6.6.0)\n",
      "Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (1.0.2)\n",
      "Requirement already satisfied: nbformat>=4.2.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (5.1.3)\n",
      "Requirement already satisfied: widgetsnbextension~=3.5.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (3.5.2)\n",
      "Requirement already satisfied: ipython-genutils~=0.2.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (0.2.0)\n",
      "Requirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from jsonschema<4,>=3.0.1->kfp<2,>=1.8.5->tfx[kfp]<2) (4.9.0)\n",
      "Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.7/site-packages (from jsonschema<4,>=3.0.1->kfp<2,>=1.8.5->tfx[kfp]<2) (0.18.0)\n",
      "Requirement already satisfied: pyasn1>=0.1.7 in /opt/conda/lib/python3.7/site-packages (from oauth2client>=1.4.12->google-apitools<1,>=0.5->tfx[kfp]<2) (0.4.8)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0,>=2.24.0->apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (2.0.9)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0,>=2.24.0->apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (3.1)\n",
      "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.6.1)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (3.3.6)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (1.8.0)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.4.6)\n",
      "Requirement already satisfied: werkzeug>=1.0.1 in /opt/conda/lib/python3.7/site-packages (from tensorboard->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (2.0.2)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from requests-oauthlib->kubernetes<13,>=10.0.1->tfx[kfp]<2) (3.1.1)\n",
      "Requirement already satisfied: cffi>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3,>=2.26.0->tfx[kfp]<2) (1.15.0)\n",
      "Requirement already satisfied: argcomplete>=1.12.3 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (1.12.3)\n",
      "Requirement already satisfied: debugpy<2.0,>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (1.5.1)\n",
      "Requirement already satisfied: jupyter-client<8.0 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (7.1.0)\n",
      "Requirement already satisfied: tornado<7.0,>=4.2 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (6.1)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->jsonschema<4,>=3.0.1->kfp<2,>=1.8.5->tfx[kfp]<2) (3.6.0)\n",
      "Requirement already satisfied: parso<0.9.0,>=0.8.0 in /opt/conda/lib/python3.7/site-packages (from jedi>=0.16->ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.8.3)\n",
      "Requirement already satisfied: typing-inspect>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from libcst>=0.2.5->google-cloud-bigquery-storage>=2.6.3->apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (0.7.1)\n",
      "Requirement already satisfied: jupyter-core in /opt/conda/lib/python3.7/site-packages (from nbformat>=4.2.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (4.9.1)\n",
      "Requirement already satisfied: typing-utils>=0.0.3 in /opt/conda/lib/python3.7/site-packages (from overrides<7.0.0,>=6.0.1->google-cloud-pubsublite<2,>=1.2.0->apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (0.1.0)\n",
      "Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/lib/python3.7/site-packages (from pexpect>4.3->ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.7.0)\n",
      "Requirement already satisfied: wcwidth in /opt/conda/lib/python3.7/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython->keras-tuner<2,>=1.0.4->tfx[kfp]<2) (0.2.5)\n",
      "Requirement already satisfied: notebook>=4.4.1 in /opt/conda/lib/python3.7/site-packages (from widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (6.4.6)\n",
      "Requirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi>=1.0.0->google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3,>=2.26.0->tfx[kfp]<2) (2.21)\n",
      "Requirement already satisfied: pyzmq>=13 in /opt/conda/lib/python3.7/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (22.3.0)\n",
      "Requirement already satisfied: entrypoints in /opt/conda/lib/python3.7/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (0.3)\n",
      "Requirement already satisfied: nest-asyncio>=1.5 in /opt/conda/lib/python3.7/site-packages (from jupyter-client<8.0->ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (1.5.4)\n",
      "Requirement already satisfied: argon2-cffi in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (21.1.0)\n",
      "Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (0.12.0)\n",
      "Requirement already satisfied: Send2Trash>=1.8.0 in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (1.8.0)\n",
      "Requirement already satisfied: terminado>=0.8.3 in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (0.12.1)\n",
      "Requirement already satisfied: nbconvert in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (6.3.0)\n",
      "Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/lib/python3.7/site-packages (from typing-inspect>=0.4.0->libcst>=0.2.5->google-cloud-bigquery-storage>=2.6.3->apache-beam[gcp]<3,>=2.38->tfx[kfp]<2) (0.4.3)\n",
      "Requirement already satisfied: testpath in /opt/conda/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (0.5.0)\n",
      "Requirement already satisfied: jupyterlab-pygments in /opt/conda/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (0.1.2)\n",
      "Requirement already satisfied: mistune<2,>=0.8.1 in /opt/conda/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (0.8.4)\n",
      "Requirement already satisfied: pandocfilters>=1.4.1 in /opt/conda/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (1.5.0)\n",
      "Requirement already satisfied: defusedxml in /opt/conda/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (0.7.1)\n",
      "Requirement already satisfied: bleach in /opt/conda/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (4.1.0)\n",
      "Requirement already satisfied: nbclient<0.6.0,>=0.5.0 in /opt/conda/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (0.5.9)\n",
      "Requirement already satisfied: webencodings in /opt/conda/lib/python3.7/site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.41,>=0.40.0->tfx[kfp]<2) (0.5.1)\n"
     ]
    }
   ],
   "source": [
    "# Use the latest version of pip.\n",
    "!pip install --upgrade pip\n",
    "!pip install --upgrade \"tfx[kfp]<2\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EwT0nov5QO1M"
   },
   "source": [
    "#### Restart the runtime\n",
    "\n",
    "Restart the runtime to ensure the following cells use the updated versions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-CRyIL4LVDlQ"
   },
   "source": [
    "You can restart the runtime with following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "KHTSzMygoBF6"
   },
   "outputs": [],
   "source": [
    "# docs_infra: no_execute\n",
    "import sys\n",
    "if not 'google.colab' in sys.modules:\n",
    "  # Automatically restart kernel after installs\n",
    "  import IPython\n",
    "  app = IPython.Application.instance()\n",
    "  app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3_SveIKxaENu"
   },
   "source": [
    "Check the package versions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "Xd-iP9wEaENu"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tfx import v1 as tfx\n",
    "import kfp\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aDtLdSkvqPHe"
   },
   "source": [
    "### Set up variables\n",
    "\n",
    "We will set up some variables used to customize the pipelines below. Following\n",
    "information is required:\n",
    "\n",
    "* GCP Project id. You can find your Project ID in the panel with your lab instructions.\n",
    "* GCP Region to run pipelines. For more information about the regions that\n",
    "Vertex Pipelines is available in, see the\n",
    "[Vertex AI locations guide](https://cloud.google.com/vertex-ai/docs/general/locations#feature-availability).\n",
    "* Google Cloud Storage Bucket to store pipeline outputs.\n",
    "\n",
    "**Enter required values in the cell below before running it**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "EcUseqJaE2XN"
   },
   "outputs": [],
   "source": [
    "GOOGLE_CLOUD_PROJECT = 'qwiklabs-gcp-01-4485f6259b3e'     # <--- ENTER THIS\n",
    "GOOGLE_CLOUD_REGION = 'us-east1'     \n",
    "GCS_BUCKET_NAME = GOOGLE_CLOUD_PROJECT + '-gcs'\n",
    "\n",
    "if not (GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_REGION and GCS_BUCKET_NAME):\n",
    "    from absl import logging\n",
    "    logging.error('Please set all required parameters.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GAaCPLjgiJrO"
   },
   "source": [
    "Set `gcloud` to use your project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "VkWdxe4TXRHk"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\n"
     ]
    }
   ],
   "source": [
    "!gcloud config set project {GOOGLE_CLOUD_PROJECT}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "CPN6UL5CazNy"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PIPELINE_ROOT: gs://qwiklabs-gcp-01-4485f6259b3e-gcs/pipeline_root/penguin-vertex-pipelines\n"
     ]
    }
   ],
   "source": [
    "PIPELINE_NAME = 'penguin-vertex-pipelines'\n",
    "\n",
    "# Path to various pipeline artifact.\n",
    "PIPELINE_ROOT = 'gs://{}/pipeline_root/{}'.format(\n",
    "    GCS_BUCKET_NAME, PIPELINE_NAME)\n",
    "\n",
    "# Paths for users' Python module.\n",
    "MODULE_ROOT = 'gs://{}/pipeline_module/{}'.format(\n",
    "    GCS_BUCKET_NAME, PIPELINE_NAME)\n",
    "\n",
    "# Paths for input data.\n",
    "DATA_ROOT = 'gs://{}/data/{}'.format(GCS_BUCKET_NAME, PIPELINE_NAME)\n",
    "\n",
    "# This is the path where your model will be pushed for serving.\n",
    "SERVING_MODEL_DIR = 'gs://{}/serving_model/{}'.format(\n",
    "    GCS_BUCKET_NAME, PIPELINE_NAME)\n",
    "\n",
    "print('PIPELINE_ROOT: {}'.format(PIPELINE_ROOT))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8F2SRwRLSYGa"
   },
   "source": [
    "### Prepare example data\n",
    "The dataset we are using is the\n",
    "[Palmer Penguins dataset](https://allisonhorst.github.io/palmerpenguins/articles/intro.html).\n",
    "\n",
    "There are four numeric features in this dataset:\n",
    "\n",
    "* culmen_length_mm\n",
    "* culmen_depth_mm\n",
    "* flipper_length_mm\n",
    "* body_mass_g\n",
    "\n",
    "All features were already normalized\n",
    "to have range [0,1]. We will build a classification model which predicts the\n",
    "`species` of penguins."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "11J7XiCq6AFP"
   },
   "source": [
    "We need to make our own copy of the dataset. Because TFX ExampleGen reads\n",
    "inputs from a directory, we need to create a directory and copy dataset to it\n",
    "on GCS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "4fxMs6u86acP"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying gs://download.tensorflow.org/data/palmer_penguins/penguins_processed.csv [Content-Type=application/octet-stream]...\n",
      "/ [1 files][ 25.0 KiB/ 25.0 KiB]                                                \n",
      "Operation completed over 1 objects/25.0 KiB.                                     \n"
     ]
    }
   ],
   "source": [
    "!gsutil cp gs://download.tensorflow.org/data/palmer_penguins/penguins_processed.csv {DATA_ROOT}/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ASpoNmxKSQjI"
   },
   "source": [
    "Take a quick look at the CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "-eSz28UDSnlG"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "species,culmen_length_mm,culmen_depth_mm,flipper_length_mm,body_mass_g\n",
      "0,0.2545454545454545,0.6666666666666666,0.15254237288135594,0.2916666666666667\n",
      "0,0.26909090909090905,0.5119047619047618,0.23728813559322035,0.3055555555555556\n",
      "0,0.29818181818181805,0.5833333333333334,0.3898305084745763,0.1527777777777778\n",
      "0,0.16727272727272732,0.7380952380952381,0.3559322033898305,0.20833333333333334\n",
      "0,0.26181818181818167,0.892857142857143,0.3050847457627119,0.2638888888888889\n",
      "0,0.24727272727272717,0.5595238095238096,0.15254237288135594,0.2569444444444444\n",
      "0,0.25818181818181823,0.773809523809524,0.3898305084745763,0.5486111111111112\n",
      "0,0.32727272727272727,0.5357142857142859,0.1694915254237288,0.1388888888888889\n",
      "0,0.23636363636363636,0.9642857142857142,0.3220338983050847,0.3055555555555556\n",
      "Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>\n",
      "BrokenPipeError: [Errno 32] Broken pipe\n"
     ]
    }
   ],
   "source": [
    "!gsutil cat {DATA_ROOT}/penguins_processed.csv | head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should be able to see five values. `species` is one of 0, 1 or 2, and all other features should have values between 0 and 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nH6gizcpSwWV"
   },
   "source": [
    "## Create a pipeline\n",
    "\n",
    "TFX pipelines are defined using Python APIs. We will define a pipeline which\n",
    "consists of three components:\n",
    "\n",
    "* CsvExampleGen: Reads in data files and convert them to TFX internal format for further processing. There are multiple ExampleGens for various formats. In this tutorial, we will use CsvExampleGen which takes CSV file input.\n",
    "* Trainer: Trains an ML model. Trainer component requires a model definition code from users. You can use TensorFlow APIs to specify how to train a model and save it in a _savedmodel format.\n",
    "* Pusher: Copies the trained model outside of the TFX pipeline. Pusher component can be thought of an deployment process of the trained ML model.\n",
    "\n",
    "Our pipeline will be almost identical to a basic [TFX pipeline](https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple).\n",
    "\n",
    "The only difference is that we don't need to set `metadata_connection_config`\n",
    "which is used to locate\n",
    "[ML Metadata](https://www.tensorflow.org/tfx/guide/mlmd) database. Because\n",
    "Vertex Pipelines uses a managed metadata service, users don't need to care\n",
    "of it, and we don't need to specify the parameter.\n",
    "\n",
    "Before actually define the pipeline, we need to write a model code for the\n",
    "Trainer component first."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lOjDv93eS5xV"
   },
   "source": [
    "### Write model code.\n",
    "\n",
    "We will create a simple DNN model for classification using TensorFlow Keras API. This model training code will be saved to a separate file.\n",
    "\n",
    "In this tutorial we will use __Generic Trainer__ of TFX which support Keras-based models. You need to write a Python file containing run_fn function, which is the entrypoint for the `Trainer` component."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "aES7Hv5QTDK3"
   },
   "outputs": [],
   "source": [
    "_trainer_module_file = 'penguin_trainer.py'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "Gnc67uQNTDfW"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting penguin_trainer.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile {_trainer_module_file}\n",
    "\n",
    "# Copied from https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple\n",
    "\n",
    "from typing import List\n",
    "from absl import logging\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow_transform.tf_metadata import schema_utils\n",
    "\n",
    "\n",
    "from tfx import v1 as tfx\n",
    "from tfx_bsl.public import tfxio\n",
    "\n",
    "from tensorflow_metadata.proto.v0 import schema_pb2\n",
    "\n",
    "_FEATURE_KEYS = [\n",
    "    'culmen_length_mm', 'culmen_depth_mm', 'flipper_length_mm', 'body_mass_g'\n",
    "]\n",
    "_LABEL_KEY = 'species'\n",
    "\n",
    "_TRAIN_BATCH_SIZE = 20\n",
    "_EVAL_BATCH_SIZE = 10\n",
    "\n",
    "# Since we're not generating or creating a schema, we will instead create\n",
    "# a feature spec.  Since there are a fairly small number of features this is\n",
    "# manageable for this dataset.\n",
    "_FEATURE_SPEC = {\n",
    "    **{\n",
    "        feature: tf.io.FixedLenFeature(shape=[1], dtype=tf.float32)\n",
    "           for feature in _FEATURE_KEYS\n",
    "       },\n",
    "    _LABEL_KEY: tf.io.FixedLenFeature(shape=[1], dtype=tf.int64)\n",
    "}\n",
    "\n",
    "\n",
    "def _input_fn(file_pattern: List[str],\n",
    "              data_accessor: tfx.components.DataAccessor,\n",
    "              schema: schema_pb2.Schema,\n",
    "              batch_size: int) -> tf.data.Dataset:\n",
    "  \"\"\"Generates features and label for training.\n",
    "\n",
    "  Args:\n",
    "    file_pattern: List of paths or patterns of input tfrecord files.\n",
    "    data_accessor: DataAccessor for converting input to RecordBatch.\n",
    "    schema: schema of the input data.\n",
    "    batch_size: representing the number of consecutive elements of returned\n",
    "      dataset to combine in a single batch\n",
    "\n",
    "  Returns:\n",
    "    A dataset that contains (features, indices) tuple where features is a\n",
    "      dictionary of Tensors, and indices is a single Tensor of label indices.\n",
    "  \"\"\"\n",
    "  return data_accessor.tf_dataset_factory(\n",
    "      file_pattern,\n",
    "      tfxio.TensorFlowDatasetOptions(\n",
    "          batch_size=batch_size, label_key=_LABEL_KEY),\n",
    "      schema=schema).repeat()\n",
    "\n",
    "\n",
    "def _make_keras_model() -> tf.keras.Model:\n",
    "  \"\"\"Creates a DNN Keras model for classifying penguin data.\n",
    "\n",
    "  Returns:\n",
    "    A Keras Model.\n",
    "  \"\"\"\n",
    "  # The model below is built with Functional API, please refer to\n",
    "  # https://www.tensorflow.org/guide/keras/overview for all API options.\n",
    "  inputs = [keras.layers.Input(shape=(1,), name=f) for f in _FEATURE_KEYS]\n",
    "  d = keras.layers.concatenate(inputs)\n",
    "  for _ in range(2):\n",
    "    d = keras.layers.Dense(8, activation='relu')(d)\n",
    "  outputs = keras.layers.Dense(3)(d)\n",
    "\n",
    "  model = keras.Model(inputs=inputs, outputs=outputs)\n",
    "  model.compile(\n",
    "      optimizer=keras.optimizers.Adam(1e-2),\n",
    "      loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "      metrics=[keras.metrics.SparseCategoricalAccuracy()])\n",
    "\n",
    "  model.summary(print_fn=logging.info)\n",
    "  return model\n",
    "\n",
    "\n",
    "# TFX Trainer will call this function.\n",
    "def run_fn(fn_args: tfx.components.FnArgs):\n",
    "  \"\"\"Train the model based on given args.\n",
    "\n",
    "  Args:\n",
    "    fn_args: Holds args used to train the model as name/value pairs.\n",
    "  \"\"\"\n",
    "\n",
    "  # This schema is usually either an output of SchemaGen or a manually-curated\n",
    "  # version provided by pipeline author. A schema can also derived from TFT\n",
    "  # graph if a Transform component is used. In the case when either is missing,\n",
    "  # `schema_from_feature_spec` could be used to generate schema from very simple\n",
    "  # feature_spec, but the schema returned would be very primitive.\n",
    "  schema = schema_utils.schema_from_feature_spec(_FEATURE_SPEC)\n",
    "\n",
    "  train_dataset = _input_fn(\n",
    "      fn_args.train_files,\n",
    "      fn_args.data_accessor,\n",
    "      schema,\n",
    "      batch_size=_TRAIN_BATCH_SIZE)\n",
    "  eval_dataset = _input_fn(\n",
    "      fn_args.eval_files,\n",
    "      fn_args.data_accessor,\n",
    "      schema,\n",
    "      batch_size=_EVAL_BATCH_SIZE)\n",
    "\n",
    "  model = _make_keras_model()\n",
    "  model.fit(\n",
    "      train_dataset,\n",
    "      steps_per_epoch=fn_args.train_steps,\n",
    "      validation_data=eval_dataset,\n",
    "      validation_steps=fn_args.eval_steps)\n",
    "\n",
    "  # The result of the training should be saved in `fn_args.serving_model_dir`\n",
    "  # directory.\n",
    "  model.save(fn_args.serving_model_dir, save_format='tf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-LsYx8MpYvPv"
   },
   "source": [
    "Copy the module file to GCS which can be accessed from the pipeline components.\n",
    "Because model training happens on GCP, we need to upload this model definition. \n",
    "\n",
    "Otherwise, you might want to build a container image including the module file\n",
    "and use the image to run the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "rMMs5wuNYAbc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying file://penguin_trainer.py [Content-Type=text/x-python]...\n",
      "/ [1 files][  3.8 KiB/  3.8 KiB]                                                \n",
      "Operation completed over 1 objects/3.8 KiB.                                      \n"
     ]
    }
   ],
   "source": [
    "!gsutil cp {_trainer_module_file} {MODULE_ROOT}/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w3OkNz3gTLwM"
   },
   "source": [
    "### Write a pipeline definition\n",
    "\n",
    "We will define a function to create a TFX pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "M49yYVNBTPd4"
   },
   "outputs": [],
   "source": [
    "# Copied from https://www.tensorflow.org/tfx/tutorials/tfx/penguin_simple and\n",
    "# slightly modified because we don't need `metadata_path` argument.\n",
    "\n",
    "def _create_pipeline(pipeline_name: str, pipeline_root: str, data_root: str,\n",
    "                     module_file: str, serving_model_dir: str,\n",
    "                     ) -> tfx.dsl.Pipeline:\n",
    "  \"\"\"Creates a three component penguin pipeline with TFX.\"\"\"\n",
    "  # Brings data into the pipeline.\n",
    "  example_gen = tfx.components.CsvExampleGen(input_base=data_root)\n",
    "\n",
    "  # Uses user-provided Python function that trains a model.\n",
    "  trainer = tfx.components.Trainer(\n",
    "      module_file=module_file,\n",
    "      examples=example_gen.outputs['examples'],\n",
    "      train_args=tfx.proto.TrainArgs(num_steps=100),\n",
    "      eval_args=tfx.proto.EvalArgs(num_steps=5))\n",
    "\n",
    "  # Pushes the model to a filesystem destination.\n",
    "  pusher = tfx.components.Pusher(\n",
    "      model=trainer.outputs['model'],\n",
    "      push_destination=tfx.proto.PushDestination(\n",
    "          filesystem=tfx.proto.PushDestination.Filesystem(\n",
    "              base_directory=serving_model_dir)))\n",
    "\n",
    "  # Following three components will be included in the pipeline.\n",
    "  components = [\n",
    "      example_gen,\n",
    "      trainer,\n",
    "      pusher,\n",
    "  ]\n",
    "\n",
    "  return tfx.dsl.Pipeline(\n",
    "      pipeline_name=pipeline_name,\n",
    "      pipeline_root=pipeline_root,\n",
    "      components=components)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mJbq07THU2GV"
   },
   "source": [
    "## Run the pipeline on Vertex Pipelines.\n",
    "\n",
    "TFX provides multiple orchestrators to run your pipeline. In this tutorial we\n",
    "will use the Vertex Pipelines together with the Kubeflow V2 dag runner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7mp0AkmrPdUb"
   },
   "source": [
    "We need to define a runner to actually run the pipeline. You will compile\n",
    "your pipeline into our pipeline definition format using TFX APIs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "fAtfOZTYWJu-"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "running bdist_wheel\n",
      "running build\n",
      "running build_py\n",
      "creating build\n",
      "creating build/lib\n",
      "copying penguin_trainer.py -> build/lib\n",
      "installing to /tmp/tmp8d5h2c3f\n",
      "running install\n",
      "running install_lib\n",
      "copying build/lib/penguin_trainer.py -> /tmp/tmp8d5h2c3f\n",
      "running install_egg_info\n",
      "running egg_info\n",
      "creating tfx_user_code_Trainer.egg-info\n",
      "writing tfx_user_code_Trainer.egg-info/PKG-INFO\n",
      "writing dependency_links to tfx_user_code_Trainer.egg-info/dependency_links.txt\n",
      "writing top-level names to tfx_user_code_Trainer.egg-info/top_level.txt\n",
      "writing manifest file 'tfx_user_code_Trainer.egg-info/SOURCES.txt'\n",
      "reading manifest file 'tfx_user_code_Trainer.egg-info/SOURCES.txt'\n",
      "writing manifest file 'tfx_user_code_Trainer.egg-info/SOURCES.txt'\n",
      "Copying tfx_user_code_Trainer.egg-info to /tmp/tmp8d5h2c3f/tfx_user_code_Trainer-0.0+afa8d5dfaf2942b08385545b5e095850331200a5d7a669b1791b3e7efcc5fdd9-py3.7.egg-info\n",
      "running install_scripts\n",
      "creating /tmp/tmp8d5h2c3f/tfx_user_code_Trainer-0.0+afa8d5dfaf2942b08385545b5e095850331200a5d7a669b1791b3e7efcc5fdd9.dist-info/WHEEL\n",
      "creating '/tmp/tmpfbnkzdjz/tfx_user_code_Trainer-0.0+afa8d5dfaf2942b08385545b5e095850331200a5d7a669b1791b3e7efcc5fdd9-py3-none-any.whl' and adding '/tmp/tmp8d5h2c3f' to it\n",
      "adding 'penguin_trainer.py'\n",
      "adding 'tfx_user_code_Trainer-0.0+afa8d5dfaf2942b08385545b5e095850331200a5d7a669b1791b3e7efcc5fdd9.dist-info/METADATA'\n",
      "adding 'tfx_user_code_Trainer-0.0+afa8d5dfaf2942b08385545b5e095850331200a5d7a669b1791b3e7efcc5fdd9.dist-info/WHEEL'\n",
      "adding 'tfx_user_code_Trainer-0.0+afa8d5dfaf2942b08385545b5e095850331200a5d7a669b1791b3e7efcc5fdd9.dist-info/top_level.txt'\n",
      "adding 'tfx_user_code_Trainer-0.0+afa8d5dfaf2942b08385545b5e095850331200a5d7a669b1791b3e7efcc5fdd9.dist-info/RECORD'\n",
      "removing /tmp/tmp8d5h2c3f\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.\n",
      "  setuptools.SetuptoolsDeprecationWarning,\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "PIPELINE_DEFINITION_FILE = PIPELINE_NAME + '_pipeline.json'\n",
    "\n",
    "runner = tfx.orchestration.experimental.KubeflowV2DagRunner(\n",
    "    config=tfx.orchestration.experimental.KubeflowV2DagRunnerConfig(),\n",
    "    output_filename=PIPELINE_DEFINITION_FILE)\n",
    "# Following function will write the pipeline definition to PIPELINE_DEFINITION_FILE.\n",
    "_ = runner.run(\n",
    "    _create_pipeline(\n",
    "        pipeline_name=PIPELINE_NAME,\n",
    "        pipeline_root=PIPELINE_ROOT,\n",
    "        data_root=DATA_ROOT,\n",
    "        module_file=os.path.join(MODULE_ROOT, _trainer_module_file),\n",
    "        serving_model_dir=SERVING_MODEL_DIR))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fWyITYSDd8w4"
   },
   "source": [
    "The generated definition file can be submitted using kfp client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "tI71jlEvWMV7"
   },
   "outputs": [],
   "source": [
    "# docs_infra: no_execute\n",
    "from google.cloud import aiplatform\n",
    "from google.cloud.aiplatform import pipeline_jobs\n",
    "\n",
    "aiplatform.init(project=GOOGLE_CLOUD_PROJECT, location=GOOGLE_CLOUD_REGION)\n",
    "\n",
    "job = pipeline_jobs.PipelineJob(template_path=PIPELINE_DEFINITION_FILE,\n",
    "                                display_name=PIPELINE_NAME)\n",
    "job.run(sync=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L3k9f5IVQXcQ"
   },
   "source": [
    "Visit __Vertex AI > Pipelines__ in your Google Cloud Console page to see the progress.\n",
    "\n",
    "Click on your `penguin-vertex-pipelines-xxx` run:\n",
    "\n",
    "![pipeline_start](01_pipeline_start.png)\n",
    "\n",
    "Explore the information displayed in each step while you wait for the job to progress.\n",
    "\n",
    "On completion, your pipeline UI should look similar to this:\n",
    "\n",
    "![pipeline_end](02_pipeline_completed.png)\n",
    "\n",
    "This job will take about 15 minutes in total to complete. Once complete, return to the lab to check your progress."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "pknVo1kM2wI2"
   ],
   "name": "Simple TFX Pipeline for Vertex Pipelines",
   "provenance": [],
   "toc_visible": true
  },
  "environment": {
   "kernel": "python3",
   "name": "tf2-cpu.2-7.m87",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-cpu.2-7:m87"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}