t81_558_class_04_5_rapids.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AkmpPvetDXgd"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_5_rapids.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L-BkYZiFDXge"
      },
      "source": [
        "# T81-558: Applications of Deep Neural Networks\n",
        "\n",
        "**Module 4: Training for Tabular Data**\n",
        "\n",
        "- Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
        "- For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WaTjAAfkDXgf"
      },
      "source": [
        "# Module 4 Material\n",
        "\n",
        "- Part 4.1: Using K-Fold Cross-validation with PyTorch [[Video]](https://www.youtube.com/watch?v=Q8ZQNvZwsNE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_1_kfold.ipynb)\n",
        "- Part 4.2: Training Schedules for PyTorch  [[Video]](https://www.youtube.com/watch?v=lMMlbmfvKDQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_2_schedule.ipynb)\n",
        "- Part 4.3: Dropout Regularization [[Video]](https://www.youtube.com/watch?v=4ixjgw6Q42U&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_3_dropout.ipynb)\n",
        "- Part 4.4: Batch Normalization [[Video]](https://www.youtube.com/watch?v=1U5nOKh9OLQ&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_4_batch_norm.ipynb)\n",
        "- **Part 4.5: RAPIDS for Tabular Data** [[Video]](https://www.youtube.com/watch?v=KgoXuhG_kfs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/app_deep_learning/blob/main/t81_558_class_04_5_rapids.ipynb)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SaFKgP8DDXgf"
      },
      "source": [
        "# Installing Rapids on Google Colab\n",
        "\n",
        "In this section, we're going to explore the use of NVIDIA RAPIDS, a suite of software libraries and APIs which enables us to execute end-to-end data science and analytics pipelines entirely on GPUs. This powerful tool leverages the compute capabilities of GPUs to provide significant performance improvements over traditional CPU-based workflows. However, it's important to note that to use RAPIDS effectively, a compatible GPU is necessary. As we progress, we suggest that you run the example provided in Google Colab. Google Colab is a free cloud service that supports GPU-based computation, and our example has been crafted to run optimally in this environment.\n",
        "\n",
        "If you wish to utilize RAPIDS in a different environment, be aware that this requires some specialized installation steps. This might involve installing appropriate GPU drivers, configuring CUDA (a parallel computing platform and API by NVIDIA), and installing RAPIDS libraries. While it is entirely possible to set up RAPIDS in such environments, it does require a certain level of familiarity with system administration and is beyond the scope of this book. We encourage you to follow the latest installation guides provided on the official RAPIDS website if you choose this route.\n",
        "\n",
        "The following code installs RAPIDS into Google Colab.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true,
          "base_uri": "https://localhost:8080/"
        },
        "id": "p4jug6R2DXgf",
        "outputId": "18976818-5ba5-4333-97de-a54ced2f1114"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Cloning into 'rapidsai-csp-utils'...\n",
            "remote: Enumerating objects: 390, done.\u001b[K\n",
            "remote: Counting objects: 100% (121/121), done.\u001b[K\n",
            "remote: Compressing objects: 100% (70/70), done.\u001b[K\n",
            "remote: Total 390 (delta 89), reused 51 (delta 51), pack-reused 269\u001b[K\n",
            "Receiving objects: 100% (390/390), 107.11 KiB | 5.95 MiB/s, done.\n",
            "Resolving deltas: 100% (191/191), done.\n",
            "Collecting pynvml\n",
            "  Downloading pynvml-11.5.0-py3-none-any.whl (53 kB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.1/53.1 kB 1.5 MB/s eta 0:00:00\n",
            "Installing collected packages: pynvml\n",
            "Successfully installed pynvml-11.5.0\n",
            "***********************************************************************\n",
            "Woo! Your instance has the right kind of GPU, a Tesla V100-SXM2-16GB!\n",
            "We will now install RAPIDS cuDF, cuML, and cuGraph via pip! \n",
            "Please stand by, should be quick...\n",
            "***********************************************************************\n",
            "\n",
            "Looking in indexes: https://pypi.org/simple, https://pypi.nvidia.com\n",
            "Collecting cudf-cu11\n",
            "  Downloading https://pypi.nvidia.com/cudf-cu11/cudf_cu11-23.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (489.3 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 489.3/489.3 MB 3.2 MB/s eta 0:00:00\n",
            "Collecting cuml-cu11\n",
            "  Downloading https://pypi.nvidia.com/cuml-cu11/cuml_cu11-23.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1079.0 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 GB 1.2 MB/s eta 0:00:00\n",
            "Collecting cugraph-cu11\n",
            "  Downloading https://pypi.nvidia.com/cugraph-cu11/cugraph_cu11-23.6.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1160.0 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 GB 1.4 MB/s eta 0:00:00\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (3.8.5)\n",
            "Requirement already satisfied: cachetools in /usr/local/lib/python3.10/dist-packages (from cudf-cu11) (5.3.1)\n",
            "Collecting cubinlinker-cu11 (from cudf-cu11)\n",
            "  Downloading https://pypi.nvidia.com/cubinlinker-cu11/cubinlinker_cu11-0.3.0.post1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.8 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.8/8.8 MB 120.1 MB/s eta 0:00:00\n",
            "Collecting cuda-python<12.0,>=11.7.1 (from cudf-cu11)\n",
            "  Downloading cuda_python-11.8.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.5 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.5/16.5 MB 62.9 MB/s eta 0:00:00\n",
            "Collecting cupy-cuda11x>=12.0.0 (from cudf-cu11)\n",
            "  Downloading cupy_cuda11x-12.1.0-cp310-cp310-manylinux2014_x86_64.whl (89.3 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.3/89.3 MB 19.1 MB/s eta 0:00:00\n",
            "Requirement already satisfied: fsspec>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from cudf-cu11) (2023.6.0)\n",
            "Collecting numba>=0.57 (from cudf-cu11)\n",
            "  Downloading numba-0.57.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.6 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 108.4 MB/s eta 0:00:00\n",
            "Requirement already satisfied: numpy>=1.21 in /usr/local/lib/python3.10/dist-packages (from cudf-cu11) (1.22.4)\n",
            "Collecting nvtx>=0.2.1 (from cudf-cu11)\n",
            "  Downloading nvtx-0.2.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (553 kB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 553.1/553.1 kB 37.3 MB/s eta 0:00:00\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from cudf-cu11) (23.1)\n",
            "Requirement already satisfied: pandas<1.6.0dev0,>=1.3 in /usr/local/lib/python3.10/dist-packages (from cudf-cu11) (1.5.3)\n",
            "Collecting protobuf<4.22,>=4.21.6 (from cudf-cu11)\n",
            "  Downloading protobuf-4.21.12-cp37-abi3-manylinux2014_x86_64.whl (409 kB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 409.8/409.8 kB 38.0 MB/s eta 0:00:00\n",
            "Collecting ptxcompiler-cu11 (from cudf-cu11)\n",
            "  Downloading https://pypi.nvidia.com/ptxcompiler-cu11/ptxcompiler_cu11-0.7.0.post1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.8 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.8/8.8 MB 101.5 MB/s eta 0:00:00\n",
            "Collecting pyarrow==11.* (from cudf-cu11)\n",
            "  Downloading pyarrow-11.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.9 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.9/34.9 MB 46.4 MB/s eta 0:00:00\n",
            "Collecting rmm-cu11==23.6.* (from cudf-cu11)\n",
            "  Downloading https://pypi.nvidia.com/rmm-cu11/rmm_cu11-23.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.7 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 91.1 MB/s eta 0:00:00\n",
            "Requirement already satisfied: typing-extensions>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from cudf-cu11) (4.7.1)\n",
            "Collecting dask-cuda==23.6.* (from cuml-cu11)\n",
            "  Downloading dask_cuda-23.6.0-py3-none-any.whl (125 kB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 125.2/125.2 kB 15.4 MB/s eta 0:00:00\n",
            "Collecting dask-cudf-cu11==23.6.* (from cuml-cu11)\n",
            "  Downloading https://pypi.nvidia.com/dask-cudf-cu11/dask_cudf_cu11-23.6.0-py3-none-any.whl (79 kB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.6/79.6 kB 10.4 MB/s eta 0:00:00\n",
            "Collecting dask==2023.3.2 (from cuml-cu11)\n",
            "  Downloading dask-2023.3.2-py3-none-any.whl (1.2 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 67.2 MB/s eta 0:00:00\n",
            "Collecting distributed==2023.3.2.1 (from cuml-cu11)\n",
            "  Downloading distributed-2023.3.2.1-py3-none-any.whl (957 kB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 957.1/957.1 kB 66.6 MB/s eta 0:00:00\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.10/dist-packages (from cuml-cu11) (1.3.1)\n",
            "Collecting raft-dask-cu11==23.6.* (from cuml-cu11)\n",
            "  Downloading https://pypi.nvidia.com/raft-dask-cu11/raft_dask_cu11-23.6.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (214.7 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 214.7/214.7 MB 3.3 MB/s eta 0:00:00\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from cuml-cu11) (1.10.1)\n",
            "Collecting treelite==3.2.0 (from cuml-cu11)\n",
            "  Downloading treelite-3.2.0-py3-none-manylinux2014_x86_64.whl (1.0 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 67.4 MB/s eta 0:00:00\n",
            "Collecting treelite-runtime==3.2.0 (from cuml-cu11)\n",
            "  Downloading treelite_runtime-3.2.0-py3-none-manylinux2014_x86_64.whl (198 kB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 198.2/198.2 kB 24.8 MB/s eta 0:00:00\n",
            "Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.10/dist-packages (from dask==2023.3.2->cuml-cu11) (8.1.6)\n",
            "Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from dask==2023.3.2->cuml-cu11) (2.2.1)\n",
            "Requirement already satisfied: partd>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from dask==2023.3.2->cuml-cu11) (1.4.0)\n",
            "Requirement already satisfied: pyyaml>=5.3.1 in /usr/local/lib/python3.10/dist-packages (from dask==2023.3.2->cuml-cu11) (6.0.1)\n",
            "Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.10/dist-packages (from dask==2023.3.2->cuml-cu11) (0.12.0)\n",
            "Collecting importlib-metadata>=4.13.0 (from dask==2023.3.2->cuml-cu11)\n",
            "  Downloading importlib_metadata-6.8.0-py3-none-any.whl (22 kB)\n",
            "Collecting pynvml<11.5,>=11.0.0 (from dask-cuda==23.6.*->cuml-cu11)\n",
            "  Downloading pynvml-11.4.1-py3-none-any.whl (46 kB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.0/47.0 kB 6.2 MB/s eta 0:00:00\n",
            "Requirement already satisfied: zict>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from dask-cuda==23.6.*->cuml-cu11) (3.0.0)\n",
            "Requirement already satisfied: jinja2>=2.10.3 in /usr/local/lib/python3.10/dist-packages (from distributed==2023.3.2.1->cuml-cu11) (3.1.2)\n",
            "Requirement already satisfied: locket>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from distributed==2023.3.2.1->cuml-cu11) (1.0.0)\n",
            "Requirement already satisfied: msgpack>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from distributed==2023.3.2.1->cuml-cu11) (1.0.5)\n",
            "Requirement already satisfied: psutil>=5.7.0 in /usr/local/lib/python3.10/dist-packages (from distributed==2023.3.2.1->cuml-cu11) (5.9.5)\n",
            "Requirement already satisfied: sortedcontainers>=2.0.5 in /usr/local/lib/python3.10/dist-packages (from distributed==2023.3.2.1->cuml-cu11) (2.4.0)\n",
            "Requirement already satisfied: tblib>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from distributed==2023.3.2.1->cuml-cu11) (2.0.0)\n",
            "Requirement already satisfied: tornado>=6.0.3 in /usr/local/lib/python3.10/dist-packages (from distributed==2023.3.2.1->cuml-cu11) (6.3.1)\n",
            "Requirement already satisfied: urllib3>=1.24.3 in /usr/local/lib/python3.10/dist-packages (from distributed==2023.3.2.1->cuml-cu11) (1.26.16)\n",
            "Collecting pylibraft-cu11==23.6.* (from raft-dask-cu11==23.6.*->cuml-cu11)\n",
            "  Downloading https://pypi.nvidia.com/pylibraft-cu11/pylibraft_cu11-23.6.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (471.7 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 471.7/471.7 MB 2.7 MB/s eta 0:00:00\n",
            "Collecting ucx-py-cu11==0.32.* (from raft-dask-cu11==23.6.*->cuml-cu11)\n",
            "  Downloading https://pypi.nvidia.com/ucx-py-cu11/ucx_py_cu11-0.32.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.9 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.9/7.9 MB 116.5 MB/s eta 0:00:00\n",
            "Collecting pylibcugraph-cu11==23.6.* (from cugraph-cu11)\n",
            "  Downloading https://pypi.nvidia.com/pylibcugraph-cu11/pylibcugraph_cu11-23.6.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1159.0 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 GB 1.5 MB/s eta 0:00:00\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp) (23.1.0)\n",
            "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp) (2.0.12)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp) (6.0.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp) (4.0.2)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp) (1.9.2)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp) (1.4.0)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp) (1.3.1)\n",
            "Requirement already satisfied: cython in /usr/local/lib/python3.10/dist-packages (from cuda-python<12.0,>=11.7.1->cudf-cu11) (0.29.36)\n",
            "Requirement already satisfied: fastrlock>=0.5 in /usr/local/lib/python3.10/dist-packages (from cupy-cuda11x>=12.0.0->cudf-cu11) (0.8.1)\n",
            "Collecting llvmlite<0.41,>=0.40.0dev0 (from numba>=0.57->cudf-cu11)\n",
            "  Downloading llvmlite-0.40.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (42.1 MB)\n",
            "     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.1/42.1 MB 45.1 MB/s eta 0:00:00\n",
            "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas<1.6.0dev0,>=1.3->cudf-cu11) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas<1.6.0dev0,>=1.3->cudf-cu11) (2022.7.1)\n",
            "Requirement already satisfied: idna>=2.0 in /usr/local/lib/python3.10/dist-packages (from yarl<2.0,>=1.0->aiohttp) (3.4)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata>=4.13.0->dask==2023.3.2->cuml-cu11) (3.16.2)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2>=2.10.3->distributed==2023.3.2.1->cuml-cu11) (2.1.3)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas<1.6.0dev0,>=1.3->cudf-cu11) (1.16.0)\n",
            "Installing collected packages: ptxcompiler-cu11, nvtx, cubinlinker-cu11, pynvml, pyarrow, protobuf, llvmlite, importlib-metadata, cupy-cuda11x, cuda-python, ucx-py-cu11, treelite-runtime, treelite, numba, dask, rmm-cu11, distributed, pylibraft-cu11, dask-cuda, cudf-cu11, raft-dask-cu11, pylibcugraph-cu11, dask-cudf-cu11, cuml-cu11, cugraph-cu11\n",
            "  Attempting uninstall: pynvml\n",
            "    Found existing installation: pynvml 11.5.0\n",
            "    Uninstalling pynvml-11.5.0:\n",
            "      Successfully uninstalled pynvml-11.5.0\n",
            "  Attempting uninstall: pyarrow\n",
            "    Found existing installation: pyarrow 9.0.0\n",
            "    Uninstalling pyarrow-9.0.0:\n",
            "      Successfully uninstalled pyarrow-9.0.0\n",
            "  Attempting uninstall: protobuf\n",
            "    Found existing installation: protobuf 3.20.3\n",
            "    Uninstalling protobuf-3.20.3:\n",
            "      Successfully uninstalled protobuf-3.20.3\n",
            "  Attempting uninstall: llvmlite\n",
            "    Found existing installation: llvmlite 0.39.1\n",
            "    Uninstalling llvmlite-0.39.1:\n",
            "      Successfully uninstalled llvmlite-0.39.1\n",
            "  Attempting uninstall: importlib-metadata\n",
            "    Found existing installation: importlib-metadata 4.6.4\n",
            "    Uninstalling importlib-metadata-4.6.4:\n",
            "      Successfully uninstalled importlib-metadata-4.6.4\n",
            "  Attempting uninstall: cupy-cuda11x\n",
            "    Found existing installation: cupy-cuda11x 11.0.0\n",
            "    Uninstalling cupy-cuda11x-11.0.0:\n",
            "      Successfully uninstalled cupy-cuda11x-11.0.0\n",
            "  Attempting uninstall: numba\n",
            "    Found existing installation: numba 0.56.4\n",
            "    Uninstalling numba-0.56.4:\n",
            "      Successfully uninstalled numba-0.56.4\n",
            "  Attempting uninstall: dask\n",
            "    Found existing installation: dask 2022.12.1\n",
            "    Uninstalling dask-2022.12.1:\n",
            "      Successfully uninstalled dask-2022.12.1\n",
            "  Attempting uninstall: distributed\n",
            "    Found existing installation: distributed 2022.12.1\n",
            "    Uninstalling distributed-2022.12.1:\n",
            "      Successfully uninstalled distributed-2022.12.1\n",
            "ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "pandas-gbq 0.17.9 requires pyarrow<10.0dev,>=3.0.0, but you have pyarrow 11.0.0 which is incompatible.\n",
            "Successfully installed cubinlinker-cu11-0.3.0.post1 cuda-python-11.8.2 cudf-cu11-23.6.1 cugraph-cu11-23.6.2 cuml-cu11-23.6.0 cupy-cuda11x-12.1.0 dask-2023.3.2 dask-cuda-23.6.0 dask-cudf-cu11-23.6.0 distributed-2023.3.2.1 importlib-metadata-6.8.0 llvmlite-0.40.1 numba-0.57.1 nvtx-0.2.6 protobuf-4.21.12 ptxcompiler-cu11-0.7.0.post1 pyarrow-11.0.0 pylibcugraph-cu11-23.6.2 pylibraft-cu11-23.6.2 pynvml-11.4.1 raft-dask-cu11-23.6.2 rmm-cu11-23.6.0 treelite-3.2.0 treelite-runtime-3.2.0 ucx-py-cu11-0.32.0\n",
            "Requirement already satisfied: cupy-cuda11x in /usr/local/lib/python3.10/dist-packages (12.1.0)\n",
            "Requirement already satisfied: numpy<1.27,>=1.20 in /usr/local/lib/python3.10/dist-packages (from cupy-cuda11x) (1.22.4)\n",
            "Requirement already satisfied: fastrlock>=0.5 in /usr/local/lib/python3.10/dist-packages (from cupy-cuda11x) (0.8.1)\n",
            "\n",
            "          ***********************************************************************\n",
            "          The pip install of RAPIDS is complete.\n",
            "          \n",
            "          Please do not run any further installation from the conda based installation methods, as they may cause issues!  \n",
            "          \n",
            "          Please ensure that you're pulling from the git repo to remain updated with the latest working install scripts. \n",
            "r          \n",
            "          Troubleshooting:\n",
            "             - If there is an installation failure, please check back on RAPIDSAI owned templates/notebooks to see how to update your personal files. \n",
            "             - If an installation failure persists when using the latest script, please make an issue on https://github.com/rapidsai-community/rapidsai-csp-utils\n",
            "          ***********************************************************************\n",
            "          \n"
          ]
        }
      ],
      "source": [
        "!git clone https://github.com/rapidsai/rapidsai-csp-utils.git\n",
        "!python rapidsai-csp-utils/colab/pip-install.py"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_9IVGfg7DXgg"
      },
      "source": [
        "# Part 4.5: RAPIDS for Tabular Data\n",
        "\n",
        "In this chapter where we delve into the powerful world of NVIDIA RAPIDS for machine learning with tabular data. As we increasingly engage with large volumes of data in modern machine learning, efficient processing and manipulation become critical. This is where RAPIDS steps in. This suite of open-source software libraries, developed by NVIDIA, accelerates data science pipelines by leveraging the power of GPUs. In this chapter, we'll explore how RAPIDS can be used to transform, model, and infer from tabular data in a significantly reduced timeframe compared to traditional CPU-based methods. Whether you're predicting sales, analyzing customer behavior, or detecting anomalies, using RAPIDS with tabular data can dramatically streamline your workflow and enhance your machine learning models' performance.\n",
        "\n",
        "## Deep Learning and Tabular Data: Not Always a Perfect Match\n",
        "\n",
        "While deep learning has brought transformative changes in numerous fields, such as image recognition, natural language processing, and even playing board games, it has sometimes fallen short when applied to tabular data, especially in comparison to traditional machine learning algorithms.\n",
        "\n",
        "It's important to understand that neural networks, the core of deep learning, are entirely capable of performing regression and classification tasks on tabular data. By applying various network architectures and tuning hyperparameters, you can certainly train a deep learning model on such datasets.\n",
        "\n",
        "However, in many instances, more traditional models, such as XGBoost, Random Forests, and other Gradient Boosted Machines (GBMs) often outperform deep learning models on purely tabular datasets. These traditional models have shown their efficiency in handling tabular data due to their ability to better capture certain kinds of relationships and dependencies between variables that neural networks sometimes miss.\n",
        "\n",
        "Real-world evidence of this trend can be found in the results of various Kaggle competitions. Kaggle, a platform for predictive modelling and analytics competitions, offers a wealth of data on machine learning model performance. Many winners of these competitions, particularly those focused on tabular data, frequently utilize GBMs. For instance, the XGBoost algorithm, a scalable and accurate implementation of gradient boosting machines, has been a part of numerous winning solutions.\n",
        "\n",
        "One reason for this is that tabular data often has structured and hierarchical relationships, and gradient boosting algorithms excel at capturing these interactions. On the other hand, deep learning models, particularly those without specific architecture designs, might struggle with such data unless provided with large amounts of training data and carefully selected features.\n",
        "\n",
        "This is not to say that deep learning should be disregarded when dealing with tabular data. Rather, it's important to understand the strengths and weaknesses of each approach. In some cases, combining deep learning with traditional methods can result in very powerful models. However, for most tabular datasets, the traditional models have proven to be more reliable, accurate, and computationally efficient.\n",
        "\n",
        "It's essential to remember that in machine learning, there is no \"one size fits all\" solution. The effectiveness of a model depends on the nature of the problem, the quality and quantity of the data, and the computational resources available. As practitioners, our role is to navigate this landscape and select or design the best models for our specific needs.\n",
        "\n",
        "## NVIDIA RAPIDS and Random Forests for Tabular Prediction\n",
        "\n",
        "As we a completly GPU-based pipeline, we're going to adapt a previous PyTorch neural network example to make use of NVIDIA RAPIDS. While neural networks have their strengths, there are cases where traditional models like Random Forests can outperform them, especially when dealing with certain types of tabular data. In our case, we've found that for the dataset we're working with, a Random Forest model indeed surpasses the previously used neural network in performance.\n",
        "\n",
        "In this section, we'll guide you step-by-step on how to replace our neural network with a Random Forest, demonstrating the flexibility and versatility that RAPIDS offers when it comes to choosing and changing machine learning models.\n",
        "\n",
        "A highlight of this adaptation will be the use of RAPIDS' cudf library. This is a powerful GPU-accelerated data manipulation library that allows us to perform all necessary preprocessing of our dataset entirely within GPU memory. One of the significant advantages here is that we can directly hand off our processed data to the Random Forest model without leaving GPU memory, enhancing the efficiency of our data science pipeline.\n",
        "\n",
        "We'll demonstrate how to take full advantage of these RAPIDS features, optimizing our model's performance and speed. By the end of this section, you'll not only have a stronger understanding of how to adapt different machine learning models within RAPIDS but also have gained hands-on experience in leveraging GPU memory for efficient data processing and modeling. This code is demonstrated here.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "2dsPRhe0DXgg",
        "outputId": "63e4b40d-f922-4672-9a08-02b57bd9e5dd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Final score (RMSE): 0.531491756439209\n"
          ]
        }
      ],
      "source": [
        "import cudf as pd\n",
        "from cuml import train_test_split\n",
        "from cuml.ensemble import RandomForestRegressor\n",
        "from cuml.metrics import mean_squared_error\n",
        "from cuml.preprocessing import StandardScaler\n",
        "\n",
        "\n",
        "# Read the data set\n",
        "df = pd.read_csv(\n",
        "    \"https://data.heatonresearch.com/data/t81-558/jh-simple-dataset.csv\",\n",
        "    na_values=[\"NA\", \"?\"],\n",
        ")\n",
        "\n",
        "# Generate dummies for job\n",
        "df = pd.concat([df, pd.get_dummies(df[\"job\"], prefix=\"job\", dtype=int)], axis=1)\n",
        "df.drop(\"job\", axis=1, inplace=True)\n",
        "\n",
        "# Generate dummies for area\n",
        "df = pd.concat([df, pd.get_dummies(df[\"area\"], prefix=\"area\", dtype=int)], axis=1)\n",
        "df.drop(\"area\", axis=1, inplace=True)\n",
        "\n",
        "# Generate dummies for product\n",
        "df = pd.concat([df, pd.get_dummies(df[\"product\"], prefix=\"product\", dtype=int)], axis=1)\n",
        "df.drop(\"product\", axis=1, inplace=True)\n",
        "\n",
        "# Missing values for income\n",
        "med = df[\"income\"].median()\n",
        "df[\"income\"] = df[\"income\"].fillna(med)\n",
        "\n",
        "# Standardize ranges\n",
        "scaler = StandardScaler()\n",
        "df[\"income\"] = scaler.fit_transform(df[\"income\"].to_frame())\n",
        "df[\"aspect\"] = scaler.fit_transform(df[\"aspect\"].to_frame())\n",
        "df[\"save_rate\"] = scaler.fit_transform(df[\"save_rate\"].to_frame())\n",
        "df[\"subscriptions\"] = scaler.fit_transform(df[\"subscriptions\"].to_frame())\n",
        "\n",
        "# Convert to cuDF DataFrame\n",
        "x_columns = df.columns.drop([\"age\", \"id\"])\n",
        "x = df[x_columns]\n",
        "y = df[\"age\"]\n",
        "x = x.astype(\"float32\")\n",
        "y = y.astype(\"float32\")\n",
        "\n",
        "# Set random seed for reproducibility\n",
        "import numpy as np\n",
        "\n",
        "np.random.seed(42)\n",
        "\n",
        "# Split into training and test datasets\n",
        "x_train, x_test, y_train, y_test = train_test_split(\n",
        "    x, y, test_size=0.2, random_state=42\n",
        ")\n",
        "\n",
        "# Train the model with RandomForestRegressor\n",
        "model = RandomForestRegressor(n_estimators=500, random_state=42, n_streams=1)\n",
        "model.fit(x_train, y_train)\n",
        "\n",
        "# Test the model\n",
        "y_pred = model.predict(x_test)\n",
        "\n",
        "# Compute RMSE\n",
        "score = np.sqrt(mean_squared_error(y_test, y_pred))\n",
        "\n",
        "print(f\"Final score (RMSE): {score}\")"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "anaconda-cloud": {},
    "colab": {
      "machine_shape": "hm",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3.9 (torch)",
      "language": "python",
      "name": "pytorch"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.16"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}