Hands-on 1. Data Discovery using Azure REST API.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Hands on - Data Discovery using Azure REST API\n",
        "\n",
        "__Notebook Version:__ 1.0<br>\n",
        "__Python Version:__ Python 3.8 - AzureML<br>\n",
        "__Required Packages:__ No<br>\n",
        "__Platforms Supported:__  Azure Machine Learning Notebooks\n",
        "     \n",
        "__Data Source Required:__ Log Analytics tables \n",
        "    \n",
        "### Description\n",
        "This notebook will provide step-by-step instructions and sample code to guide you through Azure authentication, Microsoft Sentinel data discovery by using Azure REST API.<br>\n",
        "*** No need to download and install any other Python modules. ***<br>\n",
        "*** Please run the cells sequentially to avoid errors. *** <br>\n",
        "\n",
        "## Table of Contents\n",
        "1. Warm-up\n",
        "2. Azure Authentication\n",
        "3. List Microsoft Sentinel Watchlists Using API\n",
        "4. List Microsoft Sentinel Incidents Using API"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 1. Warm-up"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# If you need to know what Python modules are available, you may run this:\n",
        "# help(\"modules\")"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "gather": {
          "logged": 1627596066714
        },
        "jupyter": {
          "outputs_hidden": false,
          "source_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Loading Python libraries\n",
        "from azure.identity import AzureCliCredential\n",
        "\n",
        "import requests\n",
        "import json\n",
        "import pandas\n",
        "from IPython.display import display, HTML, Markdown"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "gather": {
          "logged": 1685047925340
        },
        "jupyter": {
          "outputs_hidden": false,
          "source_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Functions will be used in this notebook\n",
        "def read_config_values(file_path):\n",
        "    \"This loads pre-generated parameters for Microsoft Sentinel Workspace\"\n",
        "    with open(file_path) as json_file:\n",
        "        if json_file:\n",
        "            json_config = json.load(json_file)\n",
        "            return (json_config[\"tenant_id\"],\n",
        "                    json_config[\"subscription_id\"],\n",
        "                    json_config[\"resource_group\"],\n",
        "                    json_config[\"workspace_id\"],\n",
        "                    json_config[\"workspace_name\"],\n",
        "                    json_config[\"user_alias\"],\n",
        "                    json_config[\"user_object_id\"])\n",
        "    return None\n",
        "\n",
        "def has_valid_token():\n",
        "    \"Check to see if there is a valid AAD token\"\n",
        "    try:\n",
        "        error = \"Please run 'az login'\"\n",
        "        expired = \"AADSTS70043: The refresh token has expired or is invalid\"\n",
        "        failed = \"failed\"\n",
        "        validator = !az account get-access-token\n",
        "        \n",
        "        if any(expired in item for item in validator.get_list()):\n",
        "            return '**The refresh token has expired. <br> Please continue your login process. Then: <br> 1. If you plan to run multiple notebooks on the same compute instance today, you may restart the compute instance by clicking \"Compute\" on left menu, then select the instance, clicking \"Restart\"; <br> 2. Otherwise, you may just restart the kernel from top menu. <br> Finally, close and re-load the notebook, then re-run cells one by one from the top.**'\n",
        "        elif any(error in item for item in validator.get_list()) or any(failed in item for item in validator.get_list()):\n",
        "            return \"Please run 'az login' to setup account\"\n",
        "        else:\n",
        "            return None\n",
        "    except:\n",
        "        return \"Please login\"\n",
        "        \n",
        "# Calling Microsoft Sentinel API for List, the same template can be used for calling other Azure REST APIs with different parameters.\n",
        "# For different environments, such as national clouds, you may need to use different root_url, please contact with your admins.\n",
        "# It can be ---.azure.us, ---.azure.microsoft.scloud, ---.azure.eaglex.ic.gov, etc.\n",
        "def call_azure_rest_api_for_list(token, resource_name, api_version):\n",
        "    \"Calling Microsoft Sentinel REST API\"\n",
        "    headers = {\"Authorization\": token, \"content-type\":\"application/json\" }\n",
        "    provider_name = \"Microsoft.OperationalInsights\"\n",
        "    provider2_name = \"Microsoft.SecurityInsights\"\n",
        "    target_resource_name = resource_name\n",
        "    api_version = api_version\n",
        "    root_url = \"https://management.azure.com\"\n",
        "    arm_rest_url_template_for_list = \"{0}/subscriptions/{1}/resourceGroups/{2}/providers/{3}/workspaces/{4}/providers/{5}/{6}?api-version={7}\"\n",
        "    arm_rest_url = arm_rest_url_template_for_list.format(root_url, subscription_id, resource_group, provider_name, workspace_name, provider2_name, target_resource_name, api_version)\n",
        "    response = requests.get(arm_rest_url, headers=headers, verify=True)\n",
        "    return response\n",
        "\n",
        "def display_result_name(response):\n",
        "    \"Default to display column - name, you may change it to other columns\"\n",
        "    column_name = \"name\"\n",
        "    if response != None:\n",
        "        entries = [item[column_name] for item in response.json()[\"value\"]] \n",
        "        display(entries)\n",
        "\n",
        "def display_result(response):\n",
        "    \"Display the result set as pandas.DataFrame\"\n",
        "    if response != None:\n",
        "        df = pandas.DataFrame(response.json()[\"value\"])\n",
        "        display(df)"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "gather": {
          "logged": 1685047927231
        },
        "jupyter": {
          "outputs_hidden": false,
          "source_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Calling the above function to populate Microsoft Sentinel workspace parameters\n",
        "# The file, config.json, was generated by the system, however, you may modify the values, or manually set the variables\n",
        "tenant_id, subscription_id, resource_group, workspace_id, workspace_name, user_alias, user_object_id = read_config_values('config.json');"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "gather": {
          "logged": 1685047930472
        },
        "jupyter": {
          "outputs_hidden": false,
          "source_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2. Azure Authentication"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Azure CLI is used to get device code to login into Azure, you need to copy the code and open the DeviceLogin site.\n",
        "# You may add [--tenant $tenant_id] to the command\n",
        "if has_valid_token() != None:\n",
        "    message = '**The refresh token has expired. <br> Please continue your login process. Then: <br> 1. If you plan to run multiple notebooks on the same compute instance today, you may restart the compute instance by clicking \"Compute\" on left menu, then select the instance, clicking \"Restart\"; <br> 2. Otherwise, you may just restart the kernel from top menu. <br> Finally, close and re-load the notebook, then re-run cells one by one from the top.**'\n",
        "    display(Markdown(message))\n",
        "    !echo -e '\\e[42m'\n",
        "    !az login --tenant $tenant_id --use-device-code"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1685047932761
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Extract access token, which will be used to access Microsoft Sentinel Watchlist API for your Watchlist data.  \n",
        "token = AzureCliCredential().get_token('https://management.azure.com')\n",
        "access_token = token.token\n",
        "header_token_value = \"Bearer {}\".format(access_token)"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "gather": {
          "logged": 1685047935295
        },
        "jupyter": {
          "outputs_hidden": false,
          "source_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 3. List Microsoft Sentinel Watchlists Using API"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Calling Microsoft Sentinel Watchlist API\n",
        "# If you don't have Watchlist, you may create one, or try to access different features, such as Bookmarks.\n",
        "response_watchlist = call_azure_rest_api_for_list(header_token_value, \"watchlists\", \"2023-02-01\")"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "gather": {
          "logged": 1685047938197
        },
        "jupyter": {
          "outputs_hidden": false,
          "source_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Display the result\n",
        "display_result_name(response_watchlist)"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "gather": {
          "logged": 1685047939840
        },
        "jupyter": {
          "outputs_hidden": false,
          "source_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 4. List Microsoft Sentinel Incidents Using API"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Calling Microsoft Sentinel Incident API\n",
        "# If you don't have incidents, you may create one through Azure Portal.\n",
        "response_incident = call_azure_rest_api_for_list(header_token_value, \"incidents\", \"2020-01-01\")"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "gather": {
          "logged": 1632434901252
        },
        "jupyter": {
          "outputs_hidden": false,
          "source_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Display the result\n",
        "display_result(response_incident)"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "gather": {
          "logged": 1632434904269
        },
        "jupyter": {
          "outputs_hidden": false,
          "source_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Thanks for coming along all the way to the end.  In the next Hands-on notebook, I will show you how to access data using Azure SDK for Python.  And keep one of the watchlist name, it will be used in the next notebook.  A la prochaine."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    }
  ],
  "metadata": {
    "celltoolbar": "Tags",
    "kernel_info": {
      "name": "python38-azureml"
    },
    "kernelspec": {
      "name": "python38-azureml",
      "language": "python",
      "display_name": "Python 3.8 - AzureML"
    },
    "language_info": {
      "name": "python",
      "version": "3.8.5",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "microsoft": {
      "host": {
        "AzureML": {
          "notebookHasBeenCompleted": true
        }
      },
      "ms_spell_check": {
        "ms_spell_check_language": "en"
      }
    },
    "nteract": {
      "version": "nteract-front-end@1.0.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}