Keyword_Extraction_Tutorial.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/petermr/semanticClimate/blob/main/Keyword_Extraction_Tutorial.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Step 1: Install the github repository and the required libraries."
      ],
      "metadata": {
        "id": "llp-tRtRwt0Y"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Let's code...\n",
        "**To Do 1:**\n",
        "\n",
        "Can you uncomment all the lines 1 below\n",
        "\n",
        "**To Do 2:** \n",
        "\n",
        "Write the code to install pandas package."
      ],
      "metadata": {
        "id": "Rq7wdxJPP9XU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# !git clone https://github.com/petermr/semanticClimate.git\n",
        "!pip install transformers\n",
        "!pip install numpy\n",
        "# write the code to install numpy package.\n",
        "\n",
        "\n",
        "print('\\033[1;32m We have successfully finished running this cell.')"
      ],
      "metadata": {
        "id": "MvSIJTn0mr9f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "155c3584-a6f6-4241-cbad-861e4726e28f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'semanticClimate'...\n",
            "remote: Enumerating objects: 38030, done.\u001b[K\n",
            "remote: Counting objects: 100% (5822/5822), done.\u001b[K\n",
            "remote: Compressing objects: 100% (2126/2126), done.\u001b[K\n",
            "remote: Total 38030 (delta 3375), reused 5725 (delta 3323), pack-reused 32208\u001b[K\n",
            "Receiving objects: 100% (38030/38030), 653.20 MiB | 15.13 MiB/s, done.\n",
            "Resolving deltas: 100% (14736/14736), done.\n",
            "Updating files: 100% (28953/28953), done.\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (1.5.3)\n",
            "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas) (2022.7.1)\n",
            "Requirement already satisfied: numpy>=1.21.0 in /usr/local/lib/python3.10/dist-packages (from pandas) (1.22.4)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas) (1.16.0)\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (1.22.4)\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting transformers\n",
            "  Downloading transformers-4.29.2-py3-none-any.whl (7.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.1/7.1 MB\u001b[0m \u001b[31m57.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.12.0)\n",
            "Collecting huggingface-hub<1.0,>=0.14.1 (from transformers)\n",
            "  Downloading huggingface_hub-0.14.1-py3-none-any.whl (224 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m224.5/224.5 kB\u001b[0m \u001b[31m22.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (1.22.4)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (23.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2022.10.31)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.27.1)\n",
            "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers)\n",
            "  Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m37.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.65.0)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.14.1->transformers) (2023.4.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.14.1->transformers) (4.5.0)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (1.26.15)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2022.12.7)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.0.12)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.4)\n",
            "Installing collected packages: tokenizers, huggingface-hub, transformers\n",
            "Successfully installed huggingface-hub-0.14.1 tokenizers-0.13.3 transformers-4.29.2\n",
            "\u001b[1;32m We have successfully finished running this cell.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Step 2: Import the libraries onto your colab page and make sure you've gotten the correct version."
      ],
      "metadata": {
        "id": "P00CNUSa4UCl"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Keyword Extraction**"
      ],
      "metadata": {
        "id": "uTk_ViCimmCK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "**To Do 3:**\n",
        "\n",
        "Can you import pytorch, pandas and numpy libraries."
      ],
      "metadata": {
        "id": "xY7SISIKUOUi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import json\n",
        "import requests\n",
        "# write the code here.\n",
        "print('\\033[1;32m We have successfully finished running this cell.')"
      ],
      "metadata": {
        "id": "VCunoWJHpcGD",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a4e3212f-cb18-42cb-da91-bb25a670b735"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[1;32m We have successfully finished running this cell.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "Nhj6tZ9KMQvt"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Step 3** \n",
        "We define the extraction function"
      ],
      "metadata": {
        "id": "G7Y7LleCIgPV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**To Do 4:** \n",
        "Write a function that takes the text and post the query to the deployed model in huggingface.\n",
        "\n",
        "**To Do 5:** \n",
        "Try to write a for loop which iterates over the output and append the generated words in the list keywords"
      ],
      "metadata": {
        "id": "MtNv2ECdZ1bB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "### Write a function that takes the text and post the query to the deployed model in huggingface.\n",
        "def query(API_URL,headers, payload):\n",
        "            response = requests.post(API_URL, headers=headers, json=payload)\n",
        "            return response.json()\n",
        "\n",
        "def keyword_extraction(text):\n",
        "    keywords = []\n",
        "    API_URL = \"https://api-inference.huggingface.co/models/ml6team/keyphrase-extraction-kbir-inspec\"\n",
        "    headers = {\"Authorization\": \"Bearer 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\"}\n",
        "    output = query({\n",
        "                      \"inputs\": text,\n",
        "                        \"options\": \"wait_for_model=true\"\n",
        "                        }) \n",
        "    \n",
        "    # Output is a list of dictionaries where the word is one of the key. \n",
        "    # Try to write a for loop which iterates over the output and append the generated words in the list keywords\n",
        "    # Write code here.\n",
        "\n",
        "    keyphrases = [*set(keywords)]\n",
        "\n",
        "    return keyphrases\n",
        "print('\\033[1;32m We have successfully finished running this cell.')"
      ],
      "metadata": {
        "id": "griYxUw-oeLB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f527f630-39f7-4b62-f0b2-f3caabc523ba"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[1;32m We have successfully finished running this cell.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**To Do 6:** \n",
        "Apply a the keyword_extraction function to generated the keywords for the given text and print them.\n",
        "\n",
        "---\n",
        "\n"
      ],
      "metadata": {
        "id": "wl0LOdYye2Bc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text = \"Tumor initiating cells (TICs) possessing cancer stemness were shown to be enriched after therapy, resulting in the relapse and metastasis of head and neck squamous cell carcinomas (HNC). An effective therapeutic approach suppressing the HNC-TICs would be a potential method to improve the treatments for HNC. We observed that the treatment of silibinin (SB) dose dependently down-regulated the ALDH1 activity, CD133 positivity, stemness signatures expression, self-renewal property, and chemoresistance in ALDH1+CD44+ HNC-TICs. Using miRNA-microarray and mechanistic studies, SB increased the expression of microRNA-494 (miR-494) and both Bmi1 and ADAM10 were identified as the novel targets of miR-494. Moreover, overexpression of miR-494 results in a reduction in cancer stemness. However, knockdown of miR-494 in CD44−ALDH1- non-HNC-TICs enhanced cancer stemness and oncogenicity, while co-knockdown of Bmi1 and ADAM10 effectively reversed these phenomena. Mice model showed that SB treatment by oral gavage to xenograft tumors reduced tumor growth and prolonged the survival time of tumor-bearing mice by activation of miR-494-inhibiting Bmi1/ADAM10 expression. Survival analysis indicated that a miR494highBmi1lowADAM10low phenotype predicted a favourable clinical outcome. We conclude that the inhibition of tumor aggressiveness in HNC-TICs by SB was mediated by up-regulation miR-494, suggesting that SB would be a valuable anti-cancer drug for treatment of HNC.\"\n",
        "\n",
        "### Apply a the keyword_extraction function to generated the keywords for the given text and print them."
      ],
      "metadata": {
        "id": "e2c-Mjw4oeJ_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###**Step 6**\n",
        "Extract Keywords from csv entires and add it to the csv under the 'anchor_keywords' and 'target_keywords' columns."
      ],
      "metadata": {
        "id": "o1ZwFu-JJMW_"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**To Do 6:** \n",
        "Repeat the same as we did for target_keywords using target text."
      ],
      "metadata": {
        "id": "NcIOPDnhdaP1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df = pd.read_csv('/content/semanticClimate/paragraphLinking/total_pages_groups_table.csv')\n",
        "df[\"anchor_keywords\"] = df[\"anchor_text\"].apply(lambda x: keyword_extraction(x))\n",
        "\n",
        "### repeat the same for the target_text\n",
        "### write code here.\n",
        "df.to_csv('/content/total_pages_groups_table.csv', index=None)\n",
        "print('\\033[1;32m We have successfully finished running this cell.')"
      ],
      "metadata": {
        "id": "8StiI-1d9Hkn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fb81bce8-d223-43c5-9f72-1b95f51afdfa"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[1;32m We have successfully finished running this cell.\n"
          ]
        }
      ]
    }
  ]
}