Cross_Lingual_Similarity_and_Semantic_Search_Engine_with_TF_Hub_Multilingual_Universal_Encoder.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "TPU",
    "colab": {
      "name": "Cross-Lingual Similarity and Semantic Search Engine with TF-Hub Multilingual Universal Encoder",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "machine_shape": "hm"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RUymE2l9GZfO"
      },
      "source": [
        "**Copyright 2019 The TensorFlow Hub Authors.**\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "code",
        "id": "JMyTNwSJGGWg"
      },
      "source": [
        "# Copyright 2019 The TensorFlow Hub Authors. All Rights Reserved.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# =============================================================================="
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "co7MV6sX7Xto"
      },
      "source": [
        "# Cross-Lingual Similarity and Semantic Search Engine with Multilingual Universal Sentence Encoder\n",
        "\n",
        "\n",
        "<table align=\"left\"><td>\n",
        "  <a target=\"_blank\"  href=\"https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/cross_lingual_similarity_with_tf_hub_multilingual_universal_encoder.ipynb\">\n",
        "    <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab\n",
        "  </a>\n",
        "</td><td>\n",
        "  <a target=\"_blank\"  href=\"https://github.com/tensorflow/hub/blob/master/examples/colab/cross_lingual_similarity_with_tf_hub_multilingual_universal_encoder.ipynb\">\n",
        "    <img width=32px src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "</td></table>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eAVQGidpL8v5"
      },
      "source": [
        "This notebook illustrates how to access the Multilingual Universal Sentence Encoder module and use it for sentence similarity across multiple languages. This module is an extension of the [original Universal Encoder module](https://tfhub.dev/google/universal-sentence-encoder/2).\n",
        "\n",
        "The notebook is divided as follows:\n",
        "\n",
        "*   The first section shows a visualization of sentences between pair of languages. This is a more academic exercise.\n",
        "*   In the second section, we show how to build a semantic search engine from a sample of a Wikipedia corpus in multiple languages."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UvNRbHGarYeR"
      },
      "source": [
        "## Citation\n",
        "\n",
        "*Research papers that make use of the models explored in this colab should cite:*\n",
        "\n",
        "### [Multilingual universal sentence encoder for semantic retrieval](https://arxiv.org/abs/1907.04307)\n",
        "Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2019.\n",
        " arXiv preprint arXiv:1907.04307"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pOTzp8O36CyQ"
      },
      "source": [
        "# Getting Started\n",
        "\n",
        "This section sets up the environment for access to the Multilingual Universal Sentence Encoder Module and also prepares a set of English sentences and their translations. In the following sections, the multilingual module will be used to compute similarity *across languages*."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "63Pd3nJnTl-i"
      },
      "source": [
        "**IMPORTANT**Note: Pleaseelect \"**Python 3**\" _and_ \"**GPU**\" in the ***Runtime->Change Runtime type*** dropdown menu above _before_ running this notebook for faster execution."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lVjNK8shFKOC"
      },
      "source": [
        "%%capture\n",
        "#@title Setup Environment\n",
        "# Install the latest Tensorflow version.\n",
        "!pip3 install tensorflow_text\n",
        "!pip3 install --upgrade tensorflow-gpu\n",
        "!pip install tensorflow-hub\n",
        "!pip install bokeh\n",
        "!pip install simpleneighbors\n",
        "!pip install tqdm\n",
        "!pip install nltk"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MSeY-MUQo2Ha",
        "outputId": "5d305bae-1ead-4cb7-bc11-c70aa9093c19",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        }
      },
      "source": [
        "#@title Setup common imports and functions\n",
        "import bokeh\n",
        "import bokeh.models\n",
        "import bokeh.plotting\n",
        "import numpy as np\n",
        "import os\n",
        "import pandas as pd\n",
        "import tensorflow.compat.v2 as tf\n",
        "import tensorflow_hub as hub\n",
        "from tensorflow_text import SentencepieceTokenizer\n",
        "import sklearn.metrics.pairwise\n",
        "import nltk\n",
        "nltk.download('punkt')\n",
        "from simpleneighbors import SimpleNeighbors\n",
        "from tqdm import tqdm\n",
        "from tqdm import trange\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gk2IRjZFGDsK"
      },
      "source": [
        "This is additional boilerplate code where we import the pre-trained ML model we will use to encode text throughout this notebook."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mkmF3w8WGLcM"
      },
      "source": [
        "# The 16-language multilingual module is the default but feel free\n",
        "# to pick others from the list and compare the results.\n",
        "module_url = 'https://tfhub.dev/google/universal-sentence-encoder-multilingual/3' #@param ['https://tfhub.dev/google/universal-sentence-encoder-multilingual/3', 'https://tfhub.dev/google/universal-sentence-encoder-multilingual-large/3']\n",
        "\n",
        "model = hub.load(module_url)\n",
        "\n",
        "def embed_text(input):\n",
        "  return model(input)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jhLPq6AROyFk"
      },
      "source": [
        "# Visualize Text Similarity Between Languages\n",
        "With the sentence embeddings now in hand, we can visualize semantic similarity across different languages."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8xdAogbxJDTD"
      },
      "source": [
        "## Computing Text Embeddings\n",
        "\n",
        "We first define a set of sentences translated to various languages in parallel. Then, we precompute the embeddings for all of our sentences."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0M8eP8RQLeqK"
      },
      "source": [
        "# get cosine similairty matrix\n",
        "def cos_sim_data(data_feature_vectors,query_feature_vector):\n",
        "    similarities_scores = sklearn.metrics.pairwise.cosine_similarity(data_feature_vectors,query_feature_vector)\n",
        "    return similarities_scores"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sbe2aNAtIMcg",
        "outputId": "bbe1330b-3991-4f22-9848-54a0e1402e66",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 396
        }
      },
      "source": [
        "# read hadith file\n",
        "import pandas as pd\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer \n",
        "\n",
        "df = pd.read_excel(\"Sahih Bukhari Without_Tashkel.xlsx\")\n",
        "df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "FileNotFoundError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-6-ca1ca5ed0e87>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0msklearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeature_extraction\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtext\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mTfidfVectorizer\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mdf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_excel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Sahih Bukhari Without_Tashkel.xlsx\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0mdf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhead\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/excel/_base.py\u001b[0m in \u001b[0;36mread_excel\u001b[0;34m(io, sheet_name, header, names, index_col, usecols, squeeze, dtype, engine, converters, true_values, false_values, skiprows, nrows, na_values, keep_default_na, verbose, parse_dates, date_parser, thousands, comment, skipfooter, convert_float, mangle_dupe_cols, **kwds)\u001b[0m\n\u001b[1;32m    302\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    303\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mio\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mExcelFile\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 304\u001b[0;31m         \u001b[0mio\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mExcelFile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mio\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mengine\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mengine\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    305\u001b[0m     \u001b[0;32melif\u001b[0m \u001b[0mengine\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mengine\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mio\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mengine\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    306\u001b[0m         raise ValueError(\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/excel/_base.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, io, engine)\u001b[0m\n\u001b[1;32m    822\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_io\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstringify_path\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mio\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    823\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 824\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_reader\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_engines\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mengine\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_io\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    825\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    826\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__fspath__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/excel/_xlrd.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, filepath_or_buffer)\u001b[0m\n\u001b[1;32m     19\u001b[0m         \u001b[0merr_msg\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"Install xlrd >= 1.0.0 for Excel support\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     20\u001b[0m         \u001b[0mimport_optional_dependency\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"xlrd\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mextra\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0merr_msg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 21\u001b[0;31m         \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__init__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     22\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     23\u001b[0m     \u001b[0;34m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/excel/_base.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, filepath_or_buffer)\u001b[0m\n\u001b[1;32m    351\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbook\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload_workbook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    352\u001b[0m         \u001b[0;32melif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 353\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbook\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload_workbook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    354\u001b[0m         \u001b[0;32melif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbytes\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    355\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbook\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload_workbook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mBytesIO\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/pandas/io/excel/_xlrd.py\u001b[0m in \u001b[0;36mload_workbook\u001b[0;34m(self, filepath_or_buffer)\u001b[0m\n\u001b[1;32m     34\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mopen_workbook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfile_contents\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     35\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 36\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mopen_workbook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     37\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     38\u001b[0m     \u001b[0;34m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/xlrd/__init__.py\u001b[0m in \u001b[0;36mopen_workbook\u001b[0;34m(filename, logfile, verbosity, use_mmap, file_contents, encoding_override, formatting_info, on_demand, ragged_rows)\u001b[0m\n\u001b[1;32m    114\u001b[0m         \u001b[0mpeek\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfile_contents\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mpeeksz\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    115\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 116\u001b[0;31m         \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"rb\"\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    117\u001b[0m             \u001b[0mpeek\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpeeksz\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    118\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mpeek\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34mb\"PK\\x03\\x04\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# a ZIP file\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'Sahih Bukhari Without_Tashkel.xlsx'"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9jTKO9PDubbk"
      },
      "source": [
        "# def make a column that have uni and bi and tri gram for each hadith\n",
        "import json\n",
        "\n",
        "# this functions make a mgrams from 2 : 5 gram by Tf-idf for each row 'hadith'\n",
        "# make a dict for save index for each hadith and all compination of gram's\n",
        "# return all grams for each hadith as a dict {index: grams}\n",
        "# make a json file to save dict and to saving time\n",
        "\n",
        "def make_grams():\n",
        "  all_grams_clusters = {}\n",
        "  vectorizer_1_2_3_gram = TfidfVectorizer(analyzer='word', ngram_range=(2,5))\n",
        "  for i in range(1000): # df.shape[0]\n",
        "    grams_for_each_row = []\n",
        "    vectors_1_2_3_gram = vectorizer_1_2_3_gram.fit_transform([df['Sahih Bukhari Without_Tashkel'][i]])\n",
        "    feature_names_1_2_3_gram = vectorizer_1_2_3_gram.get_feature_names()\n",
        "    grams_for_each_row.append(feature_names_1_2_3_gram)\n",
        "    all_grams_clusters[str(i)] = grams_for_each_row\n",
        "\n",
        "    # # save all similarities with indeces as json\n",
        "    # with open('similarities.txt', 'w') as json_file:\n",
        "    #   json.dump(all_grams_clusters, json_file)\n",
        "\n",
        "  # print(all_grams_clusters)\n",
        "  return all_grams_clusters"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WL_Kq8mEjMBW"
      },
      "source": [
        "# cos_sim_data\n",
        "# get top N similarities for similar gram by embedding query and all grams for hadith\n",
        "def top_n_similarities(query, grams_clusters, n = 3):\n",
        "  all_sim_clusters = {}\n",
        "  top_n_similarities_list = []\n",
        "  for hadith_num, grams_list in grams_clusters.items():\n",
        "    similarities_array = cos_sim_data(np.array(embed_text(query)),embed_text(grams_list))\n",
        "    all_sim_clusters[hadith_num] = np.max(np.array(similarities_array).tolist())\n",
        "    all_sim_clusters = {k: v for k, v in sorted(all_sim_clusters.items(), key=lambda item: item[1] , reverse= True)}\n",
        "  for index, element in enumerate(all_sim_clusters.items()):\n",
        "    # print(index)\n",
        "    if index == n-1:\n",
        "      break    \n",
        "    top_n_similarities_list.append(element)\n",
        "  return top_n_similarities_list\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uAI0EvRnvDs4"
      },
      "source": [
        "query = 'إنما الأعمال بالنيات'\n",
        "grams_clusters = make_grams() # read json file\n",
        "print(top_n_similarities(query, grams_clusters , 5))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1QuMoVC7gchm"
      },
      "source": [
        "list_of_similarities_scores = top_n_similarities(query, grams_clusters , 5)\n",
        "for index, score in list_of_similarities_scores:\n",
        "  print('Hadith number in Bukhari: {}\\nHadiths is:\\n{}\\nScore: {}\\n{}'.format(int(index)+1, df['Sahih Bukhari Without_Tashkel'][int(index)], score, '-'*200))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hAgBBowV5a6e"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9_ubJKZAgPjI"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}