anserini_robust04_demo.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/castorini/anserini-notebooks/blob/master/anserini_robust04_demo.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vUpmSFflwzR1"
      },
      "source": [
        "# Anserini Demo on Robust04"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-2PzgU4YxIl3"
      },
      "source": [
        "This notebook provides a brief overview of how to use [Anserini](http://anserini.io) to perform an _ad hoc_ retrieval run over the test collection from the TREC 2004 Robust Track.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XR8bFJc0DgaE"
      },
      "source": [
        "## Setup\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zgn5DHN5DsmM"
      },
      "source": [
        "First, install Maven (Java 11 comes pre-installed already):\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NOOi0C5WDxKU"
      },
      "source": [
        "%%capture\n",
        "!apt-get install maven -qq"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mw8bngwQD9ix"
      },
      "source": [
        "Clone and build Anserini:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xyfg_YqjZrsP"
      },
      "source": [
        "%%capture\n",
        "!git clone --recurse-submodules https://github.com/castorini/anserini.git\n",
        "%cd anserini\n",
        "!cd tools/eval && tar xvfz trec_eval.9.0.4.tar.gz && cd trec_eval.9.0.4 && make && cd ../../..\n",
        "!mvn clean package appassembler:assemble -DskipTests -Dmaven.javadoc.skip=true"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FlBNRSESxfVJ"
      },
      "source": [
        "If all goes well, you should be able to see   `anserini-X.Y.Z-SNAPSHOT-fatjar.jar` in `target/`:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "txoRXWICcU6o",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "593dfcd2-7d6a-4f2f-bab9-9630a1bfb644"
      },
      "source": [
        "!ls target"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "anserini-0.21.1-SNAPSHOT-fatjar.jar   classes\t\t      maven-status\n",
            "anserini-0.21.1-SNAPSHOT.jar\t      generated-sources       test-classes\n",
            "anserini-0.21.1-SNAPSHOT-sources.jar  generated-test-sources\n",
            "appassembler\t\t\t      maven-archiver\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xbXmstbGypef"
      },
      "source": [
        "Let's grab the pre-built index:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JT_OJKftdqGP"
      },
      "source": [
        "%%capture\n",
        "!wget https://rgw.cs.uwaterloo.ca/pyserini/indexes/lucene-index.robust04.20221005.252b5e.tar.gz\n",
        "!tar xvfz lucene-index.robust04.20221005.252b5e.tar.gz -C indexes/"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EwVSvHBz9RMM"
      },
      "source": [
        "Sanity check of index size:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KVXWA6WS0aqJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b02085e5-b1ad-47db-dc89-3e656327d3f5"
      },
      "source": [
        "!du -h indexes/lucene-index.robust04.20221005.252b5e"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2.0G\tindexes/lucene-index.robust04.20221005.252b5e\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RJ2MrEUuEk4D"
      },
      "source": [
        "## Batch Retrieval and Evaluation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XiajTcO29Xjn"
      },
      "source": [
        "Let's run the queries from the TREC 2004 Robust Track, with BM25 as the ranking model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YZm8e5LrwIt5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3f6174ee-3438-4a00-bdd7-74a53520a8c2"
      },
      "source": [
        "!target/appassembler/bin/SearchCollection \\\n",
        " -index indexes/lucene-index.robust04.20221005.252b5e \\\n",
        " -topics src/main/resources/topics-and-qrels/topics.robust04.txt \\\n",
        " -topicreader Trec \\\n",
        " -output runs/run.robust04.bm25.txt \\\n",
        " -bm25"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2023-04-02 15:38:45,018 INFO  [main] search.SearchCollection (SearchCollection.java:928) - ============ Initializing Searcher ============\n",
            "2023-04-02 15:38:45,021 INFO  [main] search.SearchCollection (SearchCollection.java:929) - Index: indexes/lucene-index.robust04.20221005.252b5e\n",
            "2023-04-02 15:38:45,258 INFO  [main] search.SearchCollection (SearchCollection.java:933) - Fields: []\n",
            "2023-04-02 15:38:45,259 INFO  [main] search.SearchCollection (SearchCollection.java:691) - Using DefaultEnglishAnalyzer\n",
            "2023-04-02 15:38:45,260 INFO  [main] search.SearchCollection (SearchCollection.java:692) - Stemmer: porter\n",
            "2023-04-02 15:38:45,261 INFO  [main] search.SearchCollection (SearchCollection.java:693) - Keep stopwords? false\n",
            "2023-04-02 15:38:45,262 INFO  [main] search.SearchCollection (SearchCollection.java:694) - Stopwords file: null\n",
            "2023-04-02 15:38:45,344 INFO  [main] search.SearchCollection (SearchCollection.java:1208) - runtag: Anserini\n",
            "2023-04-02 15:38:45,348 INFO  [main] search.SearchCollection (SearchCollection.java:1214) - ============ Launching Search Threads ============\n",
            "2023-04-02 15:38:57,421 INFO  [pool-3-thread-8] search.SearchCollection$SearcherThread (SearchCollection.java:843) - ranker: bm25(k1=0.9,b=0.4), reranker: default: 100 queries processed\n",
            "2023-04-02 15:39:04,624 INFO  [pool-3-thread-7] search.SearchCollection$SearcherThread (SearchCollection.java:843) - ranker: bm25(k1=0.9,b=0.4), reranker: default: 200 queries processed\n",
            "2023-04-02 15:39:07,609 INFO  [pool-2-thread-1] search.SearchCollection$SearcherThread (SearchCollection.java:861) - ranker: bm25(k1=0.9,b=0.4), reranker: default: 250 queries processed in 00:00:22 = ~11.25 q/s\n",
            "2023-04-02 15:39:07,795 INFO  [main] search.SearchCollection (SearchCollection.java:1420) - Total run time: 00:00:22\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NkD0kKxW9mHP"
      },
      "source": [
        "Finally, let's use `trec_eval` to determine how good the results are:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yFZlcqEX0t1f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "20ca2468-d3ec-4709-ebc2-7cd0f4b5904e"
      },
      "source": [
        "!tools/eval/trec_eval.9.0.4/trec_eval -c src/main/resources/topics-and-qrels/qrels.robust04.txt runs/run.robust04.bm25.txt"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "runid                 \tall\tAnserini\n",
            "num_q                 \tall\t249\n",
            "num_ret               \tall\t241339\n",
            "num_rel               \tall\t17412\n",
            "num_rel_ret           \tall\t10272\n",
            "map                   \tall\t0.2531\n",
            "gm_map                \tall\t0.1456\n",
            "Rprec                 \tall\t0.2924\n",
            "bpref                 \tall\t0.2603\n",
            "recip_rank            \tall\t0.6769\n",
            "iprec_at_recall_0.00  \tall\t0.7158\n",
            "iprec_at_recall_0.10  \tall\t0.5286\n",
            "iprec_at_recall_0.20  \tall\t0.4268\n",
            "iprec_at_recall_0.30  \tall\t0.3541\n",
            "iprec_at_recall_0.40  \tall\t0.2789\n",
            "iprec_at_recall_0.50  \tall\t0.2299\n",
            "iprec_at_recall_0.60  \tall\t0.1786\n",
            "iprec_at_recall_0.70  \tall\t0.1404\n",
            "iprec_at_recall_0.80  \tall\t0.0866\n",
            "iprec_at_recall_0.90  \tall\t0.0553\n",
            "iprec_at_recall_1.00  \tall\t0.0281\n",
            "P_5                   \tall\t0.5004\n",
            "P_10                  \tall\t0.4382\n",
            "P_15                  \tall\t0.3987\n",
            "P_20                  \tall\t0.3631\n",
            "P_30                  \tall\t0.3102\n",
            "P_100                 \tall\t0.1837\n",
            "P_200                 \tall\t0.1250\n",
            "P_500                 \tall\t0.0689\n",
            "P_1000                \tall\t0.0413\n"
          ]
        }
      ]
    }
  ]
}