Skip to content

Implemented a search engine on the wikipedia dump of size 73.4 GB. In order to retrieve result faster and relevant, indexing and ranking is implemented. Relevance ranking algorithm is implemented using TF-IDF score to rank documents. Creating index takes around 14 hr on a given wikipedia dump. Result is retrieved in less than 1 second.

Notifications You must be signed in to change notification settings

VatsalSoni301/Wikipedia_Search_Engine

Repository files navigation

Wikipedia_Search_Engine

Description

  • Implemented a search engine on the wikipedia dump of size 73.4 GB. In order to retrieve result faster and relevant, indexing and ranking is implemented. Relevance ranking algorithm is implemented using TF-IDF score to rank documents. Creating index takes around 14 hr on a given wikipedia dump. Result is retrieved in less than 1 second.

Prerequisites

  • python3
  • For preprocessing and Stemming, I have used nltk library.
  • To install nltk pip3 install nltk
  • Install etree to parse wikipedia dump xml file
  • To install etree pip3 install elementpath
  • stop_words.txt file must be present in the same directory to remove stop words

To run the project

  • Change wikipedia dump filepath in create_index.py file
  • Change index file path to store index in create_index.py file
  • Command to run create_index file
  • python3 create_index.py
  • Change filepath of a index file to load index in the memory in search file.
  • Command to run search file
  • python3 search.py

Format of the query

  • It supports two types of query.
  • Normal query e.g. new york , gandhi , 1981 world cup
  • Field query e.g. title:gandhi body:arjun infobox:gandhi category:gandhi ref:gandhi
  • Top 10 results will be printed.

About

Implemented a search engine on the wikipedia dump of size 73.4 GB. In order to retrieve result faster and relevant, indexing and ranking is implemented. Relevance ranking algorithm is implemented using TF-IDF score to rank documents. Creating index takes around 14 hr on a given wikipedia dump. Result is retrieved in less than 1 second.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published