Skip to content

Latest commit

 

History

History
155 lines (129 loc) · 23.7 KB

datasets.md

File metadata and controls

155 lines (129 loc) · 23.7 KB
Raw

Persian Language Processing Datasets

Table Of Contents

Part-of-Speech Tagger

  • Bijankhan Corpus - Bijankhan corpus is a tagged corpus that is suitable for natural language processing research on the Persian (Farsi) language. This collection is gathered form daily news and common texts. In this collection all documents are categorized into different subjects such as political, cultural and so on. Totally, there are 4300 different subjects. The Bijankhan collection contains about 2.6 millions manually tagged words with a tag set that contains 40 Persian POS tags.
  • Mojgan Seraji Corpus - Uppsala Persian Corpus (UPC) is a large, freely available Persian corpus. The corpus is a modified version of the Bijankhan corpus with additional sentence segmentation and consistent tokenization containing 2,704,028 tokens and annotated with 31 part-of-speech tags. The part-of-speech tags are listed with explanations in this table.
  • Large-Scale Colloquial Persian - Large Scale Colloquial Persian Dataset (LSCP) is hierarchically organized in asemantic taxonomy that focuses on multi-task informal Persian language understanding as a comprehensive problem. LSCP includes 120M sentences from 27M casual Persian tweets with its dependency relations in syntactic annotation, Part-of-speech tags, sentiment polarity and automatic translation of original Persian sentences in English (EN), German (DE), Czech (CS), Italian (IT) and Hindi (HI) spoken languages. Learn more about this project at LSCP webpage.

Named Entity Recognition

  • ArmanPersoNERCorpus - The dataset includes 250,015 tokens and 7,682 Persian sentences in total. It is available in 3 folds to be used in turn as training and test sets. Each file contains one token, along with its manually annotated named-entity tag, per line. Each sentence is separated with a newline. The NER tags are in IOB format.
  • FarsiYar PersianNER - The dataset includes about 25,000,000 tokens and about 1,000,000 Persian sentences in total based on Persian Wikipedia Corpus. The NER tags are in IOB format. More than 1000 volunteers contributed tag improvements to this dataset via web panel or android app. They release updated tags every two weeks.
  • Workshop on NLP Solutions for Under Resourced Languages (NSURL) 2019 - Task 7 dataset - contains a medium size NER corpus with 7 classes of named entities (person, location and organization, money, percent, dates, and time). This corpus contains more than 700 news documents.

Relation Extraction

  • PERLEX - The first Persian dataset for relation extraction, which is an expert translated version of the “Semeval-2010-Task-8” dataset. Link to the relevant publication.

Dependency Parsing

  • Persian Syntactic Dependency Treebank - This treebank is supplied for free noncommercial use. For commercial uses feel free to contact us. The number of annotated sentences is 29,982 sentences including samples from almost all verbs of the Persian valency lexicon.
  • Uppsala Persian Dependency Treebank: UPDT - Dependency-based syntactically annotated corpus.
  • Pretrained model
  • Universal Dependencies 1.3 - Multi lingual corpus that holds IOB gold data for dependency parsing
  • HamleDT 3.0 - HArmonized Multi-LanguagE Dependency Treebank is a compilation of existing dependency treebanks (or dependency conversions of other treebanks), transformed so that they all conform to the same annotation style. This version uses Universal Dependencies as the common annotation style.
  • Large-Scale Colloquial Persian - Large Scale Colloquial Persian Dataset (LSCP) is hierarchically organized in asemantic taxonomy that focuses on multi-task informal Persian language understanding as a comprehensive problem. LSCP includes 120M sentences from 27M casual Persian tweets with its dependency relations in syntactic annotation, Part-of-speech tags, sentiment polarity and automatic translation of original Persian sentences in English (EN), German (DE), Czech (CS), Italian (IT) and Hindi (HI) spoken languages. Learn more about this project at LSCP webpage.
  • MULTEXT-East - The MULTEXT-East morphosyntactic lexicons have a simple structure, where each line is a lexical entry with three tab-separated fields: (1) the word-form, the inflected form of the word; (2) the lemma, the base-form of the word; (3) the MSD, the morphosyntactic description of the word-form, i.e., its fine-grained PoS tag, as defined in the MULTEXT-East morphosyntactic specifications.

Text Categorization and Classification

  • Hamshahri - Hamshahri collection is a standard reliable Persian text collection that was used at Cross Language Evaluation Forum (CLEF) during years 2008 and 2009 for evaluation of Persian information retrieval systems.
  • Bijankhan Corpus - Bijankhan corpus is a tagged corpus that is suitable for natural language processing research on the Persian (Farsi) language. This collection is gathered form daily news and common texts. In this collection all documents are categorized into different subjects such as political, cultural and so on. Totally, there are 4300 different subjects. The Bijankhan collection contains about 2.6 millions manually tagged words with a tag set that contains 40 Persian POS tags.
  • Digikala Magazine (DigiMag) - A total of 8,515 articles scraped from Digikala Online Magazine. This dataset includes seven different classes Video Games, Shopping Guide, Health Beauty, Science Technology, General, Art Cinema, and Books Literature.
  • Persian News - A dataset of various news articles scraped from different online news agencies’ websites. The total number of articles is 16,438, spread over eight different classes, Economic, International, Political, Science Technology, Cultural Art, Sport, and Medical.

Spell Checking

  • FAspell - FASpell dataset was developed for the evaluation of spell checking algorithms. It contains a set of pairs of misspelled Persian words and their corresponding corrected forms similar to the ASpell dataset used for English.
  • Persian-Spell-checker - We're collecting persian words' dictionary (verbs, nouns, and etc.) for Persian spell checker.
  • Grammar and context sensitive spell checker - It is a real-world test set for grammatical errors and context sensitive spelling errors for Persian language. This test set contains 1100 context sensitive errors and was collected from Persian Blogs.
  • Spell Checker - Test set for spelling errors for Persian language.​
  • CPG: Corpus of Persian Grammatical Errors - It is a fully-annotated corpus of grammatical errors collected from 700 essays written by learners of Persian language in Dehkhoda Lexicon Institute & International Centre for Persian Studies and Imam Khomeini International University.
  • PerSpellData - Comprehensive parallel dataset for persian non-word and real-word errors
  • HeKasre - Detect and correct misspelled "e" sound in Persian (aka Farsi) writing (especially in an informal setting)
  • Lilak - Persian Spell Checking Dictionary

Textual Entailment

  • FarsTail - FarsTail is a dataset of textual entailment (also known as natural language inference, NLI) and it includes 10,367 samples in the Persian language. Here is the relevant paper.

Textual Thematic Similarity

  • Wikipedia Section Sentences - It implements the idea from Dor et al., 2018, Learning Thematic Similarity Metric Using Triplet Networks for Persian. The dataset includes 205,768 examples that covered 21,515 articles and 34,298 sections.
  • Wiki Triplet - A triplet-objective dataset extracted from Wikipedia Section Sentences into a triplet-form of anchor (), positive () and negative () examples. It covers 191,929 samples.
  • Wiki D-Similar - Wiki D-Similar is another form of thematic similarity dataset with 137,402 records that tags pairs of sentences into a form of similar or dissimilar.

Persian Poems And Classic Texts

  • Farsi Poem Corpus - This corpus consists of text documents for 48 Persian poets. The corpus comes in three formats; original, normalized (only 32 main Farsi alphabet), and stop words removed. The corpus consists of 1,216,286 mesras of Farsi poems and 8,102,119 words from which 148,588 are unique.

Sentiment Analysis

  • NRC-Persian-Lexicon - NRC Word-Emotion Association Lexicon useful for persian sentiment analysis.
  • Digikala OpenData Sentiment - data contains 100,000 samples of users' comments in favor of products labeled as no_idea, not_recommended , and recommended .
  • Pars-ABSA - Manually annotated Persian dataset, verified by 3 native Persian speakers. The dataset consists of 5,114 positive, 3,061 negative and 1,827 neutral data samples from 5,602 unique reviews.
  • PerSent - This dataset presents real-valued polarity labels, in the range from -1 to 1, for thousands of Persian words and expressions.
  • DeepSentiPers - Novel Deep Learning Models Trained Over Proposed Augmented Persian Sentiment Corpus
  • SentiPers - Documents in SentiPers are manually annotated at different levels.
  • LexiPers - An ontology based sentiment lexicon for Persian.
  • SnappFood - Snappfood (an online food delivery company) user comments containing 70,000 comments with two labels (i.e. polarity classification), Happy and Sad.
  • MirasOpinion - This repository contains information about MirasOpinion dataset, which is the largest Persian sentiment analysis dataset up to this date, alongside a demo file which contains 20 documents with their corresponding labels.
  • EmoPars - A Collection of 30K Emotion-Annotated Persian Social MediaTexts
  • ArmanEmo - A non-commercial version of ArmanEMO,A high quality human-labeled emotion dataset of more than 7000 Persian sentences labeled for seven categories, Labels are based on Ekman's six basic emotions (Anger, Fear, Happiness, Hatred, Sadness, Wonder) and another category (Other).
  • Persian tweets emotional dataset

Summarization

  • Wiki Summary - Wiki Summary is a summarization dataset extracted from Persian Wikipedia into the form of articles and highlights.

Question Answering

  • PersianQA - Persian Question Answering (PersianQA) Dataset is a reading comprehension dataset on Persian Wikipedia. The crowd-sourced dataset consists of more than 9,000 entries. Each entry can be either an impossible-to-answer or a question with one or more answers spanning in the passage (the context) from which the questioner proposed the question. Much like the SQuAD2.0 dataset, the impossible or unanswerable questions can be utilized to create a system which "knows that it doesn't know the answer".
  • ParsVQA-Caps - A Benchmark for Visual Question Answering and Image Captioning in Persian
  • MeDiaPQA - A Question-Answering Dataset on Persian Medical Dialogues
  • Persian QA Wikipedia - A Question Anwering Dataset on wikipedia paragraphs

Irony - Insult

  • MirasIrony - The irony dataset is constructed from Persian tweets. 2942 tweets are labeled in total.

Corpus

  • TEP: Tehran English-Persian Parallel Corpus - First free English-Persian corpus.

  • OPUS: the open parallel corpus - OPUS is a growing collection of translated texts from the web. In the OPUS project we try to convert and align free online data, to add linguistic annotation, and to provide the community with a publicly available parallel corpus. OPUS is based on open source products and the corpus is also delivered as an open content package. We used several tools to compile the current collection. All pre-processing is done automatically. No manual corrections have been carried out.

  • Tanzil project - Tanzil project is a collection of Quran translations to many languages, including Persian.

  • Large-Scale Colloquial Persian - Large Scale Colloquial Persian Dataset (LSCP) is hierarchically organized in asemantic taxonomy that focuses on multi-task informal Persian language understanding as a comprehensive problem. LSCP includes 120M sentences from 27M casual Persian tweets with its dependency relations in syntactic annotation, Part-of-speech tags, sentiment polarity and automatic translation of original Persian sentences in English (EN), German (DE), Czech (CS), Italian (IT) and Hindi (HI) spoken languages. Learn more about this project at LSCP webpage.

  • MIZAN - a Persian-English parallel corpus with about 1 million sentence pairs collected from masterpieces of literature. Here is the relevant paper.

  • PEPC: Parallel English-Persian Corpus Extracted from Wikipedia - a collection of parallel sentences in English and Persian languages extracted from Wikipedia documents using a bidirectional translation method.

  • Bible corpus - A multilingual parallel corpus created from translations of the Bible. The corpus also contains a Persian Bibble.

  • Transliteration - Transliteration extracted from a Persian novel book which written in both Arabic and Dabire. According to the first and last words of each sentence, we manually aligned the sentences of this book. Then for checking and eliminating the errors, the length of the sentences in both sides was compared. Finally, our parallel corpus with 13933 sentence pairs, 155623 words in Persian text and 170702 Dabire words was created.

  • TMC: Tehran Monolingual Corpus - The Tehran Monolingual Corpus (TMC) is a large-scale Persian monolingual corpus. TMC is suited for Language Modeling and relevant research areas in Natural Language Processing. The corpus is extracted from Hamshahri Corpus and ISNA news agency website. The quality of Hamshahri corpus is improved for language modeling purpose by a series of tokenization and spell-checking steps.

  • VOA Persian Corpus - A medium-sized corpus of 7.9 million words, 2003-2008. The corpus is in the public domain, so no copyright restrictions.

  • MirasText: Automatically Extracted Text Persian Corpus (about 12GB).

  • A large collection of Persian raw text - About 80GB Persian raw text, collected from a variety of sources, particularly CommonCrawl.

  • W2C – Web to Corpus – Corpora - A set of corpora for 120 languages automatically collected from wikipedia and the web.

  • dotIR Collection - dotIR is a standard Persian test collection that is suitable for evaluation of web information retrieval algorithms in Iranian web.dotIR Contains many Persian web pages including their text, links, metadata, etc that are stored in XML format. It is prepared in such a way to be a good representative of Iranian web.It is A good test bed for evaluation of link based information retrieval algorithms. It includes enough Queries and relevance judgments for a valid evaluation.It is not very large, so that it does not require high processing resources.

  • Hamshahri - Hamshahri collection is a standard reliable Persian text collection that was used at Cross Language Evaluation Forum (CLEF) during years 2008 and 2009 for evaluation of Persian information retrieval systems.

  • Prallel Gold Data from Wikipedia - This dataset contains parallel sentences, which are tagged from 33 wikipedia pages.

  • PREDICT - Persian REverse DICTionary

  • Iranian politicians twitter dataset persian

  • iPerUDT - Informal Persian Universal Dependency Treebank

  • Tasnim News - Tasnim news dataset

  • Asriran News - Asriran news dataset

  • Isna News Agency - Isna news agency dataset

  • Ensani.ir Abstrats - Abstracts extracted from ensani.ir

  • Tarjoman Articles - Extracted Tarjoman Articles

Stop Word Lists

Knowledge Bases

  • FarsBase - FarsBase the first Persian multi-source knowledge graph, which isspecifically designed for semantic search engines to support Persian knowledge. FarsBase uses a diverse set of hybrid and flexible techniques to extract and integrate knowledge from various sources, such as Wikipedia, Web tables and unstructured texts. Here is the relevant paper.

Intent Detection & Slot filling

  • Persian-ATIS - A Persian Benchmark for Joint Intent Detection and Slot Filling

Paraphrasing

  • ExaPPC - A Large-Scale Persian Paraphrase Detection Corpus

Text-to-Speech(TTS)

MISC

  • ParsiNLU - A collection natural language understanding datasets for Persian. Here is the relevant paper.
  • PersianStemmingDataset - PersianStemmingDataset is consist of two manually stemmed persian corpora and an evalution tools in order to compute stemming evaluatin metrics.
  • PersPred - PersPred, is the first online multilingual syntactic and semantic database of Persian compound verbs (complex predicates), developed by the members of the research unit Mondes iranien et indien (CNRS, Sorbonne Nouvelle, Inalco, EPHE) within the ANR-DFG project PERGRAM (2008-2012) and the LR4.1 work package of the Strand 6 of the Labex Empirical Foundations of Linguistics (EFL).
  • Popularity Prediction - It is Tabnak and Alef Datasets which are the most famous online news agencies in Iran. This dataset includes content, title, date, category and number of comments per each news. Besides popularity of these websites, the wide range of news categories they cover and they have the multilevel commenting structure.
  • Conversation Threads Prediction - It consists of five Datasets. These datasets have been crawled from 5 websites, including Thestandard , Alef , ENENews , Russianblog and Courantblogs Datasets (XML format). Thay are selected due to several reasons:
  • Iran and COVID-19 on Social Media - Content analysis of Persian Tweets during COVID-19 pandemic in Iran using NLP
  • SBU-WSD-Corpus - A Sense Annotated Corpus for Persian All-Words Word Sense Disambiguation