Find the extracted, ready-to-use corpus at github.com/lenakmeth/Wikinflection-Corpus.
Massive semi-supervised generation of multilingual inflectional corpus from Wiktionary. Corresponding publication: Metheniti, E., & Neumann, G. (2018, December). Wikinflection: Massive Semi-Supervised Generation of Multilingual Inflectional Corpus from Wiktionary. In Proceedings of the 17th International Workshop on Treebanks and Linguistic Theories (TLT 2018), December 13–14, 2018, Oslo University, Norway (No. 155, pp. 147-161). Linköping University Electronic Press.
- Download the XML wiki dump for the Englsih version of Wiktionary.
- Clone the project.
- Python3
- BeautifulSoup
- Pandas
From command line: python main.py [dump file]
- The first time running, module
pull_templates.pywill pull 7K text files from en.wiktionary.org. This process will take several hours, because of pauses between requests, and will use up ~210MB. - The output will be a .JSON file with inflectional paradigms, evaluated and corrected, separated per lemma, then separated by template ID. Every word in the inflectional paradigm has: template ID, [list of morphological features], POS, [prefixes, suffixes, [infixes]], stem.
- Currently, there are problems with the HTML table parsing (merged cells, nested tables in tables), and as a result not all features are identified per wordform. This is a known issue and is under work.
- Eleni Metheniti - Initial work - lenakmeth