Language detector

This is an implementation of a software which can detect language of texts. It's designed for detecting the language in longer texts such as scientific-papers, which can come in *.txt or *.pdf format. The language detector introduces a custom algorithm which is referred as for detecting the "spatial language identificator (sli) comparison". It aims to use as less external libraries for the classification as possible.

Program overview

Test Results

There we're 12 documents checked for testing. The document pool exists of scientific papers, wikipedia-articles and brochures on countries in different languages. In the pool there we're 3 english, 3 german, 3 french, 2 thai and 1 spanish document. These documents are not provided here because of licensing. All of these documents were classified correctly. The system was trained with input data in the same languages.

The first document was a scientific paper on sentiment-analysis, which is availible here. Result output can be seen below.

Results______________________________________________________
Input:     io_data/langdet/document_rsocher.pdf
Det. lang: en                       
_____________________________________________________________
language   distance                  likeliness               
en         6375.422232185437         68.11193438031768        
fr         7032.559338922403         64.82511973218834        
sp         7586.859986309798         62.05266407777083        
de         7835.992906497068         60.8065714085122         
th         19993.129430373243        0.0            

Pro's and Con's

Pro's:

• learning language datasets is fast, under 3 minutes for 6 languages
• the saved sli's for identification take nearly no disk space (for 6 languages it's below 85 kb)
• making comparison is fast and doesn't take much cpu to process
• learn dataset is availible freely in nearly any language (bible)
• detection with test dataset of 12 documents in different languages was 100% accurate
• simple algorithm, which could be easily adapted in other programming languages and even used on microcontrollers

Con's:

• much text required compared to dictionary approach
• no measurement that a text isn't of any of the supported languages

Usage

• Adapt read in file config in "configurations/language_detector.conf".
• Execute 'langedet_create_dataset.py' for creating some comparison sli's.
• Execute 'langdet_check_language.py' for language identification of your specified documents, results appear in stdout

IDE for development was PyChar Community Edition by Jetbrains

Possible future improvements

• instead of using least mean square comparison, train a neural classificator for distinquishing input texts. The training could be with chapter wise info from the training data
• pre-filter non-text info from pdf-data
• filter the charset used for comparison in the sli-objects
• provide more language support, by learning in more bibles
• make an adaptive web-interface, which allows to add correctly classified sli's to the comparison dataset
• create an n-gram based sli, to take character follow up sequences to account
• take one international version of the bible in many languages as learn in dataset
• find treshold or algorithm to detect texts which are not any of the supported languages
• find alternative for tika webservice for pdf-reading

References used

• book graphic in diagram by Abilngeorge and under CC-License here
• tika library is used to get *.pdf-file content. The library contacts a web service for that
• for creating documentation diagrams yED was used
• bibles for learning in can be obtained from ebible.org