The goal is to identify the language in which a document, message, or sentence is written. To achieve this goal, I propose the following methodology. First, to transform a text into character n-gram features. Second, to build a machine-learned classifier to predict the language in which a text is written, using those character n-gram features to support the predictive power. Last, to evaluate the performance of the approach proposed here on a variety of datasets and against some of the most popular language identification libraries in the Python ecosystem.
Moreover, it is worth making clear that the language identification system will be limited to 14 out of 24 official languages of the European Union, namely:
- Czech
- Danish
- Dutch
- English
- Finnish
- French
- German
- Hungarian
- Italian
- Polish
- Portuguese
- Romanian
- Spanish
- Swedish
The above languages were chosen due to data availability matters, but mostly because their alphabet is based on modern Latin.
The entire solution approach is available as a Jupyter notebook. The notebook shows the source code and a more detailed description of the experimentation. Likewise, the author invites the reader to visit this short description of the proposed approach to language identification.