Skip to content

Latest commit

 

History

History
2115 lines (1984 loc) · 153 KB

README.md

File metadata and controls

2115 lines (1984 loc) · 153 KB

lingua

Build Status codecov supported languages Go Reference Go Report Card license


1. What does this library do?

Its task is simple: It tells you which language some text is written in. This is very useful as a preprocessing step for linguistic data in natural language processing applications such as text classification and spell checking. Other use cases, for instance, might include routing e-mails to the right geographically located customer service department, based on the e-mails' languages.

2. Why does this library exist?

Language detection is often done as part of large machine learning frameworks or natural language processing applications. In cases where you don't need the full-fledged functionality of those systems or don't want to learn the ropes of those, a small flexible library comes in handy.

So far, the only other comprehensive open source library in the Go ecosystem for this task is Whatlanggo. Unfortunately, it has two major drawbacks:

  1. Detection only works with quite lengthy text fragments. For very short text snippets such as Twitter messages, it does not provide adequate results.
  2. The more languages take part in the decision process, the less accurate are the detection results.

Lingua aims at eliminating these problems. She nearly does not need any configuration and yields pretty accurate results on both long and short text, even on single words and phrases. She draws on both rule-based and statistical methods but does not use any dictionaries of words. She does not need a connection to any external API or service either. Once the library has been downloaded, it can be used completely offline.

3. Which languages are supported?

Compared to other language detection libraries, Lingua's focus is on quality over quantity, that is, getting detection right for a small set of languages first before adding new ones. Currently, the following 75 languages are supported:

  • A
    • Afrikaans
    • Albanian
    • Arabic
    • Armenian
    • Azerbaijani
  • B
    • Basque
    • Belarusian
    • Bengali
    • Norwegian Bokmal
    • Bosnian
    • Bulgarian
  • C
    • Catalan
    • Chinese
    • Croatian
    • Czech
  • D
    • Danish
    • Dutch
  • E
    • English
    • Esperanto
    • Estonian
  • F
    • Finnish
    • French
  • G
    • Ganda
    • Georgian
    • German
    • Greek
    • Gujarati
  • H
    • Hebrew
    • Hindi
    • Hungarian
  • I
    • Icelandic
    • Indonesian
    • Irish
    • Italian
  • J
    • Japanese
  • K
    • Kazakh
    • Korean
  • L
    • Latin
    • Latvian
    • Lithuanian
  • M
    • Macedonian
    • Malay
    • Maori
    • Marathi
    • Mongolian
  • N
    • Norwegian Nynorsk
  • P
    • Persian
    • Polish
    • Portuguese
    • Punjabi
  • R
    • Romanian
    • Russian
  • S
    • Serbian
    • Shona
    • Slovak
    • Slovene
    • Somali
    • Sotho
    • Spanish
    • Swahili
    • Swedish
  • T
    • Tagalog
    • Tamil
    • Telugu
    • Thai
    • Tsonga
    • Tswana
    • Turkish
  • U
    • Ukrainian
    • Urdu
  • V
    • Vietnamese
  • W
    • Welsh
  • X
    • Xhosa
  • Y
    • Yoruba
  • Z
    • Zulu

4. How good is it?

Lingua is able to report accuracy statistics for some bundled test data available for each supported language. The test data for each language is split into three parts:

  1. a list of single words with a minimum length of 5 characters
  2. a list of word pairs with a minimum length of 10 characters
  3. a list of complete grammatical sentences of various lengths

Both the language models and the test data have been created from separate documents of the Wortschatz corpora offered by Leipzig University, Germany. Data crawled from various news websites have been used for training, each corpus comprising one million sentences. For testing, corpora made of arbitrarily chosen websites have been used, each comprising ten thousand sentences. From each test corpus, a random unsorted subset of 1000 single words, 1000 word pairs and 1000 sentences has been extracted, respectively.

Given the generated test data, I have compared the detection results of Lingua and Whatlanggo running over the data of Lingua's supported 75 languages. Additionally, I have added Google's CLD3 to the comparison with the help of the gocld3 bindings. Languages that are not supported by CLD3 or Whatlanggo are simply ignored during the detection process.

Each of the following sections contains two plots. The bar plot shows the detailed accuracy results for each supported language. The box plot illustrates the distributions of the accuracy values for each classifier. The boxes themselves represent the areas which the middle 50 % of data lie within. Within the colored boxes, the horizontal lines mark the median of the distributions.

4.1 Single word detection


Single Word Detection Performance


Bar plot Single Word Detection Performance



4.2 Word pair detection


Word Pair Detection Performance


Bar plot Word Pair Detection Performance



4.3 Sentence detection


Sentence Detection Performance


Bar plot Sentence Detection Performance



4.4 Average detection


Average Detection Performance


Bar plot Average Detection Performance



4.5 Mean, median and standard deviation

The table below shows detailed statistics for each language and classifier including mean, median and standard deviation.

Open table
Language Average Single Words Word Pairs Sentences
Lingua
(high accuracy mode)
Lingua
(low accuracy mode)
  CLD3   Whatlang Lingua
(high accuracy mode)
Lingua
(low accuracy mode)
  CLD3   Whatlang Lingua
(high accuracy mode)
Lingua
(low accuracy mode)
  CLD3   Whatlang Lingua
(high accuracy mode)
Lingua
(low accuracy mode)
  CLD3   Whatlang
Afrikaans 79 64 55 51 58 38 22 21 81 62 46 39 97 93 98 92
Albanian 88 80 55 - 69 54 18 - 95 86 48 - 100 99 98 -
Arabic 98 94 90 89 96 88 79 77 99 96 92 91 100 99 100 99
Armenian 100 100 99 - 100 100 100 - 100 100 100 - 100 100 97 -
Azerbaijani 90 82 81 64 77 71 62 45 92 78 82 58 99 96 99 91
Basque 84 75 62 - 71 56 33 - 87 76 62 - 93 92 92 -
Belarusian 97 92 84 81 92 80 67 64 99 95 86 80 100 100 100 98
Bengali 100 100 99 100 100 100 98 100 100 100 99 100 100 100 99 100
Bokmal 58 50 - 34 39 27 - 15 59 47 - 28 77 75 - 60
Bosnian 35 29 33 - 29 23 19 - 35 29 28 - 41 36 52 -
Bulgarian 87 78 70 61 70 56 45 37 91 81 66 57 99 96 98 89
Catalan 70 58 48 - 51 33 19 - 74 60 42 - 87 82 84 -
Chinese 100 100 92 100 100 100 92 100 100 100 83 100 100 100 100 100
Croatian 73 60 42 55 53 36 26 28 74 57 42 44 90 86 58 91
Czech 80 71 64 50 66 54 39 31 84 72 65 46 91 87 88 71
Danish 81 70 58 47 61 45 26 24 84 70 54 38 98 95 95 79
Dutch 77 64 58 47 55 36 29 22 81 61 47 36 96 94 97 82
English 81 63 54 49 55 29 22 17 89 62 44 35 99 97 97 94
Esperanto 84 66 57 52 67 44 22 25 85 61 51 45 98 93 98 88
Estonian 92 83 70 61 80 62 41 36 96 88 69 53 100 99 99 94
Finnish 96 91 80 71 90 77 58 45 98 95 84 70 100 100 99 98
French 89 77 55 64 74 52 22 37 94 83 49 59 99 98 94 97
Ganda 91 84 - - 79 65 - - 95 87 - - 100 100 - -
Georgian 100 100 98 100 100 100 99 100 100 100 100 100 100 100 96 100
German 89 80 66 65 74 57 40 38 94 84 62 60 100 99 98 97
Greek 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
Gujarati 100 100 100 100 100 100 99 100 100 100 100 100 100 100 100 100
Hebrew 100 100 - 90 100 100 - 76 100 100 - 94 100 100 - 99
Hindi 73 33 58 52 61 11 34 27 64 20 45 40 95 67 95 88
Hungarian 95 90 76 62 87 77 53 37 98 94 76 53 100 100 99 95
Icelandic 93 88 71 - 83 72 42 - 97 92 70 - 100 99 99 -
Indonesian 61 47 46 67 39 25 26 39 61 46 45 66 83 71 66 95
Irish 91 85 67 - 82 70 42 - 94 90 66 - 96 95 94 -
Italian 87 71 62 56 69 42 31 25 92 74 57 47 100 98 98 96
Japanese 100 100 98 99 100 100 97 100 100 100 96 100 100 100 100 97
Kazakh 92 90 82 - 80 78 62 - 96 93 83 - 99 99 99 -
Korean 100 100 99 100 100 100 100 100 100 100 100 100 100 100 98 100
Latin 87 73 62 - 72 49 44 - 93 76 58 - 97 94 83 -
Latvian 93 87 75 59 85 75 51 36 97 90 77 54 99 97 98 87
Lithuanian 95 87 72 62 86 76 42 38 98 89 75 56 100 98 99 92
Macedonian 84 72 60 62 66 52 30 39 86 70 54 55 99 95 97 94
Malay 31 31 22 - 26 22 11 - 38 36 22 - 28 35 34 -
Maori 91 82 52 - 82 62 22 - 92 87 43 - 99 98 91 -
Marathi 85 39 84 73 74 16 69 52 85 30 84 74 96 72 98 93
Mongolian 97 95 83 - 93 89 63 - 99 98 87 - 99 99 99 -
Nynorsk 66 52 - 34 41 25 - 10 66 49 - 24 91 81 - 69
Persian 90 80 76 70 78 62 57 46 94 80 70 66 100 98 99 99
Polish 95 90 77 66 85 77 51 45 98 93 80 59 100 99 99 94
Portuguese 81 69 53 57 59 42 21 26 85 70 40 48 99 95 97 96
Punjabi 100 100 100 100 100 100 99 100 100 100 100 100 100 100 100 100
Romanian 87 72 53 59 69 49 24 34 92 74 48 52 99 94 88 90
Russian 90 78 71 53 76 59 48 40 95 84 72 52 98 92 93 68
Serbian 88 78 78 57 74 62 63 34 90 80 75 51 99 91 95 86
Shona 91 81 76 68 78 56 51 44 96 86 79 65 100 100 99 95
Slovak 84 75 63 - 64 49 32 - 90 78 61 - 99 97 96 -
Slovene 82 67 63 48 61 39 29 25 87 68 60 38 99 93 99 81
Somali 92 85 69 68 82 64 38 38 96 90 70 66 100 100 100 99
Sotho 86 72 49 - 67 43 15 - 90 75 33 - 100 97 98 -
Spanish 70 56 48 48 44 26 16 19 69 49 32 33 97 94 96 93
Swahili 81 70 57 - 60 43 25 - 84 68 49 - 98 97 98 -
Swedish 84 72 61 49 64 46 30 24 88 76 56 40 99 94 96 83
Tagalog 78 66 - 52 52 36 - 23 83 67 - 43 98 96 - 90
Tamil 100 100 100 100 100 100 100 100 100 100 100 100 100 100 99 100
Telugu 100 100 99 100 100 100 99 100 100 100 100 100 100 100 99 100
Thai 100 100 99 100 100 100 100 100 100 100 100 100 100 100 98 99
Tsonga 84 72 - - 66 46 - - 89 73 - - 98 97 - -
Tswana 84 71 - - 65 44 - - 88 73 - - 99 96 - -
Turkish 94 87 69 54 84 71 41 26 98 91 70 44 100 100 97 92
Ukrainian 92 86 81 72 84 75 62 53 97 92 83 71 95 93 98 93
Urdu 91 80 61 57 80 65 39 31 94 78 53 46 98 96 92 94
Vietnamese 91 87 66 73 79 76 26 36 94 87 74 85 99 98 99 97
Welsh 91 82 69 - 78 61 43 - 96 87 66 - 99 99 98 -
Xhosa 82 69 66 - 64 45 40 - 85 67 65 - 98 94 92 -
Yoruba 74 62 15 22 50 33 5 11 77 61 11 14 96 92 28 41
Zulu 81 70 63 70 62 45 35 44 83 72 63 68 97 94 92 98
Mean 86 77 69 67 74 61 48 48 89 78 67 63 96 93 93 91
Median 89.0 80.0 68.0 62.0 74.0 57.0 41.0 38.0 94.0 81.0 66.0 57.0 99.0 97.0 98.0 94.0
Standard Deviation 13.08 17.29 19.04 20.22 18.41 24.9 27.86 29.28 13.12 18.93 21.83 24.22 11.05 11.91 13.95 11.24

5. Why is it better than other libraries?

Every language detector uses a probabilistic n-gram model trained on the character distribution in some training corpus. Most libraries only use n-grams of size 3 (trigrams) which is satisfactory for detecting the language of longer text fragments consisting of multiple sentences. For short phrases or single words, however, trigrams are not enough. The shorter the input text is, the less n-grams are available. The probabilities estimated from such few n-grams are not reliable. This is why Lingua makes use of n-grams of sizes 1 up to 5 which results in much more accurate prediction of the correct language.

A second important difference is that Lingua does not only use such a statistical model, but also a rule-based engine. This engine first determines the alphabet of the input text and searches for characters which are unique in one or more languages. If exactly one language can be reliably chosen this way, the statistical model is not necessary anymore. In any case, the rule-based engine filters out languages that do not satisfy the conditions of the input text. Only then, in a second step, the probabilistic n-gram model is taken into consideration. This makes sense because loading less language models means less memory consumption and better runtime performance.

In general, it is always a good idea to restrict the set of languages to be considered in the classification process using the respective api methods. If you know beforehand that certain languages are never to occur in an input text, do not let those take part in the classifcation process. The filtering mechanism of the rule-based engine is quite good, however, filtering based on your own knowledge of the input text is always preferable.

6. Test report generation

If you want to reproduce the accuracy results above, you can generate the test reports yourself for both classifiers and all languages by doing:

cd cmd
go run accuracy_reporter.go

For gocld3 to run successfully, you need to install the exact version 3.17.3 of Google's protocol buffers which is a bit unfortunate. For each detector and language, a test report file is then written into /accuracy-reports. As an example, here is the current output of the Lingua German report:

##### German #####

>>> Accuracy on average: 89.23%

>> Detection of 1000 single words (average length: 9 chars)
Accuracy: 73.90%
Erroneously classified as Dutch: 2.30%, Danish: 2.10%, English: 2.00%, Latin: 1.90%, Bokmal: 1.60%, Basque: 1.20%, French: 1.20%, Italian: 1.20%, Esperanto: 1.10%, Swedish: 1.00%, Afrikaans: 0.80%, Tsonga: 0.70%, Nynorsk: 0.60%, Portuguese: 0.60%, Yoruba: 0.60%, Finnish: 0.50%, Sotho: 0.50%, Welsh: 0.50%, Estonian: 0.40%, Irish: 0.40%, Polish: 0.40%, Spanish: 0.40%, Swahili: 0.40%, Tswana: 0.40%, Bosnian: 0.30%, Icelandic: 0.30%, Tagalog: 0.30%, Albanian: 0.20%, Catalan: 0.20%, Croatian: 0.20%, Indonesian: 0.20%, Lithuanian: 0.20%, Maori: 0.20%, Romanian: 0.20%, Xhosa: 0.20%, Zulu: 0.20%, Latvian: 0.10%, Malay: 0.10%, Slovak: 0.10%, Slovene: 0.10%, Somali: 0.10%, Turkish: 0.10%

>> Detection of 1000 word pairs (average length: 18 chars)
Accuracy: 94.10%
Erroneously classified as Dutch: 0.90%, Latin: 0.80%, English: 0.70%, Swedish: 0.60%, Danish: 0.50%, French: 0.40%, Bokmal: 0.30%, Irish: 0.20%, Tagalog: 0.20%, Afrikaans: 0.10%, Esperanto: 0.10%, Estonian: 0.10%, Finnish: 0.10%, Italian: 0.10%, Maori: 0.10%, Nynorsk: 0.10%, Somali: 0.10%, Swahili: 0.10%, Tsonga: 0.10%, Turkish: 0.10%, Welsh: 0.10%, Zulu: 0.10%

>> Detection of 1000 sentences (average length: 111 chars)
Accuracy: 99.70%
Erroneously classified as Dutch: 0.20%, Latin: 0.10%

7. How to add it to your project?

go get github.com/pemistahl/lingua-go

8. How to build?

Lingua requires at least Go version 1.18.

git clone https://github.com/pemistahl/lingua-go.git
cd lingua-go
go build

The source code is accompanied by an extensive unit test suite. To run the tests, simply say:

go test

9. How to use?

9.1 Basic usage

package main

import (
    "fmt"
    "github.com/pemistahl/lingua-go"
)

func main() {
    languages := []lingua.Language{
        lingua.English,
        lingua.French,
        lingua.German,
        lingua.Spanish,
    }

    detector := lingua.NewLanguageDetectorBuilder().
        FromLanguages(languages...).
        Build()

    if language, exists := detector.DetectLanguageOf("languages are awesome"); exists {
        fmt.Println(language)
    }

    // Output: English
}

9.2 Minimum relative distance

By default, Lingua returns the most likely language for a given input text. However, there are certain words that are spelled the same in more than one language. The word prologue, for instance, is both a valid English and French word. Lingua would output either English or French which might be wrong in the given context. For cases like that, it is possible to specify a minimum relative distance that the logarithmized and summed up probabilities for each possible language have to satisfy. It can be stated in the following way:

package main

import (
    "fmt"
    "github.com/pemistahl/lingua-go"
)

func main() {
    languages := []lingua.Language{
        lingua.English,
        lingua.French,
        lingua.German,
        lingua.Spanish,
    }

    detector := lingua.NewLanguageDetectorBuilder().
        FromLanguages(languages...).
        WithMinimumRelativeDistance(0.9).
        Build()

    language, exists := detector.DetectLanguageOf("languages are awesome")

    fmt.Println(language)
    fmt.Println(exists)

    // Output:
    // Unknown
    // false
}

Be aware that the distance between the language probabilities is dependent on the length of the input text. The longer the input text, the larger the distance between the languages. So if you want to classify very short text phrases, do not set the minimum relative distance too high. Otherwise Unknown will be returned most of the time as in the example above. This is the return value for cases where language detection is not reliably possible. This value is not meant to be included in the set of input languages when building the language detector. If you include it, it will be automatically removed from the set of input languages.

9.3 Confidence values

Knowing about the most likely language is nice but how reliable is the computed likelihood? And how less likely are the other examined languages in comparison to the most likely one? These questions can be answered as well:

package main

import (
    "fmt"
    "github.com/pemistahl/lingua-go"
)

func main() {
    languages := []lingua.Language{
        lingua.English,
        lingua.French,
        lingua.German,
        lingua.Spanish,
    }

    detector := lingua.NewLanguageDetectorBuilder().
        FromLanguages(languages...).
        Build()

    confidenceValues := detector.ComputeLanguageConfidenceValues("languages are awesome")

    for _, elem := range confidenceValues {
        fmt.Printf("%s: %.2f\n", elem.Language(), elem.Value())
    }

    // Output:
    // English: 0.93
    // French: 0.04
    // German: 0.02
    // Spanish: 0.01
}

In the example above, a slice of ConfidenceValue is returned containing all possible languages sorted by their confidence value in descending order. Each value is a probability between 0.0 and 1.0. The probabilities of all languages will sum to 1.0. If the language is unambiguously identified by the rule engine, the value 1.0 will always be returned for this language. The other languages will receive a value of 0.0.

There is also a method for returning the confidence value for one specific language only:

confidence := detector.ComputeLanguageConfidence("languages are awesome", lingua.French)
fmt.Printf("%.2f", confidence)

// Output:
// 0.04

The value that this method computes is a number between 0.0 and 1.0. If the language is unambiguously identified by the rule engine, the value 1.0 will always be returned. If the given language is not supported by this detector instance, the value 0.0 will always be returned.

9.4 Eager loading versus lazy loading

By default, Lingua uses lazy-loading to load only those language models on demand which are considered relevant by the rule-based filter engine. For web services, for instance, it is rather beneficial to preload all language models into memory to avoid unexpected latency while waiting for the service response. If you want to enable the eager-loading mode, you can do it like this:

lingua.NewLanguageDetectorBuilder().
    FromAllLanguages().
    WithPreloadedLanguageModels().
    Build()

Multiple instances of LanguageDetector share the same language models in memory which are accessed asynchronously by the instances.

9.5 Low accuracy mode versus high accuracy mode

Lingua's high detection accuracy comes at the cost of being noticeably slower than other language detectors. The large language models also consume significant amounts of memory. These requirements might not be feasible for systems running low on resources. If you want to classify mostly long texts or need to save resources, you can enable a low accuracy mode that loads only a small subset of the language models into memory:

lingua.NewLanguageDetectorBuilder().
    FromAllLanguages().
    WithLowAccuracyMode().
    Build()

The downside of this approach is that detection accuracy for short texts consisting of less than 120 characters will drop significantly. However, detection accuracy for texts which are longer than 120 characters will remain mostly unaffected.

In high accuracy mode (the default), the language detector consumes approximately 1,800 MB of memory if all language models are loaded. In low accuracy mode, memory consumption is reduced to approximately 110 MB. The goal is to further reduce memory consumption in later releases.

An alternative for a smaller memory footprint and faster performance is to reduce the set of languages when building the language detector. In most cases, it is not advisable to build the detector from all supported languages. When you have knowledge about the texts you want to classify you can almost always rule out certain languages as impossible or unlikely to occur.

9.6 Detection of multiple languages in mixed-language texts

In contrast to most other language detectors, Lingua is able to detect multiple languages in mixed-language texts. This feature can yield quite reasonable results but it is still in an experimental state and therefore the detection result is highly dependent on the input text. It works best in high-accuracy mode with multiple long words for each language. The shorter the phrases and their words are, the less accurate are the results. Reducing the set of languages when building the language detector can also improve accuracy for this task if the languages occurring in the text are equal to the languages supported by the respective language detector instance.

package main

import (
    "fmt"
    "github.com/pemistahl/lingua-go"
)

func main() {
    languages := []lingua.Language{
        lingua.English,
        lingua.French,
        lingua.German,
    }

    detector := lingua.NewLanguageDetectorBuilder().
        FromLanguages(languages...).
        Build()

    sentence := "Parlez-vous français? " + 
        "Ich spreche Französisch nur ein bisschen. " +
        "A little bit is better than nothing."

    for _, result := range detector.DetectMultipleLanguagesOf(sentence) {
        fmt.Printf("%s: '%s'\n", result.Language(), sentence[result.StartIndex():result.EndIndex()])
    }

    // Output:
    // French: 'Parlez-vous français? '
    // German: 'Ich spreche Französisch nur ein bisschen. '
    // English: 'A little bit is better than nothing.'
}

In the example above, a slice of DetectionResult is returned. Each entry in the slice describes a contiguous single-language text section, providing start and end indices of the respective substring.

9.7 Methods to build the LanguageDetector

There might be classification tasks where you know beforehand that your language data is definitely not written in Latin, for instance. The detection accuracy can become better in such cases if you exclude certain languages from the decision process or just explicitly include relevant languages:

// Include all languages available in the library.
lingua.NewLanguageDetectorBuilder().FromAllLanguages()

// Include only languages that are not yet extinct (= currently excludes Latin).
lingua.NewLanguageDetectorBuilder().FromAllSpokenLanguages()

// Include only languages written with Cyrillic script.
lingua.NewLanguageDetectorBuilder().FromAllLanguagesWithCyrillicScript()

// Exclude only the Spanish language from the decision algorithm.
lingua.NewLanguageDetectorBuilder().FromAllLanguagesWithout(lingua.Spanish)

// Only decide between English and German.
lingua.NewLanguageDetectorBuilder().FromLanguages(lingua.English, lingua.German)

// Select languages by ISO 639-1 code.
lingua.NewLanguageDetectorBuilder().FromIsoCodes639_1(lingua.EN, lingua.DE)

// Select languages by ISO 639-3 code.
lingua.NewLanguageDetectorBuilder().FromIsoCodes639_3(lingua.ENG, lingua.DEU)

10. What's next for version 1.5.0?

Take a look at the planned issues.

11. Contributions

Any contributions to Lingua are very much appreciated. Please read the instructions in CONTRIBUTING.md for how to add new languages to the library.