Why am I forced to specify 2 languages for the detection to work with?

[DavidTheExplorer]

I have a project where players can choose their language, and then only messages in their language would be sent to their chat.
This is impossible(or unreliable) to implement because this code is illegal:

String message = ...;
Language playerLanguage = ...;

//don't send the message if it's not in the player's language
if(!LanguageDetectorBuilder.fromLanguages(playerLanguage).build().detectLanguageOf(message) != playerLanguage)
    return;

[pemistahl]

You need to choose at least two languages because the confidence score for each language is calculated relatively to the other languages. It's difficult to come up with a confidence metric for a single language that works independently of any other language. I'm working on it but it may take a while.

Please note that you should build the detector only once and reuse this instance throughout your program. Rebuilding the detector for each text is costly and should be avoided.

[DavidTheExplorer]

What do you then suggest me to do? I absolutely can't specify a dummy language along with the player's language, because it can be returned.

[pemistahl]

What do you mean by 'can be returned'? It's certainly possible to avoid the dummy language being exposed to an outside caller.

Simply build the language detector from all languages that might occur in your game. Then the chance is high that you can reliably detect the player's language.

[DavidTheExplorer]

Your approach would(as you had mentioned) be super slow, and from testings it is indeed horribly slow.
I still don't understand why 1 language isn't enough.

The code I tried:

// if not all languages, at least all spoken ones. I don't want to limit my users.
LanguageDetector detector = LanguageDetectorBuilder.fromAllSpokenLanguages().build();
		
long before = System.currentTimeMillis();
Language language = detector.detectLanguageOf("Hello from the other side");
long after = System.currentTimeMillis();
		
System.out.println("The language is " + language.name() + ", took " + (after-before) + "ms.");

Result:

[pemistahl]

@DavidTheExplorer I'm sorry for the late response but better late than never.

Your benchmark does not only measure the time for the actual language detection but also the time for the initial loading of the language models into memory. As mentioned in the README, you can preload all languages models when creating the LanguageDetector by writing LanguageDetectorBuilder.fromAllSpokenLanguages().withPreloadedLanguageModels().build(). Then your benchmark will only measure the time it takes to detect the language of the given string which will be significantly faster than your previous measurement.

[gauvins2]

I would also like a single language mode. I have a (large) corpus where 165 different languages have been identified by polyglot. At this time, I want better accuracy of ENGLISH entries, so I am trying lingua. I've set up a two language model (ENGLISH and SPANISH) as they are the two most frequent languages, but:

• I don't know which "other" language I should use as a comparison benchmark
• Processing speed appears to be linearly related to the number of languages being inferenced. Perhaps a single language mode with a probability threshold (ex: 0.5) against the null would work as well as a two-language model, at twice the speed.

This being said, very impressed by the accuracy, so far.

[pemistahl]

This being said, very impressed by the accuracy, so far.

Thank you @gauvins2, that is the actual goal of this library. I'm happy that it works for you.

A single-language mode is not as easy as it sounds due to how the underlying statistical model works. I don't have much free time currently to come up with something. Pull requests are very much welcome if you want to try to implement something in this regard.