Python Binding

[omarmhaimdat]

Hey Peter,

Really love you work.

I was wondering if you ever considered creating a python binding by leveraging the performance of the rust version?

Best,
OM

[pemistahl]

Hey Omar, thank you. :)

Yes, I did consider creating Python bindings from the Rust code but then decided against it. PyO3 does not yet support converting Rust enums to Python enums which would have resulted in a clumsy API. That's why I rewrote the library in pure Python to have a proper API.

This way, I was also able to use NumPy arrays for storing the language models. NumPy arrays are very memory-efficient. All language models only consume about 700 MB in memory which is far less than with the other implementations in Rust, Go and Kotlin.

While PyO3 evolves, I will re-evaluate regularly whether it makes sense to use PyO3 to replace the pure Python implementation. But I don't see that possibility in the near future.

[omarmhaimdat]

It's a bummer that enums are not yet supported, but I guess it would be a nightmare to convert all the complex pattern matching possible in Rust to Python.

Who would have thought the "Python" version is actually more memory efficient than the Rust version, I wonder if it would be possible to achieve the same memory efficiency by using NumPy arrays in Rust?

[serega]

I was able to reduce memory consumption of the Rust version by switching to small_str. With the change the rust version uses about 970M from about 1400M before the change. The performance also increased slightly, probably because of less memory loads. small_str is immutable and stores small strings inline without heap allocation, fits perfectly for ngrams.

[pemistahl]

I guess it would be a nightmare to convert all the complex pattern matching possible in Rust to Python.

Rust enums are much more powerful than Python enums, so it's not possible to convert every Rust enum to Python, of course. But it would be enough to support the minimal subset of features that both enum implementations have in common. The enums I use in my library are nothing special, so they could be translated to Python enums without any hassle. Perhaps PyO3 will support that someday.

@serega, I was not aware of the small_str crate. Looks very promising, so I think I will use it in the next Lingua release. Thanks for letting me know. By the way, I have not forgotten your other pull requests. I'm just not having enough time at the moment to continue the implementation. I hope it will get better soon.