Find more memory-efficient data structure for language models

[pemistahl]

Currently, the language models are loaded into simple maps at runtime. Even though accessing the maps is pretty fast, they consume a significant amount of memory. The goal is to investigate whether there are more suitable data structures available that require less storage space in memory, something like NumPy for Python.

One promising candidate could be Gonum.

[goldsam]

I think you will benefit from using a trie (not tree) data structure. Here is a Go implementation you may be able to use as a drop in replacement to map.