Language Modeling

[iliakur]

Ok, after getting some feedback on my previous attempt, I re-worked things a bit.
This time there's tests a-plenty and I've tried to add documentation as well.

I have regression tests for:

• Unseen token error in nltk.model.NgramModel.entropy #167
• Error in NgramModel backoff smoothing calculation? #367
• Weird result using NgramModel.prob() function. #380

Since I didn't add the Simple Good Turing estimator yet, can't say anything about the issues related to that. Which brings me to the next point. I spent most of my effort making sure the basic datastructures are useable and robust, so didn't have as much time to add a lot of model classes. I'm hoping that the infrastructure the module provides, however, makes it easy for folks to create models on their own.

Would be nice to merge this into the next release :)

Happy modeling!

[iliakur]

@alvations @stevenbird The tests that fail currently all have to do with python 2 lacking itertools.accumulate. functools.singledispatch is also probably an issue.

I can easily compensate for accumulate but could we add singledispatch as a dependency? It's a small module, easily installed.

[stevenbird]

@copper-head thanks for all this... fine to add this dependency.

[iliakur]

Ok, it looks like there's more tweaking that I'll have to do to make python 2.7 happy. I'll get to it soon

[iliakur]

Looks like python 2 is now happy :)
What's the drill with adding oneself to the contributors list these days?

[stevenbird]

Wow, what a big contribution!

[stevenbird]

[CI: retest]

[stevenbird]

Thanks @copper-head!

[alvations]

Thanks @copper-head!! An awesome update to the model package!!

[iliakur]

lol, only took me... 4 years ;)

[stevenbird]

How about announcing it on nltk-dev?

[iliakur]

Sure!
Which release will this appear in?

[stevenbird]

The next one... once it's packaged I hope you can announce your work on nltk-dev