Bigram Grammer Generator with Add One Smoothing

Domain : Natural Language Processing

Sub-Domain : Language Processing, Text Processing, Language Modeling

Techniques : Language Modeling, Bigram, Add-one Smoothing, Zip Law

Application Domain : Language Generation

Description

1. From the SnapshotBROWN.pos.all.txt file extracted all word types and their frequencies.
2. Sorted the list of word types in decreasing order based on their frequency. Drew a chart showing the relationship between the rank in the ordered list and the frequency (Zipf's Law). (Do not stem but do ignore punctuation.)
3. Generated a Bigram Grammar from the above file.
4. Performed add-one smoothing.
5. Showed the grammar before and after smoothing for the sentence "A similar resolution passed in the Senate".

Languages : Python

Tools/IDE : Anaconda

Libraries :

Duration :

Current Version : v1.0.0.3

Last Update : 04.07.2018 (Time : 05:45 P.M)