This is a simple implementation of a Byte Pair Encoding Algorithm, used to tokenize text, and a Bigram Word Model.
These were created as part of my research word in Language Models and are my original implementation!
This is my simple and readable implementation of the Byte Pair Encoding Algorithm and a Bigram Model.
License
pvalle6/Tokenizer_and_Bigram
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
This is my simple and readable implementation of the Byte Pair Encoding Algorithm and a Bigram Model.
Topics
Resources
License
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published