·
1 commit
to main
since this release
- Fully functional BPE tokenizer
- Decoder to convert tokens into text
- Import and export functions for tokens
- UTF-8 Support
- Case-Sensitivity Support