Skip to content

Tokenizer 1.37.1

Latest
Latest
Compare
Choose a tag to compare
@guillaumekln guillaumekln released this 01 Mar 13:09
· 3 commits to master since this release
v1.37.1
e52317c

Fixes and improvements

  • Consider escaped characters as single characters in BPE
  • Ignore undefined scripts when resolving inherited or common scripts
Assets 2