gpt-bpe

An implementation of GPT2 Byte Pair Encoding Encoder/Decoder in golang. Generally very fast, bottlenecked by the regex engine for whitespace separation.

Name		Name	Last commit message	Last commit date
Latest commit History 283 Commits
.github/workflows		.github/workflows
cmd		cmd
js		js
lib		lib
resources		resources
.gitattributes		.gitattributes
.gitignore		.gitignore
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
encoder.json		encoder.json
go.mod		go.mod
go.sum		go.sum
gpt_bpe.go		gpt_bpe.go
gpt_bpe_test.go		gpt_bpe_test.go
prose.go		prose.go
prose_js.go		prose_js.go
runetree.go		runetree.go
runetree_test.go		runetree_test.go
utils.go		utils.go
vocab.bpe		vocab.bpe

License

coreweave/gpt_bpe

Folders and files

Latest commit

History

Repository files navigation

gpt-bpe

About

Resources

License

Stars

Watchers

Forks

Languages