Skip to content

SimonWang9610/gpt_tokenizer

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

22 Commits

assets

assets

 
 

packages

packages

 
 

scripts

scripts

 
 

.gitignore

.gitignore

 
 

Cargo.toml

Cargo.toml

 
 

README.md

README.md

 
 

analysis_options.yaml

analysis_options.yaml

 
 

melos.yaml

melos.yaml

 
 

pubspec.yaml

pubspec.yaml

 
 

Repository files navigation

  • cargo build only re-run tokenizer/build.rs when the symbols are changed

  • cocopods 1.12.0 supports slice_type: .dylib. see github

  • build-apple.sh

    • if LIBNAME=libgpt_tokenizer.a, everything works well

About

BPE tokenizer used for Dart/Flutter applications when calling ChatGPT APIs

pub.dev/packages/flutter_gpt_tokenizer

Topics

flutter-plugin bpe dart-package tokenizer-nlp openai-chatgpt

Resources

Readme
Activity

Stars

6 stars

Watchers

2 watching

Forks

5 forks
Report repository

Releases 3

gpt_tokenizer-v0.1.1 Latest
Feb 7, 2024
+ 2 releases

Packages

No packages published

Languages