fix showing unknown token at gpt_tokenize #801
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As for current implementation, gpt_tokenize() shows each byte of multi bytes character if unknown token is existed, like below
I fixed with stopping show each bytes as below.
please confirm this.
-- detail --
original:
$ ./240407up/gpt-neox.org --repeat-last-n 256 --repeat-penalty 1.2 -m models/cyberagent/ggml-calm-1b-q4_0.bin -s 7654321 -p "日本で一番高い山は何ですか?"
main: seed = 7654321
gpt_neox_model_load: loading model from 'models/cyberagent/ggml-calm-1b-q4_0.bin' - please wait ...
gpt_neox_model_load: n_vocab = 52096
gpt_neox_model_load: n_ctx = 2048
gpt_neox_model_load: n_embd = 2048
gpt_neox_model_load: n_head = 16
gpt_neox_model_load: n_layer = 24
gpt_neox_model_load: n_rot = 128
gpt_neox_model_load: par_res = 0
gpt_neox_model_load: ftype = 2002
gpt_neox_model_load: qntvr = 2
gpt_neox_model_load: ggml ctx size = 1917.12 MB
gpt_neox_model_load: memory_size = 384.00 MB, n_mem = 49152
gpt_neox_model_load: .................................... done
gpt_neox_model_load: model size = 764.92 MB / num tensors = 292
extract_tests_from_file : No test file found.
test_gpt_tokenizer : 0 tests failed out of 0 tests.
gpt_tokenize: unknown token ' '
gpt_tokenize: unknown token ' '
gpt_tokenize: unknown token ' '
main: number of tokens in prompt = 6
main: token[0] = 5619, 日本で
main: token[1] = 3300, 一番
main: token[2] = 1737, 高い
main: token[3] = 14218, 山は
main: token[4] = 37814, 何で
main: token[5] = 24250, すか
日本で一番高い山は何ですか?」。そんな質問を何度か受けてきましたが、 ...
fixed:
$ ./240407up/gpt-neox.mod --repeat-last-n 256 --repeat-penalty 1.2 -m models/cyberagent/ggml-calm-1b-q4_0.bin -s 7654321 -p "日本で一番高い山は何ですか?"
main: seed = 7654321
gpt_neox_model_load: loading model from 'models/cyberagent/ggml-calm-1b-q4_0.bin' - please wait ...
gpt_neox_model_load: n_vocab = 52096
gpt_neox_model_load: n_ctx = 2048
gpt_neox_model_load: n_embd = 2048
gpt_neox_model_load: n_head = 16
gpt_neox_model_load: n_layer = 24
gpt_neox_model_load: n_rot = 128
gpt_neox_model_load: par_res = 0
gpt_neox_model_load: ftype = 2002
gpt_neox_model_load: qntvr = 2
gpt_neox_model_load: ggml ctx size = 1917.12 MB
gpt_neox_model_load: memory_size = 384.00 MB, n_mem = 49152
gpt_neox_model_load: .................................... done
gpt_neox_model_load: model size = 764.92 MB / num tensors = 292
extract_tests_from_file : No test file found.
test_gpt_tokenizer : 0 tests failed out of 0 tests.
gpt_tokenize: unknown token '?'
main: number of tokens in prompt = 6
main: token[0] = 5619, 日本で
main: token[1] = 3300, 一番
main: token[2] = 1737, 高い
main: token[3] = 14218, 山は
main: token[4] = 37814, 何で
main: token[5] = 24250, すか
日本で一番高い山は何ですか?」。そんな質問を何度か受けてきましたが、 ...