New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
这个库在chat模型的token计算上有诸多错误 #3
Comments
感谢,的确不能保证和官方返回一模一样。 可以看到官方的cookbookHow_to_count_tokens_with_tiktoken中也是提到 也是一个预估值,这个计算方法主要是用来预估发送的一些limit,所以有一点误差不会影响逻辑。 |
token计算不支持 gpt-3.5-turbo-0301 |
我按照官方Demo的计算规则,更新了计算方法。
更新了funciton的计算规则,具体可参考FunctionFormat |
More issue about token calculation can be found issue #4. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
与openai返回的token对比发现,几乎各chat模型都有计算方式错误或结果偏差,于是我自己从零建模和编写了token计算工具
你的部分token计算代码有严重错误,这里列举部分:
encodeOrdinary
来跳过special tokens由于精力有限我无法在开源代码上提交修改,本issue只是告知绝大部分token计算都有误,请你自己有精力时研究下
The text was updated successfully, but these errors were encountered: