New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Llama3 chat prompt is clearly wrong #1222
Comments
I tried the following in 'config.ts'; it seems to fix the model
WARNING!! My ECMAScript is extremely rusty, and I know absolutely nothing about TypeScript... function llama3TemplateMessages(msgs: ChatMessage[]): string {
if (msgs.length === 0) {
return "";
}
let systemMessage = (msg) => `<|begin_of_text|><|start_header_id|>${msg.role}<|end_header_id|>\n${msg.content}<|eot_id|>\n`;
let userPrompt = "<|start_header_id|>user<|end_header_id|>\n";
let assistantPrompt = "<|start_header_id|>assistant<|end_header_id|>\n";
let separator = "<|eot_id|>\n";
let prompt = "";
// Skip assistant messages at the beginning
while (msgs.length > 0 && msgs[0].role === "assistant") {
msgs.shift();
}
if (msgs.length > 0 && msgs[0].role === "system") {
prompt += systemMessage(msgs.shift()!);
}
for (let i = 0; i < msgs.length; i++) {
const msg = msgs[i];
prompt += msg.role === "user" ? userPrompt : assistantPrompt;
prompt += msg.content;
if (i < msgs.length - 1) {
prompt += separator;
}
}
if (msgs.length > 0 && msgs[msgs.length - 1].role === "user") {
prompt += separator;
prompt += assistantPrompt;
}
return prompt;
};
export function modifyConfig(config: Config): Config {
const model = config.models.find(
(model) => model.title.toLowerCase().includes("llama3"),
);
if (model) {
model.templateMessages = llama3TemplateMessages;
}
return config;
} |
Just so we're crystal clear on this: consider the above code a donation to the public domain OR as falling under Continue's license if public domain is problematic. It's not exactly profound coding magic, but I don't want to unnecessarily create licensing issues... |
@ben-alkov thank you for taking the time to write this out! I've just incorporated it in this commit, and I'll release later today |
You are very welcome - I'm just glad that it made enough sense to be useful 😆 |
I tried the fix by editing Another chat I got this at the end |
@ryancurrah; Probably better to wait for the next Preview release, where it will be incorporated into Continue itself. Don't forget to reset your 'config.ts'! |
Ack I will wait. Thanks kindly. |
AFAICT, fixed in https://github.com/continuedev/continue/releases/tag/v0.9.125-vscode |
Thanks for posting this issue and the solution. The new Llama 3 8B Instruct model now works fine for me, for example to create docstrings for Python functions using Continue in the code OSS app. However, I also downloaded all the changes Meta made to these model config files since release. @ryancurrah: |
Validations
Problem
Meta's definition of the correct chat template for Llama 3 is in https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3, section "Meta Llama 3 chat", which is not like anything in 'core/llm/templates/chat.ts' (except for a passing resemblance to
chatmlTemplateMessages
).Worse still, the content of
llama2TemplateMessages
looks nothing like the recommended prompt...(Also, AFAIU, the Instruct model's prompt should be slightly different, but I can no longer find a reference.)
Solution
Implement
llama3TemplateMessages
in 'chat.ts' based on https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3The text was updated successfully, but these errors were encountered: