Convert directly from llama3 #4268
Conversation
mxyng
force-pushed
the
pdevine/llama3
branch
8 times, most recently
from
9b83ecb
to
27588a7
Compare
mxyng
force-pushed
the
mxyng/cache-intermediate-layers
branch
from
39efb30
to
8d807d7
Compare
mxyng
force-pushed
the
mxyng/cache-intermediate-layers
branch
from
8d807d7
to
0aba2d5
Compare
mxyng
changed the base branch from
mxyng/cache-intermediate-layers
to
mxyng/fix-quantize
|
Updated the safetensors and pytorch conversion interfaces to take F32, F16, and BF16 inputs. This allows this change to convert llama3 derivatives such as nvidia's ChatQA and NousResearch's Hermes 2 Pro |
| } | ||
| } | ||
|
|
||
| switch digest := fmt.Sprintf("%x", sha256sum.Sum(nil)); digest { |
There was a problem hiding this comment.
I meant more that we have to look up these hex digests this way. I'm OK w/ the Sprintf()
There was a problem hiding this comment.
I agree it's not ideal but it's dead simple way to differentiate between different pretokenizers
mxyng
force-pushed
the
pdevine/llama3
branch
3 times, most recently
from
0a7d4fb
to
2bf8089
Compare
There was a problem hiding this comment.
this test is intended to run locally and runs iff -tags slow and the model testdata exists
| "github.com/ollama/ollama/llm" | ||
| ) | ||
|
|
||
| func convertFull(t *testing.T, p string) (llm.KV, llm.Tensors) { |
There was a problem hiding this comment.
There's room for improvement here. Ideally there's a single function call to convert or at least set up for writing the binary. I missed calling GetTensors on the first pass but the write succeeded without writing out any tensors
| tensors int | ||
| layers int | ||
| }{ | ||
| {"Meta-Llama-3-8B-Instruct", "llama", 291, 35}, |
There was a problem hiding this comment.
how does the test data get populated?
There was a problem hiding this comment.
I symlinked it into the directory since I have in another directory but you can clone or unpack as well
This change allows you to convert directly from a llama3 derived safetensors model into Ollama.
It is currently missing:
This will work with most llama3 derivatives if they are using safetensors including
dolphin-2.9-llama3, nous research's hermes 2 pro, and nvidia's chatqa.