You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Sadly the OpenAI logprobs and top_logprobs parameters didn't work either. It looks like it's because they are not mapped here in the OpenAI compatibility function:
That should emulate the OpenAI behaviour described here.
Please consider supporting this as it would be very convenient. Manually calling /completion with chat templates is how I'm working around it at the moment.
The text was updated successfully, but these errors were encountered:
Hi, first of all thank you so much for llamafile. I am very conscious of data privacy and wary of being locked-in to OpenAI, so llamafile is amazing.
There is a small disparity between the
/completions
endpoint and/v1/chat/completions
, which is that the latter doesn't seem to supportn_probs
.Here's an example of `n_probs` not working.
Sadly the OpenAI
logprobs
andtop_logprobs
parameters didn't work either. It looks like it's because they are not mapped here in the OpenAI compatibility function:https://github.com/Mozilla-Ocho/llamafile/blob/main/llama.cpp/server/oai.h#L20
I'm not brave or competent enough to try and make the change myself, but I think the necessary logic would be:
That should emulate the OpenAI behaviour described here.
Please consider supporting this as it would be very convenient. Manually calling
/completion
with chat templates is how I'm working around it at the moment.The text was updated successfully, but these errors were encountered: