You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add grammar as additional parameter to chat/completions.
Motivation
Constrained decoding with grammars is a killer feature for many users. Exposing grammar parameter would allows users to guide chat completions output with regex or json schemes.
The openai client has an extra_body argument, so that we could access TGI endpoints as follows
from openai import OpenAI
from pydantic import BaseModel, conint
from typing import List
class Animals(BaseModel):
location: str
activity: str
animals_seen: conint(ge=1, le=5) # Constrained integer type
animals: List[str]
+grammar = {+ "type": "json",+ "value": Animals.schema()+}
client = OpenAI(
base_url="http://localhost:3000/v1",
api_key="_",
)
chat_completion = client.chat.completions.create(
model="tgi",
messages=[
{
"role": "system",
"content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.",
},
{
"role": "user",
"content": "What's the weather like the next 3 days in San Francisco, CA?",
},
],
+ extra_body={"grammar"=grammar},
max_tokens=500,
)
This exposes grammars in all frameworks based on openai client. E.g., grammars could also be used via bind(extrabody=...) in LangChain OpenAIChatModels hooked on TGI Messages API.
Your contribution
Discuss implementations, test, documentation
The text was updated successfully, but these errors were encountered:
Feature request
Add grammar as additional parameter to chat/completions.
Motivation
Constrained decoding with grammars is a killer feature for many users. Exposing
grammar
parameter would allows users to guide chat completions output with regex or json schemes.The openai client has an extra_body argument, so that we could access TGI endpoints as follows
This exposes grammars in all frameworks based on openai client. E.g., grammars could also be used via
bind(extrabody=...)
in LangChain OpenAIChatModels hooked on TGI Messages API.Your contribution
Discuss implementations, test, documentation
The text was updated successfully, but these errors were encountered: