[Bug]: logprobs=True with stream=True is broken (OpenAI and Azure OpenAI)

[Manouchehri]

What happened?

curl "${OPENAI_API_BASE}/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {
        "role": "user",
        "content": "Hello!"
      }
    ],
    "logprobs": true,
    "top_logprobs": 2, "stream": true
  }' -v

data: {"error": {"message": "Error serializing to JSON: TypeError: 'MockValSer' object cannot be converted to 'SchemaSerializer'\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/litellm/proxy/proxy_server.py\", line 3039, in async_data_generator\n    chunk = chunk.model_dump_json(exclude_none=True)\n            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/pydantic/main.py\", line 398, in model_dump_json\n    return self.__pydantic_serializer__.to_json(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\npydantic_core._pydantic_core.PydanticSerializationError: Error serializing to JSON: TypeError: 'MockValSer' object cannot be converted to 'SchemaSerializer'\n", "type": "None", "param": "None", "code": 500}}

This should work, as upstream OpenAI supports it.

curl https://api.openai.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [
      {
        "role": "user",
        "content": "Hello!"
      }
    ],
    "logprobs": true,
    "top_logprobs": 2, "stream": true
  }' -v

data: {"id":"chatcmpl-9HHNYcYduE0cB4hXkWGLJ4i3Af3BL","object":"chat.completion.chunk","created":1713905408,"model":"gpt-3.5-turbo-0125","system_fingerprint":"fp_c2295e73ad","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":{"content":[]},"finish_reason":null}]}

data: {"id":"chatcmpl-9HHNYcYduE0cB4hXkWGLJ4i3Af3BL","object":"chat.completion.chunk","created":1713905408,"model":"gpt-3.5-turbo-0125","system_fingerprint":"fp_c2295e73ad","choices":[{"index":0,"delta":{"content":"Hi"},"logprobs":{"content":[{"token":"Hi","logprob":-2.1578777,"bytes":[72,105],"top_logprobs":[{"token":"Hello","logprob":-0.12464776,"bytes":[72,101,108,108,111]},{"token":"Hi","logprob":-2.1578777,"bytes":[72,105]}]}]},"finish_reason":null}]}

data: {"id":"chatcmpl-9HHNYcYduE0cB4hXkWGLJ4i3Af3BL","object":"chat.completion.chunk","created":1713905408,"model":"gpt-3.5-turbo-0125","system_fingerprint":"fp_c2295e73ad","choices":[{"index":0,"delta":{"content":" there"},"logprobs":{"content":[{"token":" there","logprob":-0.078327894,"bytes":[32,116,104,101,114,101],"top_logprobs":[{"token":" there","logprob":-0.078327894,"bytes":[32,116,104,101,114,101]},{"token":"!","logprob":-2.6185825,"bytes":[33]}]}]},"finish_reason":null}]}

data: {"id":"chatcmpl-9HHNYcYduE0cB4hXkWGLJ4i3Af3BL","object":"chat.completion.chunk","created":1713905408,"model":"gpt-3.5-turbo-0125","system_fingerprint":"fp_c2295e73ad","choices":[{"index":0,"delta":{"content":"!"},"logprobs":{"content":[{"token":"!","logprob":-0.00015335932,"bytes":[33],"top_logprobs":[{"token":"!","logprob":-0.00015335932,"bytes":[33]},{"token":",","logprob":-8.818896,"bytes":[44]}]}]},"finish_reason":null}]}

data: {"id":"chatcmpl-9HHNYcYduE0cB4hXkWGLJ4i3Af3BL","object":"chat.completion.chunk","created":1713905408,"model":"gpt-3.5-turbo-0125","system_fingerprint":"fp_c2295e73ad","choices":[{"index":0,"delta":{"content":" How"},"logprobs":{"content":[{"token":" How","logprob":-0.000037146747,"bytes":[32,72,111,119],"top_logprobs":[{"token":" How","logprob":-0.000037146747,"bytes":[32,72,111,119]},{"token":" What","logprob":-10.773976,"bytes":[32,87,104,97,116]}]}]},"finish_reason":null}]}

data: {"id":"chatcmpl-9HHNYcYduE0cB4hXkWGLJ4i3Af3BL","object":"chat.completion.chunk","created":1713905408,"model":"gpt-3.5-turbo-0125","system_fingerprint":"fp_c2295e73ad","choices":[{"index":0,"delta":{"content":" can"},"logprobs":{"content":[{"token":" can","logprob":-0.001394046,"bytes":[32,99,97,110],"top_logprobs":[{"token":" can","logprob":-0.001394046,"bytes":[32,99,97,110]},{"token":" may","logprob":-7.042742,"bytes":[32,109,97,121]}]}]},"finish_reason":null}]}

data: {"id":"chatcmpl-9HHNYcYduE0cB4hXkWGLJ4i3Af3BL","object":"chat.completion.chunk","created":1713905408,"model":"gpt-3.5-turbo-0125","system_fingerprint":"fp_c2295e73ad","choices":[{"index":0,"delta":{"content":" I"},"logprobs":{"content":[{"token":" I","logprob":-1.9816675e-6,"bytes":[32,73],"top_logprobs":[{"token":" I","logprob":-1.9816675e-6,"bytes":[32,73]},{"token":" i","logprob":-13.487041,"bytes":[32,105]}]}]},"finish_reason":null}]}

data: {"id":"chatcmpl-9HHNYcYduE0cB4hXkWGLJ4i3Af3BL","object":"chat.completion.chunk","created":1713905408,"model":"gpt-3.5-turbo-0125","system_fingerprint":"fp_c2295e73ad","choices":[{"index":0,"delta":{"content":" assist"},"logprobs":{"content":[{"token":" assist","logprob":-0.047100168,"bytes":[32,97,115,115,105,115,116],"top_logprobs":[{"token":" assist","logprob":-0.047100168,"bytes":[32,97,115,115,105,115,116]},{"token":" help","logprob":-3.0792236,"bytes":[32,104,101,108,112]}]}]},"finish_reason":null}]}

data: {"id":"chatcmpl-9HHNYcYduE0cB4hXkWGLJ4i3Af3BL","object":"chat.completion.chunk","created":1713905408,"model":"gpt-3.5-turbo-0125","system_fingerprint":"fp_c2295e73ad","choices":[{"index":0,"delta":{"content":" you"},"logprobs":{"content":[{"token":" you","logprob":-3.888926e-6,"bytes":[32,121,111,117],"top_logprobs":[{"token":" you","logprob":-3.888926e-6,"bytes":[32,121,111,117]},{"token":" or","logprob":-12.868921,"bytes":[32,111,114]}]}]},"finish_reason":null}]}

data: {"id":"chatcmpl-9HHNYcYduE0cB4hXkWGLJ4i3Af3BL","object":"chat.completion.chunk","created":1713905408,"model":"gpt-3.5-turbo-0125","system_fingerprint":"fp_c2295e73ad","choices":[{"index":0,"delta":{"content":" today"},"logprobs":{"content":[{"token":" today","logprob":-0.00008363618,"bytes":[32,116,111,100,97,121],"top_logprobs":[{"token":" today","logprob":-0.00008363618,"bytes":[32,116,111,100,97,121]},{"token":"?","logprob":-9.400616,"bytes":[63]}]}]},"finish_reason":null}]}

data: {"id":"chatcmpl-9HHNYcYduE0cB4hXkWGLJ4i3Af3BL","object":"chat.completion.chunk","created":1713905408,"model":"gpt-3.5-turbo-0125","system_fingerprint":"fp_c2295e73ad","choices":[{"index":0,"delta":{"content":"?"},"logprobs":{"content":[{"token":"?","logprob":-0.00002129266,"bytes":[63],"top_logprobs":[{"token":"?","logprob":-0.00002129266,"bytes":[63]},{"token":"?\n","logprob":-11.212309,"bytes":[63,10]}]}]},"finish_reason":null}]}

data: {"id":"chatcmpl-9HHNYcYduE0cB4hXkWGLJ4i3Af3BL","object":"chat.completion.chunk","created":1713905408,"model":"gpt-3.5-turbo-0125","system_fingerprint":"fp_c2295e73ad","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}]}

data: [DONE]

Relevant log output

No response

Twitter / LinkedIn details

https://twitter.com/DaveManouchehri

[krrishdholakia]

this is serious. thanks for raising this @Manouchehri aiming to tackle this after merging in the no-cache pr today.

[krrishdholakia]

hi @Manouchehri unable to repro - i just called the proxy with the params

[krrishdholakia]

closing out as this looks related to user's pydantic version - tiangolo/fastapi#9709

[Manouchehri]

It's not my pydantic though, I'm using the Dockerfile.database.

[krrishdholakia]

reopening - able to repro on deployed endpoint - https://litellm-main-v1-35-20.onrender.com/

[krrishdholakia]

Locally - i'm running pydantic v2.6.3.

On docker, it's - v2.7.0

Testing if changing version on hosted endpoint fixes this.

[krrishdholakia]

This looks like it came from a new openai release - openai/openai-python#1306

[krrishdholakia]

fixed - df60e47

by pinning the openai version to 1.14.3

Testing added as well e31eb6a

[Manouchehri]

Confirmed fixed in v1.35.21 for me as well, thanks!