original model response not available when setting fullResponse to true in streamText().

[jakedetels]

Hey there. Love the ModelFusion library. Thanks for creating it, Lars!

According to the docs, setting fullResponse to true in the options for generateText(model, messages, options) is supposed to include the model's original response. In my case, I want to use streamText() instead of generateText(). I see in this example that streamText() also supports the fullResponse property. When setting that prop to true, I do observe an additional metadata object is included, but that metadata object doesn't appear to include the expected original response from the selected model provider (e.g., OpenAI or Mistral).

In my case, I'm using Mistral and their docs indicate an expected response looks like this:

{
  "id": "cmpl-e5cc70bb28c444948073e77776eb30ef",
  "object": "chat.completion",
  "created": 1702256327,
  "model": "mistral-tiny",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "I don't have a favorite condiment as I don't consume food..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 93,
    "total_tokens": 107
  }
}

When I set fullResponse to true, I don't see any of the above expected response properties. Instead, ModelFusion returns a response with a metadata object that looks like this:

This seems like a bug, based on the wording of the docs which suggest one would get the original response (presumably from whichever LLM provider is selected).

Here's the ModelFusion docs I was referencing regarding fullResponse:

Rich Responses

For more advanced use cases, you might want to access the full response from the model, or the metadata about the call.
Model functions return rich results that include the original response and metadata when you set the fullResponse option to true.

// access the full response (needs to be typed) and the metadata:
const { text, texts, response, metadata } = await generateText(
  openai.CompletionTextGenerator({
    model: "gpt-3.5-turbo-instruct",
    maxGenerationTokens: 1000,
    n: 2, // generate 2 completions
  }),
  "Write a short story about a robot learning to love:\n\n"
  { fullResponse: true }
);

console.log(metadata);

// cast to the response type:
for (const choice of (response as OpenAICompletionResponse).choices) {
  console.log(choice.text);
}

Thanks in advance! I'm hoping we can get the original LLM provider's response.

[lgrammel]

Hey @jakedetels - thanks for the bug report & the kind words!

I've added an example for getting the full Mistral response: https://github.com/lgrammel/modelfusion/blob/main/examples/basic/src/model-provider/mistral/mistral-chat-generate-text-full-response-example.ts

Let know if this works for you.

[lgrammel]

For streamText I still need to add this. I plan to do so in the next few days, since it's a bit more complicated than in the basic case.