Chat completion stream returns empty total usage

[JosePina98]

Confirm this is a Node library issue and not an underlying OpenAI API issue

• This is an issue with the Node library

Describe the bug

Using openai.beta.chat.completions.stream() and then calling totalUsage function returns the object with all values set to zero.

To Reproduce

1. Call the chat completions API with response stream
2. Log the result of totalUsage function
3. Check that every value is zero

Code snippets

const response = openai.beta.chat.completions.stream({
    messages: [
        {
            role: "user",
            content: "hello"
        }
    ],
    model: "gpt-3.5-turbo-1106",
    response_format: { type: "text" },
    max_tokens: 300,
    n: 1,
    temperature: 1.2,
    stream: true
});

console.log(await response.totalUsage())

OS

Ubuntu

Node version

Node v16.13.0

Library version

open v4.19.0

[clickeo-dev]

I can confirm that this issue still exist. @JosePina98 did you find any workaround on this?

[JosePina98]

For now I have created this Typescript function to estimate the tokens used in each call. According to the tests I've done, it makes a rough estimate, but on the high end.

const tokenEstimation = (prompt: string, output: string): {
    prompt_tokens: number,
    completion_tokens: number,
    total_tokens: number
} => {

    const tokenLength = 3.8;

    const promptTokens = Math.ceil(prompt.length / tokenLength);
    const completionTokens = Math.ceil(output.length / tokenLength);
    return {
        prompt_tokens: promptTokens,
        completion_tokens: completionTokens,
        total_tokens: promptTokens + completionTokens
    }
}

[clickeo-dev]

For now I have created this Typescript function to estimate the tokens used in each call. According to the tests I've done, it makes a rough estimate, but on the high end.

const tokenEstimation = (prompt: string, output: string): {
    prompt_tokens: number,
    completion_tokens: number,
    total_tokens: number
} => {

    const tokenLength = 3.8;

    const promptTokens = Math.ceil(prompt.length / tokenLength);
    const completionTokens = Math.ceil(output.length / tokenLength);
    return {
        prompt_tokens: promptTokens,
        completion_tokens: completionTokens,
        total_tokens: promptTokens + completionTokens
    }
}

I do kinda same. Using the tokenizer package since the package doesn't support new models I hard coded 'gpt-4' as model. I assume that they are all using cl100k_base encoding model under the hood.

 // Calculate Token Sizes
const inputTokenSize = tokenizer.encodeChat(messages, "gpt-4").length;
const outputTokenSize = tokenizer.encodeChat([{role: "assistant", content: output,},], "gpt-4").length;

[rattrayalex]

Sorry about this, usage events are not sent in streaming just yet; OpenAI hopes to add that to the API soon. Once that happens, this function will return numbers as you'd expect.

It was arguably a mistake to include totalUsage() on that interface before it was ready in the API, sorry for the confusion.

[yauri-io]

Hitting this issue as well. I see this issue's status is Closed, but I couldn't see any new release with the fix.
Any ETA?

[rattrayalex]

cc @athyuttamre

[athyuttamre]

Hi folks, we pre-emptively added the totalUsage() function but unfortunately have not released support in the API just yet. We plan to do so early next year. Sorry for the confusion, but feel free to subscribe to this GitHub issue and we'll post here when it is launched.

[talos]

Glad to hear it! Would it be possible to re-open the ticket to make it clearer when this has been resolved?

[jay-pulsifi]

Hi team, any ETA when this ticket will be resolved?

[aar2dee2]

following this for updates

P.S. thanks for the stream events - have made it easy to work with db calls when streaming completions!

[Mickeygogo]

When are you going to put this issue on to resolution?

[nai-kon]

I am really looking forward to usage support on stream mode.