[BUG] DSPy connector not showing the token count, although it is returned in the request

[DamianB-BitFlipper]

Describe the bug
I am querying DSPy with Phoenix as the UI to visualize the queries. This tool has been a real game changer! I cannot help but notice that the token counter is stuck at 0 at the top of the UI.

See:

This is although the response dict from my model has the information:

See:

To Reproduce
Here is the code I use to setup Phoenix with DSPy after installing both arize-phoenix v3.25.0 and openinference-instrumentation-dspy v0.1.7.

# Phoenix Setup
import phoenix as px
from openinference.instrumentation.dspy import DSPyInstrumentor
from opentelemetry import trace as trace_api
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk import trace as trace_sdk
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace.export import SimpleSpanProcessor

PHOENIX_ENDPOINT = "http://127.0.0.1:6006/v1/traces"
TRACER_PROVIDER = trace_sdk.TracerProvider()
SPAN_OTLP_EXPORTER = OTLPSpanExporter(endpoint=PHOENIX_ENDPOINT)
TRACER_PROVIDER.add_span_processor(
    SimpleSpanProcessor(span_exporter=SPAN_OTLP_EXPORTER)
)

trace_api.set_tracer_provider(tracer_provider=TRACER_PROVIDER)
DSPyInstrumentor().instrument()

 # Start the Phoenix app
 px.launch_app()

Expected behavior
The token count would be pulled from telemetry and displayed in the UI.

Environment (please complete the following information):

• OS: MacOS
• Browser Brave

Thanks again for this amazing tool!

[dosubot]

It looks like the issue with the token count not displaying in the Phoenix UI might be related to how the token count data is being handled and passed to the UI components. The Phoenix UI expects tokenCountTotal, tokenCountPrompt, and tokenCountCompletion properties to be of type number and explicitly passed to the SpanItem component. If tokenCountTotal is not a number, the token count will not be displayed.

Ensure that the data from your model's response is correctly formatted and passed to the Phoenix backend. The backend should update the token count in the _Spans class using the SpanAttributes.LLM_TOKEN_COUNT_TOTAL attribute. This update happens in the _update_cached_statistics method, where it adds the token count from the span's attributes to the _token_count_total.

If the token count is correctly updated on the backend but still not displayed, check the data passed to the SpanItem component to ensure it's correctly formatted and not null or undefined. This verification will help identify if the issue lies in the data handling or the UI component rendering logic.

Details

Sources

• app/src/components/trace/SpanItem.tsx
• app/src/pages/project/ProjectPageHeader.tsx
• src/phoenix/core/project.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[axiomofjoy]

@DamianB-BitFlipper Thanks for trying Phoenix. For DSPy, we do not grab token counts from the DSPy native spans themselves. Can you try instrumenting your application with our OpenAI instrumentation in addition to the DSPy instrumentation and see if that helps? https://github.com/Arize-ai/openinference/tree/main/python/instrumentation/openinference-instrumentation-openai#openinference-openai-instrumentation

[DamianB-BitFlipper]

Thanks for the reply. What do you mean by "instrumenting your application". Do you mean simply calling: OpenAIInstrumentor().instrument()?

[DamianB-BitFlipper]

Follow up: Adding OpenAIInstrumentor().instrument() does not affect anything.

[dosubot]

To address the issue with the token count not displaying in the Phoenix UI, ensure the following steps are correctly implemented:

1. Model's Response Formatting: Verify that your model's response includes the token count as an integer value. This is crucial for Phoenix to process the data correctly.

2. Passing Token Count to Phoenix Backend: When sending data to the Phoenix backend, include the token count in the span's attributes using the LLM_TOKEN_COUNT_TOTAL, LLM_TOKEN_COUNT_PROMPT, and LLM_TOKEN_COUNT_COMPLETION attributes. For example:

   span_attributes[SpanAttributes.LLM_TOKEN_COUNT_TOTAL] = model_response_token_count

3. Updating the _Spans Class: Ensure that the token count is correctly updated in the _Spans class. When creating or updating a Span instance, include the token count from your model's response:

   span.token_count_total = model_response_token_count

4. Phoenix UI Integration: The Phoenix UI is designed to display the token count if it's provided as a number. Make sure your backend correctly passes the tokenCountTotal, tokenCountPrompt, and tokenCountCompletion as part of the SpanItemProps to the SpanItem component. This ensures the UI component can render the token count correctly.

By following these steps, you should be able to resolve the issue with the token count not displaying in the Phoenix UI. Ensure that each step is correctly implemented and that the token count values are passed as numbers from the backend to the UI components.

Details

Sources

• src/phoenix/server/api/types/Span.py
• app/src/components/trace/SpanItem.tsx
• tutorials/tracing/dspy_tracing_tutorial.ipynb
• docs/tracing/how-to-tracing/instrumentation/dspy.md
• src/phoenix/trace/openai/instrumentor.py
• docs/tracing/how-to-tracing/instrumentation/openai.md

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[axiomofjoy]

Follow up: Adding OpenAIInstrumentor().instrument() does not affect anything.

@DamianB-BitFlipper Here is a code snippet.

import os

import dspy
from openinference.instrumentation.dspy import DSPyInstrumentor
from openinference.instrumentation.openai import OpenAIInstrumentor
from opentelemetry import trace as trace_api
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk import trace as trace_sdk
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor

# Logs to the Phoenix Collector if running locally
endpoint = "http://localhost:6006/v1/traces"
if os.environ.get("PHOENIX_COLLECTOR_ENDPOINT"):
    endpoint = os.environ["PHOENIX_COLLECTOR_ENDPOINT"] + "/v1/traces"

resource = Resource(attributes={})
tracer_provider = trace_sdk.TracerProvider(resource=resource)
tracer_provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter(endpoint)))
tracer_provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))

trace_api.set_tracer_provider(tracer_provider=tracer_provider)


DSPyInstrumentor().instrument()
OpenAIInstrumentor().instrument()


class BasicQA(dspy.Signature):
    """Answer questions with short factoid answers."""

    question = dspy.InputField()
    answer = dspy.OutputField(desc="often between 1 and 5 words")


if __name__ == "__main__":
    turbo = dspy.OpenAI(model="gpt-3.5-turbo")

    dspy.settings.configure(lm=turbo)

    # Define the predictor.
    generate_answer = dspy.Predict(BasicQA)

    # Call the predictor on a particular input.
    pred = generate_answer(
        question="What is the capital of the united states?"  # noqa: E501
    )  # noqa: E501
    print(f"Predicted Answer: {pred.answer}")

[axiomofjoy]

Requirements:

openai
openinference-instrumentation-dspy
openinference-instrumentation-openai
opentelemetry-exporter-otlp-proto-http

[DamianB-BitFlipper]

Thanks for the quick reply. I tried exactly your code. I had to add phoenix.launch_app() however to get the application up and running. It still does not work.

See:

I fear that there may not be a ready solution, and that this may be an issue that needs to be addressed in the DSPy instrumentation. What do you think.

I am running the latest versions of everything:

arize-phoenix==3.25.0
openai==1.26.0
openinference-instrumentation-openai==0.1.4
dspy-ai==2.4.9
openinference-instrumentation-dspy==0.1.7
opentelemetry-exporter-otlp-proto-http==1.24.0

[axiomofjoy]

When you expand the trace, do you see a new ChatCompletion LLM span?

[DamianB-BitFlipper]

I believe yes:

And it started showing the total tokens, for some reason. Thanks for the assistance! Maybe this warrants an update to the docs of DSPy integration? If you look closely, even the video you show on the docs shows 0 tokens.

Thanks once again!

[DamianB-BitFlipper]

For posterity's record, the changes I made were adding:

tracer_provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
OpenAIInstrumentor().instrument()

[axiomofjoy]

Thanks @DamianB-BitFlipper, glad you're unblocked. I've changed the label from bug to enhancement. We probably want those token counts showing up without the OpenAI instrumentation, as you pointed out.

[DamianB-BitFlipper]

Thank you!

[duxtinto]

Hi!

I'm using dspy-ai[chromadb]==2.4.9 in Ubuntu, and even thought at the beginning I could not see any ChatCompletion traces after enabling the openai instrumentation, then suddenly, it started to work.

After some research, I think that dspy has a cache for the requests to the openai gpt api, and no trace is recorded whenever the cache is hit.

[DamianB-BitFlipper]

Yes. This is true. DamianOn 19. May 2024, at 06:36, David Hernando Sancha ***@***.***> wrote: Hi! I'm using dspy-ai[chromadb]==2.4.9 in Ubuntu, and even thought at the beginning I could not see the ChatCompletion trace after enabling the openapi instrumentation, then suddenly, it started to work. After some research, I think that dspy has a cache for the requests to the openai gpt api, and no trace is recorded whenever the cache is hit. image.png (view on web) —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>