Runtime logging with transform messages confusion

[Nathan-Intergral]

Hey everyone,

I'm a bit puzzled and hope someone can shed some light on this. I've set up runtime logging in my group chat, which uses TransfromMessages to limit message tokens and history. Everything seems fine in the console and debug logs—the limits are being applied as they should. But when I check the runtime logs, it looks like the token count for each message keeps increasing. Upon closer inspection, it seems like every "request" field includes the whole chat history.

So now I'm questioning the accuracy of my logs. I'm not sure if the token counts and costs I'm getting from group_chat_manager.get_total_usage() are reliable or if they even match what's shown in the chat completion logs. Does anyone else see this happening? Am I missing something?

Is it known that the runtime logger doesnt take in to account your transforms?

[ekzhu]

runtime_logger should log the exact request sent to LLM. So it looks like the transformation is not applied to some of your agents. Can you post your code snippet? I suspect the transformation is not applied to the group chat select speaker messages.

cc @WaelKarkoub @joshkyh @gagb @cheng-tan

[ekzhu]

I don't think we need to expand the scope of that PR more.

[Nathan-Intergral]

Hey, so to update, i tried applying the message transform to the group chat manager and it didnt seem to work. I hat a little dig and i can see that the select_speaker function in groupchat.py uses conversable_agent.py's generate_oai_reply. Which unlike generate_reply does not call process_all_messages_before_reply so the hook for transform messages is never run from what i can tell. Am i understanding this correctly?

[WaelKarkoub]

@Nathan-Intergral I believe your understanding is correct, good catch! Whenever you have time, can you open an issue and tag us? I can start working on it after #2304 gets merged. Like @ekzhu said, we can start applying message transforms to the inner agent once that PR goes in and it might fix your problems.

[Nathan-Intergral]

@WaelKarkoub I have created issue #2499

[marklysze]

Hi guys, I have a PR for updating select speaker for auto mode that uses a two-way chat and changes the select speaker code. Can you please check that before changing select_speaker.

See #2304

Any questions just ask :)