Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ChatAccordion and ChatCard #6598

Open
ahuang11 opened this issue Mar 27, 2024 · 10 comments · May be fixed by #6617
Open

ChatAccordion and ChatCard #6598

ahuang11 opened this issue Mar 27, 2024 · 10 comments · May be fixed by #6617
Assignees
Labels
chat A label for chat type: feature A major new feature

Comments

@ahuang11
Copy link
Contributor

ahuang11 commented Mar 27, 2024

Edit: I realized I have the ChatAccordion and ChatStatusCard reversed below in the design, i.e. ChatCard should wrap ChatAccordion

The design should also mention how to integrate this functionality seamlessly in callback. My initial thought is yielding/returning a dict with intermediate=True {"The text to stream", "intermediate": True} creates a ChatMessage with these chat status cards. Maybe context manager too:

def callback(contents, user, instance):
    with ChatCard(title="Thinking...") as cc:
        yield cc
        cc.append(...)
        cc.append(...)
        cc.title = "Completed result"
Screen.Recording.2024-03-27.at.11.05.46.PM.mov

--

To improve user experience while LLMs are performing intermediate steps, I'd like to propose two new chat components, namely, ChatAccordion and ChatStatusCard.

Unlike streaming text, character by character, these two components will improve perception of runtime by outputting each intermediate steps as ChatStatusCards.

Here's an example:
image

image

The LLM could output a step by step plan to achieve the outcome:

  • as each step is determined, a new ChatStatusCard is appended to ChatAccordion
  • as each step is completed, the results are nested under the ChatStatusCard which users may open at any time to view
  • The status emojis will change from yellow to green if succeeded or red if failed, and perhaps another emoji if needs user input to continue

Once completed, it'll show the final result at the bottom.

image

The intermediate steps can also be hidden.

image

Here's the abstract version:

image

I believe the API would look like:

class ChatAccordion(Accordion):

    objects = param.List()

    status = param.Selector()

    result = param.Parameter()

    def send()  #  to match feed
        ...

    def clear()
        ...
class ChatStatusCard(Card):

    objects = param.List()

    status = param.Selector(...)

    default_emojis = param.Dict()  # maybe?

At the current moment, I'm having a hard time to figuring out:

  • how to stream text inside the ChatStatusCard
    • do we use ChatMessage as an object inside without an avatar and all the other things
    • do we add yet another component ChatText for streaming text?)
  • the names of these components; I don't really like them

Any feedback appreciated

@ahuang11 ahuang11 added the chat A label for chat label Mar 27, 2024
@ahuang11 ahuang11 added the type: feature A major new feature label Mar 27, 2024
@ahuang11
Copy link
Contributor Author

ahuang11 commented Mar 27, 2024

If there's a ChatText, besides ability to stream text, maybe it can additionally support the floating top right copy icon.

image

@ahuang11
Copy link
Contributor Author

Actually I think I have it reversed; the card should be the outer object while the accordion is the inner object.

Screen.Recording.2024-03-27.at.11.05.46.PM.mov
import time
import panel as pn

pn.extension()


def callback(contents, user, instance):
    time.sleep(0.5)

    card = pn.Card(
        title="Thinking...", sizing_mode="stretch_width",
    )

    accordion = pn.Accordion(sizing_mode="stretch_width", active=[0], margin=0)
    card.append(accordion)
    yield card

    first_card = pn.pane.Markdown("Thinking...")
    accordion.append(("What are my meetings today?", first_card))
    time.sleep(2)
    first_card.object = "You have a meeting at 10:00 AM with the team and another at 2:00 PM with the client."
    
    second_card = pn.pane.Markdown("Thinking...")
    accordion.append(("What time is it right now", second_card))
    time.sleep(1)
    second_card.object = "It is currently 9:30 AM."

    third_card = pn.pane.Markdown("Which meeting is next?")
    accordion.append(("Which meeting is next?", third_card))
    time.sleep(1)
    third_card.object = "Your next meeting is at 10:00 AM with the team."

    fourth_card = pn.pane.Markdown("Thinking...")
    accordion.append(("How long until the next meeting?", fourth_card))
    time.sleep(1)
    fourth_card.object = "Your next meeting is in 30 minutes."

    return "Your next meeting is in 30 minutes."

chat = pn.chat.ChatInterface(callback=callback)
chat.show()
chat.send("How long until my next meeting?")

@ahuang11 ahuang11 changed the title ChatAccordion and ChatStatusCard ChatAccordion and ChatCard Mar 28, 2024
@MarcSkovMadsen
Copy link
Collaborator

MarcSkovMadsen commented Mar 28, 2024

Why do we need new components? Why can we not just use the existing components as shown in the post just above?

ps. I don't think you can mix yield and return as you do in your example above. I only think you can return (None) ??

@philippjfr
Copy link
Member

Agreed, if existing components don't do what we need them to we should first look to see if we can improve them to achieve what we need rather than moving straight to adding new components.

@MarcSkovMadsen
Copy link
Collaborator

I'm all for making it easy to show intermediate steps via Cards and Accordions. That is very powerful.

@ahuang11
Copy link
Contributor Author

I think the context manager part is definitely a desired feature, which pn.Card does not support natively, but I'll explore this idea more with applicable examples and identify what's missing.

@philippjfr
Copy link
Member

Oh definitely not saying there aren't various improvements we could make but I just wanted to emphasize that we should always try to generalize existing components and concepts instead of adding new ones. That's not to say that's necessarily true here and I owe this issue a more detailed look and analysis.

@ahuang11
Copy link
Contributor Author

ahuang11 commented Mar 29, 2024

Here's a more refined version from before. I'll need to explore it with actual LLM inputs and outputs.

Screen.Recording.2024-03-28.at.5.52.29.PM.mov

Some initial thoughts from exploration:

  • we should make streaming text to panel object a util function, e.g. stream_to_markdown; no need for ChatText
  • card / accordion CSS is a bit clunky looking; would like to remove the background, make fontsize smaller, no outline, specifically for chat use
  • I need to redesign the layout of the ChatMessage; margins are not ideal (I noticed this when I was styling the backgrounds)
image
import time
import panel as pn
from contextlib import contextmanager

pn.extension()


def append_step(accordion, question, response, placeholder="Thinking..."):
    card = pn.pane.Markdown(placeholder)
    accordion.append((question, card))
    time.sleep(1)
    accordion.active = [len(accordion.objects) - 1]  # configurable
    time.sleep(1)
    for i, char in enumerate(response):
        if i == 0:
            card.object = char
        else:
            time.sleep(0.001)
            card.object += char


@contextmanager
def show_steps():
    try:
        card = pn.Card(
            title="Thinking...",
            sizing_mode="stretch_width",
        )
        accordion = pn.Accordion(sizing_mode="stretch_width", margin=0, toggle=True)
        card.append(accordion)
        yield card, accordion
    finally:
        card.title = accordion.objects[-1].object


def callback(contents, user, instance):
    time.sleep(0.5)
    with show_steps() as (card, accordion):
        yield accordion

        # in actual class it would be
        # accordion.append(markdown_obj, markdown_obj)
        append_step(
            accordion,
            "What are my meetings today?",
            "You have a meeting at 10:00 AM with the team and another at 2:00 PM with the client.",
        )

        append_step(
            accordion,
            "What time is it right now",
            "It is currently 9:30 AM.",
        )

        append_step(
            accordion,
            "Which meeting is next?",
            "Your next meeting is at 10:00 AM with the team.",
        )

        append_step(
            accordion,
            "How long until the next meeting?",
            "Your next meeting is in 30 minutes.",
        )


chat = pn.chat.ChatInterface(callback=callback, callback_exception="verbose")
chat.show()

@ahuang11
Copy link
Contributor Author

ahuang11 commented Mar 29, 2024

Okay, here's an actual version using an LLM:

Screen.Recording.2024-03-28.at.8.47.57.PM.mov

Two things I discovered:

  • Accordion doesn't update titles easily without replacing the entire accordion object, so I have cards nested in a card for now Updating Accordions' titles / objects names #5377
  • I'd also like a custom ChatCard to serialize() the chat history and cleanup half this code
from typing import List
from pydantic import Field, BaseModel


class Query(BaseModel):
    """Class representing a single question in a query plan."""

    id: int = Field(..., description="Unique id of the query")
    question: str = Field(
        ...,
        description="Question asked using a question answering system",
    )
    dependencies: List[int] = Field(
        default_factory=list,
        description="List of sub questions that need to be answered before asking this question",
    )


class QueryPlan(BaseModel):
    """Container class representing a tree of questions to ask a question answering system."""

    query_graph: List[Query] = Field(
        ..., description="The query graph representing the plan"
    )

    def _dependencies(self, ids: List[int]) -> List[Query]:
        """Returns the dependencies of a query given their ids."""
        return [q for q in self.query_graph if q.id in ids]


import instructor
from openai import OpenAI

# Apply the patch to the OpenAI client
# enables response_model keyword
client = instructor.patch(OpenAI())


def query_planner(question: str) -> QueryPlan:
    PLANNING_MODEL = "gpt-4-0613"

    messages = [
        {
            "role": "system",
            "content": "You are a world class query planning algorithm capable ofbreaking apart questions into its dependency queries such that the answers can be used to inform the parent question. Do not answer the questions, simply provide a correct compute graph with good specific questions to ask and relevant dependencies. Before you call the function, think step-by-step to get a better understanding of the problem.",
        },
        {
            "role": "user",
            "content": f"Consider: {question}\nGenerate the correct query plan.",
        },
    ]

    root = client.chat.completions.create(
        model=PLANNING_MODEL,
        temperature=0,
        response_model=instructor.Partial[QueryPlan],
        messages=messages,
        max_tokens=1000,
        stream=True,
    )
    return root


import time
import panel as pn
from contextlib import contextmanager

pn.extension()


@contextmanager
def show_steps(instance):
    try:
        card = pn.Card(
            title="Thinking...",
            sizing_mode="stretch_width",
        )
        instance.stream(card)
        yield card
    finally:
        card.title = card.objects[-1].objects[-1].object


def callback(contents, user, instance):

    queries = {}
    with show_steps(instance) as card:
        chunked_plan = query_planner(
            "What is the difference in populations of Canada and the Andrew's home country?"
        )
        for chunk in chunked_plan:
            for query in chunk.model_dump()["query_graph"] or []:
                if query.get("id") is None or hasattr(query, "question"):
                    continue

                if query["id"] not in queries:
                    answer_md = pn.pane.Markdown("")
                    sub_card = pn.Card(
                        answer_md,
                        title=query["question"] or "",
                        sizing_mode="stretch_width",
                    )
                    queries[query["id"]] = sub_card
                    card.append(sub_card)
                else:
                    sub_card = queries[query["id"]]
                    sub_card.title = query["question"] or ""

        history = []
        for sub_card in card:
            # serialize is desperately needed here:
            question_dict = {"role": "user", "content": sub_card.title}
            history.append(question_dict)
            messages = [
                {
                    "role": "system",
                    "content": "Answer the following question to the best of your abilities.",
                },
                {
                    "role": "user",
                    "content": "Andrew lives in the US"
                },
                *history,
            ]
            response = client.chat.completions.create(
                model="gpt-3.5-turbo",
                temperature=0,
                messages=messages,
                max_tokens=1000,
                stream=True,
            )
            for chunk in response:
                if chunk:
                    sub_card.objects[0].object += chunk.choices[0].delta.content or ""
            history.append({"role": "assistant", "content": sub_card.objects[0].object})


chat = pn.chat.ChatInterface(callback=callback, callback_exception="verbose")
chat.show()

@ahuang11
Copy link
Contributor Author

ahuang11 commented Apr 2, 2024

Okay, I have a draft API available at #6617 now

Screen.Recording.2024-04-01.at.6.00.35.PM.mov

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
chat A label for chat type: feature A major new feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants