Streaming complex object fast in custom component makes bug

[MobisParkHeekang]

Describe the bug

I define a complex object like below:

class TextSection(GradioModel):
    section_type: Literal["text"]
    section_name: Optional[str]
    section_state: Optional[Literal["initial", "running", "complete"]]
    section_id: str
    feedback: Optional[bool]
    section_content: str

class Bubble(GradioModel):
    role: Literal["assistant", "user", "system"]
    sections: List[TextSection]

class ChatbotData(GradioModel):
    bubbles: List[Bubble]
    status: Literal["initial", "running", "complete"]

class GradioData(GradioRootModel):
    root: ChatbotData

When I stream data with yield, errors emerge.

def run(query, chat):
    chat = {
        "bubbles": [
            {
                "role": "user",
                "sections": [
                    {
                        "section_type": "text",
                        "section_name": null,
                        "section_state": "complete",
                        "section_id": "b1152ee1-1349-11ef-be91-8f639724379a",
                        "feedback": null,
                        "section_content": ""
                    }
                ]
            },
            (assistant_bubble := {
                "role": "assistant",
                "sections": [
                    {
                        "section_type": "text",
                        "section_name": null,
                        "section_state": "running",
                        "section_id": "b11535eb-1349-11ef-ba87-8f639724379a",
                        "feedback": null,
                        "section_content": ""
                    }
                ]
            })
        ],
        "status": "running"
    }
    yield chat
    
    text = "Lorem ipsum dolor sit amet"
    for c in text:
        assistant_bubble["sections"][0]["section_content"] += c
        yield chat
    
    #time.sleep(0.1)
    assistant_bubble["sections"][0]["section_status"] = "complete"
    chat["status"] = "complete"
    yield chat

SSE data suddenly changes from below:

{
    "msg": "process_generating",
    "event_id": "2c9694b94b054f76a254d05e143e7f6f",
    "output": {
        "data": [
            {
                "bubbles": [
                    {
                        "role": "user",
                        "sections": [
                            {
                                "section_type": "text",
                                "section_name": null,
                                "section_state": "complete",
                                "section_id": "b1152ee1-1349-11ef-be91-8f639724379a",
                                "feedback": null,
                                "section_content": ""
                            }
                        ]
                    },
                    {
                        "role": "assistant",
                        "sections": [
                            {
                                "section_type": "text",
                                "section_name": null,
                                "section_state": "running",
                                "section_id": "b11535eb-1349-11ef-ba87-8f639724379a",
                                "feedback": null,
                                "section_content": ""
                            }
                        ]
                    }
                ],
                "status": "running"
            }
        ],
        "is_generating": true,
        "duration": 0.0014843940734863281,
        "average_duration": 0.0003377993901570638
    },
    "success": true
}

to below:

{
    "msg": "process_generating",
    "event_id": "2c9694b94b054f76a254d05e143e7f6f",
    "output": {
        "data": [
            [
                [
                    "replace",
                    [
                        "bubbles",
                        0,
                        "sections",
                        0,
                        "section_id"
                    ],
                    "b1155e6f-1349-11ef-ba0d-8f639724379a"
                ],
                [
                    "replace",
                    [
                        "bubbles",
                        1,
                        "sections",
                        0,
                        "section_id"
                    ],
                    "b11561a3-1349-11ef-a2a4-8f639724379a"
                ],
                [
                    "append",
                    [
                        "bubbles",
                        1,
                        "sections",
                        0,
                        "section_content"
                    ],
                    "L"
                ]
            ]
        ],
        "is_generating": true,
        "duration": 0.0003814697265625,
        "average_duration": 0.0003392081106862714
    },
    "success": true
}

However, if I put time.sleep()(currently commented out), errors do not emerge.

Have you searched existing issues? 🔎

• I have searched and found no existing issues

Reproduction

types

class TextSection(GradioModel):
    section_state: Optional[Literal["initial", "running", "complete"]]
    section_content: str

class Bubble(GradioModel):
    role: Literal["assistant", "user", "system"]
    sections: List[TextSection]

class ChatbotData(GradioModel):
    bubbles: List[Bubble]
    status: Literal["initial", "running", "complete"]

class GradioData(GradioRootModel):
    root: ChatbotData

streaming function

def run(query, chat):
    chat = {
        "bubbles": [
            {
                "role": "user",
                "sections": [
                    {
                        "section_type": "text",
                        "section_content": query
                    }
                ]
            },
            (assistant_bubble := {
                "role": "assistant",
                "sections": [
                    {
                        "section_type": "text",
                        "section_content": ""
                    }
                ]
            })
        ],
        "status": "running"
    }
    yield chat
    
    text = "Lorem ipsum dolor sit amet"
    for c in text:
        assistant_bubble["sections"][0]["section_content"] += c
        yield chat
    
    #time.sleep(0.1)
    assistant_bubble["sections"][0]["section_status"] = "complete"
    chat["status"] = "complete"
    yield chat

Screenshot

No response

Logs

No response

System Info

Gradio Environment Information:
------------------------------
Operating System: Linux
gradio version: 4.25.0
gradio_client version: 0.15.0

------------------------------------------------
gradio dependencies in your environment:

aiofiles: 23.2.1
altair: 5.2.0
fastapi: 0.110.0
ffmpy: 0.3.2
gradio-client==0.15.0 is not installed.
httpx: 0.27.0
huggingface-hub: 0.20.3
importlib-resources: 6.1.2
jinja2: 3.1.2
markupsafe: 2.1.3
matplotlib: 3.8.3
numpy: 1.26.4
orjson: 3.9.15
packaging: 23.1
pandas: 2.2.0
pillow: 9.5.0
pydantic: 2.6.1
pydub: 0.25.1
python-multipart: 0.0.9
pyyaml: 6.0.1
ruff: 0.2.2
semantic-version: 2.10.0
tomlkit==0.12.0 is not installed.
typer: 0.9.0
typing-extensions: 4.8.0
uvicorn: 0.27.1
authlib; extra == 'oauth' is not installed.
itsdangerous; extra == 'oauth' is not installed.


gradio_client dependencies in your environment:

fsspec: 2023.10.0
httpx: 0.27.0
huggingface-hub: 0.20.3
packaging: 23.1
typing-extensions: 4.8.0
websockets: 11.0.3

Severity

Blocking usage of gradio

[freddyaboulton]

Hi @MobisParkHeekang ! What exactly is the error that's happening? It's expected that the data format sent from the server to change because we will only send the diff. Can you please share the custom component you're developing?

[MobisParkHeekang]

Thank you for your reply. Unfortunately, the component is on the company's private server, so it cannot be shared. However, your expectation sounds really plausible. Can you please tell me in which file the diff calculation and sending occur?

[abidlabs]

Hi @MobisParkHeekang you may want to check this out:

gradio/gradio/blocks.py

Line 1792 in 8f46556

def handle_streaming_diffs(

[MobisParkHeekang]

Can you please tell me in which js(or svelte) file ‘rebuild’ value from diffs sent from FastAPI server?

[freddyaboulton]

The js client will rebuild the value based on the diff:

gradio/client/js/src/utils/submit.ts

Line 597 in 35905c5

apply_diff_stream(pending_diff_streams, event_id!, data);