Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Serve] FastAPI based deployment doesn't respect ASGI scope state. #45395

Open
douglasdavis opened this issue May 16, 2024 · 1 comment
Open

[Serve] FastAPI based deployment doesn't respect ASGI scope state. #45395

douglasdavis opened this issue May 16, 2024 · 1 comment
Labels
bug Something that is supposed to be working; but isn't P2 Important issue, but not time-critical serve Ray Serve Related Issue

Comments

@douglasdavis
Copy link

douglasdavis commented May 16, 2024 •
edited

What happened + What you expected to happen

Gist: If I write a middleware that attaches a long lived object to the ASGI scope state then the middleware works with a regular FastAPI + Uvicorn setup. The middleware crashes with KeyError (the scope state dictionary is missing what I added) in a Ray Serve deployment; the request has an empty scope state where a shallow copy is expected (according to the ASGI spec).

More info before dumping the code: I'm writing a pure ASGI middleware that works on two connection scope types: lifespan and http;

  • during a lifespan connection, I check for lifespan.start and intialize a long lived "Producer" object on the scope["state"] dictionary at key ["my.producer"]. During lifespan.shutdown I call a flush method on that producer still living at scope["state"] dictionary.
  • during a http connection, I use the scope["state"] to retrieve the existing long-lived Producer object with scope["state"]["my.producer"]: in a pure FastAPI + Uvicorn setup the "my.producer" key is conserved, with Ray Serve the key is not there.

Related part of the ASGI spec: https://asgi.readthedocs.io/en/latest/specs/lifespan.html#lifespan-state

Versions / Dependencies

ray, version 2.22.0

Reproduction script

from fastapi import FastAPI
from starlette.requests import Request
from starlette.types import ASGIApp, Message, Receive, Scope, Send
from ray import serve


class Producer:
    def __init__(self, servers: str) -> None:
        print(f"producer created at {servers}")

    def send(self, message: bytes) -> None:
        print(f"producer sending message {message}")

    def flush(self):
        print("producer flushed")


class MyMiddleware:
    def __init__(self, app: ASGIApp, producer_servers: str) -> None:
        self.app = app
        self.producer_servers = producer_servers

    def start_producer(self, scope: Scope) -> None:
        scope["state"]["my.producer"] = Producer(self.producer_servers)
        return None

    def stop_producer(self, scope: Scope) -> None:
        scope["state"]["my.producer"].flush()
        return None

    async def process_lifespan_cxn(
        self, scope: Scope, receive: Receive, send: Send
    ) -> None:
        async def wrap_receive():
            message = await receive()
            if message["type"] == "lifespan.startup":
                self.start_producer(scope)
            elif message["type"] == "lifespan.shutdown":
                self.stop_producer(scope)

            return message

        await self.app(scope, wrap_receive, send)
        return None

    async def process_http_cxn(
        self, scope: Scope, receive: Receive, send: Send
    ) -> None:
        async def wrap_send(message):
            if message["type"] == "http.response.body":
                scope["state"]["my.producer"].send(message.get("body", b""))
            await send(message)

        await self.app(scope, receive, wrap_send)
        return None

    async def __call__(self, scope: Scope, receive: Receive, send: Send):
        if scope["type"] == "lifespan":
            await self.process_lifespan_cxn(scope, receive, send)
        elif scope["type"] == "http":
            await self.process_http_cxn(scope, receive, send)
        else:
            await self.app(scope, receive, send)
        return None

If I run a pure FastAPI+Uvicorn setup:

app = FastAPI()

app.add_middleware(MyMiddleware, producer_servers="some-ip")


@app.post("/hit")
async def hit(request: Request, data: dict) -> dict:
    print("scope[state] in endpoint", request.scope["state"])
    return data

and hit it with

$ curl http://localhost:8000/hit -H "Content-Type: application/json" -d '{"a":"b"}'

I see (with print statements in the expected places, including after Ctrl+Cing after the single request):

(ray-test) ddavis@charm ~/sandbox $ uvicorn repro:app
INFO:     Started server process [50670]
INFO:     Waiting for application startup.
producer created at some-ip
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
scope[state] in endpoint {'my.producer': <repro.Producer object at 0x10667fdd0>}
INFO:     127.0.0.1:65196 - "POST /hit HTTP/1.1" 200 OK
producer sending message b'{"a":"b"}'
^CINFO:     Shutting down
INFO:     Waiting for application shutdown.
producer flushed
INFO:     Application shutdown complete.
INFO:     Finished server process [50670]

But with a serve deployment the scope state doesn't appear to make it to the http connection:

def class_deco(app: FastAPI, servers: str):
    def deco(cls):
        app.add_middleware(MyMiddleware, producer_servers=servers)
        return cls

    return deco


app = FastAPI()


@serve.deployment
@serve.ingress(app)
@class_deco(app, "some-ip")
class MyDep:
    def __init__(self):
        pass

    @app.post("/hit")
    async def hit(self, request: Request, data: dict) -> dict:
        return data


dep = MyDep.bind()

If I start the deployment and shut it down without sending any requests I see expected (notice the producer start up and flush):

(ray-test) ddavis@charm ~/sandbox $ serve run repro:dep
2024-05-16 15:45:02,546	INFO scripts.py:499 -- Running import path: 'repro:dep'.
2024-05-16 15:45:04,291	INFO worker.py:1740 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265 
2024-05-16 15:45:05,812	INFO handle.py:126 -- Created DeploymentHandle 'kw66yfxy' for Deployment(name='MyDep', app='default').
2024-05-16 15:45:05,812	INFO handle.py:126 -- Created DeploymentHandle 'sknrvodv' for Deployment(name='MyDep', app='default').
(ProxyActor pid=51654) INFO 2024-05-16 15:45:05,789 proxy 127.0.0.1 proxy.py:1161 - Proxy starting on node f48e16224fcc31c6471ecb8c776665ea2bd1ea569a2b80380a7e0789 (HTTP port: 8000).
(ServeController pid=51652) INFO 2024-05-16 15:45:05,910 controller 51652 deployment_state.py:1598 - Deploying new version of Deployment(name='MyDep', app='default') (initial target replicas: 1).
(ServeController pid=51652) INFO 2024-05-16 15:45:06,012 controller 51652 deployment_state.py:1844 - Adding 1 replica to Deployment(name='MyDep', app='default').
(ServeReplica:default:MyDep pid=51657) producer created at some-ip
2024-05-16 15:45:06,822	INFO handle.py:126 -- Created DeploymentHandle 'dpug7vv7' for Deployment(name='MyDep', app='default').
2024-05-16 15:45:06,822	INFO api.py:584 -- Deployed app 'default' successfully.
^C2024-05-16 15:45:09,924	WARNING api.py:592 -- Got KeyboardInterrupt, exiting...
2024-05-16 15:45:09,924	INFO scripts.py:579 -- Got KeyboardInterrupt, shutting down...
(ServeController pid=51652) INFO 2024-05-16 15:45:10,001 controller 51652 deployment_state.py:1860 - Removing 1 replica from Deployment(name='MyDep', app='default').
(ServeController pid=51652) INFO 2024-05-16 15:45:12,062 controller 51652 deployment_state.py:2182 - Replica(id='n3e7ug8f', deployment='MyDep', app='default') is stopped.
(ServeReplica:default:MyDep pid=51657) producer flushed

But if I send a request the scope state is not as expected (notice the KeyError)

(ray-test) ddavis@charm ~/sandbox $ serve run repro:dep
2024-05-16 15:46:23,048	INFO scripts.py:499 -- Running import path: 'repro:dep'.
2024-05-16 15:46:24,791	INFO worker.py:1740 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265 
2024-05-16 15:46:26,350	INFO handle.py:126 -- Created DeploymentHandle 'q4e9jxcf' for Deployment(name='MyDep', app='default').
(ProxyActor pid=51923) INFO 2024-05-16 15:46:26,318 proxy 127.0.0.1 proxy.py:1161 - Proxy starting on node ab5627b9fdcef2afabbb491e7dab0d646d985ed7819ecb5a9148bea1 (HTTP port: 8000).
2024-05-16 15:46:26,350	INFO handle.py:126 -- Created DeploymentHandle '5shlrfvd' for Deployment(name='MyDep', app='default').
(ServeController pid=51922) INFO 2024-05-16 15:46:26,454 controller 51922 deployment_state.py:1598 - Deploying new version of Deployment(name='MyDep', app='default') (initial target replicas: 1).
(ServeController pid=51922) INFO 2024-05-16 15:46:26,557 controller 51922 deployment_state.py:1844 - Adding 1 replica to Deployment(name='MyDep', app='default').
(ServeReplica:default:MyDep pid=51925) producer created at some-ip
2024-05-16 15:46:27,365	INFO handle.py:126 -- Created DeploymentHandle 'e217zo6r' for Deployment(name='MyDep', app='default').
2024-05-16 15:46:27,365	INFO api.py:584 -- Deployed app 'default' successfully.
(ProxyActor pid=51923) ERROR:    Exception in ASGI application
(ProxyActor pid=51923) Traceback (most recent call last):
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 411, in run_asgi
(ProxyActor pid=51923)     result = await app(  # type: ignore[func-returns-value]
(ProxyActor pid=51923)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
(ProxyActor pid=51923)     return await self.app(scope, receive, send)
(ProxyActor pid=51923)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/ray/serve/_private/proxy.py", line 1108, in __call__
(ProxyActor pid=51923)     await self.app(scope, receive, send_with_request_id)
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/ray/serve/_private/proxy.py", line 846, in __call__
(ProxyActor pid=51923)     await send(message)
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/ray/serve/_private/proxy.py", line 1106, in send_with_request_id
(ProxyActor pid=51923)     await send(message)
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 522, in send
(ProxyActor pid=51923)     raise RuntimeError(msg % message_type)
(ProxyActor pid=51923) RuntimeError: Expected ASGI message 'http.response.body', but got 'http.response.start'.
(ProxyActor pid=51923) Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::ServeReplica:default:MyDep.handle_request_with_rejection() (pid=51925, ip=127.0.0.1, actor_id=2e85ec4ea7623ff00851428e01000000, repr=<ray.serve._private.replica.ServeReplica:default:MyDep object at 0x10b2171d0>)
(ProxyActor pid=51923)     async for result in self._call_user_generator(
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/ray/serve/_private/replica.py", line 460, in _call_user_generator
(ProxyActor pid=51923)     raise e from None
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/ray/serve/_private/replica.py", line 1150, in call_user_method
(ProxyActor pid=51923)     raise e from None
(ProxyActor pid=51923) ray.exceptions.RayTaskError: ray::ServeReplica:default:MyDep.handle_request_with_rejection() (pid=51925, ip=127.0.0.1)
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/ray/serve/_private/utils.py", line 168, in wrap_to_ray_error
(ProxyActor pid=51923)     raise exception
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/ray/serve/_private/replica.py", line 1132, in call_user_method
(ProxyActor pid=51923)     await self._call_func_or_gen(
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/ray/serve/_private/replica.py", line 856, in _call_func_or_gen
(ProxyActor pid=51923)     result = await result
(ProxyActor pid=51923)              ^^^^^^^^^^^^
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/ray/serve/_private/http_util.py", line 456, in __call__
(ProxyActor pid=51923)     await self._asgi_app(
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
(ProxyActor pid=51923)     await super().__call__(scope, receive, send)
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
(ProxyActor pid=51923)     await self.middleware_stack(scope, receive, send)
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
(ProxyActor pid=51923)     raise exc
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
(ProxyActor pid=51923)     await self.app(scope, receive, _send)
(ProxyActor pid=51923)   File "/Users/ddavis/sandbox/repro.py", line 61, in __call__
(ProxyActor pid=51923)     await self.process_http_cxn(scope, receive, send)
(ProxyActor pid=51923)   File "/Users/ddavis/sandbox/repro.py", line 54, in process_http_cxn
(ProxyActor pid=51923)     await self.app(scope, receive, wrap_send)
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
(ProxyActor pid=51923)     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
(ProxyActor pid=51923)     raise exc
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
(ProxyActor pid=51923)     await app(scope, receive, sender)
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
(ProxyActor pid=51923)     await self.middleware_stack(scope, receive, send)
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
(ProxyActor pid=51923)     await route.handle(scope, receive, send)
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
(ProxyActor pid=51923)     await self.app(scope, receive, send)
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
(ProxyActor pid=51923)     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
(ProxyActor pid=51923)     raise exc
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
(ProxyActor pid=51923)     await app(scope, receive, sender)
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/routing.py", line 75, in app
(ProxyActor pid=51923)     await response(scope, receive, send)
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/responses.py", line 159, in __call__
(ProxyActor pid=51923)     await send({"type": prefix + "http.response.body", "body": self.body})
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/_exception_handler.py", line 50, in sender
(ProxyActor pid=51923)     await send(message)
(ProxyActor pid=51923)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/_exception_handler.py", line 50, in sender
(ProxyActor pid=51923)     await send(message)
(ProxyActor pid=51923)   File "/Users/ddavis/sandbox/repro.py", line 51, in wrap_send
(ProxyActor pid=51923)     scope["state"]["my.producer"].send(message.get("body", b""))
(ProxyActor pid=51923)     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
(ProxyActor pid=51923) KeyError: 'my.producer'
(ServeReplica:default:MyDep pid=51925) ERROR 2024-05-16 15:46:32,091 default_MyDep pb2dbf16 624e4140-62b2-4dc1-9460-198ff7b2de12 /hit replica.py:359 - Request failed:
(ServeReplica:default:MyDep pid=51925) ray::ServeReplica:default:MyDep.handle_request_with_rejection() (pid=51925, ip=127.0.0.1)
(ServeReplica:default:MyDep pid=51925) INFO 2024-05-16 15:46:32,091 default_MyDep pb2dbf16 624e4140-62b2-4dc1-9460-198ff7b2de12 /hit replica.py:373 - __CALL__ ERROR 11.8ms
^C2024-05-16 15:46:35,377	WARNING api.py:592 -- Got KeyboardInterrupt, exiting...
2024-05-16 15:46:35,377	INFO scripts.py:579 -- Got KeyboardInterrupt, shutting down...
(ServeController pid=51922) INFO 2024-05-16 15:46:35,441 controller 51922 deployment_state.py:1860 - Removing 1 replica from Deployment(name='MyDep', app='default').
(ServeReplica:default:MyDep pid=51925) producer flushed
(ServeController pid=51922) INFO 2024-05-16 15:46:37,489 controller 51922 deployment_state.py:2182 - Replica(id='pb2dbf16', deployment='MyDep', app='default') is stopped.
(ServeReplica:default:MyDep pid=51925)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/responses.py", line 159, in __call__ [repeated 8x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(ServeReplica:default:MyDep pid=51925)     await send(message) [repeated 2x across cluster]
(ServeReplica:default:MyDep pid=51925)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/ray/serve/_private/replica.py", line 1132, in call_user_method
(ServeReplica:default:MyDep pid=51925)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/ray/serve/_private/utils.py", line 168, in wrap_to_ray_error
(ServeReplica:default:MyDep pid=51925)     raise exception
(ServeReplica:default:MyDep pid=51925)     await self._call_func_or_gen(
(ServeReplica:default:MyDep pid=51925)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/ray/serve/_private/replica.py", line 856, in _call_func_or_gen
(ServeReplica:default:MyDep pid=51925)     result = await result
(ServeReplica:default:MyDep pid=51925)              ^^^^^^^^^^^^
(ServeReplica:default:MyDep pid=51925)     await self._asgi_app(
(ServeReplica:default:MyDep pid=51925)     await super().__call__(scope, receive, send)
(ServeReplica:default:MyDep pid=51925)     await self.middleware_stack(scope, receive, send) [repeated 2x across cluster]
(ServeReplica:default:MyDep pid=51925)     raise exc [repeated 3x across cluster]
(ServeReplica:default:MyDep pid=51925)     await self.app(scope, receive, _send)
(ServeReplica:default:MyDep pid=51925)   File "/Users/ddavis/sandbox/repro.py", line 61, in __call__
(ServeReplica:default:MyDep pid=51925)     await self.process_http_cxn(scope, receive, send)
(ServeReplica:default:MyDep pid=51925)   File "/Users/ddavis/sandbox/repro.py", line 54, in process_http_cxn
(ServeReplica:default:MyDep pid=51925)     await self.app(scope, receive, wrap_send)
(ServeReplica:default:MyDep pid=51925)     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
(ServeReplica:default:MyDep pid=51925)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app [repeated 4x across cluster]
(ServeReplica:default:MyDep pid=51925)     await app(scope, receive, sender) [repeated 2x across cluster]
(ServeReplica:default:MyDep pid=51925)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/routing.py", line 75, in app [repeated 3x across cluster]
(ServeReplica:default:MyDep pid=51925)     await route.handle(scope, receive, send)
(ServeReplica:default:MyDep pid=51925)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
(ServeReplica:default:MyDep pid=51925)     await self.app(scope, receive, send)
(ServeReplica:default:MyDep pid=51925)     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
(ServeReplica:default:MyDep pid=51925)     await response(scope, receive, send)
(ServeReplica:default:MyDep pid=51925)     await send({"type": prefix + "http.response.body", "body": self.body})
(ServeReplica:default:MyDep pid=51925)   File "/Users/ddavis/.pyenv/versions/3.11.9/envs/ray-test/lib/python3.11/site-packages/starlette/_exception_handler.py", line 50, in sender [repeated 2x across cluster]
(ServeReplica:default:MyDep pid=51925)   File "/Users/ddavis/sandbox/repro.py", line 51, in wrap_send
(ServeReplica:default:MyDep pid=51925)     scope["state"]["my.producer"].send(message.get("body", b""))
(ServeReplica:default:MyDep pid=51925)     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
(ServeReplica:default:MyDep pid=51925) KeyError: 'my.producer'

Looks like both the proxy and the replica aren't conserving the ASGI scope state?

Issue Severity

Medium: It is a significant difficulty but I can work around it.
@douglasdavis douglasdavis added bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component) labels May 16, 2024
@anyscalesam anyscalesam added the serve Ray Serve Related Issue label May 16, 2024
@anyscalesam
Copy link
Collaborator

@douglasdavis this is a fantastically detailed issue report; thank you much for taking the time and effort! We'll review this with the Ray Serve team early next week.

@shrekris-anyscale shrekris-anyscale added P2 Important issue, but not time-critical and removed triage Needs triage (eg: priority, bug/not-bug, and owning component) labels May 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something that is supposed to be working; but isn't P2 Important issue, but not time-critical serve Ray Serve Related Issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@douglasdavis @shrekris-anyscale @anyscalesam