Scaling socketio server in FastAPI

[leestorm4520]

Hi,
I am using AsyncServer socketio in FastAPI with SQLAlchemy and Postgresql.
I have a list of sub-apps (socketio clients such as Streamlit, Vuejs, FastAPI,...) connected to the server. However, as more users use the app, the server stops accepting new connections. I checked the log of the server and I dont see the problem yet lots of client connections get kicked out so I think I'll need to scale the server.
Do you know any way to scale with my current tech stacks? I dont have Redis so I can only resort to Postgresql

[miguelgrinberg]

I can't really give you a recommendation without a complete understanding of why those connections are being refused. Things such as:

• What is the connection URL that is faling
• How does it fail (status code, response body, etc)
• How many active clients do you currently have or need to support
• You mentioned you have sub-apps that connect to the server as clients. What is the significance of this? I assume it is just a handful of apps, so not a reason to increase the load in your server in a big way?
• Are you doing any blocking calls in your server? Is the Postgres/SQLAlchemy use async friendly and done properly?

[leestorm4520]

so I have an AsyncServer and other AsyncClients. AsyncServer failing to connect with other AsyncClient was my issue. AsyncClient tried to connect but exception was raised "Fail to connect with the server". So I have an AsyncServer, and around 15 clients (both async and non async) but I am likely to expand more clients in the future.
Regarding subapps, yeah it is just a handful of apps. Those are clients that I was talking about
Regarding blocking calls, I have namespaces as class and use lots of sqlalchemy sessions to query data from and update tables in Postgresql. I use py-spy to profile fastapi uvicorn process and I dont see cpu over-utilized so I think since socketio server puts everything in memory and grows as there are more connections, it must have rejected new connections since it's overloaded

[leestorm4520]

Also, in javascript, socketio can scale using postgresql adapter to scale (https://socket.io/docs/v4/postgres-adapter/)
Do we have or intend to implement this in python?

[miguelgrinberg]

Are you using the async support in SQLAlchemy? You can't use normal SQLAlchemy calls in an async app, since these would be blocking there.

With such a small number of clients I believe it is unlikely you are at a point that you need to scale. See the message queue section of the documentation. Unfortunately there is no Postgres support, you will need to use something else or build a Postgres adapter (which I would be happy to accept as a contribution!)

[leestorm4520]

Yes, all of my operations are asynchronous. I use sqlalchemy, asyncpg, aiohttp, and aiocache. That's why I am not sure why it would suddenly block new connections, quite random. The correlation I can say is when the user use the manager app (Vuejs). This app is the front face that uses iframe to contain all other apps. This app everytime would create 4 connections with 4 different namespaces. I would see a ton of polling (GET/POST) from this manager app everytime it connects to the server though. Maybe that's the root cause I am not sure.
How does python-socketio handle memory though? At what point does it releases old memory?

[miguelgrinberg]

I'm not sure what to make of this. What are these GET/POST requests for? A Socket.IO client would only issue 3 or 4 HTTP requests and then the connection would switch to WebSocket. Couple of things I was curious about that you haven't provided any details on yet:

• What is the connection URL that is faling
• How does it fail (status code, response body, etc)

How does python-socketio handle memory though? At what point does it releases old memory?

Do you have memory issues? This package does not hang on to old memory, it just maintains state for connected clients.