Upcoming architecture changes for Langfuse 3.0

[maxdeichmann]

Hi all,

Langfuse is growing a lot, both in feature scope as well as in usage on single instances. Thus we plan for a couple of changes that will be released in Langfuse v3.

We are currently required to mature our architecture as we are working on the following challenges:
✅ Building model-based evals, which requires us to run asynchronous tasks, rate limited, with failover capabilities.
🧑‍🍳 Improve performance as instances scale out.

I wanted to give you a heads up on upcoming changes which are required to make these features work. Currently, Langfuse contains a single Docker container, which takes care of everything we do. This was fast to set up Langfuse initially, but we need more technical capabilities now. In addition to the existing components (Docker container + Postgres database), we will add the following:

• Asynchronous processing in a dedicated container
• Redis container (for queues)
• OLAP storage (likely Clickhouse) to enable lower latencies on analytical queries and scale out storage for large and multi modal inputs outputs

If you self-host Langfuse, this means that we will likely advise to change to the following setup to be able to benefit from new infra changes easily. We are happy to hear your thoughts on this:

• Migrate from a container service (e.g. GCP Cloud Run) to a Virtual Machine to run docker-compose from there. Docker-compose helps to quickly bootstrap the new components.
• Either provide connection strings to the docker-compose for Redis, Postgres, and Clickhouse for managed databases or run all of them on the virtual machine as well.

Feel free to share your thoughts below on these topics:

• Tool choice: do you have experience with the tools mentioned above and have advise for us?
• Do you have ideas on how to make self-hosting easier in a multi-container, multi-database setup than docker compose?
• Do you have ideas on how a great migration path could look like?

Find more context in the last Langfuse Townhall meeting. We will provide an easy to follow upgrade path for self-hosters once v3 is generally available. The infrastructure change does not affect public APIs, thus, users of Langfuse Cloud will not be affected by this change. Currently we pilot the async container & queue for the evals feature which is currently in public beta on Langfuse Cloud.

We plan to release v3 in June and will post updates in this thread.

[MarkEdmondson1234]

As requested from Discord, my comment:

I really do not want to move off serverless infra to a dedicated VM, I'd say a major reason I chose Langfuse was its Cloud Run deployment I could couple with my existing AlloyDB. And perhaps AlloyDB is quick enough it doesn't need help with analytical queries.

Cloud Run recently introduced side car containers so perhaps that is an option?
https://cloud.google.com/blog/products/serverless/cloud-run-now-supports-multi-container-deployments

There is a managed Redis option too but its a bit more pricy.

My serverless deployment at the moment I don't pay for Langfuse until I'm browsing the UI or its capturing traces, aside the already sunk cost of the database.

[noorvir]

The current docs advise not to use Docker Compose for production. I'm guessing that will change for v3? I would be great to have a ready to deploy docker compose that just needs an env file to get started. I'd use the cloud hosted service but have relatively strict data privacy requirements.

[mautini]

Hey, I would be please to help to migrate the Helm chart to v3 is needed!

[marcklingen]

awesome, thank you @mautini 🙏 We are currently finalizing many infra pieces, will definitely reach out!

[nanoguy0]

Our current stack uses langfuse in Kubernetes so I will be happy to assist as well!

[verdverm]

We use k8s operators for the redis, postgres, clickhouse components. Would want a way to have develop vs production setups, which is probably easy from the chart perspective, more about docs / examples

[marcklingen]

I think it's sensible to do this similar as the current chart, i.e. provide two options:

1. dev/staging, low-volume prod: deploy all components within k8s
2. high-volume prod: add api keys for dedicated/managed redis, clickhouse, postgres

[Manouchehri]

I'm hesitant on a more complicated docker-compose setup. One of the reasons we were able to open to using Langfuse to begin with, was how easy it was to deploy on a serverless platform.

[nikcaryo-super]

copying from discord, GCP offers: https://cloud.google.com/blog/products/serverless/cloud-run-now-supports-multi-container-deployments but I'm not sure exactly how those would work for asynchronous tasks (e.g. does the sidecar running an async task continue running if the request finishes and the cpu is unallocated?)

maybe there's an architecture where async tasks are just sent as webhooks to whatever "langfuse-worker" url people have setup? and then GCP people can just have a langfuse-worker serverless instance / pubsub task queue or whatever.

[MarkEdmondson1234]

That’s how the rest of my infra works, pubsub events to Cloud Run scale to 0 workers

[marcklingen]

agree, would love to figure out together how to best run this on Google Cloud Run to potentially scale to 0. The worker is necessary to power async workloads (evals, tokenization) but could also scale to zero if the queue could be in pub/sub (GCP specific). I/m unsure how to find a generalizable solution here but we'll figure it out

[fernandocamargoai]

It feels weird to deploy using docker-compose. For me, it's always have been a good tool to use during development, but not really for production.

My team is currently planning to deploy it. We haven't decided if we'll deploy to kubernetes or Cloud Run though. We'll stay tuned to see what will work best for us.

[marcklingen]

+1, if you can/want to deploy to k8s, that's the best option for production. We just feared that if this is the only option, Langfuse gets difficult to deploy for teams which do not have k8s experience. I like the effort on https://github.com/langfuse/langfuse-k8s, probably we can make something work there together to align on one recommended helm chart for v3

[Manouchehri]

As a team with k8s experience.. we try to avoid it at all costs. 😬 We like Langfuse enough that we'd probably cave and do it if needed, but not our first choice.

[marcklingen]

Agree that we’d want to make it work on GCP/CloudRun and comparable solutions as it is just much easier. There’s no way really around it as without an additional container, it’s seems close to impossible to run async workloads

[MarkEdmondson1234]

There are a few options for async tasks, like Batch, cloud run jobs, workflows? GCP specific though. k8s is most generic but yeah, I’d prefer that only as a last option

[chai3]

I want Radis and OLAP Clickhouse to be keep opt-in.

I am deploying LangFuse on AWS with a simple architecture.
Just AWS AppRunner and RDS.
AWS AppRunner only supports a single Docker image and cannot use docker compose and sidecar.
ECS, EC2, Radis, and ALB are expensive, but AppRunner only needs to pay the CPU cost when there is a request by user, and if there is no request, it is very cheap at 0.007 USD / GB-hour(If 2GB memory, 5 USD/month)

https://github.com/AI4Organization/langfuse-ecr-ecs-deployment-cdk/blob/f21d237324f9c5727227915a57523a7ead779720/lib/langfuse-ecr-apprunner-deployment-cdk-stack.ts

[qymab]

Any specific reason why not providing arm based images ?

[marcklingen]

See self-hosting docs

Are there prebuilt ARM images available? No, currently we do not publish official ARM images. However, you can build your own ARM images using the Dockerfile in the Langfuse repository.

No specific reason, open for contributions on this!

[newcworld]

when clickhouse, I think it's great

[aiakubovich]

I'm not a big fan of docker-compose and we unable to use docker-compose files for deployments. Our requirements dictate that we first build an image using an ARM template, scan it with an infosec tool like Aqua in the CI/CD pipeline, and then deploy it. It would be great if there is option to deploy each container as stand along app.