(New Construct): LangServ Support

[tech4life87]

Describe the feature

This construct deploys Langserv a FastAPI based framework that helps developers deploy LangChain's Runnables and Chains as REST API in production.

Use Case

Langserv simplifies the deployment of Large language Models from a prototype to production with industry best practices such as

1. API docs page with JSONSchema and Swagger
2. Efficient /invoke/, /batch/ and /stream/ endpoints with support for many concurrent requests on a single server
3. All built with battle-tested open-source Python libraries like FastAPI, Pydantic, uvloop and asyncio
4. FastAPI is a modern, fast (high-performance), web framework for building APIs with Python 3.8+ based on standard Python type hints.
5. Observability and Analytics
6. LLM Provider Agnostic

Proposed Solution

At a high level this construct will deploy LangServ and LangFuse on AWS using Services such as Fargate, ECS and RDS but will depend on existing constructs such as aws_ecs_patterns. Developers will be able to make custom REST api calls to FastAPI endpoints with support for streaming, batch and invoke. The construct will be LLM provider agnostic such as Bedrock, SageMaker or VLLM on EC2 Instances giving developers flexibility to experiment with different model providers.
The construct will also provide monitoring and observability capability using LangFuse giving developers the flexibility to trace and debug LLM applications, monitor latency and model evaluation capabilities.

No response

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

Tasks

Beta Give feedback

No tasks being tracked yet.

[laithalsaadoon]

Thank you for the suggestion @tech4life87 ! Do you mind describing a few things for us:

1. At a high level, how do you propose deploying and running LangServe on AWS? (i.e. what resources, services, etc.)
2. Is a LangServe construct related to or enhanced by existing constructs, and if so, how?
3. At a high level, do you propose that users should be able to make custom REST APIs with LangServe, and if so, what would be the "entrypoint" for providing their Chains and Runnables?

[tech4life87]

@laithalsaadoon I just updated the issue with these details.

[krokoko]

Hi @tech4life87 , thank your for this suggestion, looks very interesting !

1. Do you have an idea of the interface for this construct ?
2. Could you walk me through the developer workflow for this construct ?

[krokoko]

@tech4life87 any update on this one ?

[tech4life87]

@krokoko

1. Yes I have an idea of how this would work from my previous work on the same thing.

2. From a developer experience/flow perspective there are two possible paths and I am open to refining it.

   1. Developer imports the constructs which default to pulling the container from a public ecr repo.
   2. They do local development and local testing.
   3. during CDK deploy the container is packaged and deployed to ECS.

   options2:

   1. Developer create a local langserve application.
   2. The developer then does local development and testing.
   3. Using the local Dockefile the construct packages the code and deploys to ECS.

   I open to ideas of what other developers have done to improve dev experience when it comes to fargate based constructs.

[krokoko]

great @tech4life87 , thanks ! I see you mentioned you may be able to implement this feature, would you be able to create a draft of this construct in a branch ?

[tech4life87]

great @tech4life87 , thanks ! I see you mentioned you may be able to implement this feature, would you be able to create a draft of this construct in a branch ?

Yes I will start working on it

[krokoko]

Thank you @tech4life87 ! I will assign the ticket to you for now

[krokoko]

Hi @tech4life87 , any update on this ? Thank you !