POC: Explore the Vercel AI SDK

[nickytonline]

We're currently streaming responses for StarSearch that return markdown which we transform with React Markdown. This works fine, but we're limited to plain HTML tags.

We should explore the Vercel AI SDK to bring richer UI experiences for StarSearch.

This video is a great watch: https://www.youtube.com/watch?v=br2d_ha7alw

Also, @bdougie shared this AI Chatbot example, https://chat.vercel.ai/

Source code is available here, https://github.com/vercel/ai-chatbot.

[github-actions]

Thanks for the issue, our team will look into it as soon as possible! If you would like to work on this issue, please wait for us to decide if it's ready. The issue will be ready to work on once we remove the "needs triage" label.

To claim an issue that does not have the "needs triage" label, please leave a comment that says ".take". If you have any questions, please reach out to us on Discord or follow up on the issue itself.

For full info on how to contribute, please check out our contributors guide.

[nickytonline]

I believe we'll need to upgrade to Next.js 14, #2021, and the AI SDK requires the App router, so if we go ahead with this, we could start with StarSearch for using the App router.

[bdougie]

After a quick look, they use the tools/functions pattern to know when to render a component instead of text. It's pretty elegant and makes a ton of sense.

https://github.com/vercel/ai-chatbot/blob/095550d4dca22dc506cdbda815cab94cfe8fbe74/lib/chat/actions.tsx#L190-L380

We already have this pattern for the responses; the difference is mirroring this for components rendered. https://github.com/open-sauced/api/blob/beta/src/star-search/star-search-tools.service.ts

[nickytonline]

From @jpmcb, https://vercel.com/guides/openai-function-calling

[jpmcb]

Proposal - send metadata in the observable to inform the client what components to render

It seems the Vercel Next AI SDK assumes you just drop in an OpenAI token and it will call a GPT with some tools and prompts you right on the server side of a Next app to then render some components. Example:

https://github.com/vercel/ai-chatbot/blob/095550d4dca22dc506cdbda815cab94cfe8fbe74/lib/chat/actions.tsx#L151

This would work really well for an application that primiarily uses server side rendering for calling out to an AI service. But would make having more heavy lifting (like putting together a cosine similarity search for RAG or calculating data based on returned database results) much more difficult. This would also make having several AI agents that have different tasks managed by the API much more difficult to pull off, especially if we integrate more Azure services in the future.

I propose that we keep the current paradigm (where the API does all the heavy lifting, manages AI agents, has it's OpenAI tools, configured Azure services, etc.) but we include an additional field in the observable:

metadata: { json }

which would denote which OpenAI tools were called so the client can know which components to hit the API with to then render. This way, we're not grid locking us into using server side rendering for components and we keep the flexibility of using an observable sent to the client.

So, for example, say someone asked about the lottery factor on kubernetes/kuberentes. The metadata from a tool that OpenAI would use would look something like:

metadata: {
  "tool_called": "calculate_lottery_factor",
  "repo_name": "kubernetes/kubernetes"
}

(i'm still not sure if JSON is the best idea to send in an observable stream or if that'll be a mess for the client. Still, we could send something that the client understands.)

This way, we could render the component for the Lottery factor for Kubernetes and also have a little blurb generated from the LLM on what's going on with the data. I'd imagine the client would have something like:

switch tool_called:
  case (calculate_lottery_factor):
     // renders the lottery factor component and hits the API for repo used by the OpenAI tools
 
  case (gets_stars):
    // renders the star chart for the repo used by the OpenAI tool

  break;

Thoughts @nickytonline @bdougie @zeucapua ?

[bdougie]

I am all for the metadata approach. If we could do a POC using metadata to render one component alongside the markdown, perhaps the "who is username?" Question and rendering a dev card

@isabensusan had something similar design in the starsearch Figma exploration.

[nickytonline]

Proposal - send metadata in the observable to inform the client what components to render

It seems the Vercel Next AI SDK assumes you just drop in an OpenAI token and it will call a GPT with some tools and prompts you right on the server side of a Next app to then render some components. Example:

vercel/ai-chatbot@095550d/lib/chat/actions.tsx#L151

This would work really well for an application that primiarily uses server side rendering for calling out to an AI service. But would make having more heavy lifting (like putting together a cosine similarity search for RAG or calculating data based on returned database results) much more difficult. This would also make having several AI agents that have different tasks managed by the API much more difficult to pull off, especially if we integrate more Azure services in the future.

I propose that we keep the current paradigm (where the API does all the heavy lifting, manages AI agents, has it's OpenAI tools, configured Azure services, etc.) but we include an additional field in the observable:

metadata: { json }

which would denote which OpenAI tools were called so the client can know which components to hit the API with to then render. This way, we're not grid locking us into using server side rendering for components and we keep the flexibility of using an observable sent to the client.

So, for example, say someone asked about the lottery factor on kubernetes/kuberentes. The metadata from a tool that OpenAI would use would look something like:

metadata: {
  "tool_called": "calculate_lottery_factor",
  "repo_name": "kubernetes/kubernetes"
}

(i'm still not sure if JSON is the best idea to send in an observable stream or if that'll be a mess for the client. Still, we could send something that the client understands.)

This way, we could render the component for the Lottery factor for Kubernetes and also have a little blurb generated from the LLM on what's going on with the data. I'd imagine the client would have something like:

switch tool_called:
  case (calculate_lottery_factor):
     // renders the lottery factor component and hits the API for repo used by the OpenAI tools
 
  case (gets_stars):
    // renders the star chart for the repo used by the OpenAI tool

  break;

Thoughts @nickytonline @bdougie @zeucapua ?

I liked the idea of sending metadata. We can definitely POC this, but I already know if we get that JSON it's totally doable. I just wonder how it gets surfaced. I think the thing to test out is we can definitely render components with that meta data, but for parts of a response that are all text, do we return that as just text. Since we're streaming it, we'd need to stream valid JSON in chunks.

So maybe like this coming from the streamed response:

id: 1
data: { type="markdown", content: "brandon is a developer" }
id: 2
data: { type="markdown", content: "from the USA." }
id: 3
data: { type="markdown", content: "He works on Analog.js" }
id: 4
data: { "type": "calculate_lottery_factor", "data": "analogjs/analog" }
...

We could group the markdown together to render it so we don't end up with weird HTML. Data would be considered the props for the component. It could be simple data or JSON.

Or would you consider the meta data components we just show at the end of the prompt response @jpmcb?

Just spitballing. I'm still going to take a peek at the Vercel AI SDK.

[bdougie]

I know this is in progress, but looking at this question on beta now and see that this list would be a perfect candidate for returning a lotto chart component.

[nickytonline]

I know this is in progress, but looking at this question on beta now and see that this list would be a perfect candidate for returning a lotto chart component.

From the application side of things, this work is complete and we have our implementation in #3394. If we want to return a lotto factor chart in the scenario proposed above, we'd need to look at it from the API side of things.