Create new AI product in website / docs

[jamesefhawkins]

to use fancy words, i think we already have the primitives needed to be a pretty solid AI product based on https://github.com/PostHog/product-internal/issues/598

the reason i think we should push here - AI is changing how products are built, whilst the market is small today, this will be a very typical use case in a few years' time so we can dominate the space given our strong adoption with AI companies. this means we don't leave ourselves vulnerable to an AI specific product analytics tool appearing

i think the way this would work is pretty much:

Product page

the page could contain something like:

• screenshot - show an experiment that is optimizing for one of the AI concepts below (see tutorials)
• features
  • Prompt management
  • Latency tracking
  • Cost tracking
  • Model parameter performance
  • Model evaluation
• pricing
  • same as experiments ("a/b testing" at the moment)
• no comparison - there isn't a major competitor

Tutorials

I think we'd do a tutorial for each of:

• testing prompts
• tracking costs
• testing model parameters
• testing which model to use

Product

I don't think we need to build anything new to launch with the above.

However, I think a simple product that steers users through the above could work well.

We could call it AI Engineering or something similar, the flow would be very similar to the experiments flow, but instead the user would choose an AI experiment type (prompts / model parameters / model itself / costs)

I'm not quite sure yet how prompts / model parameters / model are all stored/connected to - maybe this logic needs to sit in their code, but i could see it sitting in PostHog. I'll do customer calls here

[raquelmsmith]

How would this be charged with flags? How does it work?

[Lior539]

A few thoughts:

1. How do you imagine our existing LLM integrations fitting into this? (e.g. langfuse, helicone. There'll be more in future too)

   • Given we're not a specialist tool for building AI products, there'll inevitably be gaps and people will be using these other tools too.
   • I do think theres an angle here on "we support all the existing AI tools out there", making us the go to place for AI products

2. "Prompt management"

   • It's quite a stretch to say we support this in product IMO. I dont think we offer anything for this? How do you imagine this working?

3. "Latency tracking, Cost tracking"

   • Can be done with event capturing 👍🏼

4. "Model parameter performance"

   • Im not familiar with this term. What is this?

5. "pricing"

   • What exactly are we charging for here? Is this not built on top of our existing products and using their pricing?

[corywatilo]

@andyvan-ph and i were chatting about the AI product page and we’re thinking creating a top-level product page for it is wrong (at this point) for a few reasons:

• what we currently have is more of a subcategory of product analytics (or a specific use case within it) than an actual product
• infers we're more in the l@ngfuse/helic0ne world than we are
• andy: from an outsider perspective, it look like we’re going “we do AI now” with jazz hands

at this point, it feels like it could be a nice landing page, and we can link it from the product analytics page.

main points:

• correlate your AI/LLM development with product usage
• integration with other products (qualitative feedback, replays, flags?)
• good opportunity to mention the langfuse integration (if we're being generous), since between l@ngfuse and posthog, you have the full suite you need

could also be a good opportunity to link to an LLM analytics forum topic for feature requests to help us prioritize what to build here.

[corywatilo]

James idea: Add "AI engineering" to homepage slider, tease it there, link to docs (on how to use Product Analytics to set up AI engineering insights

[joethreepwood]

☝️ Works for me! Will close the other PR.

We have case studies in progress regardless, so we can launch this in earnest when we're ready