Enhancement: GPT-4o Feature support

[th3raid0r]

Per the announcement here - https://openai.com/index/spring-update/

The new GPT-4o is already available via the API and can likely already be used if defined in the .env file since it's just a different model param.

The reason for this Enhancement then is to support the following functionality:

1. Ability to interrupt a TTS output with speech.
2. Video streaming from the device to the model (webcam/smartphone cam first, then screensharing down the road)

Both of those features will be enabled as an API at a future date. So obviously this cannot be implemented until then.

[danny-avila]

Ideally, we will catch up to livestream audio/video chat.

Technically, this is/was already possible in some ways, with or without gpt-4o, albeit a very advanced problem to tackle. I personally would love to implement the screen sharing capability to this end. This "omni"-modal experience is a ChatGPT product, and is more of a software engineering problem. However, I would not expect it sooner than some other items on the roadmap.

STT/TTS will come first and very soon (hopefully next week).

I'm motivated to release STT/TTS asap, but not exclusively because of the OpenAI Spring Update. I recently learned that librechat has some real handicapped users that could benefit from this, along with other accessibility features.

This is still very much an open-source project without capital funding, so keep this in mind when it comes to development progress.

[danny-avila]

Both of those features will be enabled as an API at a future date. So obviously this cannot be implemented until then.

If OpenAI said this, I missed it. Unless they actually said this, I highly doubt it is as it's currently a "moat" for their ChatGPT product(s).

[danny-avila]

Just read this at the bottom of the blog:

We plan to launch support for GPT-4o's new audio and video capabilities to a small group of trusted partners in the API in the coming weeks
source: https://openai.com/index/hello-gpt-4o/

I can see it similar to how whisper/text-streaming works already, but as for seamless integration in a chat UI, it would still not be so simple to design the experience they have outright.

[th3raid0r]

@danny-avila - I'm here to help - despite mostly creating enhancement requests at the moment! This is more a stub that would need to be revisited once the API is released.

You can bet that once the API specs are released I'll be right back here to comment on the lift of the changes involved and potentially getting to work on it.

Like, I'm glad I asked about an invite system, it might have saved us some duplicate work.

Apologies if it all came as a barrage!

[danny-avila]

No worries, did not take anything by it and I'm stoked you're here to help!

Just wanted to lay out the current development speed. If they offer tools to the end of simulating the same experience they made, I'm all here for it and it will definitely accelerate things. I would still be interested in an "homebrew" solution that would work with any LLM, at least for audio.

[grexzen]

You are amazing Danny! And yes, thank you, lots of users would benefit with hands free interactions.