OpenAI tts with word boundary event result

[ling-k]

Describe the solution you'd like
The openAI TTS voices, such as en-US-AlloyMultilingualNeuralHD, en-US-EchoMultilingualNeuralHD, do not return word boundary event results.
Additional context

def word_boundary_handler(evt):
    print(f"My Word boundary event received: {evt.text}, audio offset in ms: {evt.audio_offset / 10000}ms")
    word_boundaries[str(evt.audio_offset / 10000)] = evt.text 
    
   For Azure voices, such as "en-US-EmmaNeural", "zh-CN-XiaoxiaoNeural", it works fine. But for openAI voices, it does not return anything.  

[BrianMouncer]

The models used for the OpenAI tts voices do not provide word level timing information, so it is currently not possible to get those events when using those voices. There are also a few limitations related to what SSML tags are supported by the OpenAI voices. You can find more information about those limitations here https://learn.microsoft.com/en-us/azure/ai-services/speech-service/openai-voices#ssml-elements-supported-by-openai-text-to-speech-voices-in-azure-ai-speech

I will work with our docs team to also document the other limitations, like word level timing data.

[yulin-li]

The wordboundary is not supported in AOAI voices yet.

We should update the doc, @Kerry-LinZhang could you help track the doc refresh?

[BrianMouncer]

@ling-k. For future planning, what data center region are you using, that you want word level timing support from OpenAI voices, and or what other regions are most important to you.

https://learn.microsoft.com/en-us/azure/ai-services/speech-service/regions

[ling-k]

@ling-k. For future planning, what data center region are you using, that you want word level timing support from OpenAI voices, and or what other regions are most important to you.

https://learn.microsoft.com/en-us/azure/ai-services/speech-service/regions

Thanks for replying. I mainly work on West US or West US 2 regions.

[pankopon]

@yulin-li @Kerry-LinZhang So I guess this needs a documentation update at least, possibly also creation of a future work item? Please update status and close when done.

[Kerry-LinZhang]

Hi @ling-k we will update our Doc related in this week and I will keep you updated once it has been released.

[Kerry-LinZhang]

Hi @ling-k We have updated our Doc, please find it at: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/language-support?tabs=tts#multilingual-voices

[github-actions]

This item has been open without activity for 19 days. Provide a comment on status and remove "update needed" label.

[pankopon]

Closed as resolved.