Different TTS models

[teledoc2]

Great work, thank you for your contribution!

For cost purposes I tried using Neets.ai and Deepgram's TTS but cant get them to work. Terminal seems to log GPT-->TTS but not TTS --> Twilio despite following the docs and matching encoding. Any advice would be greatly appreciated.

Thanks again!

Abe

[cweems]

@teledoc2 do you know what audio format you are outputting to the stream? Twilio only accepts ulaw_8000 since this is what is supported by the publicly switched telephone network. You can see where we set this with ElevenLabs: https://github.com/twilio-labs/call-gpt/blob/main/services/tts-service.js#L20

[teledoc2]

thank you so much for the prompt response, yes both neets.ai and deepgram tts can produce output format ulaw_8000 (mulaw_8000 in neets.ai) and i added it as a param in the request, per docs, but still not able to pass the audio to twilio. When using base64 there is just a loud static and without base64 just silent and in both cases the terminal is showing no activity. Any advice would be appreciated. I just looked at openAI TTS and it seems to be a feasible solution so will try it next. Any other ideas would be greatly appreciated. Thanks so much! Abe

[daniel-makarov]

@teledoc2 Have you managed to successfully implement it with openAI TTS? I have tried but only hear static on my end. If you did, could you please share your solution?

[c0dr]

Neets was fairly easy to implement as they support output in mulaw, worked without any issues. To get openAI TTS to work you would probably need to use something like wavefile to convert it from wav to mulaw.

      const response = await fetch(
        'https://api.neets.ai/v1/tts',
        {
          method: 'POST',
          headers: {
            'Content-Type': 'application/json',
            'X-API-Key': 'xxxxxx',
            accept: 'audio/wav',
          },
          body: JSON.stringify({
            text: partialResponse,
            voice_id: 'vits-eng-1',
            fmt: 'mulaw',
            params: {
              model: 'vits'
            }
          })
        }
      )

[SageWorks4925]

@c0dr Thanks! Your code works well for Neets ai API.

Do you know how we can add fillers like 'Umm', 'Sure' or 'Let me think'? I'm planning to add fillers to reduce the perception of latency between the 'human voice input' and the 'tts of gpt response'.