Is there a way to make an additional float[] output showing individual phoneme times/lengths from the native DLL?

[JasonBlain]

I'm over in Unity and trying to lipsync visemes to the audio output, and I can see where the function to generate the audio from the phonemized text is appending the bits of phonemes together, but the final output length of the audio clip is not the whole story in terms of phoneme/viseme pacing.

Is there a way to add an output array of floats to go with the Phoneme sequence of a TTS output? No need to edit the tensor, just need an additional output with the array after it has been rendered. I imagine it would have to be done at the .cpp level and then rebuilt into the DLL?

[synesthesiam]

Yes, this is fully possible but will require either (1) using the original PyTorch models or (2) re-exporting the voice models and changing the C++ code.

An intermediary product of the model is the length of every phoneme. This can be returned with the audio, but will require the changes above.

[JasonBlain]

Hey thanks for the reply.

How could I re-export with the additional outputs? I've got some python experience, kind of a generalist. I've got most of a modified overload header written in the cpp files of a fork of the piper.unity repo I made, so I have that part.. worked out, I think.

[synesthesiam]

The w_ceil variable has the phoneme lengths:

piper/src/python/piper_train/vits/models.py

Line 703 in e5cb84c

w_ceil = torch.ceil(w)

Multiplying this tensor by 256 will get you the number of audio samples per phoneme.

That w_ceil tensor needs to be returned from the infer function and then also returned with the audio here:

piper/src/python/piper_train/export_onnx.py

Line 60 in e5cb84c

audio = model_g.infer(

On the C++ side, you then need to pick apart the multiple output tensors (one audio, one phoneme samples):

piper/src/cpp/piper.cpp

Line 386 in e5cb84c

auto outputTensors = session.onnx.Run(

[JasonBlain]

Lovely, will go through that as best I can and see what I can come up with. Good cites. Thank you.

[JasonBlain]

EDIT: Found checkpoints repo on HuggingFace and will try re-export.

[JasonBlain]

All of those citations were great, I did have to do a couple more things for my system config to be able to push the new ONNX export out from python.

I'm on windows so I had to do some edits to export_onnx.py, do some POSIX path Windows-explicit things. Monotonic align was acting up and had a circular reference so I had to change the setup and init on it just a bit, mostly path related issues.

[JarbasAl]

Is it possible to expose this in Piper python api? It would help a lot in OVOS to generate mouth movements for the Mark1 device

together with the generated audio file in OVOS we need a list of phonemes + duration of each phoneme, for the most part we just use the original mimic1 TTS to generate these, but this is far from perfect as they often dont match the actual audio, if we could get piper to output this info natively it would be awesome!

[synesthesiam]

Yes, I will expose this to both the C and Python APIs in the future.

[goldyfruit]

Would help a lot indeed as for now we have to install Mimic TTS which is around 400Mb 👍

[goldyfruit]

Here is an example when using Piper with ryan-low voice on a Mark 1 device.

https://youtube.com/shorts/9m3501afUEw