KittenTTS module.

[jsett]

KittenTTS Module.

This is a speech dispatch model for running Kitten TTS. Kitten TTS is a deep learning model which provides high quality natural sounding TTS generation using models ranging from 15M to 80M parameters. Due to its small size it is able to run in real-time on CPU. The goal of this project is to integrate Kitten TTS with speech dispatch while maintaining its near real-time speech generation, with special attention being place on reading of long text's such as ebooks. To achieve this the original python code was rewrote into c and tightly integrated into a speech dispatch model.

There is a lot here so let me give a quick summary of what all is going on here.

kitten_server.c:

This file is for handling the protocol and follows some what closely to the example modules for async servers with speech dispatch handling the audio. By design I made sure very little work is done in any function in this thread. Any long running code should be handed to an async queue and ran on one of the threads in the kitten_worker.c file.

kitten_worker.c:

This is where the handling of long running tasks is done. We create two different threads here one to one to handle passing audio back to the server(since the module_tts_output_server function can block and we want the generation to continue while we are outputting audio to the server). The other thread is dedicated to the generation of audio by the model. We synchronize all this using two GasyncQueue, one for handling incoming speak requests, and the other to handle the outputted audio from our model. we also keep track of how much audio we have generated and played so that long speak commands don't run the cpu unnecessarily hard (For example I have seen that Okular will in some cases send an entire book's text in a single speak command). There is also code here for parsing ssml text using libxml.

kitten_model.c:

The handles everything we need to do to generated output from onnx using our model. The most important functions here are init_voice_style to handle loading voice styles. reload_models_and_voices: to handle changing the voice style. And kitten_speak for generating audio.

kitten_downloader.c:

This handles downloading the model+voice styles if its not already on our computer. If it download it does verify the file against a sha256, but that verification is not strongly enforce and is more of a warning.

[sthibaul]

Thanks for the nice work!

In the future, we will probably want to integrate various onnx-based voices, so code will be useful to share between modules, but your writing seems already quite well structured so that it will be convenient to do.

There are just a few changes that need to happen before we can integrate this.

[unlimitedsola]

I believe this is best to be handled by package managers instead?

As models update, we don't want to frequently update and recompile this as downstream packages might lag behind for years.

[unlimitedsola]

Also all the download links are outside of this project's control. Which means that we won't be able to fix a broken download until a new release is packaged by distributions and shipped to the end user (which as commented, might be months/years after).

[jsett]

I believe this is best to be handled by package managers

It already checks if the models/voices are in the path /usr/share/speech-dispatcher/models/kitten and uses that path first. If the package manger placed them there, then they will get used.

we won't be able to fix a broken download until a new release

I have added the ability to set the download configuration through the models dot conf file.

Here is an example dot conf on how to do that https://gist.github.com/jsett/6146eb2803f8780a1830ed816bdc94ef

[unlimitedsola]

I believe this is best to be handled by package managers

It already checks if the models/voices are in the path /usr/share/speech-dispatcher/models/kitten and uses that path first. If the package manger placed them there, then they will get used.

I'm afraid distribution would not organize their packages this way.

Like piper, kitten models are general purpose in the sense of the Text-to-Speech task. If we take example from piper, the AUR packages place the models into /usr/share/piper-voices/, because their use are not limited to speech-dispatcher, and the distributions would not want to package multiple copies of large model files just for different applications to use.

It is also likely that different distributions wouldn't come to agreement on where these models should be placed. So the distribution packagers would adjust the conf default to accommodate their packaging schemes.

I believe this is best to be handled by package managers instead?

Nevertheless, I personally wouldn't expect program like speech-dispatcher (as a daemon process) to make network requests (and I strongly believe that we should learn from the log4shell disaster where a logging library contains the code path to make network requests) . So adding this to me feels like a scope creep that I would like to push against, as indicated by the first time linking with libcurl.

[jsett]

/usr/share/speech-dispatcher was recommended in the comments above, but I really don't have any real issue changing it if that what is decided.

code path to make network requests

I'm not opposed to removing the downloader. But there is a trade off here. If you remove it you increase the complexity for users to get things working, if there distro did not include the models with the install. I would be interested in hearing what other people think.

[unlimitedsola]

If you remove it you increase the complexity for users to get things working, if there distro did not include the models with the install.

We can document what distro package the user should install to get the models in the default conf template (the distro maintainers can add the documentation themselves too). Or we could also link to one-off scripts to help users to download the models into their home directories.

Personally, what matters to me is that the downloader should be a separate program/script from the daemon process code, so it is only loaded when the user explicitly request to perform the one-off download operation.

[jsett]

Turning the downloader into a one-off script might not be a bad idea. I would like to hear from @sthibaul since i don't know what the most convenient way of doing this for speachd would be. Like would I just need to add a extra build target or something else. Also I could pretty easy rewrite the downloader as a python script or something but again, I don't know if that would be better or worse for packaging for the distros. Frankly I'm not super experienced on the intricacies of packaging.