* zonos3 init * add audio gen pipeline * fix string_format * add default voice as gguf metadata * better ref * support voice clone * add delivery params * optimize speed * nits * nits 2 * add text normalization * nits
218 lines
11 KiB
Markdown
218 lines
11 KiB
Markdown
# Text-to-speech (TTS)
|
|
|
|
llama.cpp supports audio generation via `libmtmd`. The following tools can be used:
|
|
- [llama-tts](../../tools/tts/README.md), see its README for common usage and params
|
|
- [llama-server](../../tools/server/README.md) via the `/tts` endpoint and the OpenAI-compatible `/v1/audio/speech` API
|
|
|
|
Supported models:
|
|
- [Qwen3-TTS](#qwen3-tts)
|
|
- [Pocket TTS](#pocket-tts)
|
|
- [Soprano](#soprano)
|
|
- [LFM3-Audio](#lfm3-audio)
|
|
- [Chatterbox](#chatterbox)
|
|
- [Drama 3](#drama-3)
|
|
- [ZONOS3](#zonos3)
|
|
|
|
The sections below list the params supported by each model, with `llama-tts` examples. In `llama-server`, the same options are set with `--tts-lang` and `--tts-voice`, or per request (see the [`/tts` endpoint](../../tools/server/README.md#post-tts-generate-speech-audio-from-text)).
|
|
|
|
## Qwen3-TTS
|
|
|
|
Available params:
|
|
- `--tts-lang` can be `zh`, `en`, `de`, `it`, `pt`, `es`, `ja`, `ko`, `fr`, `ru` (default: `en`)
|
|
- `--tts-speaker-file` should point to a speaker reference audio file (wav, mp3)
|
|
|
|
Example usage:
|
|
|
|
```sh
|
|
llama-tts -hf ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF \
|
|
-p "Hello world" \
|
|
--tts-lang english \
|
|
--tts-speaker-file speaker.mp3 \
|
|
--output out.wav
|
|
```
|
|
|
|
## Pocket TTS
|
|
|
|
Available params:
|
|
- `--tts-speaker-file` should point to a speaker reference audio file (wav, mp3). It is required, the model produces almost no audio without it
|
|
- `--tts-voice` selects one of the predefined voices shipped with the weights (e.g. `giovanni`), when the GGUF holds them. The converter reads them from `embeddings/*.safetensors` next to the weights, of both repositories, and logs how closely each one is reproduced (some packs ship voice states computed with other weights, the converter warns above 5% and skips a voice only when it does not belong to the model at all)
|
|
- Note: `lang` is not used, the language is a property of the weights
|
|
|
|
Example usage:
|
|
|
|
```sh
|
|
llama-tts -m pocket-tts.gguf \
|
|
-mm mmproj-pocket-tts.gguf \
|
|
-p "Hello world" \
|
|
--tts-speaker-file speaker.mp3 \
|
|
--output out.wav
|
|
```
|
|
|
|
The weights [without voice cloning](https://huggingface.co/kyutai/pocket-tts-without-voice-cloning) ship a set of predefined voices (`embeddings/*.safetensors`) next to each language; the converter stores them in the mmproj, as the voice prompt embeddings, and leaves out the zeroed voice encoder. With such a model `--tts-speaker-file` is rejected and a voice is selected with `--tts-voice` (the first one when it is not given):
|
|
|
|
```sh
|
|
llama-tts -m pocket-tts.gguf \
|
|
-mm mmproj-pocket-tts.gguf \
|
|
-p "Ciao a tutti" \
|
|
--tts-voice giovanni \
|
|
--output out.wav
|
|
```
|
|
|
|
**Note for GGUF conversion:**
|
|
|
|
The [upstream repository](https://huggingface.co/kyutai/pocket-tts) holds one complete model per language under `languages/`, next to a set of shared files at the root. Convert one of the `languages/<name>` directories, **not** the root directory:
|
|
|
|
```sh
|
|
python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --outfile pocket-tts.gguf
|
|
python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --mmproj --outfile mmproj-pocket-tts.gguf
|
|
```
|
|
|
|
## Soprano
|
|
|
|
Available params:
|
|
- `--tts-speaker-file` is not supported; the model uses a fixed voice
|
|
- Note: `lang` is not used. Supply normalized English text
|
|
|
|
Example usage:
|
|
|
|
```sh
|
|
llama-tts -m soprano.gguf \
|
|
-mm mmproj-soprano.gguf \
|
|
-p "Hello world" \
|
|
--temp 0 \
|
|
--output out.wav
|
|
```
|
|
|
|
**Note for GGUF conversion:**
|
|
|
|
Both [Soprano-80M](https://huggingface.co/ekwek/Soprano-80M) and [Soprano-1.1-80M](https://huggingface.co/ekwek/Soprano-1.1-80M) are supported. Convert both files from the same model directory, including `decoder.pth`:
|
|
|
|
```sh
|
|
python convert_hf_to_gguf.py path/to/Soprano-1.1-80M --outtype f32 --outfile soprano.gguf
|
|
python convert_hf_to_gguf.py path/to/Soprano-1.1-80M --mmproj --outtype f16 --outfile mmproj-soprano.gguf
|
|
```
|
|
|
|
## LFM3-Audio
|
|
|
|
Available params:
|
|
- `--tts-lang` can be `ar`, `da`, `de`, `el`, `en`, `en-gb`, `es`, `fi`, `fr`, `he`, `it`, `ja`, `ko`, `nl`, `pl`, `pt`, `sv`, `tr`, `zh` (default: `en`), other values are used as the language name
|
|
- `--tts-voice` can be `female` (default) or `male`
|
|
- Note: `--tts-speaker-file` is not used, the model has no voice cloning
|
|
- Note: `--top-k` and `--top-p` are not used, audio frames are sampled with the settings of the model card (LFM3-Audio: temperature 1.0, top-k 128, repetition penalty 1.1 over 4096 frames; LFM3-Omni: temperature 0.4, repetition penalty 1.2 over 64 frames), the `audio_*` keys of `generation_config.json` override them at conversion
|
|
- Each frame is 80 ms, generation stops at the end-of-audio frame
|
|
|
|
Example usage:
|
|
|
|
```sh
|
|
llama-tts -m lfm3-audio.gguf \
|
|
-mm mmproj-lfm3-audio.gguf \
|
|
-p "Hello world" \
|
|
--tts-voice male \
|
|
--output out.wav
|
|
```
|
|
|
|
The mmproj holds the audio encoder (for ASR), the depthformer and the codec decoder, the latter two are exported when the checkpoint has an `audio_tokenizer` directory:
|
|
|
|
```sh
|
|
python convert_hf_to_gguf.py path/to/lfm3-audio --outfile lfm3-audio.gguf
|
|
python convert_hf_to_gguf.py path/to/lfm3-audio --mmproj --outfile mmproj-lfm3-audio.gguf
|
|
```
|
|
|
|
## Chatterbox
|
|
|
|
Two variants are supported: [Chatterbox Turbo](https://huggingface.co/ResembleAI/chatterbox-turbo) (English) and [Chatterbox Multilingual V3](https://huggingface.co/ResembleAI/chatterbox) (23 languages).
|
|
|
|
Available params:
|
|
- `--tts-speaker-file` should point to a speaker reference audio file (wav, mp3). Optional, the predefined voice is used without it
|
|
- `--tts-voice` can be `default`, the voice shipped with the model (`conds.pt`)
|
|
- `--tts-lang` sets the language of the multilingual variant, for example `fr` (default: none). It is not used by Turbo
|
|
|
|
Example usage:
|
|
|
|
```sh
|
|
llama-tts -m chatterbox-mtl.gguf \
|
|
-mm mmproj-chatterbox-mtl.gguf \
|
|
-p "Bonjour tout le monde" \
|
|
--tts-lang fr \
|
|
--tts-speaker-file speaker.mp3 \
|
|
--output out.wav
|
|
```
|
|
|
|
Note: the multilingual variant runs classifier-free guidance with a second sequence. `llama-tts` sets this up by itself, but `llama-server` does not support it yet.
|
|
|
|
**Note for GGUF conversion:**
|
|
|
|
The upstream repositories ship raw checkpoints without `config.json`. Download the files below into a directory, then convert it as usual (the variant is detected from the talker file):
|
|
- Turbo: `t3_turbo_v1.safetensors`, `s3gen_meanflow.safetensors`, `ve.safetensors`, `conds.pt` and the GPT-2 tokenizer files (`vocab.json`, `merges.txt`, `added_tokens.json`, `special_tokens_map.json`, `tokenizer_config.json`)
|
|
- Multilingual V3: `t3_mtl23ls_v3.safetensors`, `s3gen_v3.safetensors`, `ve.safetensors`, `conds.pt`, `mtl_tokenizer.json`
|
|
|
|
```sh
|
|
python convert_hf_to_gguf.py path/to/chatterbox --outfile chatterbox.gguf
|
|
python convert_hf_to_gguf.py path/to/chatterbox --mmproj --outfile mmproj-chatterbox.gguf
|
|
```
|
|
|
|
## Drama 3
|
|
|
|
[Fish Audio Drama 3](https://huggingface.co/fishaudio/drama-3): a Qwen3.5-MoE backbone (35B, 3B active) samples each frame's main token, a codebook tower samples the 9 other codes, and the Fish DAC decodes them to 44.1 kHz audio.
|
|
|
|
Available params:
|
|
- Speaker tags `<|speaker:0|>` to `<|speaker:15|>` in the text give each character its own voice, a text without a tag starts with `<|speaker:0|>`
|
|
- The language is the one of the text, `--tts-lang` is not used
|
|
- Note: `--tts-speaker-file` is not supported yet (voice cloning)
|
|
- Note: `--top-k` and `--top-p` are not used, frames are sampled with the settings of the reference (main token: temperature 1.0, top-p 0.9, repetition-aware redraw over the last 16 frames; other codes: temperature 1.0, top-k 30, top-p 0.9)
|
|
- Each frame is 2048 samples (about 46 ms), generation stops at the end-of-speech token
|
|
- The codebook tower keeps the whole sequence in its KV cache, the prompt and the generated frames are limited to 4096 positions
|
|
|
|
Example usage:
|
|
|
|
```sh
|
|
llama-tts -m drama3.gguf \
|
|
-mm mmproj-drama3.gguf \
|
|
-p "<|speaker:0|> Did you remember to buy the bread? <|speaker:1|> Of course I did, it is on the kitchen table." \
|
|
-n 1000 \
|
|
--output out.wav
|
|
```
|
|
|
|
**Note for GGUF conversion:**
|
|
|
|
The routed experts of the checkpoint are FP8, they are dequantized at conversion. The mmproj holds the codebook tower and the decoder half of `codec.pth`, it is best kept in BF16:
|
|
|
|
```sh
|
|
python convert_hf_to_gguf.py path/to/drama-3 --outtype q8_0 --outfile drama3.gguf
|
|
python convert_hf_to_gguf.py path/to/drama-3 --mmproj --outtype bf16 --outfile mmproj-drama3.gguf
|
|
```
|
|
|
|
## ZONOS3
|
|
|
|
[Zyphra ZONOS3 Mini](https://huggingface.co/Zyphra/ZONOS3-Mini): a hybrid Gated DeltaNet / attention MoE backbone (800M, 100M active) predicts the 10 codebooks of each frame with an inter-codebook delay, and the ZAC codec decodes them to 44.1 kHz audio.
|
|
|
|
Available params:
|
|
- `--tts-voice` picks one of the predefined voices of the release (`Savannah_en-American_F_1` by default); `llama-server` lists them in `GET /props`. A voice is the prompt prefix of its reference codes, the default speaker mode of the reference
|
|
- The language is the one of the text; `--tts-lang` (`lang` in `llama-server`) only selects the text normalization
|
|
- Numbers are spelled out by the `tts` chat template of the GGUF: cardinals, decimals, money, percents, clock times, common ordinal forms (`1st`, `1er`, `1º`), English and German years. A 4-digit number is read as a year only after a word like `in`, `since`, a month name, or `im Jahr`, `seit` in German (`in 1990` -> "in nineteen ninety", `1234 people` -> "one thousand two hundred thirty-four people"). Languages: `en`, `fr`, `de`, `es`, `it`, `pt`, `ja` and `zh` (or `cmn`), the region is ignored (`fr_fr`, `pt_br`); English when the language is empty, other languages are left as written. Set the language to the one of the text: the separators differ, e.g. `1.234` is a thousand in German
|
|
- Note: the reference uses NeMo text normalization, which also covers units, dates, abbreviations and Korean; supply normalized text for them
|
|
- `--tts-speaker-file` clones the voice of a clip (the first 60 s are used) with the speaker encoder of the release, the "embedding" speaker mode of the reference
|
|
- `llama-server` also takes `speed` (relative to 15 UTF-8 bytes per second) and `loudness` (LUFS), see the `/tts` endpoint; they pick the speaking-rate and loudness condition buckets of the checkpoint, the defaults leave them unknown
|
|
- Note: `loudness` has a strong effect with a cloned voice; a predefined voice mostly keeps the loudness of its reference clip
|
|
- Note: `--top-k` and `--top-p` are not used, frames are sampled with the settings of the reference: temperature 0.75 (`--temp` changes it), top-k 80, top-p 0.69, repetition penalty 2.4 over the last 96 frames of codebooks 0 to 5
|
|
- Each frame is 2940 samples (about 67 ms), generation stops at the end-of-audio code
|
|
|
|
Example usage:
|
|
|
|
```sh
|
|
llama-tts -m zonos3-mini.gguf \
|
|
-mm mmproj-zonos3-mini.gguf \
|
|
-p "The old lighthouse keeper climbed the spiral stairs every evening." \
|
|
--tts-voice Imogen_en-British_F_1 \
|
|
--output out.wav
|
|
```
|
|
|
|
**Note for GGUF conversion:**
|
|
|
|
Convert the root of the release repository, not the `lm/` subdirectory. The mmproj holds the codebook heads, the predefined voices, the speaker encoder and the `keep2_plain` codec decoder, it is best kept in F32:
|
|
|
|
```sh
|
|
python convert_hf_to_gguf.py path/to/ZONOS3-Mini --outtype q8_0 --outfile zonos3-mini.gguf
|
|
python convert_hf_to_gguf.py path/to/ZONOS3-Mini --mmproj --outtype f32 --outfile mmproj-zonos3-mini.gguf
|
|
```
|