Files
llama.cpp/docs/multimodal/tts.md
T
Xuan-Son Nguyen 7ae3c6b231 mtmd: add zonos3-mini TTS model (#32020)
* zonos3 init

* add audio gen pipeline

* fix string_format

* add default voice as gguf metadata

* better ref

* support voice clone

* add delivery params

* optimize speed

* nits

* nits 2

* add text normalization

* nits
2026-11-15 19:44:44 +01:00

11 KiB

Text-to-speech (TTS)

llama.cpp supports audio generation via libmtmd. The following tools can be used:

  • llama-tts, see its README for common usage and params
  • llama-server via the /tts endpoint and the OpenAI-compatible /v1/audio/speech API

Supported models:

The sections below list the params supported by each model, with llama-tts examples. In llama-server, the same options are set with --tts-lang and --tts-voice, or per request (see the /tts endpoint).

Qwen3-TTS

Available params:

  • --tts-lang can be zh, en, de, it, pt, es, ja, ko, fr, ru (default: en)
  • --tts-speaker-file should point to a speaker reference audio file (wav, mp3)

Example usage:

llama-tts -hf ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF \
    -p "Hello world" \
    --tts-lang english \
    --tts-speaker-file speaker.mp3 \
    --output out.wav

Pocket TTS

Available params:

  • --tts-speaker-file should point to a speaker reference audio file (wav, mp3). It is required, the model produces almost no audio without it
  • --tts-voice selects one of the predefined voices shipped with the weights (e.g. giovanni), when the GGUF holds them. The converter reads them from embeddings/*.safetensors next to the weights, of both repositories, and logs how closely each one is reproduced (some packs ship voice states computed with other weights, the converter warns above 5% and skips a voice only when it does not belong to the model at all)
  • Note: lang is not used, the language is a property of the weights

Example usage:

llama-tts -m pocket-tts.gguf \
    -mm mmproj-pocket-tts.gguf \
    -p "Hello world" \
    --tts-speaker-file speaker.mp3 \
    --output out.wav

The weights without voice cloning ship a set of predefined voices (embeddings/*.safetensors) next to each language; the converter stores them in the mmproj, as the voice prompt embeddings, and leaves out the zeroed voice encoder. With such a model --tts-speaker-file is rejected and a voice is selected with --tts-voice (the first one when it is not given):

llama-tts -m pocket-tts.gguf \
    -mm mmproj-pocket-tts.gguf \
    -p "Ciao a tutti" \
    --tts-voice giovanni \
    --output out.wav

Note for GGUF conversion:

The upstream repository holds one complete model per language under languages/, next to a set of shared files at the root. Convert one of the languages/<name> directories, not the root directory:

python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --outfile pocket-tts.gguf
python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --mmproj --outfile mmproj-pocket-tts.gguf

Soprano

Available params:

  • --tts-speaker-file is not supported; the model uses a fixed voice
  • Note: lang is not used. Supply normalized English text

Example usage:

llama-tts -m soprano.gguf \
    -mm mmproj-soprano.gguf \
    -p "Hello world" \
    --temp 0 \
    --output out.wav

Note for GGUF conversion:

Both Soprano-80M and Soprano-1.1-80M are supported. Convert both files from the same model directory, including decoder.pth:

python convert_hf_to_gguf.py path/to/Soprano-1.1-80M --outtype f32 --outfile soprano.gguf
python convert_hf_to_gguf.py path/to/Soprano-1.1-80M --mmproj --outtype f16 --outfile mmproj-soprano.gguf

LFM3-Audio

Available params:

  • --tts-lang can be ar, da, de, el, en, en-gb, es, fi, fr, he, it, ja, ko, nl, pl, pt, sv, tr, zh (default: en), other values are used as the language name
  • --tts-voice can be female (default) or male
  • Note: --tts-speaker-file is not used, the model has no voice cloning
  • Note: --top-k and --top-p are not used, audio frames are sampled with the settings of the model card (LFM3-Audio: temperature 1.0, top-k 128, repetition penalty 1.1 over 4096 frames; LFM3-Omni: temperature 0.4, repetition penalty 1.2 over 64 frames), the audio_* keys of generation_config.json override them at conversion
  • Each frame is 80 ms, generation stops at the end-of-audio frame

Example usage:

llama-tts -m lfm3-audio.gguf \
    -mm mmproj-lfm3-audio.gguf \
    -p "Hello world" \
    --tts-voice male \
    --output out.wav

The mmproj holds the audio encoder (for ASR), the depthformer and the codec decoder, the latter two are exported when the checkpoint has an audio_tokenizer directory:

python convert_hf_to_gguf.py path/to/lfm3-audio --outfile lfm3-audio.gguf
python convert_hf_to_gguf.py path/to/lfm3-audio --mmproj --outfile mmproj-lfm3-audio.gguf

Chatterbox

Two variants are supported: Chatterbox Turbo (English) and Chatterbox Multilingual V3 (23 languages).

Available params:

  • --tts-speaker-file should point to a speaker reference audio file (wav, mp3). Optional, the predefined voice is used without it
  • --tts-voice can be default, the voice shipped with the model (conds.pt)
  • --tts-lang sets the language of the multilingual variant, for example fr (default: none). It is not used by Turbo

Example usage:

llama-tts -m chatterbox-mtl.gguf \
    -mm mmproj-chatterbox-mtl.gguf \
    -p "Bonjour tout le monde" \
    --tts-lang fr \
    --tts-speaker-file speaker.mp3 \
    --output out.wav

Note: the multilingual variant runs classifier-free guidance with a second sequence. llama-tts sets this up by itself, but llama-server does not support it yet.

Note for GGUF conversion:

The upstream repositories ship raw checkpoints without config.json. Download the files below into a directory, then convert it as usual (the variant is detected from the talker file):

  • Turbo: t3_turbo_v1.safetensors, s3gen_meanflow.safetensors, ve.safetensors, conds.pt and the GPT-2 tokenizer files (vocab.json, merges.txt, added_tokens.json, special_tokens_map.json, tokenizer_config.json)
  • Multilingual V3: t3_mtl23ls_v3.safetensors, s3gen_v3.safetensors, ve.safetensors, conds.pt, mtl_tokenizer.json
python convert_hf_to_gguf.py path/to/chatterbox --outfile chatterbox.gguf
python convert_hf_to_gguf.py path/to/chatterbox --mmproj --outfile mmproj-chatterbox.gguf

Drama 3

Fish Audio Drama 3: a Qwen3.5-MoE backbone (35B, 3B active) samples each frame's main token, a codebook tower samples the 9 other codes, and the Fish DAC decodes them to 44.1 kHz audio.

Available params:

  • Speaker tags <|speaker:0|> to <|speaker:15|> in the text give each character its own voice, a text without a tag starts with <|speaker:0|>
  • The language is the one of the text, --tts-lang is not used
  • Note: --tts-speaker-file is not supported yet (voice cloning)
  • Note: --top-k and --top-p are not used, frames are sampled with the settings of the reference (main token: temperature 1.0, top-p 0.9, repetition-aware redraw over the last 16 frames; other codes: temperature 1.0, top-k 30, top-p 0.9)
  • Each frame is 2048 samples (about 46 ms), generation stops at the end-of-speech token
  • The codebook tower keeps the whole sequence in its KV cache, the prompt and the generated frames are limited to 4096 positions

Example usage:

llama-tts -m drama3.gguf \
    -mm mmproj-drama3.gguf \
    -p "<|speaker:0|> Did you remember to buy the bread? <|speaker:1|> Of course I did, it is on the kitchen table." \
    -n 1000 \
    --output out.wav

Note for GGUF conversion:

The routed experts of the checkpoint are FP8, they are dequantized at conversion. The mmproj holds the codebook tower and the decoder half of codec.pth, it is best kept in BF16:

python convert_hf_to_gguf.py path/to/drama-3 --outtype q8_0 --outfile drama3.gguf
python convert_hf_to_gguf.py path/to/drama-3 --mmproj --outtype bf16 --outfile mmproj-drama3.gguf

ZONOS3

Zyphra ZONOS3 Mini: a hybrid Gated DeltaNet / attention MoE backbone (800M, 100M active) predicts the 10 codebooks of each frame with an inter-codebook delay, and the ZAC codec decodes them to 44.1 kHz audio.

Available params:

  • --tts-voice picks one of the predefined voices of the release (Savannah_en-American_F_1 by default); llama-server lists them in GET /props. A voice is the prompt prefix of its reference codes, the default speaker mode of the reference
  • The language is the one of the text; --tts-lang (lang in llama-server) only selects the text normalization
  • Numbers are spelled out by the tts chat template of the GGUF: cardinals, decimals, money, percents, clock times, common ordinal forms (1st, 1er, 1º), English and German years. A 4-digit number is read as a year only after a word like in, since, a month name, or im Jahr, seit in German (in 1990 -> "in nineteen ninety", 1234 people -> "one thousand two hundred thirty-four people"). Languages: en, fr, de, es, it, pt, ja and zh (or cmn), the region is ignored (fr_fr, pt_br); English when the language is empty, other languages are left as written. Set the language to the one of the text: the separators differ, e.g. 1.234 is a thousand in German
  • Note: the reference uses NeMo text normalization, which also covers units, dates, abbreviations and Korean; supply normalized text for them
  • --tts-speaker-file clones the voice of a clip (the first 60 s are used) with the speaker encoder of the release, the "embedding" speaker mode of the reference
  • llama-server also takes speed (relative to 15 UTF-8 bytes per second) and loudness (LUFS), see the /tts endpoint; they pick the speaking-rate and loudness condition buckets of the checkpoint, the defaults leave them unknown
  • Note: loudness has a strong effect with a cloned voice; a predefined voice mostly keeps the loudness of its reference clip
  • Note: --top-k and --top-p are not used, frames are sampled with the settings of the reference: temperature 0.75 (--temp changes it), top-k 80, top-p 0.69, repetition penalty 2.4 over the last 96 frames of codebooks 0 to 5
  • Each frame is 2940 samples (about 67 ms), generation stops at the end-of-audio code

Example usage:

llama-tts -m zonos3-mini.gguf \
    -mm mmproj-zonos3-mini.gguf \
    -p "The old lighthouse keeper climbed the spiral stairs every evening." \
    --tts-voice Imogen_en-British_F_1 \
    --output out.wav

Note for GGUF conversion:

Convert the root of the release repository, not the lm/ subdirectory. The mmproj holds the codebook heads, the predefined voices, the speaker encoder and the keep2_plain codec decoder, it is best kept in F32:

python convert_hf_to_gguf.py path/to/ZONOS3-Mini --outtype q8_0 --outfile zonos3-mini.gguf
python convert_hf_to_gguf.py path/to/ZONOS3-Mini --mmproj --outtype f32 --outfile mmproj-zonos3-mini.gguf