Files
llama.cpp/tools/tts
Xuan-Son NguyenandPascal 5851a8e505 model, mtmd: drama-3 tts support (#31613)
* model, mtmd: drama-3 tts support

* clean up

* mtmd: drama3 helper stops at the codebook tower limit

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-04 15:15:09 +01:00
..

llama.cpp TTS

This is a tool to demonstrate audio generation capability in llama.cpp via libmtmd. It was added via PR #26254

Note: this tool used to serve as a demo for OuteTTS, but it was converted to a more model-agnostic tool.

Common usage

Simple usage:

llama-tts -hf ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF -p "Hello world" --output out.wav

Note: it's highly recommended to always provide a speaker reference voice (via --tts-speaker-file); otherwise, the model's performance may be degraded.

Common params:

  • Sampling params such as --top-k, --top-p, --temp, etc.
  • -n <number_of_frames> limits the output length, e.g. -n 500. Note that how many milliseconds each frame represents varies by model
  • Core inference params such as -ngl, -b, -ub, etc.

Qwen3-TTS

Available params:

  • --tts-lang can be zh, en, de, it, pt, es, ja, ko, fr, ru (default: en)
  • --tts-speaker-file should point to a speaker reference audio file (wav, mp3)

Example usage:

llama-tts -hf ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF \
    -p "Hello world" \
    --tts-lang english \
    --tts-speaker-file speaker.mp3 \
    --output out.wav

Pocket TTS

Available params:

  • --tts-speaker-file should point to a speaker reference audio file (wav, mp3). It is required, the model produces almost no audio without it
  • --tts-voice selects one of the predefined voices shipped with the weights (e.g. giovanni), when the GGUF holds them. The converter reads them from embeddings/*.safetensors next to the weights, of both repositories, and logs how closely each one is reproduced (some packs ship voice states computed with other weights, the converter warns above 5% and skips a voice only when it does not belong to the model at all)
  • Note: lang is not used, the language is a property of the weights

Example usage:

llama-tts -m pocket-tts.gguf \
    -mm mmproj-pocket-tts.gguf \
    -p "Hello world" \
    --tts-speaker-file speaker.mp3 \
    --output out.wav

The weights without voice cloning ship a set of predefined voices (embeddings/*.safetensors) next to each language; the converter stores them in the mmproj, as the voice prompt embeddings, and leaves out the zeroed voice encoder. With such a model --tts-speaker-file is rejected and a voice is selected with --tts-voice (the first one when it is not given):

llama-tts -m pocket-tts.gguf \
    -mm mmproj-pocket-tts.gguf \
    -p "Ciao a tutti" \
    --tts-voice giovanni \
    --output out.wav

Note for GGUF conversion:

The upstream repository holds one complete model per language under languages/, next to a set of shared files at the root. Convert one of the languages/<name> directories, not the root directory:

python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --outfile pocket-tts.gguf
python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --mmproj --outfile mmproj-pocket-tts.gguf

Soprano

Available params:

  • --tts-speaker-file is not supported; the model uses a fixed voice
  • Note: lang is not used. Supply normalized English text

Example usage:

llama-tts -m soprano.gguf \
    -mm mmproj-soprano.gguf \
    -p "Hello world" \
    --temp 0 \
    --output out.wav

Note for GGUF conversion:

Both Soprano-80M and Soprano-1.1-80M are supported. Convert both files from the same model directory, including decoder.pth:

python convert_hf_to_gguf.py path/to/Soprano-1.1-80M --outtype f32 --outfile soprano.gguf
python convert_hf_to_gguf.py path/to/Soprano-1.1-80M --mmproj --outtype f16 --outfile mmproj-soprano.gguf

LFM3-Audio

Available params:

  • --tts-lang can be ar, da, de, el, en, en-gb, es, fi, fr, he, it, ja, ko, nl, pl, pt, sv, tr, zh (default: en), other values are used as the language name
  • --tts-voice can be female (default) or male
  • Note: --tts-speaker-file is not used, the model has no voice cloning
  • Note: --top-k and --top-p are not used, audio frames are sampled with the settings of the model card (LFM3-Audio: temperature 1.0, top-k 128, repetition penalty 1.1 over 4096 frames; LFM3-Omni: temperature 0.4, repetition penalty 1.2 over 64 frames), the audio_* keys of generation_config.json override them at conversion
  • Each frame is 80 ms, generation stops at the end-of-audio frame

Example usage:

llama-tts -m lfm3-audio.gguf \
    -mm mmproj-lfm3-audio.gguf \
    -p "Hello world" \
    --tts-voice male \
    --output out.wav

The mmproj holds the audio encoder (for ASR), the depthformer and the codec decoder, the latter two are exported when the checkpoint has an audio_tokenizer directory:

python convert_hf_to_gguf.py path/to/lfm3-audio --outfile lfm3-audio.gguf
python convert_hf_to_gguf.py path/to/lfm3-audio --mmproj --outfile mmproj-lfm3-audio.gguf

Chatterbox

Two variants are supported: Chatterbox Turbo (English) and Chatterbox Multilingual V3 (23 languages).

Available params:

  • --tts-speaker-file should point to a speaker reference audio file (wav, mp3). Optional, the predefined voice is used without it
  • --tts-voice can be default, the voice shipped with the model (conds.pt)
  • --tts-lang sets the language of the multilingual variant, for example fr (default: none). It is not used by Turbo

Example usage:

llama-tts -m chatterbox-mtl.gguf \
    -mm mmproj-chatterbox-mtl.gguf \
    -p "Bonjour tout le monde" \
    --tts-lang fr \
    --tts-speaker-file speaker.mp3 \
    --output out.wav

Note: the multilingual variant runs classifier-free guidance with a second sequence. llama-tts sets this up by itself, but llama-server does not support it yet.

Note for GGUF conversion:

The upstream repositories ship raw checkpoints without config.json. Download the files below into a directory, then convert it as usual (the variant is detected from the talker file):

  • Turbo: t3_turbo_v1.safetensors, s3gen_meanflow.safetensors, ve.safetensors, conds.pt and the GPT-2 tokenizer files (vocab.json, merges.txt, added_tokens.json, special_tokens_map.json, tokenizer_config.json)
  • Multilingual V3: t3_mtl23ls_v3.safetensors, s3gen_v3.safetensors, ve.safetensors, conds.pt, mtl_tokenizer.json
python convert_hf_to_gguf.py path/to/chatterbox --outfile chatterbox.gguf
python convert_hf_to_gguf.py path/to/chatterbox --mmproj --outfile mmproj-chatterbox.gguf

Drama 3

Fish Audio Drama 3: a Qwen3.5-MoE backbone (35B, 3B active) samples each frame's main token, a codebook tower samples the 9 other codes, and the Fish DAC decodes them to 44.1 kHz audio.

Available params:

  • Speaker tags <|speaker:0|> to <|speaker:15|> in the text give each character its own voice, a text without a tag starts with <|speaker:0|>
  • The language is the one of the text, --tts-lang is not used
  • Note: --tts-speaker-file is not supported yet (voice cloning)
  • Note: --top-k and --top-p are not used, frames are sampled with the settings of the reference (main token: temperature 1.0, top-p 0.9, repetition-aware redraw over the last 16 frames; other codes: temperature 1.0, top-k 30, top-p 0.9)
  • Each frame is 2048 samples (about 46 ms), generation stops at the end-of-speech token
  • The codebook tower keeps the whole sequence in its KV cache, the prompt and the generated frames are limited to 4096 positions

Example usage:

llama-tts -m drama3.gguf \
    -mm mmproj-drama3.gguf \
    -p "<|speaker:0|> Did you remember to buy the bread? <|speaker:1|> Of course I did, it is on the kitchen table." \
    -n 1000 \
    --output out.wav

Note for GGUF conversion:

The routed experts of the checkpoint are FP8, they are dequantized at conversion. The mmproj holds the codebook tower and the decoder half of codec.pth, it is best kept in BF16:

python convert_hf_to_gguf.py path/to/drama-3 --outtype q8_0 --outfile drama3.gguf
python convert_hf_to_gguf.py path/to/drama-3 --mmproj --outtype bf16 --outfile mmproj-drama3.gguf