Files
llama.cpp/docs/multimodal/tts.md
T
Xuan-Son Nguyen d92ea9bb12 mtmd: support Minimax-Music3 (#32435)
* mtmd: support Minimax-Music3

* fix and clean up
2026-11-17 12:53:19 +01:00

19 KiB

Text-to-speech (TTS)

llama.cpp supports audio generation via libmtmd. The following tools can be used:

  • llama-tts, see its README for common usage and params
  • llama-server via the /tts endpoint and the OpenAI-compatible /v1/audio/speech API

Supported models:

The sections below list the params supported by each model, with llama-tts examples. In llama-server, the same options are set with --tts-lang and --tts-voice, or per request (see the /tts endpoint).

Qwen3-TTS

Available params:

  • --tts-lang can be zh, en, de, it, pt, es, ja, ko, fr, ru (default: en)
  • --tts-speaker-file should point to a speaker reference audio file (wav, mp3)

Example usage:

llama-tts -hf ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF \
    -p "Hello world" \
    --tts-lang english \
    --tts-speaker-file speaker.mp3 \
    --output out.wav

Pocket TTS

Available params:

  • --tts-speaker-file should point to a speaker reference audio file (wav, mp3). It is required, the model produces almost no audio without it
  • --tts-voice selects one of the predefined voices shipped with the weights (e.g. giovanni), when the GGUF holds them. The converter reads them from embeddings/*.safetensors next to the weights, of both repositories, and logs how closely each one is reproduced (some packs ship voice states computed with other weights, the converter warns above 5% and skips a voice only when it does not belong to the model at all)
  • Note: lang is not used, the language is a property of the weights

Example usage:

llama-tts -m pocket-tts.gguf \
    -mm mmproj-pocket-tts.gguf \
    -p "Hello world" \
    --tts-speaker-file speaker.mp3 \
    --output out.wav

The weights without voice cloning ship a set of predefined voices (embeddings/*.safetensors) next to each language; the converter stores them in the mmproj, as the voice prompt embeddings, and leaves out the zeroed voice encoder. With such a model --tts-speaker-file is rejected and a voice is selected with --tts-voice (the first one when it is not given):

llama-tts -m pocket-tts.gguf \
    -mm mmproj-pocket-tts.gguf \
    -p "Ciao a tutti" \
    --tts-voice giovanni \
    --output out.wav

Note for GGUF conversion:

The upstream repository holds one complete model per language under languages/, next to a set of shared files at the root. Convert one of the languages/<name> directories, not the root directory:

python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --outfile pocket-tts.gguf
python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --mmproj --outfile mmproj-pocket-tts.gguf

Soprano

Available params:

  • --tts-speaker-file is not supported; the model uses a fixed voice
  • Note: lang is not used. Supply normalized English text

Example usage:

llama-tts -m soprano.gguf \
    -mm mmproj-soprano.gguf \
    -p "Hello world" \
    --temp 0 \
    --output out.wav

Note for GGUF conversion:

Both Soprano-80M and Soprano-1.1-80M are supported. Convert both files from the same model directory, including decoder.pth:

python convert_hf_to_gguf.py path/to/Soprano-1.1-80M --outtype f32 --outfile soprano.gguf
python convert_hf_to_gguf.py path/to/Soprano-1.1-80M --mmproj --outtype f16 --outfile mmproj-soprano.gguf

LFM3-Audio

Available params:

  • --tts-lang can be ar, da, de, el, en, en-gb, es, fi, fr, he, it, ja, ko, nl, pl, pt, sv, tr, zh (default: en), other values are used as the language name
  • --tts-voice can be female (default) or male
  • Note: --tts-speaker-file is not used, the model has no voice cloning
  • Note: --top-k and --top-p are not used, audio frames are sampled with the settings of the model card (LFM3-Audio: temperature 1.0, top-k 128, repetition penalty 1.1 over 4096 frames; LFM3-Omni: temperature 0.4, repetition penalty 1.2 over 64 frames), the audio_* keys of generation_config.json override them at conversion
  • Each frame is 80 ms, generation stops at the end-of-audio frame

Example usage:

llama-tts -m lfm3-audio.gguf \
    -mm mmproj-lfm3-audio.gguf \
    -p "Hello world" \
    --tts-voice male \
    --output out.wav

The mmproj holds the audio encoder (for ASR), the depthformer and the codec decoder, the latter two are exported when the checkpoint has an audio_tokenizer directory:

python convert_hf_to_gguf.py path/to/lfm3-audio --outfile lfm3-audio.gguf
python convert_hf_to_gguf.py path/to/lfm3-audio --mmproj --outfile mmproj-lfm3-audio.gguf

Chatterbox

Two variants are supported: Chatterbox Turbo (English) and Chatterbox Multilingual V3 (23 languages).

Available params:

  • --tts-speaker-file should point to a speaker reference audio file (wav, mp3). Optional, the predefined voice is used without it
  • --tts-voice can be default, the voice shipped with the model (conds.pt)
  • --tts-lang sets the language of the multilingual variant, for example fr (default: none). It is not used by Turbo

Example usage:

llama-tts -m chatterbox-mtl.gguf \
    -mm mmproj-chatterbox-mtl.gguf \
    -p "Bonjour tout le monde" \
    --tts-lang fr \
    --tts-speaker-file speaker.mp3 \
    --output out.wav

Note: the multilingual variant runs classifier-free guidance with a second sequence. llama-tts sets this up by itself, but llama-server does not support it yet.

Note for GGUF conversion:

The upstream repositories ship raw checkpoints without config.json. Download the files below into a directory, then convert it as usual (the variant is detected from the talker file):

  • Turbo: t3_turbo_v1.safetensors, s3gen_meanflow.safetensors, ve.safetensors, conds.pt and the GPT-2 tokenizer files (vocab.json, merges.txt, added_tokens.json, special_tokens_map.json, tokenizer_config.json)
  • Multilingual V3: t3_mtl23ls_v3.safetensors, s3gen_v3.safetensors, ve.safetensors, conds.pt, mtl_tokenizer.json
python convert_hf_to_gguf.py path/to/chatterbox --outfile chatterbox.gguf
python convert_hf_to_gguf.py path/to/chatterbox --mmproj --outfile mmproj-chatterbox.gguf

Drama 3

Fish Audio Drama 3: a Qwen3.5-MoE backbone (35B, 3B active) samples each frame's main token, a codebook tower samples the 9 other codes, and the Fish DAC decodes them to 44.1 kHz audio.

Available params:

  • Speaker tags <|speaker:0|> to <|speaker:15|> in the text give each character its own voice, a text without a tag starts with <|speaker:0|>
  • The language is the one of the text, --tts-lang is not used
  • Note: --tts-speaker-file is not supported yet (voice cloning)
  • Note: --top-k and --top-p are not used, frames are sampled with the settings of the reference (main token: temperature 1.0, top-p 0.9, repetition-aware redraw over the last 16 frames; other codes: temperature 1.0, top-k 30, top-p 0.9)
  • Each frame is 2048 samples (about 46 ms), generation stops at the end-of-speech token
  • The codebook tower keeps the whole sequence in its KV cache, the prompt and the generated frames are limited to 4096 positions

Example usage:

llama-tts -m drama3.gguf \
    -mm mmproj-drama3.gguf \
    -p "<|speaker:0|> Did you remember to buy the bread? <|speaker:1|> Of course I did, it is on the kitchen table." \
    -n 1000 \
    --output out.wav

Note for GGUF conversion:

The routed experts of the checkpoint are FP8, they are dequantized at conversion. The mmproj holds the codebook tower and the decoder half of codec.pth, it is best kept in BF16:

python convert_hf_to_gguf.py path/to/drama-3 --outtype q8_0 --outfile drama3.gguf
python convert_hf_to_gguf.py path/to/drama-3 --mmproj --outtype bf16 --outfile mmproj-drama3.gguf

ZONOS3

Zyphra ZONOS3 Mini: a hybrid Gated DeltaNet / attention MoE backbone (800M, 100M active) predicts the 10 codebooks of each frame with an inter-codebook delay, and the ZAC codec decodes them to 44.1 kHz audio.

Available params:

  • --tts-voice picks one of the predefined voices of the release (Savannah_en-American_F_1 by default); llama-server lists them in GET /props. A voice is the prompt prefix of its reference codes, the default speaker mode of the reference
  • The language is the one of the text; --tts-lang (lang in llama-server) only selects the text normalization
  • Numbers are spelled out by the tts chat template of the GGUF: cardinals, decimals, money, percents, clock times, common ordinal forms (1st, 1er, 1º), English and German years. A 4-digit number is read as a year only after a word like in, since, a month name, or im Jahr, seit in German (in 1990 -> "in nineteen ninety", 1234 people -> "one thousand two hundred thirty-four people"). Languages: en, fr, de, es, it, pt, ja and zh (or cmn), the region is ignored (fr_fr, pt_br); English when the language is empty, other languages are left as written. Set the language to the one of the text: the separators differ, e.g. 1.234 is a thousand in German
  • Note: the reference uses NeMo text normalization, which also covers units, dates, abbreviations and Korean; supply normalized text for them
  • --tts-speaker-file clones the voice of a clip (the first 60 s are used) with the speaker encoder of the release, the "embedding" speaker mode of the reference
  • llama-server also takes speed (relative to 15 UTF-8 bytes per second) and loudness (LUFS), see the /tts endpoint; they pick the speaking-rate and loudness condition buckets of the checkpoint, the defaults leave them unknown
  • Note: loudness has a strong effect with a cloned voice; a predefined voice mostly keeps the loudness of its reference clip
  • Note: --top-k and --top-p are not used, frames are sampled with the settings of the reference: temperature 0.75 (--temp changes it), top-k 80, top-p 0.69, repetition penalty 2.4 over the last 96 frames of codebooks 0 to 5
  • Each frame is 2940 samples (about 67 ms), generation stops at the end-of-audio code

Example usage:

llama-tts -m zonos3-mini.gguf \
    -mm mmproj-zonos3-mini.gguf \
    -p "The old lighthouse keeper climbed the spiral stairs every evening." \
    --tts-voice Imogen_en-British_F_1 \
    --output out.wav

Note for GGUF conversion:

Convert the root of the release repository, not the lm/ subdirectory. The mmproj holds the codebook heads, the predefined voices, the speaker encoder and the keep2_plain codec decoder, it is best kept in F32:

python convert_hf_to_gguf.py path/to/ZONOS3-Mini --outtype q8_0 --outfile zonos3-mini.gguf
python convert_hf_to_gguf.py path/to/ZONOS3-Mini --mmproj --outtype f32 --outfile mmproj-zonos3-mini.gguf

Fish Audio S2 Pro

Fish Audio S2 Pro: a Qwen3 backbone (4B) samples each frame's semantic token, a 4-layer fast AR predicts the 9 other codes, and the Fish DAC (same as Drama 3) decodes them to 44.1 kHz audio.

Available params:

  • Speaker tags <|speaker:0|>, <|speaker:1|>, ... in the text give each character its own voice, a text without a tag starts with <|speaker:0|>
  • Inline instructions in brackets control the delivery, for example [laughing], [whisper] or [pause]
  • The language is the one of the text, --tts-lang is not used
  • Voice cloning: --tts-speaker-file with the reference clip, optionally --tts-speaker-text with its transcript (it is put in the prompt as the reference does). The clip is encoded by the codec encoder half of codec.pth, it is truncated to the first 30 s
  • Note: --temp, --top-k and --top-p are not used (only --seed), frames are sampled with the settings of the reference (semantic token: repetition penalty 1.1 over the last 16 frames, top-k 30, top-p 0.8, temperature 0.8, or top-p 0.9 and temperature 1.0 when the last 4 frames repeat a token; other codes: greedy)
  • Each frame is 2048 samples (about 46 ms), generation stops at the end-of-speech token

Example usage:

llama-tts -m s2-pro.gguf \
    -mm mmproj-s2-pro.gguf \
    -p "<|speaker:0|>Did you remember to buy the bread? <|speaker:1|>Of course I did [laughing] it is on the kitchen table." \
    -n 1000 \
    --output out.wav

Voice cloning (--tts-speaker-text can be left out):

llama-tts -m s2-pro.gguf \
    -mm mmproj-s2-pro.gguf \
    --tts-speaker-file reference.wav \
    --tts-speaker-text "The words spoken in reference.wav." \
    -p "Hello, this is a cloned voice." \
    --output out.wav

Note for GGUF conversion:

The mmproj holds the fast AR and codec.pth (the encoder half for voice cloning, the decoder half for the audio), it is best kept in BF16. The codec encoder is always stored in F16/F32, small errors in it change the codes picked by its VQ:

python convert_hf_to_gguf.py path/to/s2-pro --outtype q8_0 --outfile s2-pro.gguf
python convert_hf_to_gguf.py path/to/s2-pro --mmproj --outtype bf16 --outfile mmproj-s2-pro.gguf

YuE2

m-a-p/YuE2-3B generates songs (vocals and accompaniment) from a style prompt and lyrics, as 48 kHz stereo. The backbone (a Qwen3 text model) writes an ABC score, then one semantic code per 40 ms frame; the NAR half of the checkpoint turns the codes into VAE latents by flow matching, attending to the keys and values of the backbone, and the YuE2-Vae decoder makes the audio.

The prompt is the text of the reference request:

[Tags]
City Pop, upbeat, danceable, groovy bass, electric guitar, synth
[Lyrics]
[Verse]
...

Available params:

  • --tts-voice picks the planning mode of the reference (cot): full (default, a chord-annotated score), melody (a melody-only score) or off (no score)
  • Cover mode: append the score to the prompt as \n[ABC]\n<score>, with --tts-voice melody or full; the model then goes straight to the codes
  • The generation stops at the limits of the reference (4096 score tokens, then 9000 codes, 6 minutes); -n sets a lower limit on the score tokens and the codes together
  • The context must keep the default size (24576 positions, the one of the reference): a smaller -c is rejected when the model is loaded
  • Note: --top-k and --top-p are not used, both phases are sampled with the settings of the reference (yue2_generation_config.json); --temp changes the temperature of the codes
  • Note: the backbone context must use an F16 KV cache (default): the NAR pass loads its keys and values
  • The codes are sampled with classifier-free guidance like the reference: scale 1.01 with --tts-voice off, off (1.0) otherwise. llama-server takes cfg_scale to change it, the reference suggests 1.2 for a stronger text guidance. The uncond prompt runs as a second sequence, so the context must also hold it (the instruction, the score and the codes once more)
  • The NAR pass solves the flow-matching ODE in 32 midpoint steps like the reference; llama-server takes flow_steps to change it. The NAR time is about proportional to it: 16 steps are about 35% faster for the NAR and the VAE together, with a small deviation from the reference (about 0.2 dB of spectral difference); it grows with fewer steps (about 0.4 dB at 8, 0.8 dB at 4)
  • The NAR pass runs on chunks of codes like the reference: one chunk for a song that fits in 24576 positions, so the audio arrives after the last code. When streaming, each 30 s of codes is a chunk, processed during the generation, so the audio arrives in 30 s steps; the chunks are independent (each sees the prompt, the score and its own codes), which may be audible at the boundaries

Example usage:

llama-tts -m yue2-3b.gguf \
    -mm mmproj-yue2-3b.gguf \
    -f song.txt \
    --output song.wav

Note for GGUF conversion:

The VAE is a separate repository, download it into the vae subdirectory of the model:

hf download m-a-p/YuE2-3B --local-dir YuE2-3B
hf download m-a-p/YuE2-Vae --local-dir YuE2-3B/vae
python convert_hf_to_gguf.py YuE2-3B --outfile yue2-3b.gguf
python convert_hf_to_gguf.py YuE2-3B --mmproj --outfile mmproj-yue2-3b.gguf

The mmproj holds the NAR transformer layers (loaded with libllama as a Qwen3 text model, in the --outtype type), and the latent projections and the VAE decoder, always in F32.

MiniMax Music 3

MiniMaxAI/MiniMax-Music3 generates songs (vocals and accompaniment) from a music description and lyrics, as 44.1 kHz stereo. For each 40 ms frame, the backbone (a Qwen3 text model) samples the semantic code and the RVQ depth decoder the 7 other codes; the hidden states of both go to a flow-matching DiT, then a DAC decoder makes the audio. Both use the settings of the reference (the diffusers pipeline).

The prompt has the same format as YuE2, the description goes under [Tags]:

[Tags]
Genre: acoustic pop. BPM: 96. Key: C major. Vocals: soft female lead. Arrangement: fingerpicked guitar and soft piano.
[Lyrics]
[verse]
Morning light filtering through the pine
[chorus]
Softly the world begins to breathe

Available params:

  • The description and the lyrics are cleaned up like the reference: markdown is removed, a line that starts with structure tags (e.g. [Verse]) keeps only the tags
  • The generation stops at the end-of-audio token or at the limit of the reference (9000 frames, 6 minutes); -n sets a lower limit in frames (25 per second)
  • --temp changes the temperature of the codes (default 1.0); --top-k and --top-p are not used, the codes are sampled with top-k 50 like the reference
  • The codes are sampled with classifier-free guidance, scale 1.5 like the reference; llama-server takes cfg_scale to change it (1.0 disables it). The uncond prompt runs as a second sequence that shares the context, so a song needs about 2 x (prompt tokens + frames) positions: the default context (10240) holds about 3 minutes with a short description, use a larger -c (e.g. 20480) for longer songs
  • The flow matching runs 30 Euler steps like the reference; llama-server takes flow_steps to change it
  • The audio is made in windows of 200 frames (8 s) every 100 frames, like the reference: a window is processed during the generation once the frame after it exists, so with streaming the audio arrives in steps of about 4 s

Example usage:

llama-tts -m minimax-music3.gguf \
    -mm mmproj-minimax-music3.gguf \
    -f song.txt \
    --output song.wav

Note for GGUF conversion:

The conversion reads the diffusers layout of the repository, the qwen_7B directory and the .pth files are not needed:

hf download MiniMaxAI/MiniMax-Music3 --local-dir MiniMax-Music3 --exclude "qwen_7B/*" --exclude "*.pth"
python convert_hf_to_gguf.py MiniMax-Music3 --outfile minimax-music3.gguf
python convert_hf_to_gguf.py MiniMax-Music3 --mmproj --outfile mmproj-minimax-music3.gguf

The mmproj holds the depth decoder and the DiT layers (in the --outtype type), and the condition encoder, the small DiT projections and the DAC decoder, always in F32.