* mtmd: support YuE2-3B * improve speed * add CFG * fix streaming * add cfg control from api * check ctx size * fix n_predict * add flow_steps config * nits * fix batch api * nits * adapt * mtmd: yue2 samples from its own saved logits The context logits belong to the last decode of any slot, so with several server slots the score and the unguided codes were sampled from another sequence. Keep the logits of each own decode in save_logits(), which also does the guidance mix. * mtmd: let the yue2 snake chain fuse The beta reshape landed between sqr and mul, so the mul, sin, sqr, mul, add chain was not consecutive and the backends never fused it. Expand both reshapes first, like qwen3tts does. --------- Co-authored-by: Pascal <admin@serveurperso.com>
17 KiB
Text-to-speech (TTS)
llama.cpp supports audio generation via libmtmd. The following tools can be used:
- llama-tts, see its README for common usage and params
- llama-server via the
/ttsendpoint and the OpenAI-compatible/v1/audio/speechAPI
Supported models:
The sections below list the params supported by each model, with llama-tts examples. In llama-server, the same options are set with --tts-lang and --tts-voice, or per request (see the /tts endpoint).
Qwen3-TTS
Available params:
--tts-langcan bezh,en,de,it,pt,es,ja,ko,fr,ru(default:en)--tts-speaker-fileshould point to a speaker reference audio file (wav, mp3)
Example usage:
llama-tts -hf ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF \
-p "Hello world" \
--tts-lang english \
--tts-speaker-file speaker.mp3 \
--output out.wav
Pocket TTS
Available params:
--tts-speaker-fileshould point to a speaker reference audio file (wav, mp3). It is required, the model produces almost no audio without it--tts-voiceselects one of the predefined voices shipped with the weights (e.g.giovanni), when the GGUF holds them. The converter reads them fromembeddings/*.safetensorsnext to the weights, of both repositories, and logs how closely each one is reproduced (some packs ship voice states computed with other weights, the converter warns above 5% and skips a voice only when it does not belong to the model at all)- Note:
langis not used, the language is a property of the weights
Example usage:
llama-tts -m pocket-tts.gguf \
-mm mmproj-pocket-tts.gguf \
-p "Hello world" \
--tts-speaker-file speaker.mp3 \
--output out.wav
The weights without voice cloning ship a set of predefined voices (embeddings/*.safetensors) next to each language; the converter stores them in the mmproj, as the voice prompt embeddings, and leaves out the zeroed voice encoder. With such a model --tts-speaker-file is rejected and a voice is selected with --tts-voice (the first one when it is not given):
llama-tts -m pocket-tts.gguf \
-mm mmproj-pocket-tts.gguf \
-p "Ciao a tutti" \
--tts-voice giovanni \
--output out.wav
Note for GGUF conversion:
The upstream repository holds one complete model per language under languages/, next to a set of shared files at the root. Convert one of the languages/<name> directories, not the root directory:
python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --outfile pocket-tts.gguf
python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --mmproj --outfile mmproj-pocket-tts.gguf
Soprano
Available params:
--tts-speaker-fileis not supported; the model uses a fixed voice- Note:
langis not used. Supply normalized English text
Example usage:
llama-tts -m soprano.gguf \
-mm mmproj-soprano.gguf \
-p "Hello world" \
--temp 0 \
--output out.wav
Note for GGUF conversion:
Both Soprano-80M and Soprano-1.1-80M are supported. Convert both files from the same model directory, including decoder.pth:
python convert_hf_to_gguf.py path/to/Soprano-1.1-80M --outtype f32 --outfile soprano.gguf
python convert_hf_to_gguf.py path/to/Soprano-1.1-80M --mmproj --outtype f16 --outfile mmproj-soprano.gguf
LFM3-Audio
Available params:
--tts-langcan bear,da,de,el,en,en-gb,es,fi,fr,he,it,ja,ko,nl,pl,pt,sv,tr,zh(default:en), other values are used as the language name--tts-voicecan befemale(default) ormale- Note:
--tts-speaker-fileis not used, the model has no voice cloning - Note:
--top-kand--top-pare not used, audio frames are sampled with the settings of the model card (LFM3-Audio: temperature 1.0, top-k 128, repetition penalty 1.1 over 4096 frames; LFM3-Omni: temperature 0.4, repetition penalty 1.2 over 64 frames), theaudio_*keys ofgeneration_config.jsonoverride them at conversion - Each frame is 80 ms, generation stops at the end-of-audio frame
Example usage:
llama-tts -m lfm3-audio.gguf \
-mm mmproj-lfm3-audio.gguf \
-p "Hello world" \
--tts-voice male \
--output out.wav
The mmproj holds the audio encoder (for ASR), the depthformer and the codec decoder, the latter two are exported when the checkpoint has an audio_tokenizer directory:
python convert_hf_to_gguf.py path/to/lfm3-audio --outfile lfm3-audio.gguf
python convert_hf_to_gguf.py path/to/lfm3-audio --mmproj --outfile mmproj-lfm3-audio.gguf
Chatterbox
Two variants are supported: Chatterbox Turbo (English) and Chatterbox Multilingual V3 (23 languages).
Available params:
--tts-speaker-fileshould point to a speaker reference audio file (wav, mp3). Optional, the predefined voice is used without it--tts-voicecan bedefault, the voice shipped with the model (conds.pt)--tts-langsets the language of the multilingual variant, for examplefr(default: none). It is not used by Turbo
Example usage:
llama-tts -m chatterbox-mtl.gguf \
-mm mmproj-chatterbox-mtl.gguf \
-p "Bonjour tout le monde" \
--tts-lang fr \
--tts-speaker-file speaker.mp3 \
--output out.wav
Note: the multilingual variant runs classifier-free guidance with a second sequence. llama-tts sets this up by itself, but llama-server does not support it yet.
Note for GGUF conversion:
The upstream repositories ship raw checkpoints without config.json. Download the files below into a directory, then convert it as usual (the variant is detected from the talker file):
- Turbo:
t3_turbo_v1.safetensors,s3gen_meanflow.safetensors,ve.safetensors,conds.ptand the GPT-2 tokenizer files (vocab.json,merges.txt,added_tokens.json,special_tokens_map.json,tokenizer_config.json) - Multilingual V3:
t3_mtl23ls_v3.safetensors,s3gen_v3.safetensors,ve.safetensors,conds.pt,mtl_tokenizer.json
python convert_hf_to_gguf.py path/to/chatterbox --outfile chatterbox.gguf
python convert_hf_to_gguf.py path/to/chatterbox --mmproj --outfile mmproj-chatterbox.gguf
Drama 3
Fish Audio Drama 3: a Qwen3.5-MoE backbone (35B, 3B active) samples each frame's main token, a codebook tower samples the 9 other codes, and the Fish DAC decodes them to 44.1 kHz audio.
Available params:
- Speaker tags
<|speaker:0|>to<|speaker:15|>in the text give each character its own voice, a text without a tag starts with<|speaker:0|> - The language is the one of the text,
--tts-langis not used - Note:
--tts-speaker-fileis not supported yet (voice cloning) - Note:
--top-kand--top-pare not used, frames are sampled with the settings of the reference (main token: temperature 1.0, top-p 0.9, repetition-aware redraw over the last 16 frames; other codes: temperature 1.0, top-k 30, top-p 0.9) - Each frame is 2048 samples (about 46 ms), generation stops at the end-of-speech token
- The codebook tower keeps the whole sequence in its KV cache, the prompt and the generated frames are limited to 4096 positions
Example usage:
llama-tts -m drama3.gguf \
-mm mmproj-drama3.gguf \
-p "<|speaker:0|> Did you remember to buy the bread? <|speaker:1|> Of course I did, it is on the kitchen table." \
-n 1000 \
--output out.wav
Note for GGUF conversion:
The routed experts of the checkpoint are FP8, they are dequantized at conversion. The mmproj holds the codebook tower and the decoder half of codec.pth, it is best kept in BF16:
python convert_hf_to_gguf.py path/to/drama-3 --outtype q8_0 --outfile drama3.gguf
python convert_hf_to_gguf.py path/to/drama-3 --mmproj --outtype bf16 --outfile mmproj-drama3.gguf
ZONOS3
Zyphra ZONOS3 Mini: a hybrid Gated DeltaNet / attention MoE backbone (800M, 100M active) predicts the 10 codebooks of each frame with an inter-codebook delay, and the ZAC codec decodes them to 44.1 kHz audio.
Available params:
--tts-voicepicks one of the predefined voices of the release (Savannah_en-American_F_1by default);llama-serverlists them inGET /props. A voice is the prompt prefix of its reference codes, the default speaker mode of the reference- The language is the one of the text;
--tts-lang(langinllama-server) only selects the text normalization - Numbers are spelled out by the
ttschat template of the GGUF: cardinals, decimals, money, percents, clock times, common ordinal forms (1st,1er,1º), English and German years. A 4-digit number is read as a year only after a word likein,since, a month name, orim Jahr,seitin German (in 1990-> "in nineteen ninety",1234 people-> "one thousand two hundred thirty-four people"). Languages:en,fr,de,es,it,pt,jaandzh(orcmn), the region is ignored (fr_fr,pt_br); English when the language is empty, other languages are left as written. Set the language to the one of the text: the separators differ, e.g.1.234is a thousand in German - Note: the reference uses NeMo text normalization, which also covers units, dates, abbreviations and Korean; supply normalized text for them
--tts-speaker-fileclones the voice of a clip (the first 60 s are used) with the speaker encoder of the release, the "embedding" speaker mode of the referencellama-serveralso takesspeed(relative to 15 UTF-8 bytes per second) andloudness(LUFS), see the/ttsendpoint; they pick the speaking-rate and loudness condition buckets of the checkpoint, the defaults leave them unknown- Note:
loudnesshas a strong effect with a cloned voice; a predefined voice mostly keeps the loudness of its reference clip - Note:
--top-kand--top-pare not used, frames are sampled with the settings of the reference: temperature 0.75 (--tempchanges it), top-k 80, top-p 0.69, repetition penalty 2.4 over the last 96 frames of codebooks 0 to 5 - Each frame is 2940 samples (about 67 ms), generation stops at the end-of-audio code
Example usage:
llama-tts -m zonos3-mini.gguf \
-mm mmproj-zonos3-mini.gguf \
-p "The old lighthouse keeper climbed the spiral stairs every evening." \
--tts-voice Imogen_en-British_F_1 \
--output out.wav
Note for GGUF conversion:
Convert the root of the release repository, not the lm/ subdirectory. The mmproj holds the codebook heads, the predefined voices, the speaker encoder and the keep2_plain codec decoder, it is best kept in F32:
python convert_hf_to_gguf.py path/to/ZONOS3-Mini --outtype q8_0 --outfile zonos3-mini.gguf
python convert_hf_to_gguf.py path/to/ZONOS3-Mini --mmproj --outtype f32 --outfile mmproj-zonos3-mini.gguf
Fish Audio S2 Pro
Fish Audio S2 Pro: a Qwen3 backbone (4B) samples each frame's semantic token, a 4-layer fast AR predicts the 9 other codes, and the Fish DAC (same as Drama 3) decodes them to 44.1 kHz audio.
Available params:
- Speaker tags
<|speaker:0|>,<|speaker:1|>, ... in the text give each character its own voice, a text without a tag starts with<|speaker:0|> - Inline instructions in brackets control the delivery, for example
[laughing],[whisper]or[pause] - The language is the one of the text,
--tts-langis not used - Voice cloning:
--tts-speaker-filewith the reference clip, optionally--tts-speaker-textwith its transcript (it is put in the prompt as the reference does). The clip is encoded by the codec encoder half ofcodec.pth, it is truncated to the first 30 s - Note:
--temp,--top-kand--top-pare not used (only--seed), frames are sampled with the settings of the reference (semantic token: repetition penalty 1.1 over the last 16 frames, top-k 30, top-p 0.8, temperature 0.8, or top-p 0.9 and temperature 1.0 when the last 4 frames repeat a token; other codes: greedy) - Each frame is 2048 samples (about 46 ms), generation stops at the end-of-speech token
Example usage:
llama-tts -m s2-pro.gguf \
-mm mmproj-s2-pro.gguf \
-p "<|speaker:0|>Did you remember to buy the bread? <|speaker:1|>Of course I did [laughing] it is on the kitchen table." \
-n 1000 \
--output out.wav
Voice cloning (--tts-speaker-text can be left out):
llama-tts -m s2-pro.gguf \
-mm mmproj-s2-pro.gguf \
--tts-speaker-file reference.wav \
--tts-speaker-text "The words spoken in reference.wav." \
-p "Hello, this is a cloned voice." \
--output out.wav
Note for GGUF conversion:
The mmproj holds the fast AR and codec.pth (the encoder half for voice cloning, the decoder half for the audio), it is best kept in BF16. The codec encoder is always stored in F16/F32, small errors in it change the codes picked by its VQ:
python convert_hf_to_gguf.py path/to/s2-pro --outtype q8_0 --outfile s2-pro.gguf
python convert_hf_to_gguf.py path/to/s2-pro --mmproj --outtype bf16 --outfile mmproj-s2-pro.gguf
YuE2
m-a-p/YuE2-3B generates songs (vocals and accompaniment) from a style prompt and lyrics, as 48 kHz stereo. The backbone (a Qwen3 text model) writes an ABC score, then one semantic code per 40 ms frame; the NAR half of the checkpoint turns the codes into VAE latents by flow matching, attending to the keys and values of the backbone, and the YuE2-Vae decoder makes the audio.
The prompt is the text of the reference request:
[Tags]
City Pop, upbeat, danceable, groovy bass, electric guitar, synth
[Lyrics]
[Verse]
...
Available params:
--tts-voicepicks the planning mode of the reference (cot):full(default, a chord-annotated score),melody(a melody-only score) oroff(no score)- Cover mode: append the score to the prompt as
\n[ABC]\n<score>, with--tts-voice melodyorfull; the model then goes straight to the codes - The generation stops at the limits of the reference (4096 score tokens, then 9000 codes, 6 minutes);
-nsets a lower limit on the score tokens and the codes together - The context must keep the default size (24576 positions, the one of the reference): a smaller
-cis rejected when the model is loaded - Note:
--top-kand--top-pare not used, both phases are sampled with the settings of the reference (yue2_generation_config.json);--tempchanges the temperature of the codes - Note: the backbone context must use an F16 KV cache (default): the NAR pass loads its keys and values
- The codes are sampled with classifier-free guidance like the reference: scale 1.01 with
--tts-voice off, off (1.0) otherwise.llama-servertakescfg_scaleto change it, the reference suggests 1.2 for a stronger text guidance. The uncond prompt runs as a second sequence, so the context must also hold it (the instruction, the score and the codes once more) - The NAR pass solves the flow-matching ODE in 32 midpoint steps like the reference;
llama-servertakesflow_stepsto change it. The NAR time is about proportional to it: 16 steps are about 35% faster for the NAR and the VAE together, with a small deviation from the reference (about 0.2 dB of spectral difference); it grows with fewer steps (about 0.4 dB at 8, 0.8 dB at 4) - The NAR pass runs on chunks of codes like the reference: one chunk for a song that fits in 24576 positions, so the audio arrives after the last code. When streaming, each 30 s of codes is a chunk, processed during the generation, so the audio arrives in 30 s steps; the chunks are independent (each sees the prompt, the score and its own codes), which may be audible at the boundaries
Example usage:
llama-tts -m yue2-3b.gguf \
-mm mmproj-yue2-3b.gguf \
-f song.txt \
--output song.wav
Note for GGUF conversion:
The VAE is a separate repository, download it into the vae subdirectory of the model:
hf download m-a-p/YuE2-3B --local-dir YuE2-3B
hf download m-a-p/YuE2-Vae --local-dir YuE2-3B/vae
python convert_hf_to_gguf.py YuE2-3B --outfile yue2-3b.gguf
python convert_hf_to_gguf.py YuE2-3B --mmproj --outfile mmproj-yue2-3b.gguf
The mmproj holds the NAR transformer layers (loaded with libllama as a Qwen3 text model, in the --outtype type), and the latent projections and the VAE decoder, always in F32.