* mtmd: drama3 voice cloning The Fish S2 codec encoder (fish_s2_spkenc) now also serves drama3: - conversion: drama3 writes the codec encoder and the audio encoder KV, shared with fish_s2 - fish_s2_spkenc: outputs the codes of the clip (gen-side reference) and, in a drama3_gen mmproj, builds the drama3 backbone rows (E_lm[semantic] + E_cb[1][code 1]) / sqrt(2) - drama3_gen tower: teacher forcing when the caller gives the codes of the next frame - drama3 pipeline: reference-bracket-v1 prompt with one reference (vLLM-Omni fish_omni), the reference frames go through the backbone as codec rows and through the tower with their codes - mtmd_gen_state_get_ref_codes(): read-only access to the reference codes of a gen state Claude-Session: https://claude.ai/code/session_01PpX3DmaxNSmJbxGaxnUBrX * fix correctness * clean up * rm redundant code * fish_s2: rm unused import --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
29 KiB
Text-to-speech (TTS)
llama.cpp supports audio generation via libmtmd. The following tools can be used:
- llama-tts, see its README for common usage and params
- llama-server via the
/ttsendpoint and the OpenAI-compatible/v1/audio/speechAPI
Supported models:
- Qwen3-TTS
- Pocket TTS
- Soprano
- LFM3-Audio
- Chatterbox
- Drama 3
- ZONOS3
- Fish Audio S2 Pro
- YuE2 (songs)
- MiniMax Music 3 (songs)
- Kokoro
- tiny-voca
- VoxCPM2
The sections below list the params supported by each model, with llama-tts examples. In llama-server, the same options are set with --tts-lang and --tts-voice, or per request (see the /tts endpoint).
Qwen3-TTS
Available params:
--tts-langcan bezh,en,de,it,pt,es,ja,ko,fr,ru(default:en)--tts-speaker-fileshould point to a speaker reference audio file (wav, mp3)--tts-speaker-textis the transcript of the speaker file. With it, the speech continues from the reference codes (the "ICL" mode of the reference), which clones the voice more closely. It needs a clip of 30 s or less and an mmproj converted with the speech tokenizer encoder, otherwise it falls back to the speaker embedding only
Example usage:
llama-tts -hf ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF \
-p "Hello world" \
--tts-lang english \
--tts-speaker-file speaker.mp3 \
--output out.wav
Voice cloning with the transcript of the reference:
llama-tts -hf ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF \
-p "Hello world" \
--tts-speaker-file speaker.wav \
--tts-speaker-text "The words spoken in speaker.wav." \
--output out.wav
Pocket TTS
Available params:
--tts-speaker-fileshould point to a speaker reference audio file (wav, mp3). It is required, the model produces almost no audio without it--tts-voiceselects one of the predefined voices shipped with the weights (e.g.giovanni), when the GGUF holds them. The converter reads them fromembeddings/*.safetensorsnext to the weights, of both repositories, and logs how closely each one is reproduced (some packs ship voice states computed with other weights, the converter warns above 5% and skips a voice only when it does not belong to the model at all)- Note:
langis not used, the language is a property of the weights
Example usage:
llama-tts -m pocket-tts.gguf \
-mm mmproj-pocket-tts.gguf \
-p "Hello world" \
--tts-speaker-file speaker.mp3 \
--output out.wav
The weights without voice cloning ship a set of predefined voices (embeddings/*.safetensors) next to each language; the converter stores them in the mmproj, as the voice prompt embeddings, and leaves out the zeroed voice encoder. With such a model --tts-speaker-file is rejected and a voice is selected with --tts-voice (the first one when it is not given):
llama-tts -m pocket-tts.gguf \
-mm mmproj-pocket-tts.gguf \
-p "Ciao a tutti" \
--tts-voice giovanni \
--output out.wav
Note for GGUF conversion:
The upstream repository holds one complete model per language under languages/, next to a set of shared files at the root. Convert one of the languages/<name> directories, not the root directory:
python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --outfile pocket-tts.gguf
python convert_hf_to_gguf.py path/to/pocket-tts/languages/english --mmproj --outfile mmproj-pocket-tts.gguf
Soprano
Available params:
--tts-speaker-fileis not supported; the model uses a fixed voice- Note:
langis not used. Supply normalized English text
Example usage:
llama-tts -m soprano.gguf \
-mm mmproj-soprano.gguf \
-p "Hello world" \
--temp 0 \
--output out.wav
Note for GGUF conversion:
Both Soprano-80M and Soprano-1.1-80M are supported. Convert both files from the same model directory, including decoder.pth:
python convert_hf_to_gguf.py path/to/Soprano-1.1-80M --outtype f32 --outfile soprano.gguf
python convert_hf_to_gguf.py path/to/Soprano-1.1-80M --mmproj --outtype f16 --outfile mmproj-soprano.gguf
LFM3-Audio
Available params:
--tts-langcan bear,da,de,el,en,en-gb,es,fi,fr,he,it,ja,ko,nl,pl,pt,sv,tr,zh(default:en), other values are used as the language name--tts-voicecan befemale(default) ormale- Note:
--tts-speaker-fileis not used, the model has no voice cloning - Note:
--top-kand--top-pare not used, audio frames are sampled with the settings of the model card (LFM3-Audio: temperature 1.0, top-k 128, repetition penalty 1.1 over 4096 frames; LFM3-Omni: temperature 0.4, repetition penalty 1.2 over 64 frames), theaudio_*keys ofgeneration_config.jsonoverride them at conversion - Each frame is 80 ms, generation stops at the end-of-audio frame
Example usage:
llama-tts -m lfm3-audio.gguf \
-mm mmproj-lfm3-audio.gguf \
-p "Hello world" \
--tts-voice male \
--output out.wav
The mmproj holds the audio encoder (for ASR), the depthformer and the codec decoder, the latter two are exported when the checkpoint has an audio_tokenizer directory:
python convert_hf_to_gguf.py path/to/lfm3-audio --outfile lfm3-audio.gguf
python convert_hf_to_gguf.py path/to/lfm3-audio --mmproj --outfile mmproj-lfm3-audio.gguf
Chatterbox
Two variants are supported: Chatterbox Turbo (English) and Chatterbox Multilingual V3 (23 languages).
Available params:
--tts-speaker-fileshould point to a speaker reference audio file (wav, mp3). Optional, the predefined voice is used without it--tts-voicecan bedefault, the voice shipped with the model (conds.pt)--tts-langsets the language of the multilingual variant, for examplefr(default: none). It is not used by Turbo
Example usage:
llama-tts -m chatterbox-mtl.gguf \
-mm mmproj-chatterbox-mtl.gguf \
-p "Bonjour tout le monde" \
--tts-lang fr \
--tts-speaker-file speaker.mp3 \
--output out.wav
Note: the multilingual variant runs classifier-free guidance with a second sequence. llama-tts sets this up by itself, but llama-server does not support it yet.
Note for GGUF conversion:
The upstream repositories ship raw checkpoints without config.json. Download the files below into a directory, then convert it as usual (the variant is detected from the talker file):
- Turbo:
t3_turbo_v1.safetensors,s3gen_meanflow.safetensors,ve.safetensors,conds.ptand the GPT-2 tokenizer files (vocab.json,merges.txt,added_tokens.json,special_tokens_map.json,tokenizer_config.json) - Multilingual V3:
t3_mtl23ls_v3.safetensors,s3gen_v3.safetensors,ve.safetensors,conds.pt,mtl_tokenizer.json
python convert_hf_to_gguf.py path/to/chatterbox --outfile chatterbox.gguf
python convert_hf_to_gguf.py path/to/chatterbox --mmproj --outfile mmproj-chatterbox.gguf
Drama 3
Fish Audio Drama 3: a Qwen3.5-MoE backbone (35B, 3B active) samples each frame's main token, a codebook tower samples the 9 other codes, and the Fish DAC decodes them to 44.1 kHz audio.
Available params:
- Speaker tags
<|speaker:0|>to<|speaker:15|>in the text give each character its own voice, a text without a tag starts with<|speaker:0|> - The language is the one of the text,
--tts-langis not used - Voice cloning:
--tts-speaker-filewith the reference clip and--tts-speaker-textwith its transcript (a transcript without a speaker tag is speaker 0). The clip is encoded by the codec encoder half ofcodec.pthand its codes are put in the prompt as one reference, as the reference does; the clip is truncated to the first 30 s. It needs an mmproj converted with the codec encoder - Note:
--top-kand--top-pare not used, frames are sampled with the settings of the reference (main token: temperature 1.0, top-p 0.9, repetition-aware redraw over the last 16 frames; other codes: temperature 1.0, top-k 30, top-p 0.9) - Each frame is 2048 samples (about 46 ms), generation stops at the end-of-speech token
- The codebook tower keeps the whole sequence in its KV cache, the prompt and the generated frames are limited to 4096 positions
Example usage:
llama-tts -m drama3.gguf \
-mm mmproj-drama3.gguf \
-p "<|speaker:0|> Did you remember to buy the bread? <|speaker:1|> Of course I did, it is on the kitchen table." \
-n 1000 \
--output out.wav
Note for GGUF conversion:
The routed experts of the checkpoint are FP8, they are dequantized at conversion. The mmproj holds the codebook tower and codec.pth (the encoder half for voice cloning, the decoder half for the audio), it is best kept in BF16; the codec encoder is always stored in F16/F32:
python convert_hf_to_gguf.py path/to/drama-3 --outtype q8_0 --outfile drama3.gguf
python convert_hf_to_gguf.py path/to/drama-3 --mmproj --outtype bf16 --outfile mmproj-drama3.gguf
ZONOS3
Zyphra ZONOS3 Mini: a hybrid Gated DeltaNet / attention MoE backbone (800M, 100M active) predicts the 10 codebooks of each frame with an inter-codebook delay, and the ZAC codec decodes them to 44.1 kHz audio.
Available params:
--tts-voicepicks one of the predefined voices of the release (Savannah_en-American_F_1by default);llama-serverlists them inGET /props. A voice is the prompt prefix of its reference codes, the default speaker mode of the reference- The language is the one of the text;
--tts-lang(langinllama-server) only selects the text normalization - Numbers are spelled out by the
ttschat template of the GGUF: cardinals, decimals, money, percents, clock times, common ordinal forms (1st,1er,1º), English and German years. A 4-digit number is read as a year only after a word likein,since, a month name, orim Jahr,seitin German (in 1990-> "in nineteen ninety",1234 people-> "one thousand two hundred thirty-four people"). Languages:en,fr,de,es,it,pt,jaandzh(orcmn), the region is ignored (fr_fr,pt_br); English when the language is empty, other languages are left as written. Set the language to the one of the text: the separators differ, e.g.1.234is a thousand in German - Note: the reference uses NeMo text normalization, which also covers units, dates, abbreviations and Korean; supply normalized text for them
--tts-speaker-fileclones the voice of a clip (the first 60 s are used) with the speaker encoder of the release, the "embedding" speaker mode of the referencellama-serveralso takesspeed(relative to 15 UTF-8 bytes per second) andloudness(LUFS), see the/ttsendpoint; they pick the speaking-rate and loudness condition buckets of the checkpoint, the defaults leave them unknown- Note:
loudnesshas a strong effect with a cloned voice; a predefined voice mostly keeps the loudness of its reference clip - Note:
--top-kand--top-pare not used, frames are sampled with the settings of the reference: temperature 0.75 (--tempchanges it), top-k 80, top-p 0.69, repetition penalty 2.4 over the last 96 frames of codebooks 0 to 5 - Each frame is 2940 samples (about 67 ms), generation stops at the end-of-audio code
Example usage:
llama-tts -m zonos3-mini.gguf \
-mm mmproj-zonos3-mini.gguf \
-p "The old lighthouse keeper climbed the spiral stairs every evening." \
--tts-voice Imogen_en-British_F_1 \
--output out.wav
Note for GGUF conversion:
Convert the root of the release repository, not the lm/ subdirectory. The mmproj holds the codebook heads, the predefined voices, the speaker encoder and the keep2_plain codec decoder, it is best kept in F32:
python convert_hf_to_gguf.py path/to/ZONOS3-Mini --outtype q8_0 --outfile zonos3-mini.gguf
python convert_hf_to_gguf.py path/to/ZONOS3-Mini --mmproj --outtype f32 --outfile mmproj-zonos3-mini.gguf
Fish Audio S2 Pro
Fish Audio S2 Pro: a Qwen3 backbone (4B) samples each frame's semantic token, a 4-layer fast AR predicts the 9 other codes, and the Fish DAC (same as Drama 3) decodes them to 44.1 kHz audio.
Available params:
- Speaker tags
<|speaker:0|>,<|speaker:1|>, ... in the text give each character its own voice, a text without a tag starts with<|speaker:0|> - Inline instructions in brackets control the delivery, for example
[laughing],[whisper]or[pause] - The language is the one of the text,
--tts-langis not used - Voice cloning:
--tts-speaker-filewith the reference clip, optionally--tts-speaker-textwith its transcript (it is put in the prompt as the reference does). The clip is encoded by the codec encoder half ofcodec.pth, it is truncated to the first 30 s - Note:
--temp,--top-kand--top-pare not used (only--seed), frames are sampled with the settings of the reference (semantic token: repetition penalty 1.1 over the last 16 frames, top-k 30, top-p 0.8, temperature 0.8, or top-p 0.9 and temperature 1.0 when the last 4 frames repeat a token; other codes: greedy) - Each frame is 2048 samples (about 46 ms), generation stops at the end-of-speech token
Example usage:
llama-tts -m s2-pro.gguf \
-mm mmproj-s2-pro.gguf \
-p "<|speaker:0|>Did you remember to buy the bread? <|speaker:1|>Of course I did [laughing] it is on the kitchen table." \
-n 1000 \
--output out.wav
Voice cloning (--tts-speaker-text can be left out):
llama-tts -m s2-pro.gguf \
-mm mmproj-s2-pro.gguf \
--tts-speaker-file reference.wav \
--tts-speaker-text "The words spoken in reference.wav." \
-p "Hello, this is a cloned voice." \
--output out.wav
Note for GGUF conversion:
The mmproj holds the fast AR and codec.pth (the encoder half for voice cloning, the decoder half for the audio), it is best kept in BF16. The codec encoder is always stored in F16/F32, small errors in it change the codes picked by its VQ:
python convert_hf_to_gguf.py path/to/s2-pro --outtype q8_0 --outfile s2-pro.gguf
python convert_hf_to_gguf.py path/to/s2-pro --mmproj --outtype bf16 --outfile mmproj-s2-pro.gguf
YuE2
m-a-p/YuE2-3B generates songs (vocals and accompaniment) from a style prompt and lyrics, as 48 kHz stereo. The backbone (a Qwen3 text model) writes an ABC score, then one semantic code per 40 ms frame; the NAR half of the checkpoint turns the codes into VAE latents by flow matching, attending to the keys and values of the backbone, and the YuE2-Vae decoder makes the audio.
The prompt is the text of the reference request:
[Tags]
City Pop, upbeat, danceable, groovy bass, electric guitar, synth
[Lyrics]
[Verse]
...
Available params:
--tts-voicepicks the planning mode of the reference (cot):full(default, a chord-annotated score),melody(a melody-only score) oroff(no score)- Cover mode: append the score to the prompt as
\n[ABC]\n<score>, with--tts-voice melodyorfull; the model then goes straight to the codes - The generation stops at the limits of the reference (4096 score tokens, then 9000 codes, 6 minutes);
-nsets a lower limit on the score tokens and the codes together - The context must keep the default size (24576 positions, the one of the reference): a smaller
-cis rejected when the model is loaded - Note:
--top-kand--top-pare not used, both phases are sampled with the settings of the reference (yue2_generation_config.json);--tempchanges the temperature of the codes - Note: the backbone context must use an F16 KV cache (default): the NAR pass loads its keys and values
- The codes are sampled with classifier-free guidance like the reference: scale 1.01 with
--tts-voice off, off (1.0) otherwise.llama-servertakescfg_scaleto change it, the reference suggests 1.2 for a stronger text guidance. The uncond prompt runs as a second sequence, so the context must also hold it (the instruction, the score and the codes once more) - The NAR pass solves the flow-matching ODE in 32 midpoint steps like the reference;
llama-servertakesflow_stepsto change it. The NAR time is about proportional to it: 16 steps are about 35% faster for the NAR and the VAE together, with a small deviation from the reference (about 0.2 dB of spectral difference); it grows with fewer steps (about 0.4 dB at 8, 0.8 dB at 4) - The NAR pass runs on chunks of codes like the reference: one chunk for a song that fits in 24576 positions, so the audio arrives after the last code. When streaming, each 30 s of codes is a chunk, processed during the generation, so the audio arrives in 30 s steps; the chunks are independent (each sees the prompt, the score and its own codes), which may be audible at the boundaries
Example usage:
llama-tts -m yue2-3b.gguf \
-mm mmproj-yue2-3b.gguf \
-f song.txt \
--output song.wav
Note for GGUF conversion:
The VAE is a separate repository, download it into the vae subdirectory of the model:
hf download m-a-p/YuE2-3B --local-dir YuE2-3B
hf download m-a-p/YuE2-Vae --local-dir YuE2-3B/vae
python convert_hf_to_gguf.py YuE2-3B --outfile yue2-3b.gguf
python convert_hf_to_gguf.py YuE2-3B --mmproj --outfile mmproj-yue2-3b.gguf
The mmproj holds the NAR transformer layers (loaded with libllama as a Qwen3 text model, in the --outtype type), and the latent projections and the VAE decoder, always in F32.
MiniMax Music 3
MiniMaxAI/MiniMax-Music3 generates songs (vocals and accompaniment) from a music description and lyrics, as 44.1 kHz stereo. For each 40 ms frame, the backbone (a Qwen3 text model) samples the semantic code and the RVQ depth decoder the 7 other codes; the hidden states of both go to a flow-matching DiT, then a DAC decoder makes the audio. Both use the settings of the reference (the diffusers pipeline).
The prompt has the same format as YuE2, the description goes under [Tags]:
[Tags]
Genre: acoustic pop. BPM: 96. Key: C major. Vocals: soft female lead. Arrangement: fingerpicked guitar and soft piano.
[Lyrics]
[verse]
Morning light filtering through the pine
[chorus]
Softly the world begins to breathe
Available params:
- The description and the lyrics are cleaned up like the reference: markdown is removed, a line that starts with structure tags (e.g.
[Verse]) keeps only the tags - The generation stops at the end-of-audio token or at the limit of the reference (9000 frames, 6 minutes);
-nsets a lower limit in frames (25 per second) --tempchanges the temperature of the codes (default 1.0);--top-kand--top-pare not used, the codes are sampled with top-k 50 like the reference- The codes are sampled with classifier-free guidance, scale 1.5 like the reference;
llama-servertakescfg_scaleto change it (1.0 disables it). The uncond prompt runs as a second sequence that shares the context, so a song needs about 2 x (prompt tokens + frames) positions: the default context (10240) holds about 3 minutes with a short description, use a larger-c(e.g. 20480) for longer songs - The flow matching runs 30 Euler steps like the reference;
llama-servertakesflow_stepsto change it - The audio is made in windows of 200 frames (8 s) every 100 frames, like the reference: a window is processed during the generation once the frame after it exists, so with streaming the audio arrives in steps of about 4 s
Example usage:
llama-tts -m minimax-music3.gguf \
-mm mmproj-minimax-music3.gguf \
-f song.txt \
--output song.wav
Note for GGUF conversion:
The conversion reads the diffusers layout of the repository, the qwen_7B directory and the .pth files are not needed:
hf download MiniMaxAI/MiniMax-Music3 --local-dir MiniMax-Music3 --exclude "qwen_7B/*" --exclude "*.pth"
python convert_hf_to_gguf.py MiniMax-Music3 --outfile minimax-music3.gguf
python convert_hf_to_gguf.py MiniMax-Music3 --mmproj --outfile mmproj-minimax-music3.gguf
The mmproj holds the depth decoder and the DiT layers (in the --outtype type), and the condition encoder, the small DiT projections and the DAC decoder, always in F32.
Kokoro
Available params:
--tts-voiceselects one of the voices shipped with the weights (default:af_heart);llama-ttslists them when the name is unknown. The first letter of a voice is its language:aAmerican English,bBritish English,eSpanish,fFrench,hHindi,iItalian,pBrazilian Portuguese,jJapanese,zMandarin--tts-langcan been-usoren-gb; when it is not given, the voices namedb*speak British English and those nameda*American English- Note:
--tts-speaker-fileis not supported, the model has no voice cloning - Note: sampling params are not used, the model is not autoregressive.
--seedonly changes the noise of the vocoder source speed(llama-serveronly) changes the speaking rate, e.g.1.5;loudnessis not supported
Example usage:
llama-tts -m kokoro.gguf \
-mm mmproj-kokoro.gguf \
-p "Hello world" \
--tts-voice am_michael \
--output out.wav
The model speaks phonemes. Only English has a text to phonemes step (the model's tts chat template, with the pronunciation dictionary stored in the model):
- the pronunciation of a word missing from the dictionary (mostly rare names) is predicted from its spelling, by a model that the converter learns from the dictionary. The reference uses espeak-ng for these words; the prediction is about as accurate, but it is not the same, so such a word can sound different from the reference
- a text longer than 510 phonemes is split at sentence ends and spoken segment by segment
Other languages, or your own phonemes:
The prompt of a voice of another language must already be phonemes, in the symbols of the model (see vocab in its config.json). Run the text to phonemes step in your own code, for example with espeak-ng, which the reference uses for Spanish, French, Hindi, Italian and Brazilian Portuguese:
PHONEMES=$(echo "El gobierno anunció nuevas medidas." | espeak-ng -q --ipa -v es --stdin | tr '\n' ' ')
llama-tts -m kokoro.gguf \
-mm mmproj-kokoro.gguf \
-p "$PHONEMES" \
--tts-voice ef_dora \
--output out.wav
Notes:
- espeak-ng drops the punctuation, which the model uses for pauses: phonemize the text between the marks and put them back for better prosody
- the reference also replaces a few sounds by single symbols (e.g.
tʃbyʧ), seeespeak.pyof misaki - with
llama-server, send"raw_prompt": trueto pass phonemes with any voice, including the English ones
Note for GGUF conversion:
Download config.json, kokoro-v1_0.pth and the voices/*.pt files you want from Kokoro-82M, every voice in the directory is stored in the mmproj:
python convert_hf_to_gguf.py path/to/Kokoro-82M --outfile kokoro.gguf
python convert_hf_to_gguf.py path/to/Kokoro-82M --mmproj --outfile mmproj-kokoro.gguf
The pronunciation dictionary and the part-of-speech tagger of the tts template come from the misaki and nltk python packages. The converter also learns the spelling-to-sound model for unknown words from that dictionary, which adds about 15 seconds:
pip install misaki nltk
python -m nltk.downloader averaged_perceptron_tagger_eng
Without them the converter writes no tts template, and the prompt must be a phoneme string (as produced by misaki).
tiny-voca
tiny-voca: a 4.5M-parameter English TTS model. A phoneme encoder (the text model), then a duration predictor, two frame transformers and a source-filter synthesizer (in the mmproj).
Available params:
--tts-voicecan beM2(default) orF2- Note:
--tts-speaker-fileis not supported, the model has no voice cloning - Note: sampling params are not used, the model is not autoregressive.
--seedonly changes the noise of the synthesizer speed(llama-serveronly) changes the speaking rate, from0.25to4;loudnessis not supported
Example usage:
llama-tts -m tiny-voca.gguf \
-mm mmproj-tiny-voca.gguf \
-p "Hello! The meeting moved to 3:30 pm." \
--tts-voice F2 \
--output out.wav
The model speaks phonemes. The text to phonemes step is the model's tts chat template, a port of dizione (the reference one), with its pronunciation dictionary and tables stored in the model:
- the choice of a pronunciation for each word: exceptions and names, the dictionary, function words and their sentence stress, words spelled alike and read differently ("I read it yesterday", "to read"), hyphenated words, capitals read letter by letter, mixed case ("iPhone"), stretched words ("soooo"), accents, chat spellings ("lol", "btw")
- a word missing from the dictionary is built from a known word when it can be (endings, prefixes, compounds), else it goes through the unknown-word model of dizione (stored in the mmproj) and its conversion rules
- a pronunciation can be given in the text, in CMUdict notation or as phonemes:
Say {HH AH0 L OW1} to Samir{s ə m ˈɪɹ} - the text rules are ported too: numbers, money, times, dates, telephone numbers, versions, codes, units, fractions, ranges, math signs and comparisons, web addresses and file names, abbreviations, chat spellings
With llama-server, send "raw_prompt": true to pass phonemes directly, in the format printed by dizione: symbols separated by spaces, <sp> between words, a stress mark (ˈ or ˌ) before a stressed vowel, e.g. h ə l ˈoʊ <sp> w ˈɜː l d .
Each sentence is spoken as one utterance. The template splits the text as the reference does (before a capital or a figure, not after a title or an initial), ends each sentence with <eos>, and adds a full stop when the text ends without a mark. Phonemes without <eos> are cut after each ., ?, ! or …, and a sentence without a final mark gets a full stop.
Note for GGUF conversion:
Convert the root of the repository (it needs config.json, tiny-voca.pt and phonemizer/lexicon.tsv). The other tables of the tts template come from the dizione python package; without it the converter writes no tts template, and the prompt must be phonemes:
pip install git+https://github.com/n4ze3m/dizione
python convert_hf_to_gguf.py path/to/tiny-voca --outfile tiny-voca.gguf
python convert_hf_to_gguf.py path/to/tiny-voca --mmproj --outfile mmproj-tiny-voca.gguf
VoxCPM2
VoxCPM2: a MiniCPM4 backbone (2B) makes one hidden state per frame, without an audio token. The mmproj turns it into a patch of 4 continuous latents: FSQ, an 8-layer residual LM (run with libllama, one context per sequence), then the LocDiT flow matching. The causal AudioVAE decodes the latents to 48 kHz audio.
Available params:
- The language is the one of the text,
--tts-langis not used. A voice description in parentheses at the start of the text designs the voice, for example(A young woman, gentle and warm)Hello! - Voice cloning:
--tts-speaker-filewith the reference clip (resampled to 16 kHz, truncated to the first 30 s).--tts-speaker-textis not used, the transcript of the clip is not needed --tempscales the noise of the flow matching (default 1.0),--top-kand--top-pare not used- The flow matching uses 10 Euler steps and classifier-free guidance with scale 2.0, like the reference.
llama-servertakesflow_steps(2 to 32) andcfg_scaleto change them - Each frame is 4 latents, 7680 samples (160 ms). Generation stops on the stop head of the model, at least 4 frames, at most 6 frames per text token + 10 (the limit of the reference)
- Not supported yet: the continuation mode of the reference (prompt audio with its transcript)
- Not supported yet: the bad case retry of the reference (
retry_badcase, a new seed when the audio reaches 6 frames per text token). Short prompts may then run long and repeat some words, try another seed
Example usage:
llama-tts -m VoxCPM2.gguf \
-mm mmproj-VoxCPM2.gguf \
-p "(A calm male narrator)The quick brown fox jumps over the lazy dog." \
--output out.wav
Voice cloning:
llama-tts -m VoxCPM2.gguf \
-mm mmproj-VoxCPM2.gguf \
--tts-speaker-file reference.wav \
-p "Hello, this is a cloned voice." \
--output out.wav
Note for GGUF conversion:
The mmproj holds LocEnc, LocDiT and the residual LM layers (loaded with libllama as a Granite text model with RoPE off) in the --outtype type, and the AudioVAE (audiovae.pth, encoder for voice cloning, decoder for the audio) always in F32:
hf download openbmb/VoxCPM2 --local-dir VoxCPM2
python convert_hf_to_gguf.py VoxCPM2 --outfile VoxCPM2.gguf
python convert_hf_to_gguf.py VoxCPM2 --mmproj --outfile mmproj-VoxCPM2.gguf