* jinja: impl scope-chain key lookup
* add test for call_statement lambda
* chat: add tts_dict for kokoro G2P template
* more testing
* value_float_t range check
* no json-in-meta, add llama_model_meta_arr_str/_n
* mtmd: add kokoro tts support
* detailed guide about G2P and text norm port
* refine G2P
* clean up
* migrate batch api
* nits
* mtmd: support YuE2-3B
* improve speed
* add CFG
* fix streaming
* add cfg control from api
* check ctx size
* fix n_predict
* add flow_steps config
* nits
* fix batch api
* nits
* adapt
* mtmd: yue2 samples from its own saved logits
The context logits belong to the last decode of any slot, so with
several server slots the score and the unguided codes were sampled
from another sequence. Keep the logits of each own decode in
save_logits(), which also does the guidance mix.
* mtmd: let the yue2 snake chain fuse
The beta reshape landed between sqr and mul, so the mul, sin, sqr,
mul, add chain was not consecutive and the backends never fused it.
Expand both reshapes first, like qwen3tts does.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* mtmd, server : add temperature to /tts, default Pocket TTS to 0.3
Assisted-by: Claude Opus 5.5
* feat: add temp parameter in llama-tts
* temp: fix default hardcoded values
* fix: remove too narrow test
* fix: addressing PR comments
* feat: support PocketTTS voices with no voice cloning
The kyutai/pocket-tts-without-voice-cloning weights ship predefined voices as the
backbone KV state after the voice prompt. The converter turns each state back into
the prompt embeddings and stores them in the mmproj (voice.<name>), the zeroed voice
encoder is left out, and the "voice" field of /tts and --tts-voice select one of them.
A speaker reference is rejected when the model has no voice encoder.
Assisted-by: Claude Fable 5.1
* convert : pocket-tts voices never fail the conversion, log the drift of the state
The voice states shipped by Kyutai are not always the ones the pack's
weights give for the prompt: italian, german and spanish match within the
bf16 rounding, english_2026-04 is 2% off and french 20% (its layer 0 state
comes from other weights), while the reference loads them as is and they
still sound like the same voice. The converter now re-prompts the backbone
with the recovered embeddings, compares the whole state and logs the
distance per voice, warning above 5%. A state that does not belong to the
weights at all is skipped, so the packs of kyutai/pocket-tts convert too.
Assisted-by: Claude Fable 5.1
* mtmd : list the predefined voices, expose them in GET /props
mtmd_gen_audio_info gains n_voices and mtmd_gen_audio_voice_name() the
names, with the voices worded as a feature of any audio generation model.
The server lists them under "voices" in /props when the model ships some.
Assisted-by: Claude Fable 5.1
* server : an unknown voice is a bad request
A model with predefined voices answers 400 naming the voice instead of 500
"failed to process TTS prompt". A model without them ignores "voice" with a
warning, as the other audio generation models do.
Assisted-by: Claude Fable 5.1
* refactor api
* fix conversion
* small renaming
* fix lint
* tts : a model with predefined voices speaks with the first one by default
Without a voice prompt such a model only produces near silence. When neither
a voice nor a speaker reference is given, llama-tts and the server now take
the first predefined voice and say so in the log, as the LFM3-Audio template
does with its default voice.
Assisted-by: Claude Fable 5.1
* mtmd : one comment per voice accessor, converter states the rule only
The clip.h comment described clip_gen_get_voice but sat above all three
functions; the converter comment quoted the drift measured per pack, which
goes stale with the next upload of the weights.
Assisted-by: Claude Fable 5.1
* clean up redundant code and comments
* mtmd: match the LFM3-Audio voice name case-insensitively
The tts chat template lowers the voice name, so Male speaks with
the male voice; the voice lookup lowers it the same way and names the
bitmap after the canonical voice.
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Pascal <admin@serveurperso.com>
* model : add LiquidAI/LFM3-Audio-2B and LiquidAI/LFM3-Omni-2B
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Assisted-by: Claude Opus 5.5
* convert : read the LFM3-Audio config.json as released
A transformers that knows lfm3_audio rewrites the config into another layout (no lfm/encoder keys).
Assisted-by: Claude Opus 5.5
* mtmd : disable flash attention for the LFM3-Audio depthformer
Assisted-by: Claude Opus 5.5
* tts : add a voice option for LFM3-Audio and LFM3-Omni
The model card offers a female and a male voice; --tts-voice for llama-tts, "voice" for the server /tts.
Assisted-by: Claude Opus 5.5
* clean up template handling
* convert : detect LFM3-Audio and LFM3-Omni by model_type
Assisted-by: Claude Opus 5.5
* move banned tokens --> logits bias --> mtmd_helper
* leave mtmd-cli unchanged
* no more template_asr_system
* left over
* llama : keep the LFM3-Audio encoder convs and codec unquantized
Assisted-by: Claude Opus 5.5
* common, mtmd : shorten comments
Assisted-by: Claude Opus 5.5
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* mtmd: add audio out stream api
* add /tts endpoint
* add docs
* wire up mtmd_helper_model_can_chat
* upload speaker_ref via form-data
* expose seed to api
* fix header for pcm
* wip
* fix penalty_last_n
* fix server_output_limits
* fix pocket-tts stream
* wip
* add server_schema for tts
* add server routing
* server: coerce seed and stream from multipart form fields
Form fields arrive as strings, so -F seed=42 and -F stream=true were
rejected by the schema. seed joins the integer fields, parsed as a long
long so a full uint32 seed fits, and stream is read as a boolean when
it comes as a string.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* feat: add --mmproj-device arg & backwards compatible MTMD_BACKEND_DEVICE env var
* feat: load mmproj device backend immediately, add -mmdev shortflag
* fix: its a pointer now get the name
* clean up
* gen docs
* nits
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* adapt the api
* text model ok
* working impl, need verify and clean up
* mtmd: build the pocket-tts transposed convolutions as GEMM + col2im
ggml_conv_transpose_1d has no grouped mode, so the depthwise upsample
was built as one convolution and one concat per channel, which floods
the graph with small nodes and makes kernel launches dominate the
decoder.
Fold both cases into the column form the seanet decoder already needs:
the general case reshapes the kernel to [IC, K * OC] and matmuls it
with the input, the depthwise case batches a matmul over the channels
so a step scales its own kernel. A single col2im_1d then scatter-adds
the columns back to the signal, with the same shape as before, so the
overlap-add tail, the streaming state and the bias are untouched.
Generation time per frame drops by 80% on CUDA and by 50% on CPU. The
output matches the previous implementation sample for sample, with a
correlation of 0.999994 and identical frame counts.
* flow_temp + frames_after_eos
* chunking
* mtmd: carry the remaining pocket-tts per-pack settings
The language packs also tune the end-of-speech padding and the padding
of short prompts, next to the temperature already carried in the
mmproj: french_24l asks for 8 tail frames instead of the guessed 3,
english_2026-01 asks for short prompts to be padded with spaces.
Write both in the mmproj as clip.gen.audio.frames_after_eos and
clip.gen.audio.pad_short_text, keyed on the pack in the conversion
script like the temperature. The loader keeps them optional, so a
mmproj without them behaves as before. Map semicolons to commas for
every pack instead, the reference only asks for it on three of them and
it costs nothing elsewhere.
Existing mmproj files must be converted again to carry the two keys.
On a long french text the port now lands within 2% of the reference:
22.96s against 23.44s, with the same peak level and the same amount of
silence.
* clip.gen.audio.model_variant
* clean up code comments
* nit: drop the dead flow_temp hparam, the pack table holds the default
* update docs
* address security problems
* less invasive base.py
* lint
* add mtmd_gen_inp_default
* add docs
* rm gen_flow_temp
---------
Co-authored-by: Pascal <admin@serveurperso.com>
get_output runs the waveform work the pipeline defers to it, from a
single trailing window to a full pass depending on the model. Measuring
it keeps the reported total and the audio to process ratio honest.
The build info is now only for debug, so we avoid the duplicate
with `--version`.
The UTF-8 setup at the beginning is needed to avoid logging
garbage on Windows.
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Set C locale for consistent float formatting across all binaries.
* Add C locale setting to all tools binaries
Add std::setlocale(LC_NUMERIC, "C") to all 16 binaries in the tools/
directory to ensure consistent floating-point formatting.
* Apply suggestion from @JohannesGaessler
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* sampling : optimize sorting using bucket sort in more places
ggml-ci
* sampling : do not sort in dist sampler
ggml-ci
* sampling : avoid heap allocations for sort buffers
ggml-ci
* common : add option to sort sampling candidates by probability
ggml-ci
* sampling : revert the change for preserving sort buffers
* sampling : use std::copy instead of memcpy
* sampling : clarify purpose of partial sort helpers
ggml-ci
* cont : remove wrong comment [no ci]
* common : update comment
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>