* mtmd: support YuE2-3B * improve speed * add CFG * fix streaming * add cfg control from api * check ctx size * fix n_predict * add flow_steps config * nits * fix batch api * nits * adapt * mtmd: yue2 samples from its own saved logits The context logits belong to the last decode of any slot, so with several server slots the score and the unguided codes were sampled from another sequence. Keep the logits of each own decode in save_logits(), which also does the guidance mix. * mtmd: let the yue2 snake chain fuse The beta reshape landed between sqr and mul, so the mul, sin, sqr, mul, add chain was not consecutive and the backends never fused it. Expand both reshapes first, like qwen3tts does. --------- Co-authored-by: Pascal <admin@serveurperso.com>
12 KiB
libmtmd dev guide
History
Please refer to multimodal.md for a broader context.
In short:
libmtmdstarted as a wrapper aroundlibllava/clip.cpp- Various components that used to be in
clip.cppare moved progressively to mtmd. For example, preprocessor is now part of mtmd clip.cpphas since been split intomtmd-model(hparams and weights, onemtmd_modelper modality),mtmd-subcontext(backends and compute, onemtmd_subcontextper modality) andmtmd-graph(graph builders, seemodels/). Only the GGUF format keeps theclip.name (metadata keys and thecliparchitecture), for compatibility with existing mmproj files
Terminologies
- mtmd: MulTiMoDal
- bitmap: representing a raw input data, for example: RGB image, PCM audio
- tiles / slices: for llava-uhd-style models, the preprocessor breaks a large input into smaller square images called tiles or slices
- chunk: a mtmd_input_chunk represents a preprocessed input that can then be passed through
mtmd_encode()
Structure
Overview of the libmtmd structure:
mtmd_context(mtmd.cpp): the public handlemtmd_models: one mmproj file, loaded bymtmd_model_load()mtmd_modelv/a/gen_a(mtmd-model.h): one per modality, holds hparams, weights (ctx_data+buf) and requirements
mtmd_backends(mtmd-subcontext.h): GPU/CPU backend instances, CPU threadpool and one scheduler, shared by the subcontextscompute_mutex: held by every public function that runs a compute pass, so only one pass at a time runs on the backendsmtmd_subcontextctx_v/ctx_a/ctx_gen_a(mtmd-subcontext.h): one per model, holds a const ref to it and a ref to the backendsbuf_compute_metaand the resolved flash attn of its graphsmtmd_graph(mtmd-graph.h,models/): built for eachencode()call, temporary
mtmd_image_preprocessor/mtmd_audio_preprocessor(mtmd-image.h,mtmd-audio.h): read the hparams of their model
mtmd_gen_state/mtmd_audio_stream: public, one per sequence / streammtmd_subctx_gen_state/mtmd_subctx_enc_state: allocated on the backend of a subcontext, seegen_state_init()/enc_state_init()
Rules of thumb:
mtmd_modelis read-only once loaded. Anything that depends on the backend (flash attn resolution, warmup) belongs tomtmd_subcontext- Like the pp/tg graphs of
llama_context, the subcontexts share one scheduler: each warmup reserves its worst-case graph on it, so the compute buffers fit the largest one, and switching modality re-plans inside them - A model declares its runtime needs (e.g.
max_nodes,flash_attn_auto) inload_hparams, the subcontext applies them - Models and backends must outlive the subcontexts,
mtmd_contextdeclares them first for that reason - Models having both audio and vision encoders must have different arch names (projector types), postfixed by
aandv. See Gemma 4 as an example (gemma4a/gemma4v) - Locking happens only at the public API boundary (
mtmd.cpp): a new public function that runs a graph must takecompute_mutex. The internals (subcontexts, preprocessors,mtmd_audio_log_mel()) assume it is held and never lock - No magic numbers for a model's fixed shapes (layer counts, codebook sizes, window lengths...): define them once as
static constexprin the graph struct inmodels/models.h, and refer to them from the loader or the subcontext (e.g.mtmd_graph_zonos3_gen::n_codebooks,mtmd_graph_fish_s2_spkenc::n_tfm_layer)
Pipeline
A typical pipeline of the core libmtmd is as follows:
- A bitmap (RGB image or PCM audio) is created
- Bitmap and the text prompt is provided to
mtmd_tokenize()that breaks the input into chunks- Alternatively,
mtmd_tokenize_from_parts()takes a list of pre-split text/media parts instead of a marker-based prompt - The tokenizer function first expands a "lazy" bitmap if it finds one. Typically, this is used by video, so that one media token corresponds to one input bitmap
- For models that support "fused" temporal frames like Qwen-VL, the tokenizer tries to merge pair of consecutive frames into one batch. Only bitmaps marked by
mtmd_bitmap_set_mergeable()are merged - The preprocessor will then be called, which produces a list of chunks
- Depending on the model itself, special tokens will be injected to separate image chunks (i.e. llava-uhd-style models)
- Alternatively,
- Multiple bitmaps may be batched together to form a larger
mtmd_batch() - Single image or batch is encoded, via
mtmd_encode()ormtmd_batch_encode() - Get the output embeddings
Helper
We provide a set of helper functions via mtmd_helper to make using libmtmd easier. The helper provides:
- Image, audio and video file decoding (for example, decode raw JPEG into RGB bitmap)
- Manage
llama_batchand calls tollama_decode
Audio generation support
Audio generation is added to mtmd in PR #26254
Currently, we support the 3-stage pipeline below which should cover most TTS models:
- Stage 1: Backbone / Semantic Stage: Backbone model accepts text prompt and reference voice as input
- Stage 2: Acoustic Detail Generator: A model takes the hidden state from backbone and generate audio details (usually as audio codes or mel-spectrogram)
- Stage 3: Waveform Reconstruction: Convert the semantic and acoustic data from previous stages to the final waveform
For example, Qwen3-TTS:
- Reference voice is encoded using ECAPA-TDNN speaker encoder (
speaker_encoder) - Text prompt and reference voice are processed via a backbone (
talker.model) - A model converts sampled semantic token and hidden state from stage 2 into a list of 15 acoustic codes (
talker.code_predictor) - 16 generated codes are converted into waveform (
code2wav)
API design constraints
Due to wide variety of audio generation pipelines, the mtmd_gen_audio system is designed to be flexible and reusable by new models.
mtmd_gen_audio is split into 2 main API:
- Core API
mtmd.h: handles main inference. Important:mtmd_gen_stateholds only the per-sequence data that the mmproj itself writes and reads back (RNG, decoder KV / conv state, output buffers). Data that goes to or comes from the backbone (hidden states, codes, PCM) must stay explicit inmtmd_gen_inp/mtmd_gen_out; the caller drives the backbone and handles audio frame accumulation. - Helper API
mtmd-helper.h: provides a model-agnostic stateful API. Usage example can be found in thetools/ttsdirectory.
Running a transformer stage on libllama (mtmd-owned llama_context)
Some stages are a full LLM-sized transformer that attends to the keys/values of the backbone, for example the flow-matching NAR of YuE2 (same layer shapes as the backbone, its own weights, bidirectional over its rows, and it reads the backbone KV). Do not re-implement it as an mtmd graph. Instead, mtmd can own a second llama_model + llama_context (see mtmd_yue2_nar_load() and mtmd_gen_flow_yue2() in mtmd.cpp):
- Conversion: store the layers in the mmproj under
a.gen.<stage>.with the libllama tensor names of an existing arch (e.g.a.gen.nar.blk.0.attn_q.weight) - Loading: build a synthetic
gguf_contextwith the arch hparams (general.architecture,tokenizer.ggml.model = none,vocab_size = 1, ...) and one tensor info per mmproj tensor (only the type is used, the shapes come from the hparams), then callllama_model_init_from_user(). Theset_tensor_datacallback reads the data from the mmproj file, so no copy is kept in the mtmd model- libllama also creates every optional tensor of the arch (biases,
*.scale,*.input_scale,dense_2, ...) for a model built from metadata. The callback must fill them with values that leave the graph unchanged: 0 for biases, 1 for scales, identity fordense_2. Zeros everywhere silently make every matmul output 0
- libllama also creates every optional tensor of the arch (biases,
- Backbone keys/values: the helper passes the backbone sequence state (
llama_state_seq_get_data()) throughmtmd_gen_inp, mtmd loads it into its context withllama_state_seq_set_data(). This only checks the layer count and the K/V types and row sizes, so it works across models with the same KV layout. The two contexts must agree on the KV type and on flash attention (v_trans) - Inputs and outputs are embedding rows (
llama_batch::embd,llama_get_embeddings()withLLAMA_POOLING_TYPE_NONE), the projections around them stay small mtmd graphs - Non-causal rows:
LLAMA_ATTENTION_TYPE_NON_CAUSAL, all rows in one ubatch (n_ubatch>= rows), they see every cached position of the sequence. Remove them withllama_memory_seq_rm()after each pass - Size the context per call (
n_ctx,n_ubatch) instead of keeping one for the whole session
Checklist for porting new audio generation models to mtmd
- Make sure to consult merged PRs about adding new TTS models, especially reviewer comments
- Establish a list of reusable and missing components from the current mtmd implementation.
- For GGUF conversion:
- Backbone model should be converted to a normal text model (loadable via
libllama)- If model used hard-coded embedding row ID, append them to token embeddings and assign token name for them (see
qwen3tts.py) - If model have a specific output logits head for audio codes (usually semantic code), keep the head as-is and pad the logits at inference time (see
src/models/qwen3vl.cpp)
- If model used hard-coded embedding row ID, append them to token embeddings and assign token name for them (see
- Sidecar models (code2wav, bigvgan, etc) must live inside the mmproj GGUF (but can be in different
mtmd_subcontextif necessary)- Note: it should use
ggml_build_forward_selectto select graphs if multiple graphs living in the same context
- Note: it should use
- Reuse existing GGUF metadata key name and tensor name whenever possible; think twice before adding extensive changes to GGUF writer. For example, Qwen3-TTS hard-code part of the hparams in
mtmd-model.cppas they won't likely to change. - For tensor naming:
- Prefixed with
a.*for tensors used by speaker encoder pipeline - Prefixed with
a.gen.*for generation stages (code / mel-spectrogram / PCM generation)
- Prefixed with
- For GGUF metadata:
- Reuse as many existing keys as possible
- In most cases, you can hard-code model configs in the model graph class, or in
mtmd_hparams - If some values need to be exposed to the
mtmd_helperlayer, hard-code them inmtmd_helperand distinguish by pipeline andmtmd_gen_audio_info::model_variantif necessary - Do NOT add new GGUF metadata or new fields to
mtmd_gen_audio_infounless you can prove that you absolutely need them
- If the model requires special preprocessing of the input text prompt (prompt format, text normalization), put it in
tokenizer.chat_template.ttsas a jinja program. It receives the text asmessages[0].content, pluslanguageandvoice. Seeconversion/lfm3.py(prompt format) andmodels/templates/Zyphra-ZONOS3-Mini-tts.jinja(text normalization) as examples. For text normalization, usetts_norm_scan()/tts_norm_sub()(seedocs/tts-norm-patterns.md) and add test cases totests/test-jinja.cpp, which also checks them against the python reference with-py
- Backbone model should be converted to a normal text model (loadable via
- Make sure most of the changes happen inside
mtmd-helper-gen.cpp. A good PR looks like this:- 10-20% changes is to add new backbone (text) model and conversion
- 60% changes inside
mtmd-helper-gen.cpp - 10% changes inside
libmtmdcore (mtmd-model,mtmd-subcontext,models/) - The rest downstream code (CLI, server) should have no changes at all
- Validate with several concurrent requests on
llama-server(e.g.-np 2, one plain and one streaming request at the same time), not only withllama-tts- The server interleaves the slots on one
llama_context, so a pipeline must copy what it needs (hidden state, logits) right after its own decode, not read it at the next step
- The server interleaves the slots on one
- Update usage documentation in
docs/multimodal/tts.md
IMPORTANT: If your model needs changes that don't fit the existing infrastructure, open an issue first for discussion.
No-go checklist (these will get the PR rejected and require discussion before proceeding):
- Violating the API design constraints stated above
- Adding a new model-specific binary: the API and binary surface must stay model-agnostic
- Adding new flags to conversion script: this script must stay model-agnostic