Files
llama.cpp/tools/mtmd/README-dev.md
T
Xuan-Son NguyenandPascal 92192d31e4 mtmd: support YuE2-3B (#32369)
* mtmd: support YuE2-3B

* improve speed

* add CFG

* fix streaming

* add cfg control from api

* check ctx size

* fix n_predict

* add flow_steps config

* nits

* fix batch api

* nits

* adapt

* mtmd: yue2 samples from its own saved logits

The context logits belong to the last decode of any slot, so with
several server slots the score and the unguided codes were sampled
from another sequence. Keep the logits of each own decode in
save_logits(), which also does the guidance mix.

* mtmd: let the yue2 snake chain fuse

The beta reshape landed between sqr and mul, so the mul, sin, sqr,
mul, add chain was not consecutive and the backends never fused it.
Expand both reshapes first, like qwen3tts does.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-16 20:22:23 +01:00

12 KiB

libmtmd dev guide

History

Please refer to multimodal.md for a broader context.

In short:

  • libmtmd started as a wrapper around libllava / clip.cpp
  • Various components that used to be in clip.cpp are moved progressively to mtmd. For example, preprocessor is now part of mtmd
  • clip.cpp has since been split into mtmd-model (hparams and weights, one mtmd_model per modality), mtmd-subcontext (backends and compute, one mtmd_subcontext per modality) and mtmd-graph (graph builders, see models/). Only the GGUF format keeps the clip. name (metadata keys and the clip architecture), for compatibility with existing mmproj files

Terminologies

  • mtmd: MulTiMoDal
  • bitmap: representing a raw input data, for example: RGB image, PCM audio
  • tiles / slices: for llava-uhd-style models, the preprocessor breaks a large input into smaller square images called tiles or slices
  • chunk: a mtmd_input_chunk represents a preprocessed input that can then be passed through mtmd_encode()

Structure

Overview of the libmtmd structure:

  • mtmd_context (mtmd.cpp): the public handle
    • mtmd_models: one mmproj file, loaded by mtmd_model_load()
      • mtmd_model v / a / gen_a (mtmd-model.h): one per modality, holds hparams, weights (ctx_data + buf) and requirements
    • mtmd_backends (mtmd-subcontext.h): GPU/CPU backend instances, CPU threadpool and one scheduler, shared by the subcontexts
    • compute_mutex: held by every public function that runs a compute pass, so only one pass at a time runs on the backends
    • mtmd_subcontext ctx_v / ctx_a / ctx_gen_a (mtmd-subcontext.h): one per model, holds a const ref to it and a ref to the backends
      • buf_compute_meta and the resolved flash attn of its graphs
      • mtmd_graph (mtmd-graph.h, models/): built for each encode() call, temporary
    • mtmd_image_preprocessor / mtmd_audio_preprocessor (mtmd-image.h, mtmd-audio.h): read the hparams of their model
  • mtmd_gen_state / mtmd_audio_stream: public, one per sequence / stream
    • mtmd_subctx_gen_state / mtmd_subctx_enc_state: allocated on the backend of a subcontext, see gen_state_init() / enc_state_init()

Rules of thumb:

  • mtmd_model is read-only once loaded. Anything that depends on the backend (flash attn resolution, warmup) belongs to mtmd_subcontext
  • Like the pp/tg graphs of llama_context, the subcontexts share one scheduler: each warmup reserves its worst-case graph on it, so the compute buffers fit the largest one, and switching modality re-plans inside them
  • A model declares its runtime needs (e.g. max_nodes, flash_attn_auto) in load_hparams, the subcontext applies them
  • Models and backends must outlive the subcontexts, mtmd_context declares them first for that reason
  • Models having both audio and vision encoders must have different arch names (projector types), postfixed by a and v. See Gemma 4 as an example (gemma4a / gemma4v)
  • Locking happens only at the public API boundary (mtmd.cpp): a new public function that runs a graph must take compute_mutex. The internals (subcontexts, preprocessors, mtmd_audio_log_mel()) assume it is held and never lock
  • No magic numbers for a model's fixed shapes (layer counts, codebook sizes, window lengths...): define them once as static constexpr in the graph struct in models/models.h, and refer to them from the loader or the subcontext (e.g. mtmd_graph_zonos3_gen::n_codebooks, mtmd_graph_fish_s2_spkenc::n_tfm_layer)

Pipeline

A typical pipeline of the core libmtmd is as follows:

  • A bitmap (RGB image or PCM audio) is created
  • Bitmap and the text prompt is provided to mtmd_tokenize() that breaks the input into chunks
    • Alternatively, mtmd_tokenize_from_parts() takes a list of pre-split text/media parts instead of a marker-based prompt
    • The tokenizer function first expands a "lazy" bitmap if it finds one. Typically, this is used by video, so that one media token corresponds to one input bitmap
    • For models that support "fused" temporal frames like Qwen-VL, the tokenizer tries to merge pair of consecutive frames into one batch. Only bitmaps marked by mtmd_bitmap_set_mergeable() are merged
    • The preprocessor will then be called, which produces a list of chunks
    • Depending on the model itself, special tokens will be injected to separate image chunks (i.e. llava-uhd-style models)
  • Multiple bitmaps may be batched together to form a larger mtmd_batch()
  • Single image or batch is encoded, via mtmd_encode() or mtmd_batch_encode()
  • Get the output embeddings

Helper

We provide a set of helper functions via mtmd_helper to make using libmtmd easier. The helper provides:

  • Image, audio and video file decoding (for example, decode raw JPEG into RGB bitmap)
  • Manage llama_batch and calls to llama_decode

Audio generation support

Audio generation is added to mtmd in PR #26254

Currently, we support the 3-stage pipeline below which should cover most TTS models:

  • Stage 1: Backbone / Semantic Stage: Backbone model accepts text prompt and reference voice as input
  • Stage 2: Acoustic Detail Generator: A model takes the hidden state from backbone and generate audio details (usually as audio codes or mel-spectrogram)
  • Stage 3: Waveform Reconstruction: Convert the semantic and acoustic data from previous stages to the final waveform

For example, Qwen3-TTS:

  • Reference voice is encoded using ECAPA-TDNN speaker encoder (speaker_encoder)
  • Text prompt and reference voice are processed via a backbone (talker.model)
  • A model converts sampled semantic token and hidden state from stage 2 into a list of 15 acoustic codes (talker.code_predictor)
  • 16 generated codes are converted into waveform (code2wav)

API design constraints

Due to wide variety of audio generation pipelines, the mtmd_gen_audio system is designed to be flexible and reusable by new models.

mtmd_gen_audio is split into 2 main API:

  • Core API mtmd.h: handles main inference. Important: mtmd_gen_state holds only the per-sequence data that the mmproj itself writes and reads back (RNG, decoder KV / conv state, output buffers). Data that goes to or comes from the backbone (hidden states, codes, PCM) must stay explicit in mtmd_gen_inp / mtmd_gen_out; the caller drives the backbone and handles audio frame accumulation.
  • Helper API mtmd-helper.h: provides a model-agnostic stateful API. Usage example can be found in the tools/tts directory.

Running a transformer stage on libllama (mtmd-owned llama_context)

Some stages are a full LLM-sized transformer that attends to the keys/values of the backbone, for example the flow-matching NAR of YuE2 (same layer shapes as the backbone, its own weights, bidirectional over its rows, and it reads the backbone KV). Do not re-implement it as an mtmd graph. Instead, mtmd can own a second llama_model + llama_context (see mtmd_yue2_nar_load() and mtmd_gen_flow_yue2() in mtmd.cpp):

  • Conversion: store the layers in the mmproj under a.gen.<stage>. with the libllama tensor names of an existing arch (e.g. a.gen.nar.blk.0.attn_q.weight)
  • Loading: build a synthetic gguf_context with the arch hparams (general.architecture, tokenizer.ggml.model = none, vocab_size = 1, ...) and one tensor info per mmproj tensor (only the type is used, the shapes come from the hparams), then call llama_model_init_from_user(). The set_tensor_data callback reads the data from the mmproj file, so no copy is kept in the mtmd model
    • libllama also creates every optional tensor of the arch (biases, *.scale, *.input_scale, dense_2, ...) for a model built from metadata. The callback must fill them with values that leave the graph unchanged: 0 for biases, 1 for scales, identity for dense_2. Zeros everywhere silently make every matmul output 0
  • Backbone keys/values: the helper passes the backbone sequence state (llama_state_seq_get_data()) through mtmd_gen_inp, mtmd loads it into its context with llama_state_seq_set_data(). This only checks the layer count and the K/V types and row sizes, so it works across models with the same KV layout. The two contexts must agree on the KV type and on flash attention (v_trans)
  • Inputs and outputs are embedding rows (llama_batch::embd, llama_get_embeddings() with LLAMA_POOLING_TYPE_NONE), the projections around them stay small mtmd graphs
  • Non-causal rows: LLAMA_ATTENTION_TYPE_NON_CAUSAL, all rows in one ubatch (n_ubatch >= rows), they see every cached position of the sequence. Remove them with llama_memory_seq_rm() after each pass
  • Size the context per call (n_ctx, n_ubatch) instead of keeping one for the whole session

Checklist for porting new audio generation models to mtmd

  1. Make sure to consult merged PRs about adding new TTS models, especially reviewer comments
  2. Establish a list of reusable and missing components from the current mtmd implementation.
  3. For GGUF conversion:
    • Backbone model should be converted to a normal text model (loadable via libllama)
      • If model used hard-coded embedding row ID, append them to token embeddings and assign token name for them (see qwen3tts.py)
      • If model have a specific output logits head for audio codes (usually semantic code), keep the head as-is and pad the logits at inference time (see src/models/qwen3vl.cpp)
    • Sidecar models (code2wav, bigvgan, etc) must live inside the mmproj GGUF (but can be in different mtmd_subcontext if necessary)
      • Note: it should use ggml_build_forward_select to select graphs if multiple graphs living in the same context
    • Reuse existing GGUF metadata key name and tensor name whenever possible; think twice before adding extensive changes to GGUF writer. For example, Qwen3-TTS hard-code part of the hparams in mtmd-model.cpp as they won't likely to change.
    • For tensor naming:
      • Prefixed with a.* for tensors used by speaker encoder pipeline
      • Prefixed with a.gen.* for generation stages (code / mel-spectrogram / PCM generation)
    • For GGUF metadata:
      • Reuse as many existing keys as possible
      • In most cases, you can hard-code model configs in the model graph class, or in mtmd_hparams
      • If some values need to be exposed to the mtmd_helper layer, hard-code them in mtmd_helper and distinguish by pipeline and mtmd_gen_audio_info::model_variant if necessary
      • Do NOT add new GGUF metadata or new fields to mtmd_gen_audio_info unless you can prove that you absolutely need them
    • If the model requires special preprocessing of the input text prompt (prompt format, text normalization), put it in tokenizer.chat_template.tts as a jinja program. It receives the text as messages[0].content, plus language and voice. See conversion/lfm3.py (prompt format) and models/templates/Zyphra-ZONOS3-Mini-tts.jinja (text normalization) as examples. For text normalization, use tts_norm_scan() / tts_norm_sub() (see docs/tts-norm-patterns.md) and add test cases to tests/test-jinja.cpp, which also checks them against the python reference with -py
  4. Make sure most of the changes happen inside mtmd-helper-gen.cpp. A good PR looks like this:
    • 10-20% changes is to add new backbone (text) model and conversion
    • 60% changes inside mtmd-helper-gen.cpp
    • 10% changes inside libmtmd core (mtmd-model, mtmd-subcontext, models/)
    • The rest downstream code (CLI, server) should have no changes at all
  5. Validate with several concurrent requests on llama-server (e.g. -np 2, one plain and one streaming request at the same time), not only with llama-tts
    • The server interleaves the slots on one llama_context, so a pipeline must copy what it needs (hidden state, logits) right after its own decode, not read it at the next step
  6. Update usage documentation in docs/multimodal/tts.md

IMPORTANT: If your model needs changes that don't fit the existing infrastructure, open an issue first for discussion.

No-go checklist (these will get the PR rejected and require discussion before proceeding):

  • Violating the API design constraints stated above
  • Adding a new model-specific binary: the API and binary surface must stay model-agnostic
  • Adding new flags to conversion script: this script must stay model-agnostic