14 KiB
libmtmd dev guide
History
Please refer to multimodal.md for a broader context.
In short:
libmtmdstarted as a wrapper aroundlibllava/clip.cpp- Various components that used to be in
clip.cppare moved progressively to mtmd. For example, preprocessor is now part of mtmd clip.cpphas since been split intomtmd-model(hparams and weights, onemtmd_modelper modality),mtmd-subcontext(backends and compute, onemtmd_subcontextper modality) andmtmd-graph(graph builders, seemodels/). Only the GGUF format keeps theclip.name (metadata keys and thecliparchitecture), for compatibility with existing mmproj files
Terminologies
- mtmd: MulTiMoDal
- bitmap: representing a raw input data, for example: RGB image, PCM audio
- tiles / slices: for llava-uhd-style models, the preprocessor breaks a large input into smaller square images called tiles or slices
- chunk: a mtmd_input_chunk represents a preprocessed input that can then be passed through
mtmd_encode()
Structure
Overview of the libmtmd structure:
mtmd_context(mtmd.cpp): the public handlemtmd_models: one mmproj file, loaded bymtmd_model_load()mtmd_modelv/a/gen_a(mtmd-model.h): one per modality, holds hparams, weights (ctx_data+buf) and requirements
mtmd_backends(mtmd-subcontext.h): GPU/CPU backend instances, CPU threadpool and one scheduler, shared by the subcontextscompute_mutex: held by every public function that runs a compute pass, so only one pass at a time runs on the backendsmtmd_subcontextctx_v/ctx_a/ctx_gen_a(mtmd-subcontext.h): one per model, holds a const ref to it and a ref to the backendsbuf_compute_metaand the resolved flash attn of its graphsmtmd_graph(mtmd-graph.h,models/): built for eachencode()call, temporary
mtmd_image_preprocessor/mtmd_audio_preprocessor(mtmd-image.h,mtmd-audio.h): read the hparams of their model
mtmd_gen_state/mtmd_audio_stream: public, one per sequence / streammtmd_subctx_gen_state/mtmd_subctx_enc_state: allocated on the backend of a subcontext, seegen_state_init()/enc_state_init()
Rules of thumb:
mtmd_modelis read-only once loaded. Anything that depends on the backend (flash attn resolution, warmup) belongs tomtmd_subcontext- Like the pp/tg graphs of
llama_context, the subcontexts share one scheduler: each warmup reserves its worst-case graph on it, so the compute buffers fit the largest one, and switching modality re-plans inside them - A model declares its runtime needs (e.g.
max_nodes,flash_attn_auto) inload_hparams, the subcontext applies them - Models and backends must outlive the subcontexts,
mtmd_contextdeclares them first for that reason - Models having both audio and vision encoders must have different arch names (projector types), postfixed by
aandv. See Gemma 4 as an example (gemma4a/gemma4v) - Locking happens only at the public API boundary (
mtmd.cpp): a new public function that runs a graph must takecompute_mutex. The internals (subcontexts, preprocessors,mtmd_audio_log_mel()) assume it is held and never lock - No magic numbers for a model's fixed shapes (layer counts, codebook sizes, window lengths...): define them once as
static constexprin the graph struct inmodels/models.h, and refer to them from the loader or the subcontext (e.g.mtmd_graph_zonos3_gen::n_codebooks,mtmd_graph_fish_s2_spkenc::n_tfm_layer)
Pipeline
A typical pipeline of the core libmtmd is as follows:
- A bitmap (RGB image or PCM audio) is created
- Bitmap and the text prompt is provided to
mtmd_tokenize()that breaks the input into chunks- Alternatively,
mtmd_tokenize_from_parts()takes a list of pre-split text/media parts instead of a marker-based prompt - The tokenizer function first expands a "lazy" bitmap if it finds one. Typically, this is used by video, so that one media token corresponds to one input bitmap
- For models that support "fused" temporal frames like Qwen-VL, the tokenizer tries to merge pair of consecutive frames into one batch. Only bitmaps marked by
mtmd_bitmap_set_mergeable()are merged - The preprocessor will then be called, which produces a list of chunks
- Depending on the model itself, special tokens will be injected to separate image chunks (i.e. llava-uhd-style models)
- Alternatively,
- Multiple bitmaps may be batched together to form a larger
mtmd_batch() - Single image or batch is encoded, via
mtmd_encode()ormtmd_batch_encode() - Get the output embeddings
Helper
We provide a set of helper functions via mtmd_helper to make using libmtmd easier. The helper provides:
- Image, audio and video file decoding (for example, decode raw JPEG into RGB bitmap)
- Manage
llama_batchand calls tollama_decode
Audio generation support
Audio generation is added to mtmd in PR #26254
Currently, we support the 3-stage pipeline below which should cover most TTS models:
- Stage 1: Backbone / Semantic Stage: Backbone model accepts text prompt and reference voice as input
- Stage 2: Acoustic Detail Generator: A model takes the hidden state from backbone and generate audio details (usually as audio codes or mel-spectrogram)
- Stage 3: Waveform Reconstruction: Convert the semantic and acoustic data from previous stages to the final waveform
For example, Qwen3-TTS:
- Reference voice is encoded using ECAPA-TDNN speaker encoder (
speaker_encoder) - Text prompt and reference voice are processed via a backbone (
talker.model) - A model converts sampled semantic token and hidden state from stage 2 into a list of 15 acoustic codes (
talker.code_predictor) - 16 generated codes are converted into waveform (
code2wav)
API design constraints
Due to wide variety of audio generation pipelines, the mtmd_gen_audio system is designed to be flexible and reusable by new models.
mtmd_gen_audio is split into 2 main API:
- Core API
mtmd.h: handles main inference. Important:mtmd_gen_stateholds only the per-sequence data that the mmproj itself writes and reads back (RNG, decoder KV / conv state, output buffers). Data that goes to or comes from the backbone (hidden states, codes, PCM) must stay explicit inmtmd_gen_inp/mtmd_gen_out; the caller drives the backbone and handles audio frame accumulation. - Helper API
mtmd-helper.h: provides a model-agnostic stateful API. Usage example can be found in thetools/ttsdirectory.
Running a transformer stage on libllama (mtmd-owned llama_context)
Some stages are a full LLM-sized transformer that attends to the keys/values of the backbone, for example the flow-matching NAR of YuE2 (same layer shapes as the backbone, its own weights, bidirectional over its rows, and it reads the backbone KV). Do not re-implement it as an mtmd graph. Instead, mtmd can own a second llama_model + llama_context (see mtmd_yue2_nar_load() and mtmd_gen_flow_yue2(), or mtmd_voxcpm2_res_load() and mtmd_gen_code_voxcpm2() in mtmd.cpp):
- Conversion: store the layers in the mmproj under
a.gen.<stage>.with the libllama tensor names of an existing arch (e.g.a.gen.nar.blk.0.attn_q.weight) - Loading: build a synthetic
gguf_contextwith the arch hparams (general.architecture,tokenizer.ggml.model = none,vocab_size = 1, ...) and one tensor info per mmproj tensor (only the type is used, the shapes come from the hparams), then callllama_model_init_from_user()(mtmd_owned_llama_load()does this, the caller only writes the hparams). Theset_tensor_datacallback reads the data from the mmproj file, so no copy is kept in the mtmd model- libllama also creates every optional tensor of the arch (biases,
*.scale,*.input_scale,dense_2, ...) for a model built from metadata. The callback must fill them with values that leave the graph unchanged: 0 for biases, 1 for scales andrope_freqs, identity fordense_2. Zeros everywhere silently make every matmul output 0
- libllama also creates every optional tensor of the arch (biases,
- Backbone keys/values: the helper passes the backbone sequence state (
llama_state_seq_get_data()) throughmtmd_gen_inp, mtmd loads it into its context withllama_state_seq_set_data(). This only checks the layer count and the K/V types and row sizes, so it works across models with the same KV layout. The two contexts must agree on the KV type and on flash attention (v_trans) - Inputs and outputs are embedding rows (
llama_batch::embd,llama_get_embeddings()withLLAMA_POOLING_TYPE_NONE), the projections around them stay small mtmd graphs - Non-causal rows:
LLAMA_ATTENTION_TYPE_NON_CAUSAL, all rows in one ubatch (n_ubatch>= rows), they see every cached position of the sequence. Remove them withllama_memory_seq_rm()after each pass - Size the context per call (
n_ctx,n_ubatch) instead of keeping one for the whole session - Causal stage with its own KV cache across calls (the residual LM of VoxCPM2): the
mtmd_gen_stateowns thellama_context,mtmd_gen_state_reset()clears its memory. Options of the arch may switch parts off, e.g. RoPE withrope.scaling.finetuned = falseon granite
Text preprocessing (the tts chat template)
Anything a model needs done to the input text before tokenization (prompt format, text normalization, text to phonemes) goes in tokenizer.chat_template.tts, as a jinja program. It receives the text as messages[0].content, plus language and voice.
Examples:
- Prompt format:
conversion/lfm3.py - Text normalization:
models/templates/Zyphra-ZONOS3-Mini-tts.jinja - Text to phonemes (G2P):
models/templates/hexgrad-Kokoro-82M-tts.jinja
Functions available to the template (see docs/tts-norm-patterns.md):
tts_norm_scan()/tts_norm_sub(): pattern matching for text normalizationtts_dict(): lookup in a dictionary stored in the GGUF, such as a pronunciation lexicon
Add test cases to tests/test-jinja.cpp, which also checks them against the python reference with -py.
Rules for phoneme-based models:
- C++ only holds the pieces that every such model needs (the functions above). The rules of one model (lexicon handling, stress, suffixes, number spelling, ...) live in its jinja template
- The reason is binary size: phoneme logic compiled into C++ is shipped to every user, and most of them never run a TTS model. A template only costs the users of that model
- Do not add a model-specific or language-specific G2P step in C++, and do not add a dependency for it (e.g. espeak-ng)
G2P in jinja has limits, mostly for languages other than English (no lexicon, or rules that need a real morphological analyzer). For now the escape hatch is on the application side: the user runs the text to phonemes step in their own code and sends the phonemes with raw_prompt (see the /tts endpoint of llama-server), which skips the template. Document it in docs/multimodal/tts.md when a model relies on it.
Checklist for porting new audio generation models to mtmd
- Make sure to consult merged PRs about adding new TTS models, especially reviewer comments
- Establish a list of reusable and missing components from the current mtmd implementation.
- For GGUF conversion:
- Backbone model should be converted to a normal text model (loadable via
libllama)- If model used hard-coded embedding row ID, append them to token embeddings and assign token name for them (see
qwen3tts.py) - If model have a specific output logits head for audio codes (usually semantic code), keep the head as-is and pad the logits at inference time (see
src/models/qwen3vl.cpp)
- If model used hard-coded embedding row ID, append them to token embeddings and assign token name for them (see
- Sidecar models (code2wav, bigvgan, etc) must live inside the mmproj GGUF (but can be in different
mtmd_subcontextif necessary)- Note: it should use
ggml_build_forward_selectto select graphs if multiple graphs living in the same context
- Note: it should use
- Reuse existing GGUF metadata key name and tensor name whenever possible; think twice before adding extensive changes to GGUF writer. For example, Qwen3-TTS hard-code part of the hparams in
mtmd-model.cppas they won't likely to change. - For tensor naming:
- Prefixed with
a.*for tensors used by speaker encoder pipeline - Prefixed with
a.gen.*for generation stages (code / mel-spectrogram / PCM generation)
- Prefixed with
- For GGUF metadata:
- Reuse as many existing keys as possible
- In most cases, you can hard-code model configs in the model graph class, or in
mtmd_hparams - If some values need to be exposed to the
mtmd_helperlayer, hard-code them inmtmd_helperand distinguish by pipeline andmtmd_gen_audio_info::model_variantif necessary - Do NOT add new GGUF metadata or new fields to
mtmd_gen_audio_infounless you can prove that you absolutely need them
- If the model requires special preprocessing of the input text prompt, see Text preprocessing
- Backbone model should be converted to a normal text model (loadable via
- Make sure most of the changes happen inside
mtmd-helper-gen.cpp. A good PR looks like this:- 10-20% changes is to add new backbone (text) model and conversion
- 60% changes inside
mtmd-helper-gen.cpp - 10% changes inside
libmtmdcore (mtmd-model,mtmd-subcontext,models/) - The rest downstream code (CLI, server) should have no changes at all
- Validate with several concurrent requests on
llama-server(e.g.-np 2, one plain and one streaming request at the same time), not only withllama-tts- The server interleaves the slots on one
llama_context, so a pipeline must copy what it needs (hidden state, logits) right after its own decode, not read it at the next step
- The server interleaves the slots on one
- Update usage documentation in
docs/multimodal/tts.md
IMPORTANT: If your model needs changes that don't fit the existing infrastructure, open an issue first for discussion.
No-go checklist (these will get the PR rejected and require discussion before proceeding):
- Violating the API design constraints stated above
- Adding a new model-specific binary: the API and binary surface must stay model-agnostic
- Adding new flags to conversion script: this script must stay model-agnostic