Files
llama.cpp/tools/mtmd
Piotr Wilkin (ilintar)andXuan Son Nguyen 5c008e0008 feat: add Inkling 2 support (text + vision) (#32217)
* convert : add Inkling2 text GGUF conversion

Assisted-by: Codex

* llama : implement Inkling2 text inference

Assisted-by: Codex

* tests : cover Inkling2 text support

Assisted-by: Codex

* conversion : dequantize packed Inkling2 experts in row chunks

Decode per-layer NVFP4 and MXFP8 schemes, validate companions, and use stored scale shapes with a warning for the declared MXFP8 [32,32] grid. Reorder packed w13 rows and their scale rows together. Keep normal GGUF outtype selection and 3D expert tensors.

Stream 128 rows per expert through LazyChunkedTensor. Real w13 (384x4608x4608) needs a 2.25 MiB F32 chunk instead of a 30.375 GiB expanded layer. Real-shaped sparse probes peak below 5.8 MiB of Python/NumPy chunk allocations, plus 5.2 MiB of callback metadata: budget about 16 MiB per projection including mapped pages, excluding runtime, other tensors and OS cache. Full-model chunk metadata is roughly 370 MiB.

Packed synthetic GGUF weights match the independent pinned vLLM decode bit-for-bit. CPU prefill/decode logits remain within 3.1e-6 absolute error, with 100% argmax agreement. Float fixtures, original-name conversion and CPU architecture checks pass. Real checkpoint conversion remains untested.

Assisted-by: Codex

* convert : add Inkling2 vision mmproj support

Assisted-by: Codex

* mtmd : implement Inkling2 still-image encoder

Assisted-by: Codex

* inkling2 : scope image loading

Assisted-by: Codex

* inkling2 : enable flash attention with folded bias and sinks

Assisted-by: Codex

* cuda : route per-head flash attention masks through vec

Assisted-by: Codex

* mtmd : support Inkling2 temporal frame means

Assisted-by: Codex

* style : restore shared image comment

Assisted-by: Codex

* tests : enlarge Inkling2 context fixture

Assisted-by: Codex

* inkling2 : load per-layer expert feed-forward widths

Assisted-by: Codex

* tests : add Inkling2 Metal fusion baselines

Assisted-by: Codex

* inkling2 : group and align registry entries

Assisted-by: Fable

* convert : regenerate the Inkling2 tokenizer hash with the update script

Assisted-by: Fable

* conversion : revert the print_registered_models change

Assisted-by: Fable

* conversion : sort the Inkling2 model map entries

Assisted-by: Fable

* convert : map Inkling2 vision checkpoint names in the tensor table

Assisted-by: Fable

* convert : map Inkling2 text checkpoint names in the tensor table

Assisted-by: Fable

* tests : document the Inkling2 packed expert conversion test

Assisted-by: Fable

* tests : cover Inkling2 checkpoint and transformers tensor names

Assisted-by: Fable

* llama : place Inkling2 fields and list entries by file convention

Assisted-by: Fable

* gguf-py : align Inkling2 tensor mapping comments

Assisted-by: Fable

* mtmd : place Inkling2 entries by file convention

Assisted-by: Fable

* tests : place Inkling2 cases after the existing ones

Assisted-by: Fable

* cuda : index per-head flash attention masks in the tile and mma kernels

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* inkling2 : project the relative bias before the gather, drop arch branches

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* convert : require Inkling2 hparams, add inkling2 pre-tokenizer

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* mtmd : cap Inkling2 image size, drop video frame averaging

Assisted-by: Claude
Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q

* inkling2 : fix CI build, Meta device assert and Windows conversion test

Assisted-by: Claude

* tests : use the default sliding window for the Inkling2 test model

Assisted-by: Claude

* simplify mtmd part

* give _v arch for vision

* remove alpha channel handling, it should be a follow up

* explicit rule about audio + vision models

* add audio support

* rm audio

* rm gguf mtmd rope base

* address review comment

* nits

* do not normalize embd from mtmd

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
2026-11-15 00:51:31 +01:00
..

Multimodal Support in llama.cpp

This directory provides multimodal capabilities for llama.cpp. Initially intended as a showcase for running LLaVA models, its scope has expanded significantly over time to include various other vision-capable models. As a result, LLaVA is no longer the only multimodal architecture supported.

Important

Multimodal support can be viewed as a sub-project within llama.cpp. It is under very heavy development, and breaking changes are expected.

The naming and structure related to multimodal support have evolved, which might cause some confusion. Here's a brief timeline to clarify:

  • #3436: Initial support for LLaVA 1.5 was added, introducing llava.cpp and clip.cpp. The llava-cli binary was created for model interaction.
  • #4954: Support for MobileVLM was added, becoming the second vision model supported. This built upon the existing llava.cpp, clip.cpp, and llava-cli infrastructure.
  • Expansion & Fragmentation: Many new models were subsequently added (e.g., #7599, #10361, #12344, and others). However, llava-cli lacked support for the increasingly complex chat templates required by these models. This led to the creation of model-specific binaries like qwen2vl-cli, minicpmv-cli, and gemma3-cli. While functional, this proliferation of command-line tools became confusing for users.
  • #12849: libmtmd was introduced as a replacement for llava.cpp. Its goals include providing a single, unified command-line interface, improving the user/developer experience (UX/DX), and supporting both audio and image inputs.
  • #13012: mtmd-cli was added, consolidating the various model-specific CLIs into a single tool powered by libmtmd.

Pre-quantized models

See the list of pre-quantized model here

How it works and what is mmproj?

Multimodal support in llama.cpp works by encoding images into embeddings using a separate model component, and then feeding these embeddings into the language model.

This approach keeps the multimodal components distinct from the core libllama library. Separating these allows for faster, independent development cycles. While many modern vision models are based on Vision Transformers (ViTs), their specific pre-processing and projection steps can vary significantly. Integrating this diverse complexity directly into libllama is currently challenging.

Consequently, running a multimodal model typically requires two GGUF files:

  1. The standard language model file.
  2. A corresponding multimodal projector (mmproj) file, which handles the image encoding and projection.

What is libmtmd?

As outlined in the history, libmtmd is the modern library designed to replace the original llava.cpp implementation for handling multimodal inputs.

Originally built upon clip.cpp (similar to llava.cpp), libmtmd offers several advantages:

  • Unified Interface: Aims to consolidate interaction for various multimodal models.
  • Improved UX/DX: Features a more intuitive API, inspired by the Processor class in the Hugging Face transformers library.
  • Flexibility: Designed to support multiple input types (text, audio, images) while respecting the wide variety of chat templates used by different models.

How to obtain mmproj

Multimodal projector (mmproj) files are specific to each model architecture.

For the following models, you can use convert_hf_to_gguf.py with --mmproj flag to get the mmproj file:

  • Gemma 3 ; See the guide here - Note: 1B variant does not have vision support
  • SmolVLM (from HuggingFaceTB)
  • SmolVLM2 (from HuggingFaceTB)
  • Pixtral 12B - only works with transformers-compatible checkpoint
  • Qwen 2 VL and Qwen 2.5 VL (from Qwen)
  • Mistral Small 3.1 24B
  • InternVL 2.5 and InternVL 3 from OpenGVLab (note: we don't support conversion of InternVL3-*-hf model, only non-HF version is supported ; InternLM2Model text model is not supported)
  • MiniCPM-V 4.6 ; See the guide here - requires the standard transformers v5.7.0+ checkpoint

For older models, please refer to the relevant guide for instructions on how to obtain or create them:

NOTE: conversion scripts are located under tools/mtmd/legacy-models