55 Commits

Include renames

Author	SHA1	Message	Date
Xuan-Son Nguyen	2578138397	llama: improve TENSOR_READ_LAZY handling (#27837) ... * force lazy tensor on cpu if lazy is on * llama: improve TENSOR_READ_LAZY handling	2026-08-30 16:59:48 +02:00
Xuan-Son Nguyen	50f068ffff	bench: add --tensor-read-lazy (#27881) ... * bench: add --tensor-read-lazy * rm the alias * rename to LLAMA_LAZY_MODE_*	2026-08-28 20:51:05 +02:00
6 people	6c84c7d5d8	model: add Qwen3.8-Flash-Next (qwen4exp) (#27742) ... * gguf: add qwen4exp (Qwen3.8-Flash-Next) arch and converter Adds the GGUF-side plumbing for HF model_type qwen4_exp: - MODEL_ARCH.QWEN4EXP plus tensors for the low-rank hyper-connection variant (hc_*_norm/down/up/inject) and the PLE n-gram hash embeddings. The DeepSeek-V4 hc_*_fn/base/scale tensors are a different parameterisation, so these are separate entries rather than reuse. - Reuses the existing indexer, per_layer_token_embd, SSM and compress_ratios keys unchanged. - conversion/qwen4exp.py inherits the Qwen3.5 linear-attention V-head reorder and interleaved mrope, concatenates the 128 PLE embedding shards, and splits index_qk_proj into separate indexer q/k tensors. The PLE hash multipliers reach ~2.4e13. prepare_tensors() casts every non-float dtype to float32 before modify_tensors() runs, and GGUF array writes infer INT32 from Python ints, so both paths are bypassed: the constants are read from the pre-cast lazy tensors and written as explicit UINT64 arrays. Additive only; no existing arch changes behaviour. * llama: load qwen4exp (Qwen3.8-Flash-Next) hparams and tensors Adds LLM_ARCH_QWEN4EXP with its hparams and tensor loading. The graph comes in the next commit; this makes the model load and report correct metadata. - hyper-connections set n_embd_out_impl = hc_count * n_embd, so the residual stream is 4x wide and there is no output_norm: the final mixer's hc_norm is the last norm in the model. - registered as hybrid and given the same recurrent/attention memory filters as Qwen3-Next and Qwen3.5. - reuses the existing indexer, per_layer_token_embd, SSM and compress_ratios keys as-is. - the PLE table row count is read back from the file rather than recomputing the vocab padding rule. llama-model-loader gains UINT64 array support. That branch previously threw, so no existing caller changes behaviour; it is needed because the PLE hash multipliers do not fit in int32. * qwen4exp: shorten comments * llama: qwen4exp text graph with hyper-connections, GDN and MoE Implements the decode graph for Qwen3.8-Flash-Next: the hyper-connection residual stream, gated delta net layers, the MoE block with its gated shared expert, and dense full attention. The QSA indexer and the PLE n-gram embedding are not wired up yet and land in later commits. Hyper-connections are implemented here rather than shared with deepseek4.cpp. The two formulations agree on the [n_embd, hc, n_tokens] layout and little else: DeepSeek-V4 mixes with a full-rank projection and Sinkhorn-normalises it, whereas this model uses a low-rank down/silu/up sigmoid gate and collapses by a plain mean. Only the ~10 line stream mean is genuinely common, so sharing would mean touching DSV4's hot path and its three fused CUDA ops to reuse very little. What is reused is the substantive part: the LLM_KV_HYPER_CONNECTION_* keys, the n_embd_out_impl wide-residual support already in the loader, and the layout convention. Also allows a checkpoint to carry no PLE layers at all, which makes it possible to bring the graph up and validate it in stages. Validated against vLLM, the only working reference implementation. On a scaled-down model with an init scale large enough to give non-uniform logits, agreement with vLLM sits at the numerical noise floor: llama.cpp f32 against its own bf16 gives 84.3% top-1 agreement over 255 positions, and this graph against vLLM gives 85.1%. The comparison was calibrated by seeding three deliberate bugs (silu instead of sigmoid on the delta net gate, dropping the 1/hc scale in the mix, dropping the 2x in the combine); each drops top-1 to between 0% and 11%, an order of magnitude below the floor. * llama: qwen4exp PLE n-gram hash embedding Adds the per-layer embedding: a custom I32 graph input hashes each token with its ngram_size-1 predecessors host-side and the result is a plain row gather over the shared table, the same shape gemma3n's per-layer embedding uses. The hash has to run on the host because the splitmix64-derived multipliers reach 2^45, so the products need 64-bit integers and an xor, neither of which ggml has. Predecessors that fall outside the ubatch come from a small per-sequence history on the model, mirroring the per-request ngram_context the reference carries. It is only trusted when contiguous with the incoming position, so a fresh prompt or a rewound cache falls back to EOS padding rather than hashing against stale tokens. The depthwise conv is written out as a sum of shifted, per-channel-scaled copies rather than through ggml_conv_1d_dw, which carries a correctness warning upstream. Verified two ways. The row indices match a transcription of the reference's tensor formulation exactly, 1024 of 1024 rows, including sequences with EOS tokens sprinkled through them to exercise the segment reset. Separately, with PLE placed on layer 0 so its input is just the token embedding, ple_embd and ple_gated_value match a PyTorch computation from the same checkpoint to every printed digit. End to end over 1023 scored positions the port sits the same distance from vLLM with PLE as without it, 6.3 points of top-1 against 6.0, so PLE costs no accuracy relative to the rest of the model. That common offset is vLLM's bf16 activations, which cannot be removed: its QSA kernel refuses float32. Two bugs found along the way, both caught by the row-index check. The history was read and updated in the same pass, so a token early in a ubatch could pick up an earlier token of that same ubatch as prior context; it is now snapshotted first. And an EOS token was cutting its own context, where the reference takes the last EOS strictly before the position, so a boundary only hides tokens from the positions after it. Known gap: the conv carries no state across ubatches, so it is exact only for a prefill that starts at position 0. Chunked prefill and decode need the conv state wired into the recurrent memory, and the conv branch itself is still numerically unverified because the fixture zeroes its weights. * llama: carry the qwen4exp PLE conv state across ubatches The PLE depthwise conv was zero-padding on the left, which is only right for a prefill that starts at position 0. Decode and chunked prefill saw a truncated history for the first (kernel-1)*ngram_size positions of every ubatch. The PLE module sits on a layer that is also a delta-net layer, so both need a conv history in the same recurrent row. Rather than plumb a per-layer state size through build_rs and build_conv_state, the row is widened once and each convolution addresses its own slice through a local helper. n_embd_r() gains the extra span, which is zero for every other architecture because it is derived from ple_n_heads. Verified by feeding the same 1024 token sequence in chunks instead of one shot: at 64 tokens per decode the logits are bit-identical to the single-shot run, 1023 of 1023 top-1 and a maximum logprob deviation of exactly zero. At one token per decode they differ slightly, but the no-PLE model differs more under the same test (94.6% against 97.1%), so that is the usual gemv-versus- gemm accumulation difference and not the state. The conv branch is also no longer unverified. With non-zero conv weights the port sits 6.3 points of top-1 below the numerical floor, the same distance as with the weights zeroed and as the model with no PLE at all, so the branch adds no error of its own. test-llama-archs passes every existing architecture at 0.00e+00, including the delta-net models that share this code path. * llama: fix the qwen4exp PLE conv state and unblock test-llama-archs build_rs writes into the state tensor in place, zeroing one row and copying the carried-over states, so calling it twice for the same layer let the second call clobber the first write-back. The PLE layer is also a delta-net layer, so that is exactly what happened: both convolutions gathered the same row. They now share a single gather per layer. The earlier claim that the conv state was carried correctly was tested on a fixture whose conv weights are zero, where the branch contributes nothing and chunking matches trivially. Re-running with non-zero conv weights showed the divergence, growing with the number of ubatch boundaries: 97.1% top-1 at one boundary down to 90.2% at seven. With the shared gather it is bit-identical to the single-shot run at every chunk size tried, 512, 128 and 64, with a maximum logprob deviation of exactly zero over 1023 positions. The delta-net-only model stays bit-identical too, so nothing regressed there. Also derive the delta-net conv channel count the way load_arch_tensors sizes wqkv instead of from ssm_d_inner. The two agree for this model, but n_embd_r() only bounds the row and the convolution has to match the tensor feeding it. test-llama-archs previously aborted on this architecture and took every later architecture with it. qwen4exp is marked MoE-only, given the hyper-connection keys and an ssm_d_inner consistent with its tensor derivation, and skipped for now: the hyper-connection keys written by get_gguf_ctx are not reaching the synthesised file, which needs a separate look. The suite completes again, 124 architectures at 0.00e+00. * llama: optional indexer key cache in llama_memory_hybrid Groundwork for qwen4exp's QSA sparse attention. Its indexer needs a per-token key history for the full-attention layers, but a hybrid model cannot use llama_kv_cache_dsa: that class derives from llama_memory_i rather than llama_kv_cache, and llama_memory_hybrid constructs its attention cache directly. No existing architecture pairs recurrent state with a sparse indexer, so there was nothing to reuse wholesale. llama_memory_hybrid therefore gains a third, optional cache, shaped the same way llama_kv_cache_dsa shapes its lightning-indexer cache: a copy of hparams with n_head_kv forced to 1 and n_embd_head_k_full set to indexer_head_size. It is built only when a filter_idx callback is passed, which defaults to nullptr, so every existing architecture gets exactly what it got before. The per-sequence operations and the batch preparation forward to it under a null check, matching how the DSA cache prepares its two caches over the same ubatches. test-llama-archs passes all 124 architectures at 0.00e+00, including the 12 in the hybrid family that share this code. The qwen4exp fixtures are unchanged: same logits against vLLM, and chunked evaluation still bit-identical to single-shot. * llama: QSA sparse attention for qwen4exp The full-attention layers of this model do not attend to everything. An indexer scores one mean-pooled key per block of compress_ratio tokens and keeps a budget of the best blocks, plus the tail of tokens that do not yet form a complete block. Below indexer_top_k + compress_ratio - 1 cached tokens every block fits in the budget, so the result is exactly dense. What is reused rather than rebuilt: - the mask machinery. build_attn's DSA overload already turns a list of token indices into a KQ mask via ggml_set_rows, so that block is lifted out verbatim into build_attn_mask_top_k and shared with a new overload on llm_graph_input_attn_kv. DSA's node sequence is unchanged; the new overload exists because llama_kv_cache_dsa assumes MLA and cannot be dropped into a hybrid model. - the indexer key cache, which is the optional third cache added to llama_memory_hybrid in the previous commit. It holds raw keys, because pooling happens before the norm and the rotation. The graph expands block scores rather than block indices: giving every token of a block its block's score needs only a gather, where expanding indices would need an integer multiply-add that ggml has no op for. Since the budget is a whole number of blocks and a block's members tie exactly, the cut still lands on a block boundary. Everything that depends on cache layout is computed host-side in set_input_qsa. Blocks are cuts of the position line rather than of the cell array, so nothing assumes the cache is contiguous. Measured on the tiny fixture against vLLM, comparing the selected token indices directly rather than the logits: below the budget selection identical, and 1024-token logits are bit-identical to the pre-QSA dense path above the budget mean jaccard 0.975 The direct index comparison is what made this correct. The reference rectifies each head's dot product before summing over heads, which an earlier reading of it had missed; on logits alone the resulting port looked fine, because on a randomly initialised fixture the known-correct dense path already disagrees with vLLM by more than the bug did. Comparing the indices showed 0.794, and fixing the ReLU moved it to 0.975. * llama: give the qwen4exp indexer cache the attention cache's slots The indexer cache found its own slots, independently of the attention cache. Both are the same size and see the same ubatches, so in a straight-through prefill they agree, which is why every fixture and every single-shot parity run passed. They drift once the context is being rewritten between turns, and then the QSA top-k indices, which are applied against the attention mask, point at the wrong cells. The seven-turn chat test caught it on the third turn: llama-server aborted on the assertion that the two caches report the same n_kv. The cache is a side buffer addressed by the attention cache's cells, so it now takes that cache's slot layout instead of computing one. Applying that layout also marks its cells identically, so the two agree cell for cell by construction rather than by coincidence, and the assertion can no longer fire. Inert where the caches already agreed: test-llama-archs green at 126 archs and 0.00e+00, and the 4096-token tiny fixture is unchanged at max logit delta 0.0. * tests: record what the qwen4exp arch-test skip actually observes The old note guessed that the hyper-connection keys never reach the file. They do: dumping the gguf_context handed to llama_model_init_from_user shows both among its 67 KVs, and the loader still reports one missing. * tests: cover qwen4exp in test-llama-archs The arch was skipped with a note guessing that the hyper-connection keys never reached the synthesised file. They did. The suite builds a model, then saves and reloads it, and llama_model_saver did not re-emit those keys, so the failure was in the roundtrip leg rather than the first load. Three gaps, all in shared code and all additive: - add_kv_from_model wrote no hyper-connection, compress-ratio or PLE keys. The PLE group only means anything whole, so it is written or omitted together; the rest follow the file's existing style of writing every key unconditionally, since an architecture that does not read one is unaffected by a zero. - the saver had no uint64 path at all, which the PLE hash constants need. - add_tensors_from_model enumerates model-level tensors by hand and was missing per_layer_tok_embd and the three final-mixer tensors. Two smaller fixes on the qwen4exp side, both found by running the test: - build_qsa_top_k divided by the compression ratio before asserting it was non-zero, so a file without the key crashed instead of reporting. - a layer with no compression ratio now falls back to dense attention, which is what the model computes below the budget anyway. The test then has to write a ratio to reach QSA at all, and an indexer key length no narrower than n_rot, since the indexer ropes with the main attention's rotary width. Full suite: 126 archs, qwen4exp at 0.00e+00 with roundtrip OK. The tiny fixture is unchanged, max logit delta 0.0 against the pre-QSA dense run. * convert: stream the qwen4exp PLE table instead of concatenating it The n-gram table arrives as 128 shards that were held in a dict and then torch.cat-ed, so the peak was the shards plus the concatenation: around 300 GB of RSS on the real checkpoint, which rules out machines that could otherwise convert this model. Each shard is now written straight into a memory-mapped file at its final row offset and dropped, so the resident set is one shard and the rest is the page cache's problem. The temporary file sits beside the output and is removed once the write finishes, including on failure. Shards other than the last must be uniform for direct placement, which is asserted rather than assumed, and a shard arriving before the stride is known is held instead of misplaced. Verified on the tiny fixture: the resulting GGUF is byte-identical to the one the concatenating path produced (md5 2d274efac91ad1e9a6007efb0687e597). * quantize: fall back to F16 for 32-block types with an odd ncols tensor_type_fallback demotes a tensor whose ncols is not a multiple of the target's block size, but its switch only enumerates the 256-block types. A target that is already a 32-block type (iq4_nl, q4_0, q5_0, q8_0, ...) falls into default: and throws, even though the function already knows how to answer that case: the ncols check right below the switch resolves an unrepresentable shape to F16. Route those types into that check instead of throwing. Only paths that abort today change, so no quantization that currently succeeds is affected. Found on a 4-wide depthwise conv kernel. llama-quantize reported nothing but "failed to quantize model from ...", with no tensor name and no exception text, which made a quant recipe that had simply not pinned the tensor look like a corrupt model. It now names the tensor and continues. * quantize: let --tensor-type name per_layer_token_embd per_layer_token_embd shares the TOKEN_EMBD category with token_embd.weight, so --token-embedding-type is returned for it before any --tensor-type pattern is consulted, and there is no way to give it a tier of its own. That grouping is fine as a default and stays the default. It is a poor fit for the size, though: on qwen4exp the table is 97.7 GiB of a 337.6 GiB BF16 file and about 46% of a 4-bit one, roughly eighty times token_embd.weight, and it is read by ggml_get_rows rather than a matmul so no imatrix ever covers it. Allow an explicit --tensor-type pattern to name it, and only it. Nothing changes unless such a pattern is passed, and token_embd.weight keeps the old precedence in either case. Measured on Qwen3.8-Flash-Next, Q4_K_M with an imatrix: the table lands at q8_0 (51.9 GiB, 113.5 GiB total) by following --token-embedding-type, and pinning it q4_1 gives 30.5 GiB for 92.1 GiB total, 19% off the file. * quantize: size the output buffer exactly instead of nelements * 4 The per-tensor output buffer was sized `nelements * 4`, described as an upper bound. It is a very loose one: the output is at most 2 bytes per element (f16/bf16) and usually well under 1.1 (q8_0 and below), so between 2x and 4x of it is never touched. The exact size is already known here, since it is what the quantization loop writes, what new_size sums to, and what the GGUF metadata is asserted against a few lines later. On a model whose largest tensor is a few GB none of this matters. On Qwen3.8-Flash-Next it does: per_layer_token_embd is 51.2 G elements, so the buffer was 205 GB where 54 GB is needed at q8_0 and 32 GB at q4_1. Measured on that model, VmHWM of a live llama-quantize was 485 GB per process. Three of them fit in 2 TB and five did not, which is what an OOM-killed quant ladder looks like. This removes about 150 GB of that. Byte-identical output, verified against the same binary built at the parent commit: q4_K, q8_0, q5_K, q6_K and IQ4_XS, over BF16 and F32 sources, with and without a PLE table present. Six cases, six matching md5s. * qwen4exp: hash the image placeholder for multimodal batches The PLE row indices are computed host-side from ubatch->token, and set_input returned early when that was null. A multimodal ubatch is exactly that case: the mtmd layer consumes the image placeholder ids and hands llama_decode embeddings instead. The early return left the I32 index tensor uninitialised, so ggml_get_rows indexed a 320 M row table with whatever the buffer happened to contain, and aborted: GGML_ASSERT(i01 >= 0 && i01 < ne01) failed ggml_compute_forward_get_rows mtmd_helper_decode_image_chunk -> llama_decode Every image request crashed. Nothing caught it because the vision work had only ever been verified by converting an mmproj, never by running one. The reference computes the hash over input_ids, where those positions still hold the image placeholder, so carry that id through as qwen4exp.ple.image_token_id and hash it. The key is optional: a file converted before it existed falls back to the PLE EOS token, which is defined and treats the image as a segment boundary rather than crashing. Verified end to end with llama-mtmd-cli, a Q4_K_M base and the F16 mmproj, on a generated image with known content. The model names the red circle, the blue square, the inverted green triangle and reads "UNSLOTH 42", each with the right position. * qwen4exp: support a non-unified KV cache in QSA set_input_qsa asserted n_stream == 1, so llama-server could not serve this model with more than one slot unless -kvu was passed. With a non-unified cache each sequence owns its own cells, and a cell index means a different token in each stream, so a single shared mapping is wrong. - cell_blk, blk_cells and bias gain a stream dimension. At n_stream == 1 these collapse to the shapes they had, so the unified path is unchanged. - Scoring is now batched over streams. ggml_mul_mat matches ne[2] on both operands, so stream s's queries only ever meet stream s's blocks; without this sequences would score against each other's context. - set_input_qsa loops per stream and resolves cells through v_cells[seq_to_stream[seq_id]], following set_input_kq_mask_impl, instead of hardcoding v_cells[0]. - llama_kv_cache_context::get_n_stream() is added, mirroring the ns that get_k and get_v already derive from the slot info. build_attn_mask_top_k needed no change: it already expects [n_top_k, n_batch, 1, n_stream], so the top-k result is reshaped to meet it. set_input_qsa has exactly one caller, so the blast radius is qwen4exp only. Validation, UD-Q4_K_XL on one B200: - unified cache unchanged within noise: 1802.9/68.85 -> 1807.2/69.11 t/s at batch 1, 2262.5/192.43 -> 2270.1/193.75 at batch 4. - non-unified now runs at npl 1, 4, 16 where it previously aborted, and is 22% faster than the -kvu workaround at batch 16 (1205 vs 984 t/s total), since per-stream cells avoid the cross-sequence masking a unified cache pays for. - no cross-stream contamination: four concurrent sequences each carrying a distinct secret all recall their own and no other, on both cache modes. - test-llama-archs green on qwen4exp, deepseek2, gemma3n, qwen3next, llama. Note on testing: comparing concurrent output against solo output exactly is not a valid check. It failed 0/4 with no bug present, and the unified-cache control failed the same way, because batch composition changes the floating-point reduction order and near-tied tokens flip. The contamination test above is what the exit code gates on. * llama: keep the qwen4exp top-k attention mask arch-local The QSA graph needed a build_attn that attends only to the cells named by a top_k tensor, and the first version got it by adding a llm_graph_input_attn_kv overload to llm_graph_context and factoring the mask construction out of the existing MLA sparse path into a shared build_attn_mask_top_k. That put a new arch on the shared attention path and made the deepseek32 and glm-dsa attention build depend on a helper introduced for qwen4exp. Build the mask in src/models/qwen4exp.cpp instead and leave llama-graph.{h,cpp} exactly as they were: the MLA path keeps its own copy of the same node sequence. The nodes emitted are unchanged, so this is bit-identical. * llama: hold the qwen4exp indexer cache in a new llama_memory_hybrid_idx The indexer key cache was added by extending llama_memory_hybrid with an optional third cache, and the host-side cell/block mapping that drives QSA was added as set_input_qsa on llama_kv_cache. Both are shared classes that every hybrid and every attention model goes through. Move both into a new memory type, llama_memory_hybrid_idx, following llama_kv_cache_msa: the indexer cache and the pos<->cell translation live with the sparse-attention memory rather than in the classes that serve every other architecture. llama-kv-cache.{h,cpp} and llama-memory-hybrid.{h,cpp} are restored to their unmodified state. init_batch is repeated from llama_memory_hybrid because the indexer cache has to be handed the attention cache's slot infos, and those are not reachable through the context the base returns. Allocating them separately lets the two caches drift, which is what pointed QSA's top-k at the wrong cells before. The context derives from llama_memory_hybrid_context so build_inp_mem_hybrid keeps working unchanged, and get_n_stream is computed from the slot infos exactly as llama_kv_cache_context did. Behaviour is unchanged: logits over an 8192-token sequence are bit-identical to the previous implementation, sparse and dense alike. * llama: save and restore the qwen4exp indexer KV cache llama_memory_hybrid_idx forwarded clear, seq_rm, seq_cp, seq_keep, seq_add and seq_div to the indexer cache but not state_write / state_read, so a saved session dropped the indexer keys and a restored one selected QSA top-k against an empty cache. The effect is invisible until the context passes indexer_top_k + compress_ratio - 1 cells, because QSA is exactly dense below that and the indexer contents cannot change the result. The indexer section is written last rather than next to the attention cache it mirrors. As a suffix, a reader that does not expect it stops early and the trailing bytes are caught by the size check in state_load_file; placed between the attention and recurrent sections it would instead be parsed as recurrent state, which can succeed and restore silent garbage. It follows the same LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY gate as the attention cache, since a partial checkpoint deliberately skips the token-level attention caches. The indexer restores its own cells instead of taking the attention cache's restored slots. The two caches share size, padding and every sequence operation, and init_batch hands the indexer the attention cache's slot infos, so both state_read_meta calls run find_slot over identical occupancy and land on identical cells. The overrides live on llama_memory_hybrid_idx, the only memory type that owns an indexer cache, so llama_memory_hybrid and every architecture that uses it write and read exactly the bytes they did before. The session and sequence state versions are bumped because the qwen4exp state layout changed. The session path already rejects a short read via its size check, but llama_state_seq_load_file accepts one silently, so only the version check stops a pre-fix blob from being half-restored by a fixed build. (cherry picked from commit 2721542354f8e158c3217625f4e2e7b83e51e3fe) * llama: make the qwen4exp PLE n-gram history per context and serialise it The PLE hash of a token mixes in the ple_ngram_size - 1 tokens before it, which a decode ubatch does not carry, so they were remembered in a map on llama_model_qwen4exp. That is the wrong owner twice over. A llama_model is shared by every context that loads it, and the map was keyed only by llama_seq_id, so two contexts running the same sequence id - two server instances on one model, or a draft/target pair - overwrote each other's window. The next_pos guard turned that into EOS padding instead of a crash, so it degraded quality silently. The map was also in no state blob: grep found ple_hist in neither llama-kv-cache.cpp nor llama-memory-*.cpp nor llama-context.cpp. A restored context therefore failed the next_pos check on its first ubatch and hashed the first tokens after the restore against EOS padding. This is why a session blob round-tripped byte for byte while the restored context computed different logits: the state was never in the bytes. It moves to llama_memory_hybrid_idx, which is per context, is the memory type qwen4exp always builds, and already does the per-sequence bookkeeping this needs. Every sequence operation now carries the window with it: seq_rm a rewind (p1 < 0) truncates the window to the surviving prefix and moves next_pos to p0, so a rollback keeps exact context; a hole punched in the middle leaves the window non-contiguous, so it is dropped seq_cp the destination inherits the source's window, truncated to the copied position range - a copied sequence continues with the same n-grams the source would have used seq_keep every other sequence's window is dropped, like its cells seq_add a shift that moves the whole window keeps it and moves next_pos with it, which is the context-shift case; one that cuts through it drops it seq_div positions stop being consecutive, so an overlapping window is dropped clear everything is dropped Dropping means next_pos = -1, which set_input turns into full EOS padding: the same thing a fresh sequence gets, and the same thing this code did before it followed the sequence operations at all, so no case is worse than before. The state payload is a self-delimiting list, u32 count then per entry { i32 seq_id, i32 next_pos, u32 n_toks, i32 toks[n_toks] }, so a whole-context save and a single-sequence save share one format and a single-sequence restore can retarget the window at its destination seq_id. It is written after the indexer section, last, for the same reason that one is: as a pure suffix an older reader stops early instead of parsing these bytes as something else. Unlike the indexer section it is not under LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY. The window is recurrent state - it is the input the PLE convolution's own recurrent state is derived from - and the recurrent cache beside it is written for partial checkpoints too. Gating it would leave the server's speculative decoding checkpoints restoring the conv state without the window that produced it. No further version bump: LLAMA_SESSION_VERSION 10 and LLAMA_STATE_SEQ_VERSION 3 were introduced for the indexer section in the same unreleased series, and both changes are qwen4exp-only additions to the same blob layout. Also fixes the padding of a short window. set_input pads a window shorter than ngram_size - 1 up to that length, but prev() indexes the snapshot with the most recent token last, and resize() pads at the back, so the filler EOS landed where the immediately preceding token belongs. It now pads at the front. A window is short at a sequence start after a one-token prefill, and after a seq_rm rewind, which the new bookkeeping makes common. Every architecture other than qwen4exp builds llama_memory_hybrid rather than llama_memory_hybrid_idx, has no PLE table and never asks for a history, so nothing about its graph, its sequence operations or its state bytes changes. (cherry picked from commit de170364c052c68fcf63285cc0028095edb9f23c) * qwen4exp: tidy comments and simplify image token read Rewrite the comments this series adds to the AGENTS.md rules: one or two lines, no prose hard-wrapped mid-sentence, no narrative or history, and no comment that only restates the code. Net 146 fewer comment lines, no code change. Correct the PLE image comment: mtmd does not consume the placeholder ids. An image is decoded as an embeddings-only batch, so ubatch->token is null and the per-position ids never exist here. gemma3n and gemma4 hit the same case and stand in row 0 of per_layer_token_embd; qwen4exp stands in the configured image token id instead. Read image_token_id straight from self.hparams in the converter. base.py merges text_config into the root of hparams, and the key sits at the root of config.json, so the config.json re-read was redundant. (cherry picked from commit 205840c12169057da3e8d2f65ec4ceec3e18b980) * qwen4exp: support a quantized KV cache in the QSA attention path (cherry picked from commit 4c30574f81dc1115d08078c47b6cf8c789c0a842) * llama: give qwen4exp a large-graph node budget (cherry picked from commit 37c8c194e6a30e4c46ac29bee3fb264f091596ef) * qwen4exp: drop an unused variable that breaks -Werror builds (cherry picked from commit 528d032b51fa3cf935ed3ef6e0fb1c7401df53b5) * quantize: dequantize and quantize large tensors in row bands f32_conv_buf held the whole dequantized tensor, which is 204.8 GB for per_layer_token_embd alone and dies with std::bad_alloc long before the work buffer is reached. Dequantize and quantize in bands of whole rows instead, capping the f32 staging at 1 GiB per band. Rows are independent and the imatrix is indexed by column, so band boundaries cannot change any output byte. Bands nest inside the existing per-expert loop so each expert slice keeps its own imatrix, and a band is kept to at least one quantization chunk per worker thread so the existing multithreading still has work. F32 sources still stage nothing and are banded by pointer arithmetic into the tensor. llama_tensor_dequantize_impl now takes a first element offset; the single caller is updated. (cherry picked from commit 658c22549613555dbce57a772be4de8509eba3ee) * llama: segment the qwen4exp fused QKV for tensor split qwen4exp was missing from the gated delta net branch of get_split_segments, so its attn_qkv.weight, shaped {n_embd, 2*key_dim + value_dim}, fell through to the generic fused QKV rule and tripped GGML_ASSERT(tensor->ne[axis] == n_embd + 2*n_embd_gqa) while loading with --split-mode tensor. --split-mode layer was unaffected. qwen4exp broadcasts K to the V heads by tiling, k_conv is grown with a plain ggml_repeat_4d over the head axis so that v head j pairs with k head j % n_k_heads. That is the Qwen 3.5 pattern, not the repeat interleave that Qwen 3 Next builds explicitly, so qwen4exp takes the else branch and its V is segmented on the scale of K. Reported by benklop. (cherry picked from commit 353d753f595dc81634ae6130188b31f06018f5ae) * llama: fix the qwen4exp PLE history seq_rm(-1) iterator invalidation and the fatal-warning build ple_hist_rm recursed over ple_hist with a range-based for and the recursive call erases the entry it is iterating when the whole sequence is removed (p0 <= 0, p1 < 0), so the loop then increments an invalidated iterator. It is unreachable today only because llama_memory_recurrent::seq_rm rejects seq_id < 0 before llama_memory_hybrid_idx::seq_rm reaches the history, which is a guard in another class. Advance past the entry before recursing. Two smaller things in the same area: - the n_toks sanity bound in ple_hist_state_read was the literal 64, which is the value of LLAMA_MAX_PLE_HEADS, not of the quantity being checked. The window is at most ple_ngram_size - 1 tokens, so the bound is LLAMA_MAX_PLE_NGRAM - 1, eight times tighter. - build_conv_state_at left mem_size unused, so -DLLAMA_FATAL_WARNINGS=ON does not compile. Predates this series; drop the line. (cherry picked from commit 6eba44a89d5f328eb4859b844e1d28fb564cbe3e) * qwen4exp: include llama-impl.h explicitly for llama_mul_mat_hadamard (cherry picked from commit b634fd4d250d181ef82bf78bd00c1ae3b96a7af6) * convert: fix the qwen4exp lint and type-check failures flake8 flagged an unused MmprojModel import, and ty flagged seven errors in the PLE streaming path: eos_token_id can be absent, and _ple_map, _ple_path, _ple_row_dim and _ple_rows_per_shard are all Optional at the declaration but were dereferenced without narrowing. The map is opened and the stride fixed before the first shard is written, and _finish_ple_table only runs once every shard has landed, so the invariants hold. Assert them so the checker can see it. A missing eos_token_id now raises with the reason instead of a TypeError from int(None). * llama: give the qwen4exp indexer cache its own tensor names The indexer KV cache and the attention KV cache both named their tensors cache_k_l%d, so the Meta backend matched the indexer cache against the attention split pattern and aborted in handle_set_rows. Tag the names instead, and mirror the indexer cache: it has one key head and its projections are mirrored. (cherry picked from commit a1cdc8181134659766763a17762545a1f0e5db7b) * qwen4exp: double the Q split granularity for tensor parallelism qwen4exp fuses the attention gate into attn_q.weight the same way qwen3next and qwen 3.5 do, so a device boundary must fall on a whole q+gate pair or the Q heads stop lining up with the K/V heads and attn_output rows. (cherry picked from commit 6c9a592f0a425a459ab6efae3b897cf68460e244) * qwen4exp: keep the indexer cache in step across server slots The QSA indexer keeps a side cache addressed by the cells of the attention cache, so cell j has to hold the same token in both: the top-k indices it produces are applied to the attention KQ mask. init_batch already hands the indexer the attention cache's slot layout rather than letting it look for its own, but the restore path did not. state_read called llama_kv_cache::state_read on the two caches in turn and each ran its own find_slot over its own occupancy. That agrees only for as long as nothing has already pushed the two caches apart, which is the property a restore is supposed to re-establish rather than one it can lean on. The failure path was the worse half, and it is reachable from the public API with nothing more than a short buffer. Truncating a good blob at 35 offsets and feeding it to llama_state_seq_set_data left the two caches disagreeing at 5 of them, and every one of 23 truncations of a whole-context blob did. Four of those five land inside the attention section, so the attention cache drops the sequence and the indexer keeps it; only the cut that lands in the indexer section gives the opposite direction. llama_kv_cache::state_read cleans up its own cache and rethrows, so whichever way it falls, nothing is left to bring the two back together. The server papers over this by clearing the slot when a prompt cache load fails; a caller of llama_state_seq_set_data that does not is left with an indexer addressing cells that no longer mean what it thinks. llama_kv_cache::state_read_sinfo reports the cells a restore landed in, or takes a copy of them, and state_read_meta uses a supplied layout in place of find_slot once it has checked that those cells are free here too. The indexer now adopts the attention cache's restored layout by construction instead of reproducing it by coincidence, and a layout that does not fit fails the read rather than being applied over cells that already drifted. The hybrid restore is wrapped so that any failure drops the sequence, or for a whole-context restore the context, from all three caches at once, which is a state they do agree on. * kv-cache: clear the cache once when restoring a whole context state_read walks the streams of the cache in turn, and for a whole-context restore each stream went through state_read_meta, which starts by calling clear(). clear() resets every stream at once, so each stream after the first threw away the streams already restored, and the K/V buffers with them. A non-unified cache holds one stream per sequence, so a context saved with N sequences in it came back with only the sequence in the last stream that carried any cells - the highest sequence id. A unified cache has one stream and never showed it. The cache is now emptied once, before the loop, which is what a whole-context restore means. A blob whose streams are all empty now empties the cache as well, where before it left the old contents in place. * kv-cache: check the mirrored slot layout on a whole-context restore too state_read_meta only looked at the layout it was given on the single-sequence path. A whole-context restore lays the cells out from 0 in both caches, so they agree as long as they restore the same number of cells, but nothing checked that they did: an indexer section belonging to some other context was read over cells the attention cache had filled from a different one, which is the state the indexer must never be left in. * qwen4exp: give the PLE conv history its own mirrored recurrent row n_embd_r() reserved n_conv + ple_conv_state() so that one cache_r_l row could carry both the delta-net conv state and the PLE dilated conv history, but the QWEN4EXP arm of get_split_segments only described n_conv. Under -sm tensor the segment sum came up short by ple_conv_state() and llama_memory_recurrent construction aborted in ggml_backend_meta_alloc_ctx_tensors_from_buft. Widening the segment list is not the fix. The Meta backend propagates a view's split descriptor from its parent unchanged, so a view of one sub-range of a split axis is sized as the whole row on every device; declaring the PLE tail as a second segment merely moves the abort to "shape mismatch for VIEW" at graph allocation. The two histories also want opposite policies: the delta-net state is split by head to match wqkv and ssm_conv1d, while per_layer_tok_embd, ple_conv1d and ple_norm_conv are all mirrored, so every device computes the whole dilated conv and needs the whole history. One tensor cannot be both, and the split state has no per-segment mirroring. Move the PLE history into its own cache_ple_r_l%d row, mark it MIRRORED, and return n_embd_r() to n_conv. The row is allocated only on layers where is_ple holds, so mirroring one 92160-element row per device replaces a 92160-element tail on all 36 recurrent rows: the recurrent R footprint drops rather than grows. build_conv_state_at now takes its width from the tensor it was handed and keys its gather on that tensor, which also drops a cont of a strided view. * no more ple_hist (use master version) * llama: give the qwen4exp full memory context its indexer cache graph_reserve() walks a full memory context, and qwen4exp builds its sparse attention only when the context exposes an indexer cache. the full-context constructor left ctx_idx null, so the reserved worst case was the dense fallback: a smaller graph than the one decode executes. ggml-alloc then had to grow the compute buffer on the first decode, past the size reported at load. with -np 4 -c 32768 -fa on -ctk q8_0 -ctv q8_0 on an IQ1_S qwen4exp, the reserved CUDA0 buffer was 217.00 MiB against 275.71 MiB actually used, and CUDA_Host 42.31 MiB against 191.14 MiB. reserving the sparse graph makes both match exactly, in unified and non-unified cache mode. Co-authored-by: Pascal <admin@serveurperso.com> Assisted-by: Claude * qwen4exp: shrink the PLE hparams storage llama_hparams is held by value inside llm_graph_params and every llm_graph_input_*, and llm_graph_params is a stack local in graph_reserve and process_ubatch, so its width is paid on every worker thread stack. is_ple_impl spent 2048 bytes carrying 512 bits. It is the one per-layer flag that is not moved through the loader's uint32 array templates, so a bitset costs nothing in call sites and also removes the uninitialized read that non-qwen4exp archs had, since nothing filled the array for them. The PLE head offsets and vocab sizes are token-space indices; the gather that consumes them already truncates to int32, so 64-bit storage was never reachable. The gguf arrays stay uint64 for file compatibility and are narrowed on load. sizeof(llama_hparams) 34440 -> 31944, sizeof(llm_graph_params) 34872 -> 32376. * llama: opt-in random-access mmap advice for host-resident gather tables qwen4exp keeps per_layer_token_embd on the host: 26.8 GiB at IQ4_NL, read by ggml_get_rows as 16 gathers of ~90-170 bytes per token, spread across 16 head regions ~20M rows apart. Measured over 4.75M gathers, no two consecutive gathers land on the same 4 KiB page, so the readahead the loader asks for buys nothing here and the whole table ends up cached to serve about 4% of itself. llama_mmap applies POSIX_FADV_SEQUENTIAL, MAP_POPULATE and a whole-file POSIX_MADV_WILLNEED unconditionally. Those are right for streaming the file once into buffers and wrong for whatever stays mapped afterwards. Under LLAMA_MMAP_RANDOM the eager pull-in is skipped and the mapping is advised random once every tensor has been read, so the load itself keeps its sequential readahead. That alone drops the table to 4.4% resident but serializes one NVMe latency per gather. The second half is what pays for it: the PLE input already computes every row index for the ubatch before the graph runs, so the pages those rows fall on are handed to the kernel in one batch and the reads overlap. POSIX_MADV_WILLNEED on POSIX, PrefetchVirtualMemory on Windows, which takes the discontiguous ranges in a single call. Off by default and off for every other model: the batched prefetch keys off "this mapping was advised random", which nothing sets unless the user opts in. -c 512 --chunks 60, cold, IQ1_S, mean of 3: default 35.3 s 26.82 GiB resident (100%) advice only 104.5 s 1.19 GiB resident (4.4%) advice + prefetch 34.2 s 1.19 GiB resident (4.4%) PPL 4.2346 +/- 0.07862 in all three. IQ1_S KLD is unchanged in every field, including Mean KLD 0.396070 +/- 0.001931 and Same top p 77.325%. * llama: narrow the random-access mmap advice to the gather table The advice was applied per mapping: every mapping the model kept got POSIX_MADV_RANDOM plus a whole-file POSIX_FADV_RANDOM, and the eager pull-in was skipped for every file. On qwen4exp that also hit token_embd.weight, which sits 0.33 GiB past the PLE table in the same shard and is read densely, not by sparse gathers. Measured over -c 512 --chunks 60 on IQ1_S it fell to 8.45% resident, against 100% with the feature off. A model now nominates its gather tables (qwen4exp: per_layer_tok_embd) and only those byte ranges are advised. The range is rounded out to whole pages, which on this model takes in 832 bytes before and 192 after. token_embd goes back to 86.55% resident and the PLE table still drops to 4.44%; smaps shows one VM_RAND_READ VMA of exactly the table instead of one over all 27.16 GiB that stays mapped. posix_fadvise is dropped from the narrowed path. POSIX_FADV_RANDOM ignores its offset and length and marks the whole open file, and the FMODE_RANDOM it sets is only read by page_cache_sync_ra() on the read() path, which a fault on a MADV_RANDOM vma never reaches. POSIX_FADV_ DONTNEED does take a range, so the drop mode keeps it. The eager pull-in is now skipped only for the files holding a nominated table, and re-issued as WILLNEED over the rest of such a file, so other shards load exactly as before. prefetch_rows() keys off the tensor being nominated rather than off a mapping-level flag, so the batched readahead lands only where the advice did. -c 512 --chunks 60, cold, IQ1_S, mean of 3, total wall: default 32.50 s whole mapping 30.05 s narrowed 30.35 s PPL 4.2061 in all three. IQ1_S KLD is bit-identical with the feature on and off, including Mean KLD 0.396070 +/- 0.001931 and Same top p 77.325%. tg128 73.65 +/- 0.33 narrowed against 73.49 +/- 0.34 whole. Assisted-by: Claude * llama: fold the random-access prefetch into its own feature flag LLAMA_MMAP_RANDOM_PREFETCH existed to measure the two halves of the feature apart, and the measurement is done: on a cold cache over the same wikitext run, MADV_RANDOM without the batched readahead takes 94.4 s against 36.7 s for an untouched mapping, while the pair together take 34.1 s. Suppressing the kernel's readahead only pays if we replace it, so the split let a user select a 2.6x regression through a documented switch. Keep the accessor, since the call site reads better than a mode comparison, but derive it from the mode alone. * FACP (Fewer Acronym Classes Please) * qwen4exp: bias the QSA selection per block, not per cell The QSA bias is a graph input, so it is pinned on the host and uploaded every decode, and at -c 32768 -np 4 its twelve copies were 768 of the 815 MiB of reserved host compute buffer. Only one half of it needs a cell: whether the cell sits in the always-visible tail, and whether its block was pooled. Both are properties of the block. The other half - empty, other sequence, or in the future - is the plain visible/not test the attention mask already carries over the same cells, so add that mask instead of repeating it. The bias then holds one value per block. A block sits wholly inside or wholly outside the tail because the tail starts on a block boundary, so one value per block is exact. Cells no block covers keep their -inf from the mask. The mask is F16 and the bias F32, and a mixed ggml_add reinterprets the F16 buffer as float rather than converting it, so the cast is required. reserved host compute buffer at -c 32768 -np 4: --kv-unified 814.86 -> 238.86 MiB, CUDA0 721.07 -> 421.07 MiB --no-kv-unified 214.86 -> 70.86 MiB, CUDA0 317.07 -> 265.07 MiB Selection is unchanged: over 8192 tokens, four times the budget, every QSA layer returns identical top-k indices and the logprobs are bitwise equal. Two things a reviewer should know. A cell whose position divides past the last block is guarded by an assert rather than handled, because no run reached it. And the mask's same-position M-RoPE rule cannot fire for text and was never exercised for images, so the 2D case is unverified. * clean up code comments * clean up new comments * revert LLAMA_MMAP_RANDOM * nits * replace some changes with #27795 * improve the m-rope image for get_prev_tokens * LazyChunkedTensor * fix lint * add some validations * reduce input nodes * trim output tokens * nits * some more sanity checks * fix llm_graph_input_ple reuse * exclude from webgpu test --------- Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com> Co-authored-by: danielhanchen <unslothshared@gmail.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: Pascal <admin@serveurperso.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>	2026-08-27 21:32:31 +02:00
Xuan-Son Nguyen	732707dff2	quantize: cap working memory size to avoid loading big tensors onto RAM (#27795)	2026-08-27 18:31:13 +02:00
Xuan-Son Nguyen	fac889fb38	llama: model_loader: add TENSOR_READ_LAZY (#27794) ... * llama: model_loader: add TENSOR_GET_ROW_LAZY * add --tensor-read-lazy * rename to TENSOR_READ_LAZY * gen docs * address comments	2026-08-27 15:14:34 +02:00
Ed Addario	0596704284	quant : Optimise memory usage by evicting weights after processing each layer (#22877) ... * Evict weights from memory after processing each layer * Revert changes * Move unmap to libllama * Unmap weights offloaded to backend * Change member's constness * Remove unmap weights offloaded to backend	2026-08-18 16:22:32 +02:00
Xuan-Son Nguyen	27df9199d1	fix: check gguf array type before reading (#27075) ... * fix: check gguf array type before reading * update skill	2026-08-15 11:45:30 +02:00
Fathi Boudra	0177dcc730	common: migrate the deprecated --mmap/--no-mmap to --load-mode (#26934) ... Replace the deprecated --mmap, --no-mmap, --mlock, and --direct-io flags with the unified --load-mode argument across scripts, examples, and documentation. Internal warning message and env var docs updated accordingly. Signed-off-by: Fathi Boudra <fathi.boudra@linaro.org>	2026-08-15 16:35:53 +08:00
3 people	1692f9e50b	ggml : recurrent state rollback for ggml_ssm_scan (#26623) ... * Initial changes for Recurrent state rollback for nemotron for cpu and cuda * Removing CPU RS rollback. Will enable it in subsequent PRs * addition of test case * Removing assert and calling runtime API to check if op is supported * removing extra API and updating the call sites for K * replace static cuda detection to runtime fused_op api * address review comments and fallback when SSM rollback not supprted * Adding changes for supporting RS-rollback in CPU. Also added test-backend-ops for cpu and cuda * removing memory manipulation as rs rollback is now supported in CPU * removing the static probe which is not needed now * correcting the format * address review comments * enabling test for all the backends, unsupported backends will fallback to CPU * Apply suggestions from code review Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * choose different graph based on the result of fused_ssm_op is supported or not and also handled memory->n_rs_seq >1 case incase of op is not supported * Support K > 1 in ssm_scan for all backends * Fix CI Issues --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>	2026-08-14 17:20:40 +03:00
4 people	153d324bcf	llama: add default load-mode auto, which avoids mmap on iGPUs (#26081) ... * llama: add new default load-mode auto which picks mmap unless a non-Metal iGPU is used * Update ggml/src/ggml-hexagon/ggml-hexagon.cpp Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> * set mmap_support to false on OpenCL backend * fix order of load modes * use -1 for auto * resolve load mode auto earlier to correctly pick gpu host or cpu memory * add load mode auto to llama-bench * bump virtgpu api version, regenerate docs --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2026-08-11 09:20:46 +03:00
Bar Haim	157b81fe6d	model : Granite-Switch Architecture (#25107) ... * granite-switch: add llama.cpp backend (POC, CPU) New "granite-switch" architecture: a dense, all-attention Granite-4.1 model with N embedded LoRA adapters selected per-token by control tokens. - gguf-py schema (arch, KV keys, stacked LoRA tensor names) + writer helpers - conversion/granite.py: GraniteSwitchModel converter (stacks N adapters + zero base slot into per-projection A/B tensors; emits switch metadata) - C++ arch registration (llama-arch.{h,cpp}, llama-model.{h,cpp}) - src/models/granite_switch.cpp: load + per-token switched-LoRA graph via ggml_mul_mat_id over stacked tensors; sticky per-token index + control-token substitution in llm_graph_input_switch::set_input - llm_graph_input_switch in src/models/models.h Runs end-to-end on CPU: convert 3b checkpoint (842 tensors, stacked dim 13) and generate on both base and control-token paths. Sticky switch state is single-sequence (POC); full multi-sequence machinery is a follow-up. * granite-switch: add Mac (Metal) build + mid-sequence switch demo script Self-contained script to build llama.cpp on Apple Silicon (Metal), convert the composed 3b checkpoint, and run the crisp mid-sequence adapter-switch demos verified on Vela: - answerability: <|answerability|> mid-seq -> "unanswerable" - query_rewrite: <|query_rewrite|> mid-seq -> {"rewritten_question": ...} Each demo runs the same prompt twice, differing only by a control token placed before the assistant turn, so the per-token switch is visible. * granite-switch mac demo: add -no-cnv so each run is one-shot The composed model ships a chat template, so llama-completion auto-enables interactive conversation mode and halts at a `>` prompt after generating, stalling the script. -no-cnv disables conversation mode: generate once from the raw prompt and exit (also prints special tokens, making the switch visible). * granite-switch: replace global sticky index with in-graph router attention The POC computed the per-token adapter index on the CPU and carried it across ubatches in ONE global `mutable int32_t poc_sticky_index`, reset only when a ubatch contained sequence position 0. That global had two problems: 1. Concurrency: with multiple sequences in a batch it was last-writer- wins — one sequence's adapter leaked into the others. 2. Multi-turn: an interactive `ollama run` chat continues one KV cache, so turn 2 never saw position 0 and the index never reset — the adapter stayed stuck on across turns. Port the vLLM/HF backend mechanism faithfully: a single-head causal "router" attention recovers the adapter index in-graph. Per token, only dim 0 carries signal — Q[0]=1, K[0]=+gain for a control token / -gain otherwise, V[0]=adapter slot / 0 — and the causal softmax over the single visible control token recovers that adapter's slot (readback = clamp(round(V[0]), 0, n_adapters)). gain=15 matches config.py and is F16-safe (no F32 cache). The router's K/V live in the model KV cache at an extra layer R == hparams.router_layer (== n_layer). We bump n_layer_all to n_real+1 so the cache allocator gives the router its own per-sequence slot, and set n_layer_nextn=1 so n_layer() stays n_real — the decoder loop and tensor loading are untouched and never reference layer R. The router K is exempted from the k-shift RoPE loop (its dim-0 value is a literal magnitude, not a rotation). Because the selection now lives in the per-sequence KV cache, CONCURRENT requests are isolated for free (problem 1 fixed; verified by scratch/concurrent_switch_test.cpp). set_input becomes stateless pure per-token maps; the global is gone. Single-switch contract / known limitation, identical to vLLM & HF: the gain is flat (no recency), so within one sequence there is no mechanism to revert to base mid-sequence — once an adapter fires it stays on until that sequence ends (problem 2 is therefore NOT fixed by a faithful copy; vLLM/HF avoid it only because each served request is a fresh sequence). A client continuing one KV cache across turns must start a fresh sequence per turn, or opt into a recency-biased router (a deliberate divergence, not done here). Documented in granite_switch.cpp and asserted by scratch/multiturn_leak_test.cpp. Verified (CPU): both demos unchanged (answerability -> "unanswerable", query_rewrite -> rewritten query); concurrent two-sequence isolation passes; multi-turn carry-over matches the vLLM/HF contract. * granite-switch: drop scratch tests and mac demo for upstream PR Remove the local-only development artifacts that should not ship in the upstream PR: - granite-switch-mac-demo.sh (local Metal build + demo driver) - scratch/concurrent_switch_test.cpp - scratch/multiturn_leak_test.cpp Also drop the now-dangling reference to the scratch tests from the granite_switch.cpp header comment. Leaves only the core architecture support (conversion, gguf constants, llama-arch/model/kv-cache, and the granite_switch graph). * granite-switch: trim comments to match native llama.cpp style * granite-switch: trim conversion comments to match native style * granite-switch: drop unused adapter_ranks metadata * granite-switch: rename arch to graniteswitch and drop obid alias * granite-switch: fix non-ASCII comments and document router gain assumption * granite-switch: drop section comments from constants.py to match native style * granite-switch: add functional tensor block comments matching Granite4 Vision style * granite-switch: clarify n_expert_used comment State the actual constraint: mul_mat_id needs n_expert_used == 1, and since the GGUF carries expert_count = 0 the generic loader's n_expert == 0 => n_expert_used == 0 assertion has already passed by the time load_arch_hparams runs, so it is forced to 1 here. * granite-switch: note n_layer_nextn reuse has no MTP The router carving reuses n_layer_nextn, normally the MTP/next-token count. Clarify in the comment that it is borrowed here purely as the trailing-layers lever and that there is no MTP head, to spare readers the double-take. * granite-switch: rename source file and apply review nits * granite-switch: don't force LoRA tensors to F16, follow --outtype instead * granite-switch: drop redundant _permute_qk wrapper, call LlamaModel.permute directly * granite-switch: read router gain from GGUF (control_token_gain) instead of hardcoding 15.0 * granite-switch: derive n_slots() * granite-switch: move llm_graph_input_switch into granite-switch.cpp * granite-switch: cut AI-style narration comments * granite-switch: collapse multi-line comments * granite-switch: rename control_token_* maps to adapter_token_* * granite-switch: cut noise comments * granite-switch: rename embedded LoRA tensors to <base>.lora_a/lora_b * granite-switch: GGML_ASSERT token input to avoid UB on embeddings * granite-switch: TODO for raw embedding input support * granite-switch: collapse LoRA tensor constants to .lora_a/.lora_b suffix * granite-switch: drop n_expert_used hack, guard mul_mat_id buft probe * granite-switch: stop forcing dense expert counts, read from config * granite-switch: renamed control_token_gain metadata key to router_gain * granite-switch: trim header comments to match native style * granite-switch: collapse LoRA tensors to base name + suffix * granite-switch: inline suffix checks in tensor op resolution * granite-switch: drop switch-lora struct comment * granite-switch: guard router layer index and inline n_slots * granite-switch: group adapter metadata under {arch}.adapters.* namespace * granite-switch: add hparams.has_rope(il) for KV-shift rope skipping * granite-switch: skip arch in test-llama-archs (adapter fixture missing, TODO) * granite-switch: Keys.Adapters namespace + simplify n_slots * granite-switch: validate substitute token ids against n_vocab * granite-switch: bound adapter count and lora rank from GGUF * granite-switch: reject MTP context type when router_layer is set * granite-switch: throw on bad adapter metadata instead of GGML_ASSERT * granite-switch: use ASCII +/- in router K signal comment * granite-switch: document n_layer_nextn repurpose and its leak points * granite-switch: gate lora_a/lora_b op mapping on router_layer * granite-switch: label all three preview model sizes	2026-08-10 09:53:46 +02:00
Georgi Gerganov	3db4ff877d	model-loader : fix quantized reshaped tensor strides (#26672)	2026-08-06 15:21:44 +03:00
Georgi Gerganov	1269cb1ff1	model : allow reshape of tensors during load (#26531)	2026-08-04 09:06:44 +03:00
fairydreamingandStanisław Szymczyk	82dbc4f017	llama : load MTP tensors only if they are really used (#26296) ... * llama : load MTP tensors only if they are really used * llama : skip loading MTP (if not used) in remaining models that support MTP --------- Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>	2026-07-31 14:57:02 +02:00
Aaron Teo	ad256ded30	args: add -lm mlock where it mlocks but doesnt mmap (#26135) ... * arg: add `-lm mlock` where it mlocks but doesnt mmap Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: rm unwanted docs changes Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: revert auto-formatting Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * bench: fix automated review point 3 Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * arg: revert the meaning of --mlock to non-mmap'ed mlock Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: update docs Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: remove extra changes from `llama-gen-docs` Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> --------- Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>	2026-07-27 16:44:08 +08:00
Aaron Teo	e6dd0e29a6	args: refactor mlock/mmap/directio into load-mode (#20834) ... * args: overhaul mmap/mlock/dio into single arg Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: update docs with llama-gen-docs Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * chore: satisfy code quality Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * args: make the `+` sign an actual modifier now Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * chore: general code clean up + comments Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * arg: fix deprecated flags support Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * arg: quick sanity check Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * bench: sync llama-bench argument parsing Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * fix: bugfix variable behaviour + llama-bench lm column size Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * arg: inverse commands should do the opposite instead of doing nothing Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * bench: fix incorrect dash Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * bench: fix missing modifiers for deprecated flags Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * llama: switch back to thread_local Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * arg: switch back to single enum Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: update arg docs Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * chore: fix missing `mlock` from llama_load_mode_from_str + cleanup llama-bench Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * llama: fix mlock not activating Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * arg: add deprecation warning when old and new flags are combined Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * arg: cont add comment for todo in the future Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: sync with upstream Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: re-sync with upstream again Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> --------- Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>	2026-07-23 20:32:56 +08:00
Pasha Khosravi	bec4772f6a	Add Q2_0 quantization: type definition and CPU backend (#24448)	2026-07-07 12:05:47 -07:00
Adrien Gallouët	fdb1db877c	llama : add llama_model_ftype_name() (#25134) ... * llama : add llama_model_ftype_name() Expose the model file type (quantization) name, e.g. "Q8_0" or "Q4_K - Medium", through a new public C API. The returned pointer is valid for the lifetime of the model and nullptr when the model is invalid or the file type is unknown. Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Export enum Signed-off-by: Adrien Gallouët <angt@huggingface.co> * s/llama_model_ftype_name/llama_ftype_name/ Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Move "(guessed)" to the front in llama_ftype_name Prepend the "(guessed)" label instead of appending it. This allows removing the non-thread-safe static std::string, making the function allocation-free. Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Add LLAMA_FTYPE_PREFIX Signed-off-by: Adrien Gallouët <angt@huggingface.co> * Dont check for model Signed-off-by: Adrien Gallouët <angt@huggingface.co> --------- Signed-off-by: Adrien Gallouët <angt@huggingface.co>	2026-07-02 17:26:47 +02:00
6 people	8c146a8366	DeepSeek V4 (#24162) ... * convert: add dsv4 conversion * add basic setup * add llm_graph_input_dsv4 * add save-load state * add sinkhorn eps - correction by @fairydreaming * add rope fix * cleanup dead code * fix bugs * support pro model: added by @fairydreaming * remove redundant V cache * Chat template * remove debugging leftovers * Add mechanism for inlining templates based on architecture * s/deepseek-v4-flash/deepseek4/g * s/deepseek-v4-flash/deepseek4/g continued * enable graph reuse * enable FA * fix test llama archs * rename * compatibility with antirez ds4 GGUFs * simplified set_gguf_parameters() by calling super class method, replaced moe.score_func with expert_gating_func. * reserve worst-case kv-cache * revert max split inputs * address review comments * add padding to enable FA * pad only the final value of plan.n_kv to 256 * remove built-in cpp chat template * cont: remove cpp built-in template * rm outdated test * replace ggml_view_3d() with ggml_reshape_3d() Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * only support n_seq=1 for now * remove unused var * cont: remove unused var * use scale bias * use correct ptr for can_reuse * remove gen-chat-inline-templates.py * simplify graph reuse * cont: cleanup * remove unused inputs * enable partial checkpointing * add correct shape for kq_mask + set llama_model_n_swa to 0 for dsv4 * precompute source_idx + add comment about dummy write * support multi-seq * remove restored_trim_pos * use split_equal when possible * fix indent * address review comments * use LLM_KV * fix ci --------- Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com> Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2026-06-29 16:58:51 +08:00
5 people	88a39274ec	spec: add EAGLE3 speculative decoding support (#18039) ... * llama : enable layer input extraction * spec: support eagle3 * eagle3: fix params bug * eagle3: support Gemma4 eagle3 from RedHatAI * eagle3: set sync when get features from target Co-authored-by: tnhnyzc <115956684+tnhnyzc@users.noreply.github.com> * eagle3 : fix ubatch handling in embd_layer_inp extraction and encoder Co-authored-by: Doğaç Eldenk <dogacel@gmail.com> * eagle3: adapt to upstream changes * eagle3: fix rebase issues and adapt to upstream changes * eagle3:exclude the eagle3 arch from test-llama-archs * eagle3: fix editorconfig check failures * eagle3: fix multi-seq issue in d2t vocab mapping * cont : minor style / clean-up * spec : remove `common_speculative_setup_draft_model()` * llama : clean-up unused API * eagle3: set d2t vocab mapping in decode graph * cont : assert layer inputs are configured * hparams : use n_embd_inp instead of n_embd_target_features * eagle3: make output.weight optional and inherit from target model when needed * haparams : generic norm-before-residual param * llama-ext : consistent names * cont : fix * hparams : remove target_hidden_size * cparams : rename output_layer_inp -> embeddings_layer_inp * arch : reuse ATTN_NORM_2 instead of adding new hidden norm * llama : clean-up names * cont : add assert + comment * Update conversion/llama.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: tnhnyzc <115956684+tnhnyzc@users.noreply.github.com> Co-authored-by: Doğaç Eldenk <dogacel@gmail.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>	2026-06-12 10:21:06 +03:00
Gabe Goodhart	64086f2b2f	model, mtmd: Granite4 Vision (#23545) ... * feat(convert): Get language model conversion working for 4.1 vision Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(convert): Skip multimodal tensors for GraniteMoeHybrid (vision 4.0) Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Disable vocab padding for non-hybrid models that use GraniteMoeHybrid Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Plumb python-side vision projector names and mappings There are several awkward things here: 1. Most of these are essentially identical to the audio qformer tensors. On the c++ side, that's mapped using the prefix, so the rest of the GGUF name needs to align, but on the python side there's no prefix notion, so they all get duplicated. 2. There are a couple of net-new tensors for vision, in particular PROJ_NORM. In both speech and vision, the QF_PROJ_NORM is qualified as belonging to the qformer portion, but the GGUF name is simply proj_norm which conflicts with the ideal name for this new PROJ_NORM that is not qualified as part of the qformer. To get around this, I used "proj_layernorm" as the GGUF name. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add python side architecture name Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add python-side plumbing for setting FEATURE_LAYERS hparam Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add c++ side tensor naming defines NOTE: Usage of these hasn't been updated to include prefix yet Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(mtmd): Convert vision_feature_layer to an ordered vector We need to preserve the ordering of these feature index values so that they can be mapped to the sub-tensors within the stacked projectors. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(mtmd): Add architecture label plumbing Branch: Granite4Vision AI-usage: full (OpenCode + qwen3.5:122b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(wip): Add partial conversion for mmproj This handles stacking the projector tensors and setting the new harams Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add gguf_writer and constant support for new hparams and deepstack layer arr Branch: Granite4Vision AI-usage: draft (OpenCode + qwen3.5:122b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Full conversion for mmproj w/ tensor mappings Branch: Granite4Vision AI-usage: full (OpenCode + qwen3.5:122b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Add lm_head skip for mmproj for 4.0 Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: De-alias text_config architecture in convert_lora_to_gguf.py Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add --trust-remote-code arg to convert_lora_to_gguf.py This defaults to False, but allows a user to enable it programmaticly instead of using the interactive prompt. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: De-alias model.language_model. -> model. for lora adapters Branch: Granite4Vision AI-usage: full (OpenCode + qwen3.5:122b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Extend language model tensor dealiasing in adapters Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Remove unnecessary registration for GraniteSpeech in language model Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Plumb through mm prefix formatting for qformer tensors Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: Refactor vision projector tensors to use predictor ID as the block This is cleaner than stacking them. The modeling file hard-codes single-layer qformers, so we can punt on the multiipule multi-layer projectors problem. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add spatial offests array hparam conversion Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add stub plumbing for granite vision in mtmd Branch: Granite4Vision AI-usage: draft (OpenCode + qwen3.5:122b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add new hparam and tensor naming in clip-impl.h New hparams: - KEY_PROJ_SAMPLE_QUERY_SIDE - KEY_PROJ_SAMPLE_WINDOW_SIDE - KEY_PROJ_SPATIAL_OFFSETS New tensors: - TN_MULTI_PROJ_IMG_POS - TN_MULTI_PROJ_QUERY - TN_MULTI_PROJ_LAYERNORM - TN_MULTI_PROJ_LINEAR - TN_MULTI_PROJ_NORM Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Move deepstack_layer_arr to llm hparam instead of mmproj Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Remove IS_DEEPSTACK_LAYERS This appears to have been added during Qwen3 VL (https://github.com/ggml-org/llama.cpp/pull/16780), but it was never actually used. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: n_deepstack_layers -> deepstack_layer_arr The old logic hard coded a correspondence between the first N layers of the LLM and the 1->N entries in the input embeddings. Now, that relationship is maintained at loading time if the GGUF value is single-valued. If it is multi-valued, it loads directly allowing for deepstack layers to be spaced out throughout the model. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Use try/catch for single/multi valued deepstack info The alternative would be to use get_key_or_arr, but then the single value would be populated through the entire array and we'd need to detect that and update it with the right correspondence. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add deepstack injection point for granite LLM The use of ggml_add here assumes that the elements of inp_embd will be pre- arranged to be the full embedding length with only the vision-mask'ed portions non-zero from the projector. This matches how Qwen3VL does it. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: add missing vision attn layernorm eps Branch: Granite4Vision AI-usage: full (OpenCode + Qwen 3.6-35B) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: Hoist qformer tensors into qf_block and hold a vector for multi-proj Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Fix missing prefix template for TN_QF_PROJ_LINEAR It's not strictly necessary since vision uses the blockwise version, but it makes the loading consistent. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Add embedding scale and image grid pinpoints hparams in conversion Also remove dead parsing for self._deepstack_layer_arr Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add mtmd KEY_ section for hparams shared with the LLM In this case, we need the EMBEDDING_SCALE so we can unscale the image embeddings to compensate for applying embedding scale to the input embeddings Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Implement c++ hparam parsing Branch: Granite4Vision AI-usage: draft (Claude Code) Co-authored-by: Eli Schwartz <eliyahu.schwartz@ibm.com> Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Flatten pinpoints in conversion Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Add missing break Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: No reason to have modality prefix for img_pos Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add tensor loading Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix(convert): Fix confusion between proj.norm and proj.qformer.layernorm Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Use the right portion of speech for tensor loading! Also plumb through the layernorm -> post_norm naming change Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add logging of deepstack_layers_arr if set I also changed the print_f output type to int32_t to avoid printing overflow values for -1. This could cause overflows on the other side, but I can't imagine a value for any of the current array hparams that would trigger that. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Make sure input embeddings are cont before f_embedding_scale Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add init and mmproj_embd cases for g4v The n_mmproj_embd is 1+ to make space for the text embedding and all 8 projectors Branch: Granite4Vision AI-usage: draft (Bob) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Invert (h, w) -> (w, h) pinpoints Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Reorder projectors based on llm index and skip the first injection The multi-projector stack has a strange asymmetry based on how it's currently implemented for qwen3vl: on the mmproj side, it's all N projectors, but the output of the "first" (by inp_embd index) projector is automatically consumed as if it were a standard single-projector mmproj, so the deepstack portion needs to only contain the 1-N entries. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Eli Schwartz <eliyahu.schwartz@ibm.com> * fix: Fix mmproj hparams in conversion Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Eli Schwartz <eliyahu.schwartz@ibm.com> * fix: Fix ordering/logic for deepstack injection in granite Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Eli Schwartz <eliyahu.schwartz@ibm.com> * fix: Fix preprocessing config to match what the model needs Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Eli Schwartz <eliyahu.schwartz@ibm.com> * wip: Partial port of Eli's implementation This is still pretty broken, but it's getting closer. It now happily generates tokens, but the values are quite incorrect still. I suspect it's caused by the mapping of projectors from safetensors to their respective orders here. Also, this implementation breaks encapsulation pretty badly in mtmd_encode. This will need a big refactor to put the G4V-specific encoding logic somewhere more appropriate. Branch: Granite4Vision AI-usage: draft (Claude Code, Bob) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Eli Schwartz <eliyahu.schwartz@ibm.com> * fix: Fix the pre-scaling on the input embeddings to correctly invert the scale We've got tokens! They still don't line up quite right, so something's a little off, but we're getting much closer now. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: invert embedding multiplier -> base_scale at load Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Fix setting image_resize_pad after new enum introduced Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Add G4V to mmproj mapping in conversion Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Re-add padding disable for non-hybrid hybrid models Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: Simplify G4V n_tokens computation This is slightly more efficient and flexible for when we implement the unpad cropping. IMO, it's also clearer that it is adding the number of image_newline tokens (embeddings) to the grid, rather than recomputing the entire count. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add new clip APIs for post-tile-encoding assembly Granite 4 Vision uses llava-next style pack-and-unpad which requires injecting the learned newline after each row of the tile grid. A row here is a single row of the grid which is composed of (grid_x * cols_per_tile) * (grid_y * rows_per_tile), so the result is newlines injected in between individual tile rows, thus not something that can be handled with the standard llava-uhd block-wise endcoding. Branch: Granite4Vision AI-usage: draft (Claude Code + Opus 4.7) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add model interfaces for granite 4 vision assembler I'm on the fence about the best organization of this. These free functions allow the per-architecture logic in clip.cpp to access the model-specific graph building, but they still require a fair bit of model-specific logic in clip.cpp which is not ideal. I think a better approach may be to replicate what is done with the graph builders themselves (and possibly even make the assembler part of the model's existing graph builder). Branch: Granite4Vision AI-usage: full (Claude Code + Opus 4.7) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: Remove all g4v-specific branching from mtmd.cpp in favor of clip assembler Branch: Granite4Vision AI-usage: full (Claude Code + Opus 4.7) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor(mtmd): Consolidate assembler logic into clip_assembler class family Just like `clip_graph` is the base class for building the model-specific encoder graphs, `clip_assembler` will be the base class for building the model-specific assembler graphs. This allows the assembly pattern to follow how the encoder pattern is implemented where the model-specific logic lives in a subclass co-located with the encoder graph builder that gets constructed by a simple factory method. Branch: Granite4Vision AI-usage: full (Claude Code + Opus 4.7) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: Comment improvement Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: granite_vision -> granite4_vision Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Remove dead codepath for Qwen3VL add_vision_is_deepstack These pieces were never used on the c++ side (removed there in an earlier commit), so this is just cleanup that I missed before. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Oops! I did not mean to commit one of my prompt files But now it's too far back in history to effectively rebase out, even with interactive and --rebase-merges :( Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Add missing <algorithm> include for std::find It seems that this was already pulled in on some platforms, but not on others Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Fix Flake8 warnings in granite conversion module Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: Remove clip_assembler in favor of clip_image_f32.append_token Per conversation in the PR, the clip_assembler pattern was too invasive. This is a compromise that limits model-specific blocks to add_media where each preprocessed tile is annotated with an injection type, after which all the token counting logic is generic and the newline injection itself is handled in the graph based on the value for the given tile image. Branch: Granite4Vision AI-usage: draft (Bob, OpenCode + Qwen 3.6 35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor(convert): Split n_deepstack_layers and deepstack_layers (array) Branch: Granite4Vision AI-usage: full (Bob, OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor(src): Handle n_deepstack_layers and deepstack_layers GGUF keys Branch: Granite4Vision AI-usage: draft (Bob, OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Fix GGUF key for deepstack_layers_arr Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: Remove pre-scaling embeddings and skip scaling for raw embd inputs This follows how gemma3 and gemma4 handle embedding scaling by skipping the multiplier for raw input embeddings. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: deepstack_layers(_arr) -> deepstack_mapping(_arr) Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: Fully revert changes to n_deepstack_layers and qwen3vl* Since we're going to keep the GGUF KVs separate, it makes sense to just keep the hparams separate too to limit the scope of this branch. The down side is that n_deepstack_layers and deepstack_mapping_arr are potentially conflicting. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Revert removal of "is_deepstack_layers" GGUF KV This KV is not used at all on the c++ side, so it's fully dead, but there's also no need to conflate this cleanup with the addition of G4V. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Remove unnecessary ggml_cont and build_forward_expand in cbx Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: Clean up comments Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Tighter and more flexible code for g4v_build_block This could be refactored to look a lot more like granite-speech, but the overall block constructs before/after the qformer are pretty different, so for now I'm going to leave it as is and just tighten a bit. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Remove unnecessary `unordered_set` include Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Add architecture guard on deepstack_mapping_arr printout Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Remove unnecessary AI-gen comment Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Always initialize deepstack_mapping_arr with -1 values This was causing `test-llama-archs` to fail, likely due to trying to save the uninitialized values, then re-loading them. It's safer to always initialize so that other models don't forget and end up with undefined behavior. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: Remove TODO about block/vs non-block tensor mapping Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: Move is_vision_feature_layer logic into clip_hparams Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: Use a bool for append_token Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: Remove unnecessary comment Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Remove unused get_model api yikes! Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: Rearrange helpers for g4v to be private members and use build_attn Branch: Granite4Vision AI-usage: full (Bob, OpenCode + Qwen3.6-35b) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Fix off-by-one in vision layer index This was inherited from the Claude Code implementation that pushed the negative index inversion down into the model file. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Fix norm/post_norm mixup in conversion face. palm. :( Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: More descriptive tensor names Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Apply PR cleanup for new conversion changes AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * fix(convert): Remove duplicate V_ENC_EMBD_IMGNL Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * refactor: append_token -> add_newline Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: Comment cleanup Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Cleaner error handling/checking NOTE: format_string is not available in granite.cpp (and including clip-impl.h to get it doesn't compile, so I think it violates the intended encapsulation), so std::stringstream is the simplest answer. Branch: Granite4Vision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> --------- Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>	2026-06-05 17:44:59 +02:00
Georgi GerganovandSigbjørn Skjæret	7acb4e8cd2	hparams : refactor hparams.n_layer (#24060) ... * hparams : refactor hparams.n_layer * cont : remove `n_layer_kv()`, use n_layer_all instead * cont : type consistency * pi : update SYSTEM.md * models : fix Step3.5 MTP * cont : remove duplicate switch cases * cont : explicitly set `false` to extra layers for `is_swa` and `is_recr` * cont : fix nextn layer count handling Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>	2026-06-05 11:09:36 +03:00
Georgi Gerganov	06938ac129	tests : add support for qwen3 SSM archs (#24031) ... * tests : add support for qwen3 SSM archs * arch : add LLM_KV_ATTENTION_RECURRENT_LAYERS * cont : naming + TODOs	2026-06-03 10:15:27 +03:00
Aman GuptaandGeorgi Gerganov	255582687b	llama + spec: MTP Support (#22673) ... * spec: support MTP * fix batch size * rename files * cont : simplify (#7) * MTP: clean-up (#9) * MTP: clean-up * review: use llama_context_type instead of llama_graph_type * review: remove llama_model_has_mtp * review: fix convert issues * convert: fix pycheck * review: formatting * use `mtp-` for identifying mtp models * convert: fix mtp conversion * mtp -> draft-mtp * remove unused llama_arch * add need_embd in speculative * llama: allow partial seq_rm for GDN models for speculative decoding Currently speculative checkpoint needs to restart from a checkpoint after some draft tokens are not accepted, this leads to some wastage in running the target again. This PR adds the ability to rollback upto `draft_max` by storing the GDN intermediates. * fix pending state * vulkan: add GDN partial rollback * meta: extend check to axis 1 * metal: add GDN partial rollback Extend the gated delta net kernel to store intermediate states for partial rollback support on the Metal backend. - Add K (snapshot slot count) as a function constant - Read input state from slot 0 of the 3D state tensor - Write intermediate states to different slots during token loop - For K=1, maintain backward-compatible single-slot behavior Ref: https://github.com/ggml-org/llama.cpp/commit/8c05923630110223669f069af2000e9cf10c02bc Assisted-by: llama.cpp:local pi * delta_net_base: use ggml_pad instead of new_tensor * review: add need_rs_seq * review: rename part_bounded to n_rs * review: deslop comments * review: rename, add asserts * server : adjust checkpoint logic (#11) * server : adjust checkpoint logic * cont : rm asserts * server-context: fix early exit * spec : fix compatibility with n-gram and add TODOs (#13) * metal : cleanup * llama : fix faulty bitwise check in recurrent memory * server : disable RS-based MTP in combination with other spec types * spec : add TODOs * cont : fix comment * cont : update comment * common : fix logic for ngram + mtp compat * llama-memory: enable checkpointing with partial rollback * cont: add test-case for loading into a dirty ctx * llama-memory-recurrent: clear rs_idx in clear * download: fix mtp path * llama-arch: fix enorm op * docs: update docs * conversion: fix type annotations --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2026-05-16 20:06:23 +08:00
Pasha KhosraviandSigbjørn Skjæret	2e1f0a889e	ggml: add Q1_0 1-bit quantization support (CPU) (#21273) ... * ggml: add Q1_0 and Q1_0_g128 1-bit quantization support (CPU) * add generic fallback for x86 * remove Q1_0 (group size 32) * rename Q1_0_g128 => Q1_0 * fix Q1_0 LlamaFileType Enum * Fix trailing spaces; add generic fallback for othre backends * Apply suggestions from code review Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * fix /r/n spacing + arch-fallback --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>	2026-04-06 20:55:21 +02:00
anchortense	58190cc84d	llama : correct platform-independent loading of BOOL metadata (#21428) ... * model-loader : fix GGUF bool array conversion * model-loader : fix remaining GGUF bool pointer uses	2026-04-06 01:40:38 +02:00
Aman Gupta	278521c33a	llama-model-loader: print warning when using overrides with mmap (#20978) ... * llama-model-loader: use pinned memory for tensor overrides * change to warning	2026-03-30 17:40:17 +08:00
Johannes GäßlerandSiddhesh2377	36dafba5c4	llama: fix llama-model-saver (#20503) ... * llama : add fd-based model loading via llama_model_load_from_fd * llama : address review feedback for fd-based model loading * llama : use FILE pointer instead of fd in public API * llama : use FILE pointer consistently, address review feedback * fixup * fix tensor names * fix llama-model-saver * roundtrip tests * fixup * refactor tests * fix prints * fix model saving * fix CI, disable Chameleon * print seed --------- Co-authored-by: Siddhesh2377 <siddheshsonar2377@gmail.com>	2026-03-25 12:53:16 +02:00
Richard DavisonandSigbjørn Skjæret	5eae9cb1d9	ggml : add NVFP4 quantization type support (#19769) ... * WIP: add NVFP4 quantization support * tests * improve NVFP4 dot product implementation performance and fix bad super call * typo * Use nvfp4 kvalues * vulkan : fix NVFP4 shader compilation by including kvalues_mxfp4 lookup table * vulcal and perf fixes * wip * Fix metal * fix vulcan * Rename threshold & fix wrong scale * Fix MOE * Shelf backend implementations (CUDA, Metal, Vulkan, arch-specific SIMD) Remove NVFP4 support from GPU backends and architecture-specific optimized dot products. These should be added in separate PRs so backend specialists can review them independently. Reverted files: - ggml-cuda: common.cuh, convert.cu, mmq.cu/cuh, mmvq.cu, vecdotq.cuh, quantize.cu/cuh, mma.cuh, ggml-cuda.cu, fattn-tile.cuh - ggml-metal: ggml-metal.metal, ggml-metal-device.cpp, ggml-metal-impl.h, ggml-metal-ops.cpp - ggml-vulkan: ggml-vulkan.cpp, all vulkan-shaders/* - ggml-cpu arch: arm/quants.c, x86/quants.c, powerpc/quants.c, s390/quants.c Core NVFP4 support (type definition, CPU fallback dot product, quantization, dequantization, conversion) is retained. * Fix arch-fallback.h: add NVFP4 generic fallback for all platforms After shelving backend-specific SIMD implementations, the generic CPU dot product needs to be aliased on ARM, x86, PowerPC, and s390 platforms that previously relied on arch-specific versions. * quantize: add NVFP4 as a quantization type option * Fix ggml_fp32_to_ue4m3: handle subnormal values Previously, values with ue4m3_exp <= 0 were clamped to 0, causing all small scales to underflow. This made NVFP4 quantization via llama-quantize produce garbage (PPL = 5.8M) since typical transformer weights have amax/6.0 in the range 0.001-0.01, which falls in the UE4M3 subnormal range. Now subnormals are properly encoded as man * 2^-9 (exp=0, man=1..7), matching the decode path in ggml_ue4m3_to_fp32. Result: NVFP4 requantization now produces PPL = 15.25 (vs F16 = 14.33), comparable to Q4_1 (PPL = 15.81) at slightly lower BPW (4.70 vs 5.15). * Restore ARM NEON NVFP4 dot product implementation Restores the optimized ggml_vec_dot_nvfp4_q8_0 for ARM NEON using vqtbl1q_s8 lookup and ggml_vdotq_s32 dot products. tg128 performance: 4.37 t/s (generic) -> 13.66 t/s (NEON) = 3.1x speedup * Optimize ARM NEON NVFP4 dot product: LUT + vpaddq + vfmaq - Add ue4m3_scale_lut[128] to ggml-common.h replacing branch-heavy ggml_ue4m3_to_fp32() in the hot loop - Use vpaddq_s32 for pairwise int32 reduction instead of vaddvq_s32 - Accumulate with vfmaq_f32 into float32x4_t vector accumulators tg128: 8.1 -> 31.0 t/s (3.8x speedup, 77% of Q4_1 speed) * ARM NEON NVFP4: rearrange q8 to match nibble layout Alternative approach: rearrange q8 data to match the NVFP4 lo/hi nibble layout instead of rearranging the looked-up NVFP4 values. Eliminates vcombine_s8(vget_low, vget_low) shuffles. Performance is equivalent (~18.5 t/s) - the bottleneck is the 2x block overhead from QK=16 vs QK=32, not the shuffle instructions. * CPU only backend 64 super-block layout * cleanup * Remove unused LUT * int * exclude NVFP4 from unsupported ops in metal build * remove quantization for now * store scales as native UE4M3, preserve original model bits when possible * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * correct comment * format * reduce duplication and cleanup * Address comments * move detection to prepare_tensors * Use math instead of const * Move * fix comment * Shelf quantize tests * Rebase and move check * cleanup * lint * Update gguf-py/gguf/scripts/gguf_convert_endian.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Use fallback quant config * Simplify Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * organize * Refactor * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * add quantize_nvfp4 (required for test_quants.py) * add quantize_nvfp4 (required for test_quants.py) * add quantize_nvfp4 (required for test_quants.py) * fix return type --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>	2026-03-11 21:02:54 +01:00
Xuan-Son Nguyen	59db9a357d	llama: dynamic head_dim and n_rot for SWA (#20301) ... * llama: dynamic head_dim and n_rot for SWA * also add gguf_writer wrappers * fix build * build_rope_shift arg reorder	2026-03-09 22:22:39 +01:00
Johannes Gäßler	a976ff081b	llama: end-to-end tests (#19802) ... * tests: add end-to-end tests per model architecture * fixup for rebase * fix use-after-free in llama-model-loader.cpp * fix CI * fix WebGPU * fix CI * disable CI for macOS-latest-cmake-arm64 * use expert_weights_scale only if != 0.0f * comments	2026-03-08 12:30:21 +01:00
Georgi Gerganov	c5c64f72ac	llama : disable Direct IO by default (#19109) ... * llama : disable Direct IO by default * cont : override mmap if supported	2026-01-28 09:11:13 +02:00
Julius Tischbein	287a33017b	llama : Extend fallback, fix fileno for dio file, exclude case that mmap uses dio file (#18887)	2026-01-18 18:35:57 +02:00
Sigbjørn Skjæret	2a13180100	model-loader : support bool array sliding window pattern (#18850)	2026-01-15 10:12:46 +01:00
3 people	2038101bd9	llama : add use_direct_io flag for model loading (#18166) ... * Adding --direct-io flag for model loading * Fixing read_raw() calls * Fixing Windows read_raw_at * Changing type off_t to size_t for windows and Renaming functions * disable direct io when mmap is explicitly enabled * Use read_raw_unsafe when upload_backend is available, not functional on some devices with Vulkan and SYCL * Fallback to std::fread in case O_DIRECT fails due to bad address * Windows: remove const keywords and unused functions * Update src/llama-mmap.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: jtischbein <jtischbein@gmail.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2026-01-08 08:35:30 +02:00
3 people	dfc959b886	model : Granite Embedding support (#15641) ... ModernBERT but without `head.norm` so will currently fail to convert and run any other ModernBERT models, PRs with `head.norm` support welcome! * constants and tensor mappings for modern bert support, model not supported yet but working on getting conversion to work for encoder only * conversion now working, hf -> gguf * working on support, now working on building graph * some cleanup * cleanup * continuing * correct tensor shape for qkv * fixed tensor mappings and working on buildin graph * tensor debugging now works -> (llama-eval-callback), instead of simulated gate split with views, GEGLU is now used which does exactly this * cleanup * cleanup * cleanup * more cleanup * ubatch issues, the assert for checking equal seqs in llama-graph.cpp when building attention keeps failing, setting ubatch size to 1 when running llama-embedding with --ubatch-size 1 makes it work, but needs to be looked into more * added cls token per previous modern bert attempt, still working on checking out the rest * fixed pre tokenizer and still working through previous pr * working through previous attemp, implimented more accurate conversion per previous attempt, added local sliding window attention that alternates every third layer * fixed pre tokenizer * working on swa with local and global alternating attention * some cleanup and now fails on build attn * starting to work, and some cleanup, currently failing on last layer construction in graph build * alternating rope implemented and modern bert graph build succeeds * fixed asser for equal ubatch seq * cleanup * added mask check in vocab * fixed alternating rope, the hparams.rope_freq_base_train and hparams.rope_freq_base_train_swa were the same and i set them to correct values * reuse variable * removed repeat * standard swa method can be used instead of a new enum being LLAMA_SWA_TYPE_LOCAL * correct swa layer indexing, is supposed to be 0, 3, 6 ... instead of 1, 4, 7 ... * more modular hparam setting * replaced attn out norm with ffn_norm and cosine similarity between hf embds and llama.cpp embds went way up, from 0.05 to 0.24, replaced the cacheless kv with swa todo per the previous conversion * Update gguf-py/gguf/tensor_mapping.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update convert_hf_to_gguf_update.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-model.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-vocab.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-model.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update gguf-py/gguf/tensor_mapping.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update gguf-py/gguf/tensor_mapping.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update gguf-py/gguf/tensor_mapping.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update gguf-py/gguf/tensor_mapping.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update gguf-py/gguf/tensor_mapping.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update gguf-py/gguf/tensor_mapping.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update gguf-py/gguf/tensor_mapping.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update gguf-py/gguf/tensor_mapping.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update gguf-py/gguf/tensor_mapping.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-graph.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-arch.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-model.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-model.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-model.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-model.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-model.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * removed redundant hparam set * enums for model sizes * conversion for modern-bert model supported rather than just granite-small * Update src/llama-model.cpp Co-authored-by: Gabe Goodhart <ghart@us.ibm.com> * Update src/llama-model.cpp Co-authored-by: Gabe Goodhart <ghart@us.ibm.com> * fixed ordering of enum for freq_base_swa * fixed where I added residual, now gives much much better embeddings~ * readded cacheless logic * removing whitespace * conversion now working for swa pattern - dense every n layers * modern bert put into seperate src file * removing whitespace * fixed whitespace and newline errors in editorconfig job * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * better naming convention, n_swa_pattern -> swa_period * reusing sliding_window_pattern key rather than making new dense_every_n_layers key, and adding writing and reading support * fixing pyright type-check fail * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update gguf-py/gguf/gguf_writer.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-hparams.h Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-model-saver.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/models/modern-bert.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/models/modern-bert.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/models/modern-bert.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update gguf-py/gguf/gguf_writer.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/models/modern-bert.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/models/modern-bert.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-model.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-model-loader.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-model-loader.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-model-loader.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * added descriptions in llama-model * fixed tensor mappings for conversion * Update src/llama-model.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update src/llama-model.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * mapping name for size * nits * unused --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> Co-authored-by: Gabe Goodhart <ghart@us.ibm.com>	2025-12-23 00:28:19 +01:00
Julius Tischbein	f99ef53d2a	llama : Changing off_t to size_t for Windows (#18204)	2025-12-19 16:42:46 +02:00
Julius Tischbein	4d4f4cacd1	llama : Async DirectIO model loading on Linux (#18012) ... * Uncached model read * Removing additional --mmap arg * Removing trailing whitespaces * Adding fallback when O_DIRECT is not supported * Remove branching in llama-model-loader.cpp and reduce code duplications in llama-mmap.cpp * Adding maybe unused keyword for Mac and Windows. * File seek aligned * Removing all branches for direct_io in llama-model-loader.cpp * Always use alignment from llama_file * use_mmap=true	2025-12-18 08:27:19 +02:00
Johannes Gäßler	b1f3a6e5db	llama: automatically set parameters not set by the user in such a way that maximizes GPU utilization (#16653) ... * llama: automatically fit args to free memory llama-fit-params tool * fix CI * hints for bug reports, ensure no reallocation * fix segfault with Vulkan * add llama-fit-params to CI * fix CI * fix CI * fix CI * minor adjustments * fix assignment of 1 dense layer * fix logger not being reset on model load failure * remove --n-gpu-layer hint on model load failure * fix llama-fit-params verbosity * fix edge case * fix typo [no ci]	2025-12-15 09:24:59 +01:00
4 people	34fcc5a4ac	model : Apertus model implementation (#15852) ... * First attempt * No permute during convert (fixes qk tensors), proper norm application. * RoPE = NeoX * Coherence! * Migrate xielu params from tensors to hyperparameters * Simple CUDA kernel * Revert stupid LLM refactorings * Chat template support * configchecker / flake8 errors * Reorder unary.cu * I do conclude that LLMs are, in fact, stupid. * Fix after merge * Final newline * Make xIELU an UNARY_OP * Final newline * Correctly account for parameter shift * Argh. * Update ggml/src/ggml-cpu/unary-ops.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Refactor: remove unused methods, inline and factorize softplus, add const modifiers * Revert CUDA changes, implement xIELU as a separate OP * Pesky newline * Add float2half / half2float for F16 inputs/outputs * CUDA variants, attempt 2 * Actually, attempt 3 * Update ggml/src/ggml-cuda/unary.cu Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * Missing convert header * Proper formula and reference for xIELU in the comments. * Modify unary-ops.cpp to add the functor-based logic besides the template system to retain optimizations * Apply suggestions from code review Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Add tensor mappings for Apertus to global list instead * Fix lazy on scalars * Update ggml/src/ggml-cuda/unary.cu Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * Add comment about the constraints on positive/negative alpha * Change `softplus` to `ggml_softplus` --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Johannes Gäßler <johannesg@5d6.de> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>	2025-10-02 20:43:22 +03:00
3 people	e8d99dd0b6	nvidia nemotron nano v2 (nemotronh) (#15507) ... * feat: Add NEMOTRONH to python arch enum https://github.com/ggml-org/llama.cpp/issues/nemotron-nano-15409 Branch: gabe-l-hart/nvidia-nemotron-nano-15409 Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add NEMOTRONH to c++ arch enum https://github.com/ggml-org/llama.cpp/issues/nemotron-nano-15409 Branch: gabe-l-hart/nvidia-nemotron-nano-15409 Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add NEMOTRONH to llama-arch layer map https://github.com/ggml-org/llama.cpp/issues/nemotron-nano-15409 Branch: gabe-l-hart/nvidia-nemotron-nano-15409 Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: First pass at conversion for nemotronh https://github.com/ggml-org/llama.cpp/issues/nemotron-nano-15409 Branch: gabe-l-hart/nvidia-nemotron-nano-15409 Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Add a verbose log for each tensor loaded This is really helpful for diagnosing mismatches between the expected and received tensors https://github.com/ggml-org/llama.cpp/issues/nemotron-nano-15409 Branch: gabe-l-hart/nvidia-nemotron-nano-15409 Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: First (broken) pass at nemotronh model architecture It generates tokens, just not valid ones! https://github.com/ggml-org/llama.cpp/issues/nemotron-nano-15409 Branch: gabe-l-hart/nvidia-nemotron-nano-15409 Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Explicitly enable add_bos_token during conversion The `tokenizer.json`/`tokenizer_config.json` in the model are a bit contradictory. In the config, add_bos_token is set to False, but the tokenizer model itself has a post_processor that adds the BOS token via type: TemplateProcessing https://github.com/ggml-org/llama.cpp/issues/nemotron-nano-15409 Branch: gabe-l-hart/nvidia-nemotron-nano-15409 Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Use relu2 (LLM_FFN_RELU_SQR) for activation in FFN layers https://github.com/ggml-org/llama.cpp/issues/nemotron-nano-15409 Branch: gabe-l-hart/nvidia-nemotron-nano-15409 Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Only allocate attention cache for attention layers (not non-recurrent) https://github.com/ggml-org/llama.cpp/issues/nemotron-nano-15409 Branch: gabe-l-hart/nvidia-nemotron-nano-15409 Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Move residual add to after every block https://github.com/ggml-org/llama.cpp/issues/nemotron-nano-15409 Branch: gabe-l-hart/nvidia-nemotron-nano-15409 Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * fix: Use the correct norm tensor for the MLP blocks https://github.com/ggml-org/llama.cpp/issues/nemotron-nano-15409 Branch: gabe-l-hart/nvidia-nemotron-nano-15409 Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * Nemotron-H: MLP gate cleanup (pass NULL for unused gate) This model does not use a gate in MLP blocks; pass NULLs for gate tensors to make intent clear and avoid unused-pointer noise. * SSM: respect ssm_dt_rank for dt_dim when provided Use GGUF-provided time_step_rank (ssm_dt_rank) to set dt_dim when > 0; fallback to max(64, n_embd/16). * fix: plamo2 - revert dt_dim to default (remove ssm_dt_rank usage) * Rename nemotronh to nemotron_h for consistency - Update architecture name from NEMOTRONH to NEMOTRON_H in constants.py - Change architecture string from 'nemotronh' to 'nemotron_h' in all files - Update enum LLM_ARCH_NEMOTRONH to LLM_ARCH_NEMOTRON_H - Update class name llm_build_nemotronh to llm_build_nemotron_h - Consistent naming with underscore convention (nemotron_h vs nemotronh) * feat: Support conversion for older NemotronH models https://github.com/ggml-org/llama.cpp/issues/nemotron-nano-15409 Branch: gabe-l-hart/nvidia-nemotron-nano-15409 Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> --------- Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> Co-authored-by: Maicon Domingues <dominguesm@outlook.com> Co-authored-by: weatherman <fxdstudios@gmail.com>	2025-08-28 18:39:31 -06:00
3 people	fd1234cb46	llama : add gpt-oss (#15091) ... * oai moe * compat with new checkpoint * add attn sink impl * add rope scaling yarn * logits match with latest transformers code * wip chat template * rm trailing space * use ggml_scale_bias * rm redundant is_swa_all * convert interleaved gate_up * graph : fix activation function to match reference (#7) * vocab : handle o200k_harmony special tokens * ggml : add attention sinks support (#1) * llama : add attn sinks * ggml : add attn sinks * cuda : add attn sinks * vulkan : add support for sinks in softmax remove unnecessary return * ggml : add fused swiglu_oai op (#11) * ggml : add fused swiglu_oai op * Update ggml/src/ggml-cpu/ops.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * update CUDA impl * cont : metal impl * add vulkan impl * test-backend-ops : more test cases, clean up * llama : remove unfused impl * remove extra lines --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: slaren <slarengh@gmail.com> * repack mxfp4 upon conversion * clean up a bit * enable thinking * add quick hack to render only some special tokens * fix bf16 conversion * remove vocab hack * webui ok * support chat parsing for gpt-oss * fix webui * direct mapping mxfp4, FINALLY * force using mxfp4 * properly use lazy tensor * ggml : add mxfp4 ggml : use e8m0 conversion instead of powf Co-authored-by: Diego Devesa <slarengh@gmail.com> change kvalues_mxfp4 table to match e2m1 (#6) metal : remove quantization for now (not used) cuda : fix disabled CUDA graphs due to ffn moe bias vulkan : add support for mxfp4 cont : add cm2 dequant * ggml : add ggml_add_id (#13) * ggml : add ggml_add_id * add cuda impl * llama : add weight support check for add_id * perf opt * add vulkan impl * rename cuda files * add metal impl * allow in-place ggml_add_id * llama : keep biases on CPU with --cpu-moe * llama : fix compile error ggml-ci * cuda : add fallback for __nv_cvt_e8m0_to_bf16raw ggml-ci * cleanup ggml-ci * sycl : fix supports_op for MXFP4 ggml-ci * fix Unknown reasoning format * ggml-cpu : fix AVX build ggml-ci * fix hip build ggml-ci * cuda : add mxfp4 dequantization support for cuBLAS ggml-ci * ggml-cpu : fix mxfp4 fallback definitions for some architectures ggml-ci * cuda : fix version required for __nv_cvt_e8m0_to_bf16raw --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: slaren <slarengh@gmail.com>	2025-08-05 22:10:36 +03:00
Sigbjørn Skjæret	d17a809ef0	llama : support multiple classifier outputs and labels (#13940)	2025-06-06 09:03:25 +02:00
Diego Devesa	c6a2c9e741	gguf : use ggml log system (#13571) ... * gguf : use ggml log system * llama : remove unnecessary new lines in exception messages	2025-05-15 19:13:11 +02:00
Diego Devesa	b7d2672082	llama : fix quantize with dl backends (#13539)	2025-05-14 16:12:36 +02:00
Johannes Gäßler	10d2af0eaa	llama/ggml: add LLM training support (#10544) ... * llama/ggml: add LLM training support more compact progress bar llama_save_model_to_file llama_opt_param_filter ggml_graph_dup force_grads refactor ggml_opt, fix test-opt * remove logits_all * refactor CUDA implementation for ACC * reset graph at beginning of opt period	2025-05-12 14:44:49 +02:00
Diego Devesa	27ebfcacba	llama : do not crash if there is no CPU backend (#13395) ... * llama : do not crash if there is no CPU backend * add checks to examples	2025-05-09 13:02:07 +02:00
Georgi GerganovandDiego Devesa	833e2b7409	model : print tensor size during load (#12711) ... * model : print tensor size during load * cont : fix units MB -> MiB Co-authored-by: Diego Devesa <slarengh@gmail.com> --------- Co-authored-by: Diego Devesa <slarengh@gmail.com>	2025-04-02 16:38:54 +03:00
Diego Devesa	e0e912f49b	llama : add option to override model tensor buffers (#11397) ... * llama : add option to override tensor buffers * ggml : fix possible underflow in ggml_nbytes	2025-04-02 14:52:01 +02:00
jklincn	e39e727e9a	llama : use LLM_KV_GENERAL_FILE_TYPE instead of gguf_find_key (#12672)	2025-04-01 14:54:28 +02:00