LLAMA_MOE_IDS_SKIP=0 forces the ids readback in sched_copy_experts (the
upstream path) and =log reports each skip on stderr, so validation can show
that the skip engages at prefill and that output is byte-identical with and
without it in one build. Read once (function-local statics); unset changes
nothing. Not set in production.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two ways a shutting-down server kept the VRAM arbiter's token, stalling every
other model:
- SIGTERM with a request in flight: terminate() stopped only the task queue.
The HTTP handler kept waiting for results the loop would never send, the main
thread blocked joining it, and the token was released only in the destructor
after that join - i.e. never, until the client went away. Terminate the
result queue too (readers now get "server is shutting down": a 503, or an SSE
error event), and let wait_until_no_sleep give up once stopping.
- SIGTERM while waiting for the token: the loop thread sat in a blocking flock
inside the wake callback, holding mutex_tasks, so terminate() waited for the
current holder to finish and then for a pointless restore of our weights.
Poll the lock non-blocking and abort the wait on shutdown; the aborted wake
stays cold, skips the restore, and the loop exits before update_slots.
Also release the token right after start_loop() returns rather than in the
destructor, and only after the device buffers are actually freed: the old
order unlocked before destroy(), so the next model could start uploading into
VRAM that was still occupied. Freeing at exit uses a new
llama_context_discard_device(), which drops weights, KV, MoE cache and scratch
without capturing host shadows first.
Off production: SIGTERM mid-generation or mid-wait now exits in ~0.2 s with
exit 0 and frees the token within 5-70 ms (previously never / after the
holder's whole generation). Handoff latency and greedy output are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The MoE cache (--moe-cache-mib, #29887) keeps the most recently used host
experts in device banks. The on-demand VRAM path (release_device /
restore_device, used by the server VRAM arbiter and the on-demand mmproj path)
did not know about it, so its banks were never freed on handoff. Restoring them
by reallocation alone would also be wrong: the banks come back zeroed while the
slot maps still mark experts as cached, so the model would silently read zero
weights.
Free the banks on release and empty the cache instead of shadowing it: re-init
every LRU, re-point its slot-map pointers at the host slot maps and clear those
to -1. The banks are reallocated (which also re-inits their views) and cleared
on restore, before the scheduler is rebuilt, and the cache refills from the
host experts as before. Release runs in both modes, weights-only and evict_kv.
device_resident() now covers the cache too, for the early returns in
release/restore, the auto-restore in decode() and
llama_context_weights_resident().
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
release_device(evict_kv) drops the scheduler, so a context evicted by the VRAM
arbiter and then destroyed reaches the destructor with a null sched. The upstream
"compute buffer size matches expectation" loop calls
ggml_backend_sched_get_buffer_size() unconditionally, whose GGML_ASSERT(sched)
then aborts: the server exits 134 instead of 0 on every shutdown taken while
cold, which under llama-swap makes an ordinary stop look like a crashed child and
leaves a ggml backtrace in the log each time. It also skipped the rest of the
destructor, so the cold teardown path had never actually run to completion.
Guard at the call site rather than relaxing the assert - the assert is right, and
every other caller reserves the scheduler first. Same shape as the null guards in
synchronize(), memory_breakdown() and the released-buffer iteration.
The loop is a diagnostic size comparison, already reported by sched_reserve() at
load, so skipping it when cold loses nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
On device release with KV eviction, also call ggml_backend_free_scratch()
on each backend so a cold model drops its Vulkan compute preallocations
(reallocated lazily on the next compute via restore).
Also guard llama_context::memory_breakdown() against a freed scheduler:
evict_kv release resets sched to null, so a cold model that is then torn
down (e.g. terminated by the process manager) hit GGML_ASSERT(sched) in
ggml_backend_sched_get_buffer_type. Skip the compute-buffer accounting
when sched is null.
Assisted-by: Claude
The recurrent (SSM/conv) state of hybrid models (e.g. Qwen3.5) was left
resident when a model's device buffers were released for on-demand VRAM
sharing - llama_memory_recurrent::release_device_buffers() was a no-op
default. Implement it (and restore_device_buffers) with the same
capture-host-shadow / free / reallocate pattern as llama_kv_cache, so
llama_memory_hybrid now evicts both its attention KV and its recurrent
state. The state is read-write, so its shadow is recaptured on every
release.
Assisted-by: Claude
Under LLAMA_SLEEP_EVICT_KV a cold model still held the scheduler's worst-case
compute buffer (hundreds of MiB, e.g. ~700 MiB for gemma-26B) plus the resident
experts. release_device(evict_kv) now also frees the sched (sched.reset() +
sched_need_reserve), and restore_device() rebuilds it via sched_reserve(), so a
fully-evicted model holds essentially no VRAM (gemma cold: 7147 -> 51 MiB).
Trades a heavier re-warm (weights+KV H2D + sched reserve) for the freed VRAM;
weights-only mode is unchanged (fast switch, keeps KV+compute).
Assisted-by: Claude
release_device_weights()/release_device_buffers() leave a null buffer in
ctxs_bufs while the device memory is released for on-demand VRAM sharing.
The memory-breakdown / total_size / clear paths iterated these and called
ggml_backend_buffer_get_size()/get_type()/clear() on the null buffer, tripping
GGML_ASSERT(buffer) and aborting on shutdown of a cold model. Skip null buffers
in llama_model::memory_breakdown and llama_kv_cache::{memory_breakdown,
total_size,clear}.
Assisted-by: Claude
Extend on-demand device residency to the KV cache so that when a model's KV
plus another model would not fit in VRAM, the KV can also be evicted to a host
shadow (D2H on release, H2D on restore) instead of only the weights.
- llama_memory_i: add release_device_buffers()/restore_device_buffers()
(default no-op). Implemented in llama_kv_cache (D2H shadow of the live
ctxs_bufs, freed and reallocated like the weights); llama_memory_hybrid and
llama_kv_cache_iswa delegate to their child caches.
- llama_context::release_device(evict_kv): also evict the memory's device
buffers when requested; restore_device() rebuilds them. Public API
llama_context_release_device gains an evict_kv flag.
- server: LLAMA_SLEEP_EVICT_KV=1 enables it. Off by default (weights-only),
since the KV shadow adds a D2H/H2D copy of the live cache each cycle.
Validated on RX 580 (Vulkan), 4B @ 32k ctx: weights-only cold VRAM 1750 MB
(KV stays); weights+KV cold VRAM 726 MB (KV freed, ~1 GB reclaimed). KV
survives the round-trip: prompt cache reused after the cycle (prompt_n 4 vs
42), correct output.
Assisted-by: Claude
Add release/restore of a model's GPU weight buffers (keeping a host shadow
and the KV cache) so several always-loaded llama-server processes can
time-share a single GPU without reloading or losing the prompt cache.
- llama-model: release_device_weights()/restore_device_weights() capture a
compact host shadow (stable iteration order, view-skipping) and free then
realloc the device weight buffers; weights_resident() query.
- llama-context: release_device()/restore_device() wrappers; decode() auto-
restores; public C API llama_context_release_device/restore_device.
- server: LLAMA_SLEEP_VRAM_ONLY makes idle-sleep release only the VRAM weights
(not a full unload/reload). A cross-process flock token in LLAMA_VRAM_ARENA
enforces "resident iff holds token"; an inotify doorbell forces the holder
to release on contention. The warden thread only touches the task queue, so
releases run on the loop thread and never race a decode.
Validated on RX 580 (Vulkan): two models share 8GB, never both resident,
correct output under contention, KV cache preserved (no re-prefill).
Assisted-by: Claude
Offloaded expert weights are uploaded per split, and to decide which experts to
upload the copy callback reads the routing ids back to the host. The ids are
produced on the same device we are about to upload to, so the readback forces a
full pipeline flush - 47 of them per eval on a 48-layer MoE.
Once the batch draws enough experts the readback stops telling us anything. At
2048 tokens x 10 experts over 256 experts every expert comes back used, so the
used set is all true and the copy is a single whole-tensor range anyway.
Skip the readback when ggml_nelements(ids) >= 4 * n_expert and mark every expert
used. Uploading an expert that no id selects cannot change the result, since
mul_mat_id only reads the rows the ids point at, so this stays exact. With the
MoE cache on, the all-used set still goes through moe_cache->copy_experts, which
copies the cached experts device-to-device from the host slot maps and uploads
the rest - still exact, and no readback is needed for it. Decode is unaffected:
n_ids there is the number of experts per token, far below the threshold. The
small-batch MoE cache slot path has its own readback and is not touched.
The st.ids/st.used caching is unchanged: a repeat call with the same ids tensor
reuses the all-true set, which is what the threshold would decide again.
Port of dbc449afd ("sched: skip the MoE routing-ids readback when the batch
uses every expert"), whose target block in ggml_backend_sched_compute_splits was
removed upstream (#29943) in favour of this copy callback.
Measured (original, before the port) on an RX 580 (Vulkan, Polaris) running
Laguna-S-2.1 118B IQ2_M with all experts host-resident: pp2048 +4.7% at depth
32k, +0.9% at 16k, tg32 +3%.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
* mtmd: support hibiki realtime duplex
* fix and clean up
* fix long session quality
* fix ci
* tests : add the moshi fusion baseline for Metal
The Metal fusion check found 5 RMS_NORM+MUL fusions in the new moshi
arch and no baseline for them.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
The loop slots copied the physical layers at the end of load_arch_tensors,
before the generic pass attaches the optional .scale tensors, so the later
passes through the core ran NVFP4 experts without their per-expert scales.
Add an optional post_arch_tensors hook that runs after the scale pass and
share the slots there.
The fused lightning indexer is always used now, so drop the cparams flags, the auto probe and the non-fused fallbacks.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* Conversion
* Separate model_type
* Add gate_rank to allow building from metadata while testing
* Base model
* Loop graph
* Support the new is_varloop attribute
* Adopt naming scheme from https://github.com/ggml-org/llama.cpp/pull/32588
* Rename to varying
* model: load `ffn_gate_up_exps` scales
The generic scale pass loads per-expert scales for ffn_gate_exps,
ffn_up_exps and ffn_down_exps but not for the merged ffn_gate_up_exps.
Load ~ffn_gate_up_exps_s there~ and expose it as ffn_up_exps_s ~when the layer
has no separate up tensor,~ since build_moe_ffn applies up_exps_s to the
merged gate_up output. Inkling 2 now passes the expert scales to
build_moe_ffn like the other MoE models.
* Update src/llama-model.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* Update src/llama-model.h
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
* llama : move the indexer key rotation choice into the model classes
The KV cache forced a Hadamard rotation for the lightning indexer keys
based on a list of architectures. The virtual llama_model::indexer_k_rot()
now returns true for the same models. DeepSeek4.1 inherits from DeepSeek4
but was not in the list, so it overrides it back to false.
* llama : rename indexer_k_rot() to lid_k_hdmd()
graph_max_nodes() picked the nodes-per-tensor factor from lists of
architectures. Each model class now gives it through a new virtual,
llama_model::graph_nodes_per_tensor(), which defaults to 8. The
architectures that need 32 or 64 override it in models.h, and DFlash
keeps its hparams-based rule in dflash.cpp.
LoRA adapter nodes are now added for every architecture, not only for
the ones on the default factor.
The MoE cache adds a slot lookup to every MoE layer, and with
--moe-cache-cpu-share a second expert path, but graph_max_nodes() did
not account for it. Models on the default 8*n_tensors budget with few
tensors per layer (cohere2moe: 443 tensors, 49 layers) overflow the
graph context when the cache is combined with a quantized KV cache
(K/V rotation), aborting in ggml_new_tensor with GGML_ASSERT(obj_new).
Fixes#32584
Assisted-by: Claude Opus 5.5
load_arch_hparams multiplies n_hrm_layers_per_stack, n_hrm_h_cycles and
(n_hrm_l_cycles + 1) into a uint32_t, then asserts the product equals
block_count.
Any factors whose true product is block_count + k*2^32 still pass the
assert; 1073741825 * 2 * 2 = 2^32 + 4 compares equal to block_count = 4.
load_arch_tensors iterates the same three values in int and writes past
the end of the block_count-sized layers vector.
* llama : give Inkling2 its own relative-position attention input
The Inkling2 graph built its flash attention mask from the hybrid iswa
input and passed it to build_attn through a new kq_mask argument. That
argument let a caller pair its own dense mask with the kv_rows of
llm_graph_input_attn_kv_iswa, whose mask follows the per-sequence cell
lists whenever kv_rows is enabled.
Add llm_graph_input_attn_rel_pos instead. It owns dense KQ masks and the
query-to-cell distances, fills both from the same cells, and its
build_attn overload adds the per-head bias to its own mask and never
passes kv_rows. The iswa build_attn is back to its previous signature;
its body moves to build_attn_iswa_impl, which both overloads call.
Inkling2 now folds the per-query sink into the bias with and without
flash attention, so build_attn_mha no longer needs the per-query sink
path.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
* inkling2 : make the expert weight scale and final logit softcap optional
Load both keys as optional and skip them when they are zero, as other
models do, instead of rejecting zero at load.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
* llama : build kv_rows masks for the hybrid iswa memory
llm_graph_input_mem_hybrid_iswa checked its masks with
can_reuse_attn_mask, which expects kv_rows whenever they are enabled,
but built them dense. With a unified cache and flash attention the
check always failed, and hybrid iswa models (LFM2, LFM3, Inkling2)
never reused a graph. Build the masks with build_attn_inp_attn_mask,
like the plain iswa input.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
* llama : make the relative-position input a hybrid iswa input
Address review: llm_graph_input_mem_hybrid_iswa_rel derives from
llm_graph_input_mem_hybrid_iswa and owns the attention and recurrent
inputs, so Inkling2 builds one memory input instead of two. Its
attention input is built with dense masks, can_reuse checks them as
dense, and its build_attn asserts that the mask has no kv_rows (TODO:
a kv_rows variant). build_attn_iswa_impl is now a static helper.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
* llama : compose the hybrid iswa rel input, format inkling2
Address review: llm_graph_input_mem_hybrid_iswa_rel owns its attention
and recurrent inputs like the other composite inputs instead of deriving
from llm_graph_input_mem_hybrid_iswa.
Format src/models/inkling2.cpp like the other model sources: grouped
and aligned key and tensor loading, descriptive per-layer names, blank
lines and section comments between the steps, and cb() names for the
intermediate tensors. The graph is unchanged.
Assisted-By: Claude Opus 5.5 <noreply@anthropic.com>
llm_graph_params no longer carries hparams, prec_policy and hdmd, since the model reference already provides them. llm_graph_context sources them from the model and keeps prec_policy as a reference.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* llama : fall back to buffered I/O on EIO
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* Preserve bytes already read, resumes at the current file offset
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
---------
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
n_mudd_taps comes from limite.mudd.tap_count, a scalar read separately
from the length-checked tap_indices array. mudd_tap_idx is a fixed
std::array<int32_t, 512> indexed at [i*n_taps] in both the tensor load
and the graph build, so a crafted GGUF with n_layer * n_mudd_taps > 512
reads past the array.
Reject that at hparams load; the one check covers both index sites.
* llama-bench : size the batch to the prompt for models without memory
Assisted-by: Claude Opus 5.5
* server : read up to 16 lfm3-d1 questions in one pass
Assisted-by: Claude Opus 5.5
* model:add Zaya1 8B (Zyphra)
- Added the support for Zaya.
- Fixed the SSM_CONV case, so the model runs on cuda without abort.
Please enter the commit message for your changes. Lines starting
* Made the following changes based on the bot reviewer feedback:
1) Added runtime asserts when loading wrong KV head size, unsupported conv kernel size.
2) Removed the static cast usage.
3) Fixed double load of tokenizer.
4) Updated res_scale_in to be input tensor.
* - Moved the res_scale_in back to layer 0 to fix the cuda bug
* i) Removed the EOS addition.Team agreed that updating Tokenizer is the right move.
ii) Fixed formatting
* Fixed the comment to make it clear
---------
Co-authored-by: varun <varun_subramanian@zyphra.com>
This commit adds a lambda to check if a specified byte is not a valid
continuation byte in a multibyte UTF8 sequence.
The motivation for this is to improve readability and avoid some code
duplication.
* vocab : extract bpe_rank loading into function
This commit extracts the BPE rank loading logic into a separate function
named `load_bpe_ranks`.
The motivation for this is that the came code is currently used by both
gpt2/hybriddna/whitespace tokenizer models and gemma4 and this avoids
the duplication of code to improve readability and maintainability.
* change signature of load_bpe_ranks
* jinja: impl scope-chain key lookup
* add test for call_statement lambda
* chat: add tts_dict for kokoro G2P template
* more testing
* value_float_t range check
* no json-in-meta, add llama_model_meta_arr_str/_n
* mtmd: add kokoro tts support
* detailed guide about G2P and text norm port
* refine G2P
* clean up
* migrate batch api
* nits
* llama : keep the k-pool layout prefix on sequence edits
Assisted-by: Claude Code
* llama : address review of the k-pool layout update
* Append and edit share one branch, using stale > sq.pos_min.
* LLAMA_KPOOL_VERIFY=1 checks every update against a full rebuild. The scan moved into kpool_scan() so the check can reuse it.
* The test turns the check on for its rollback context, skips -sm tensor, and drops kimi-k31.
* Two comments that were no longer true are fixed.
Assisted-by: Claude Code
* llama : check the k-pool layout update in debug builds instead of LLAMA_KPOOL_VERIFY
* llama : remove the k-pool layout verification
* llama : inline the k-pool scan again
* remove dangling new line
* do not search twice
---------
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>