* server: fix LRU hang on multiple requests same model
* server: keep a queued model out of the victim pool until its waiters leave
A waiter that gave up while its model was still loading left the
model idle with no request behind it, and nothing recounted the free
slots, so a second request queued behind it stayed queued forever.
tick() was only driven by requests: join, claim and the end of a
proxied request.
Keep the queue entry alive after a successful claim so the model
coming up is never picked as a victim before its waiters use it, and
recount the slots on every status change and whenever a waiter
abandons the queue. The model is then evicted as soon as it comes up
with nobody left to serve.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* vulkan: fix missing view-alias dependencies in ggml_vk_graph_optimize
is_src_of doesn't treat two views of one tensor as dependent, so the optimizer reorders nodes across aliased reads and writes.
Result: silently wrong tokens under greedy decoding, different output on every server start, and invalid speculative-decoding acceptance, with nothing logged.
Hits Qwen3.8's recurrent state (and any model with view-aliased state) on AMD and NVIDIA Vulkan. CUDA is clean.
Compare view_src bases on both sides.
Fixes#27805
* vulkan: don't treat view/no-op nodes as aliasing dependencies
Nodes whose op is NONE, RESHAPE, TRANSPOSE, VIEW or PERMUTE execute nothing, so aliasing through them is not a real dependency. The previous base comparison matched them anyway, which only costs the optimizer reordering freedom.
Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
* vulkan: make the lambda parameter const and capture is_empty in is_src_of
Code will not compile without these changes.
is_src_of has an empty capture list, so is_empty was not visible inside it, and is_empty took a non-const pointer, while is_src_of receives const ones. Other call sites pass non-const pointers, which still convert as usual.
---------
Co-authored-by: Jeff Bolz <jbolz@nvidia.com>
* spec : add DFlash2 support (local convolution + candidate selector) (#27342)
* support DFlash2
* Add p_min in DFlash2
Assisted-by: Claude Opus 5
* Revert unnecessary changes
Assisted-by: Claude Opus 5
* Revert draft sampling in rejection sampling
Assisted-by: Claude Opus 5
* Refactor code structure
Assisted-by: Claude Opus 5
* Delete embedding scaling
Assisted-by: Claude Opus 5
* Gate output transforms on DFlash2
Assisted-by: Claude Opus 5
* Optimize Dflash 2 cost
Assisted-by: Claude Opus 5
* Avoid using atoi
Assisted-by: Claude Opus 5
* Modify comments
Assisted-by: Claude Opus 5
* Move llama_model_dflash_selector_top_k to llama-ext.h
Assisted-by: Claude Opus 5
* Formatting
Assisted-by: Claude Opus 5
* Apply patch to fix the mrope bug
Assisted-by: Claude Opus 5
* fix ci
Assisted-by: Claude Opus 5
* Fix graph number calculation
Assisted-by: Claude Opus 5
* rename hid and unary
Assisted-by: Claude Opus 5
---------
Co-authored-by: Jian Chen <jianchen0311@gmail.com>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
* revert top-k.cu changes
---------
Co-authored-by: Zihan Zhang <tiancaizhangdaxian@sjtu.edu.cn>
Co-authored-by: Jian Chen <jianchen0311@gmail.com>
* common : dedupe --n-cpu-moe / --spec-draft-n-cpu-moe override loops
* common : add --n-cpu-ffn to CPU-offload dense FFN weights of first N layers
* common : generalize llm_ffn_block_regex over the FFN regex, drop TODO
* fit: also take into account n_streams
* server: make the draft context follow the target context
With a non-unified KV cache the target context now holds n_ctx_train
tokens per sequence, while the draft context was still created with
n_ctx = 0 and fell back to n_ctx_train / n_streams per sequence. A slot
filled beyond that point makes the draft batch fail to decode, and the
server answers 500 on the request.
The draft context now takes its size from the target context, so both
hold the same number of tokens per sequence. Contexts that share their
cells with the target no longer need the kv_size override.
The memory reserved for the draft model before fitting is measured at
the largest context the target can take, since the draft context grows
with the target and a fixed byte margin cannot express that.
* fit: take an optional second model into account
Illustrates the alternative discussed on the draft context fix. The
memory of a draft or MTP context is currently handed to the fit as a
fixed byte margin, which cannot express a memory that grows with the
context the fit is still deciding on.
common_fit_params now takes an optional second model that shares the
devices of the main one. Its context follows the main context and its
memory is measured again whenever that context changes, so the reduce
path stays exact instead of conservative. A model that cannot be
measured on its own, such as a shared cell MTP context, is skipped with
a warning and the main model is fitted alone.
This drops the reservation block in the server, which no longer has to
probe the trained context size of the target to guess an upper bound.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* feat: add --mmproj-device arg & backwards compatible MTMD_BACKEND_DEVICE env var
* feat: load mmproj device backend immediately, add -mmdev shortflag
* fix: its a pointer now get the name
* clean up
* gen docs
* nits
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* add params
* cpu kernel
* metal kernel
* add test backend ops
* gate other backends
* ggml: (cuda) support ggml_rope_set_offset (#27121)
* rm cuda supports_op guard, fix webgpu clang-format
* ggml: support ggml_rope_set_offset on vulkan (#27344)
* ggml: support ggml_rope_set_offset on vulkan
* remove inplace optimization
* ui: move get_datetime tool to frontend
* clarify docs
* server: drop the now unused ctime include
strftime() and gmtime_r() were the only users, both went away with the
get_datetime tool. Also make the renderer's catch inert: the browser
executor always emits JSON, so a non-JSON result is no longer a date to
display.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* server : save serialized image chunks at the end of the llama state
* server : support multimodal slot state save/restore with packed payload
* server : refine image slot state serialization
* server : support media slot state and centralize media validation
* server : remove unnecessary comment
* server : remove defensive media checks and move the chunk type check to validate()
* vulkan: TQ2_0 (ternary) support — dequant + dedicated mul_mat_vec + matmul via dequant_funcs
First Vulkan ternary type in ggml. Correctness: OM-125m TQ2_0 vs F16 top-12
logprobs identical to 4 decimals fully offloaded (float dequant path, no Q8_K
activation quant). Speed at 125m ~= F16 (overhead-bound at this scale); the
bandwidth win targets larger BitNet SKUs. MMQ/int-dot path intentionally not
wired yet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* tests: enable TQ2_0 in backend-ops type lists
Vulkan now implements TQ2_0 (dequant, mul_mat_vec, mul_mm, get_rows); backends
without support skip via not-supported as usual. TQ1_0 stays disabled.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Michael Trabalka <michael.trabalka@sqv.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* adapt the api
* text model ok
* working impl, need verify and clean up
* mtmd: build the pocket-tts transposed convolutions as GEMM + col2im
ggml_conv_transpose_1d has no grouped mode, so the depthwise upsample
was built as one convolution and one concat per channel, which floods
the graph with small nodes and makes kernel launches dominate the
decoder.
Fold both cases into the column form the seanet decoder already needs:
the general case reshapes the kernel to [IC, K * OC] and matmuls it
with the input, the depthwise case batches a matmul over the channels
so a step scales its own kernel. A single col2im_1d then scatter-adds
the columns back to the signal, with the same shape as before, so the
overlap-add tail, the streaming state and the bias are untouched.
Generation time per frame drops by 80% on CUDA and by 50% on CPU. The
output matches the previous implementation sample for sample, with a
correlation of 0.999994 and identical frame counts.
* flow_temp + frames_after_eos
* chunking
* mtmd: carry the remaining pocket-tts per-pack settings
The language packs also tune the end-of-speech padding and the padding
of short prompts, next to the temperature already carried in the
mmproj: french_24l asks for 8 tail frames instead of the guessed 3,
english_2026-01 asks for short prompts to be padded with spaces.
Write both in the mmproj as clip.gen.audio.frames_after_eos and
clip.gen.audio.pad_short_text, keyed on the pack in the conversion
script like the temperature. The loader keeps them optional, so a
mmproj without them behaves as before. Map semicolons to commas for
every pack instead, the reference only asks for it on three of them and
it costs nothing elsewhere.
Existing mmproj files must be converted again to carry the two keys.
On a long french text the port now lands within 2% of the reference:
22.96s against 23.44s, with the same peak level and the same amount of
silence.
* clip.gen.audio.model_variant
* clean up code comments
* nit: drop the dead flow_temp hparam, the pack table holds the default
* update docs
* address security problems
* less invasive base.py
* lint
* add mtmd_gen_inp_default
* add docs
* rm gen_flow_temp
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* ci: Add support for CUDA 13.4 ARM64 builds for Windows
Added an architecture-specific CUDA 13.4 Windows build entry targeting ARM64.
Added a CMake configuration to enable ARM64 CUDA cross-compilation from an x64 Windows environment using the x64-hosted CUDA and MSVC toolchain while linking against the ARM64 CUDA import libraries to produce ggml-cuda.dll.
Validated the self-hosted Windows x64 workflow, including toolkit acquisition, CMake configuration, ARM64 CUDA cross-compilation, and packaging. Runtime validation was performed separately on a native ARM64 RTX Spark system using TinyLlama 1.1B Q4_K_M to verify the generated binaries.
The ARM64 CUDA job builds only the ggml-cuda.dll backend (LLAMA_BUILD_SERVER=OFF). The release consists of two packages: the main ARM64 release package, which combines the existing ARM64 CPU outputs with ggml-cuda.dll, and a separate runtime package containing the required CUDA runtime libraries (cudart64_13.dll, cublas64_13.dll, and cublasLt64_13.dll).
The CUDA 13.4 setup uses NVIDIA Developer Preview component archives instead of the GA component downloads used by the existing CUDA setups and will require updates once CUDA 13.4 reaches GA.
* ci: cleans up to align with x64 CUDA setup
- Moves CUDA-specific CMake options into matrix defines.
- Keeps the CUB 3DOT2 option only for CUDA 12.4.
- Removes runtime argument construction and the unnecessary server option.
- Aligns ARM64 CUDA runtime packaging with the existing robocopy approach.
- Generalizes the ARM64 release label from CUDA 13.4 to CUDA 13.
* ci: Set CUDA job name as version-architecture pair
* mark as preview
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ggml : add fallback definitions for missing aarch64 HWCAP bits
* ggml : require HWCAP_ASIMDHP for the aarch64 fp16 cpu variants
Also rename has_fp16_va to has_fp16, the field gates the whole FEAT_FP16
extension, scalar and vector half-precision arithmetic together.
* mtmd: support multi-row batching for deepseek-ocr
* mtmd: weave deepseek-ocr rows in one shot instead of per row (#26615)
---------
Co-authored-by: Saba Fallah <sabafallah@gmail.com>
* server: add get_info tool
* fix --rpc in docs
* server: harden get_info probe result handling
Report the OS as unknown when the probe process fails to spawn or times
out, so the diagnostic text from run() is never returned as an OS name.
Strip the probe output on both ends, which also drops the blank line
that ver prints before the version on Windows. Name the output and
timeout limits, and report an unreadable working directory as unknown
instead of an empty string.
* server: simplify get_info result handling
Drop the named limits and the working directory error branch, keeping
the probe result handling to a single expression.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* add common/subproc.h|cpp
* add compile flag LLAMA_SUBPROCESS
* disabled by default on android and ios
* test-jinja: use common subproc
* mtmd: disable video if subproc is not set
* disable subproc on wasm
* make is_created atomic
* migrate server-mcp
* move server_pipe to common
* init impl
* vendor: update subprocess.h
* add server_mcp_stdio
* stderr drain
* server_mcp_transport
* server_mcp_stdio is now framing-only, no json
* internal/mcp-stdio: integration + tests + fixes (#26075)
* server-mcp: harden transport and wire up the tool integration
Builds on the transport/manager architecture (server_mcp_transport + server_pipe)
with the hardening and integration the draft did not yet have.
Hardening:
* Reader and stderr pumps are polled (running-aware) instead of blocking on a read
that only ends at EOF. subprocess_terminate() SIGKILLs only the direct child, so a
grandchild the MCP server spawned that inherited the pipe would otherwise keep the
write end open and hang teardown (both warmup shutdown at startup and process
shutdown). The writer is likewise non-blocking + polled.
* Windows: resolve the command through PATHEXT so "npx" (npm ships npx.cmd, never
npx.exe) spawns, matching POSIX's PATH search; and enumerate the parent environment
as UTF-8 (GetEnvironmentStringsW) instead of the active code page.
* server_pipe gains an opt-in max_size (default unbounded, so the router's streaming
use is unchanged); the MCP reply queue uses it so a server that streams unsolicited
notifications between requests cannot grow it without bound.
Integration:
* --mcp-servers-config / --mcp-servers-json flags; enabling MCP restricts default CORS
to localhost, same as --tools.
* MCP tools are exposed through /tools (and chat-completions) as <server>_<tool>,
skipping names that collide with a built-in or another MCP tool.
* Manager lifecycle wired into llama_server(): warmup at start, shutdown() from the
signal handler before the HTTP server drains, blocking teardown in clean_up().
* SIGPIPE ignored so a child dying mid-write yields EPIPE rather than killing us.
Assisted-By: Claude Opus 4.8 <noreply@anthropic.com>
* server-mcp: add MCP test suite with grandchild deadlock regression test
21 tests over the /tools endpoint: tool discovery/invocation, timeouts, crash
recovery and respawn cooldown, warmup partial failure, malformed and batched
notification+response output, tool-definition shape, and prompt shutdown during a
slow call.
The last test spawns an MCP server that leaves a grandchild inheriting its
stdout/stderr and asserts the server both starts and stops promptly. Verified it
fails (5s SIGKILL fallback on a deadlocked reader-join) when the pump is made to
ignore the running flag, and passes with the polled reader.
Assisted-By: Claude Opus 4.8 <noreply@anthropic.com>
* clean up
* clean up 2
* even stricter life cycle
* nits
* nits 2
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* fix some edge cases
* fix last_error data race
* fix response schema + docs
* server: fix MCP zombie leak and timeout-induced transport teardown
join_pumps() never reaped the child, leaking one zombie per spawn:
call subprocess_join() before subprocess_destroy().
A per-call timeout permanently closed from_server and got a healthy
transport evicted: add close_on_stop to server_pipe::read() and pass
false from send_rpc(), where should_stop is a per-request deadline
and a late reply is already skipped on id mismatch.
Also drop the unreachable disconnect cancellation in
server_mcp_tool::invoke(): support_stream is false, st is always null.
(cherry picked from commit e6de1ec043174fd0570b1e60d47f06c7c19d620d)
Assisted-by: Claude Opus 4.8
* server: make MCP test fixtures JSON-RPC 2.0 compliant
Add the missing notification guard to mcp_malformed_server.py and
mcp_burst_server.py (the latter treated id 0 as a notification and
replied to unknown ones; its notification table is now unused).
Return -32602 instead of -32601 for unknown tools: tools/call is a
valid method, the tool name is the invalid parameter.
Also fix the test module docstring: tools are named <server>_<tool>.
(cherry picked from commit 74a08e8c311dabf3b49d06cc6d754b0097ae7a38)
Assisted-by: Claude Opus 4.8
---------
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
Co-authored-by: Pascal <admin@serveurperso.com>
