Commit Graph
12344 Commits
Author SHA1 Message Date
Hamish M. Blair d8c8cc4d9b server : cancel MCP tool calls on client disconnect or timeout (#32025)
* server : cancel MCP tool calls on client disconnect or timeout

* server : shorten MCP cancellation tests and fix a type-check error
2026-11-12 17:19:26 +01:00
Oliver Simons 3ba1ce0e28 qwen3.5: merge tensors on convert also (#32102)
* qwen3.5: merge tensors on convert also

* Remove conversion test for qwen3.5
2026-11-12 15:54:58 +01:00
anujj 0fb8f15770 Title: CUDA: few-row MMQ for BS 2-24 (#31533)
* CUDA: few-row MMQ for 2-24 src1 columns

New int8 MMA kernel for quantized mat-muls with 2-24 src1 columns, e.g. speculative decoding verify passes.
Supports Q8_0, Q3_K, Q4_K, Q5_K, Q6_K, IQ3_S, IQ4_NL and IQ4_XS on NVIDIA GPUs from Ampere on.

* nit fixes

* CUDA-only guard for few-row MMQ, weights flag in test_mul_mat

* few-row quantize: DS4 sum from the quantized values, as in #31247

* few-row MMQ: clearer HIP/MUSA abort, test_mul_mat two-context build

* few-row MMQ: dispatch order MMVQ, few-row, MMQ
2026-11-12 21:12:45 +08:00
Sigbjørn Skjæret 827466b9ef ci : build ui on non-slim runner (#32138) 2026-11-12 12:35:31 +01:00
Xuan-Son Nguyen 8954f19483 server: fix SSRF for CORS proxy and media URLs (#32115) 2026-11-12 12:33:48 +01:00
Xuan-Son Nguyen a1a68eb797 server: explicit opt-in tools via non-loopback addr, check cors host port (#32113)
* server: explicit opt-in tools via non-loopback addr, check host port

* fix ci
2026-11-12 12:16:50 +01:00
Xuan-Son Nguyen e182fdad4a server: disallow direct access child instance from browser (#32120) 2026-11-12 12:16:29 +01:00
PascalandVenkata Vineel 09452cdedd server: keep the server awake while a request uses its state (#32133)
With --sleep-idle-seconds, a handler passes wait_until_no_sleep()
while the server is awake, then keeps using the server state, for
example to parse the body and tokenize. If the idle timer expires in
that window, the main loop enters sleep and the callbacks destroy the
model: the handler reads freed vocab memory, and a task posted after
that stays in the queue with nobody awake to process it.

Count the requests that passed wait_until_no_sleep() and are not done
yet, and do not enter sleep while the count is positive. The response
generator releases its count in its destructor, so every return path
of a handler gives it back.

Co-authored-by: Venkata Vineel <vyalamar@users.noreply.github.com>
2026-11-12 12:16:16 +01:00
Pascal cfb9d4b754 mtmd: require an F32 position embedding for HunyuanVL (#32131)
HunyuanVL resamples its position embedding on the host into a float
buffer sized for F32, then reads ggml_nbytes() of the tensor into it.
The graph never uses the tensor, so any type loads, and an F64 tensor
overflows the buffer at the first image request.

Require the position embedding to be present and F32 when loading the
HunyuanVL tensors.
2026-11-12 12:15:28 +01:00
Pascal 72f4603c7c mtmd: read the MiniCPM-V projection width from ne[1] (#32132)
The MiniCPM-V resampler ends with build_mm(mm_model_proj), whose output
width is ne[1], while n_mmproj_embd() returns ne[0]. Both are equal in
real models, the projection is square. A non square projection passes
the text model n_embd check and only aborts at the encode, on the
output buffer size check.

Return ne[1], so the width matches the graph output.
2026-11-12 12:14:50 +01:00
Pascal 8c388364e1 mtmd: require the MLP projector and patch embedding tensors (#32129)
* mtmd: require the MLP projector and patch embedding tensors

The loader marks the patch embedding and the LLaVA projector tensors
as optional, while the graphs and n_mmproj_embd() dereference them
unconditionally. A GGUF without them crashes on a null tensor during
the warmup.

Load the patch embedding as required for the vision encoders that
start with it, which is all of them except GEMMA3NV, DEEPSEEKOCR and
DEEPSEEKOCR2. Load mm.0 and mm.2 as required for MLP, and mm.0, mm.1,
mm.3 and mm.4 for the Yi-type MLP_NORM variant, selected by mm.3.

The graph no longer tests mm.2 and the yasa2 assert on the patch
embedding is covered by the loader.

* mtmd: keep the Yi-type llava TODO

The Yi-type variant is still detected by the presence of mm.3, the
TODO on that detection stays.
2026-11-12 12:14:36 +01:00
Pascal 08f49598c1 mtmd: require a 2x2 spatial merge for the Qwen-VL projectors (#32130)
The Qwen2-VL, Qwen2.5-VL, Qwen3-VL and Ling3-VL graphs build their
input with a temporal merge that assumes a 2x2 patch grid, and the
Qwen2-VL merger reshapes by 4. Any other spatial_merge_size from the
GGUF reaches a GGML_ASSERT at the first image request and aborts the
server.

Reject a spatial_merge_size other than 2 when loading the hparams.
2026-11-12 12:14:10 +01:00
Kong Jiale a8575a3ef4 ggml : route GATED_DELTA_NET to the SpacemiT RVV kernel (#31897)
The SpacemiT compute case was commented out, so the op stayed on the scalar CPU path. Type dispatch stays with the other ops, and use_ref keeps the reference path on the generic CPU kernel.

Assisted-by: Codex

Signed-off-by: gitGurugu <2049447146@qq.com>
Signed-off-by: Wang Yang <yangwang@iscas.ac.cn>
2026-11-12 12:20:25 +02:00
sliu39 a963f65957 vulkan : Intel FA kernel for prefill. (#29357)
* vulkan : Intel customized FA shader for prefill usage

* vulkan : adding check of max_bias/logit_softcap for Intel FA kernel path

* vulkan : add m16n16k16/SIMD32 support for Xe FA prefill shader

* vulkan : Intel FA shader and host refinement, with accuracy issues fixes

* vulkan : Limit the SLM usage to 48 KiB for Intel prefill FA shader

* vulkan : Update Intel FA shaders to handle cases with large head dimension K, unaligned Q head stride and multi head mask cases

* vulkan : Intel FA shader and host refinement to align with existing design, extreme cases handling, shape check relaxing.

* vulkan : Intel FA shader and host update to support q as tensor view, unaligned batch stride q. Add env variable for enforcing XE FA path for other vendor GPUs with 16x16x16 CM1 and SIMD32 support

* vulkan : Inte FA shader and host update to fix the potential SLM over 48 KiB issue, enhance the buffer access robust and add asymmetric qk/pv ration support.

* vulkan : Inte FA shader and host update to add SLM 48 KiB size check, remove unnecessary extensions and finetune the unroll/dont_unroll loops

* vulkan : Intel FA shader and host update to add SLM size check function, use real device max SLM size in shader specialization constants.

* vulkan : Intel FA shader and host update to consolidate host helper function, remove redundant buffer sync, bypass unused pipeline and buffer, fix subgroup shmem conflict and add negative case handling.

* vulkan : Intel FA host update to correct get_xe_fa_tuning_params() function log message, and add workgroupMemoryExplicitLayout/workgroupMemoryExplicitLayout16BitAccess prerequisite‌ feature check.
2026-11-12 11:19:17 +01:00
Marco Colombo ab4a77b9ad ggml-cpu: use the 4x4 simd_gemm tile on ARM NEON (#32128)
Assisted-by: Claude Fable 5.2
2026-11-12 12:13:59 +02:00
Konrad Moren 0d69255c52 CUDA: Optimize prefil via fuse of w_s scale in epilogue MMQ for nvfp4 checkpoints (#26048)
* ggml-cuda: Apply NVFP4 weight scale in MMQ write-back

* ggml-cuda: Fuse mul_mat(_id) + scale into MMQ

* ggml-cuda: Tests for the fused scale

* ggml-cuda: reuse existing graph to test fusion in mmq epiloque

* ggml-cuda: rename test_mul_mat_vec_fusion to test_mul_mat_fusion for consistency

* ggml-cuda: refine comments for clarity on MMQ dispatch and fusion checks in mul_mat functions

* ggml-cuda: refactor fusion checks for mul_mat operations and remove unused function
2026-11-12 10:28:36 +01:00
Aman Gupta 3bce8bd50c ggml-backend : prefetch only inputs computed by earlier splits (#32100)
* ggml-backend : prefetch only inputs computed by earlier splits

The copy callback of a split runs after the prefetch and can write tensors that a later split reads, e.g. the MoE cache uploads missed experts into its bank. A prefetch of such a tensor reads it before the write.

* llama : skip the MoE cache when its device cannot run the cached mul_mat_id

Otherwise the scheduler runs the matmul on another backend and copies the whole bank every ubatch.
2026-11-12 17:24:40 +08:00
Kong Jiale 81f21f4b27 ggml : dispatch SpacemiT Q4_1 32x256 through the HP kernel (#31929)
The 32x256 pack already keeps a separate uint8 zero point. Complete the M1 path and use the same HP kernel as Q4_0.

Assisted-by: Codex

Signed-off-by: gitGurugu <2049447146@qq.com>
Signed-off-by: Wang Yang <yangwang@iscas.ac.cn>
2026-11-12 10:59:09 +02:00
Xuan-Son NguyenandAleksander Grygier b19f1d9e63 ui: add search_chats tool (#32051)
* ui: add search_chats tool

* add from_timestamp / to_timestamp

* ui: search_chats review fixes

- keep search_chats disabled on import, copy tool policy on fork
- stream messages without attachment payloads, one transaction for time ranges
- keep the output within budget, cut whole messages from the older end
- reject impossible date-times, quote titles and attachment names
- typed readOnly tables, move localIsoString to formatters as formatLocalIso

Assisted-by: Claude (Opus 5.5 Medium)

---------

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
2026-11-12 09:44:35 +01:00
Aleksander Grygier d90cfd4a9b ui: install llama-brand from a GitHub tarball instead of git (#32112)
npm 11.20 refuses git dependencies by default (EALLOWGIT), so npm ci
failed on fresh installs. Pin llama-brand to the same commit through
an HTTPS tarball, which also adds an integrity hash to the lockfile.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-12 09:36:28 +01:00
Xuan-Son Nguyen 60d46a5abe server: when --tools-runtime is specified, ignore x-tool-runtime (#32119) 2026-11-12 09:34:16 +01:00
Aldehir Rojas 93cf736a8d chat : refactor gemma4 to build loose schema rules and clean up (#31915)
* chat : build a schema-shaped PEG parser for Gemma 4 tool arguments

* chat : enforce only the key structure of Gemma 4 tool arguments

* chat : remove old gemma4 workarounds

* cont : add node.as<>() to simplify static casts

* cont : add depth_guard and use numerical rule names

* cont : clean up

* cont : extract gemma 4 parsers

* cont : clean up

* cont : clean up

* cont : remove dynamic cast
2026-11-12 09:44:42 +02:00
Joel Fernandes 4fffab623e llama : avoid touching unused graph metadata pages (#32043)
Each graph result reserves metadata for max_nodes, but many graphs use
only a fraction of that capacity.  The vector resize initializes the
whole reserve and makes otherwise unused pages resident.  Allocate the
reserve with new uint8_t[] instead.
2026-11-12 09:42:45 +02:00
Titaniumtown 09338b85e8 sycl: make oneDNN FA cache thread-safe and memoize runtime failures (#31138) 2026-11-12 08:58:26 +02:00
Neo Zhang dcbff21661 sycl : fix AOT for more than 4GB (#31873)
* fix aot for >4g garbage output issue; add --aot in build script

* correct the aot flag

* support no-xmx build, update the code for no-xmx building, rename nossl to no-ssl
2026-11-12 08:54:58 +02:00
zhaoyshineandzhaoyshine f3d7725a77 sycl : use cached device-id in set_main_device (#32013)
ggml_sycl_set_main_device() runs at the start of every graph compute. Its
early-return check called dpct::get_current_device_id(), which locks the
device manager mutex and looks up the thread-to-device map. Use the
thread-local cached id that the rest of the backend already uses.

Assisted-by: Claude Code

Co-authored-by: zhaoyshine <zhaoyshine@users.noreply.github.com>
2026-11-12 08:50:55 +02:00
Joseph Patricio 7021f7e099 CUDA: add cuda debug buffer pool (#31919)
* CUDA: add cuda debug buffer pool

* CUDA: add code comment for debug pool with compute-sanitizer
2026-11-12 08:50:32 +02:00
shaofeiqiandLi He e0949d09b1 opencl: add bin kernel kernel_gemm_noshuffle_q4_0_f32_32b_trans_ila_a7_bin (#30947)
* opencl: add A7 Q4_0 non-MoE binary kernel

* opencl: refactor

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
2026-11-11 21:53:51 -08:00
Feiteng 5d4c8eeb24 CUDA: skip empty MoE tiles in MMQ stream-k fixup (#32067)
* CUDA: skip empty MoE tiles in MMQ stream-k fixup

mul_mat_q_stream_k_fixup loads ids_dst[col_low + j] for all j < J before
checking j_sup. mul_mat_q returns early on col_low == col_high, the fixup
kernel did not. mm_ids_helper emits one extra tile per expert that may be
empty; for trailing experts col_low == ncols_dst, so the load reads one
element past the J_best-1 padding added in #29953. The kernel returns at
j >= j_sup before storing anything, so results are unaffected.

Return early when j_sup <= 0, like the main kernel. The return is
block-uniform (tsj is loaded once per block) and precedes __syncthreads().

Add test_mul_mat_id_unused_tail to test-backend-ops: only the first
n_mats/2 experts are selected, so the trailing experts have zero rows and
their tiles are empty; m=33, k=512 keeps tile efficiency low enough that
stream-k splits along K and launches the fixup kernel.

* CUDA: drop the extra MUL_MAT_ID test case and the explanatory comment (review)
2026-11-12 11:45:56 +08:00
Aleksander Grygier dd6b945be2 server: use the moved test model in the router CORS test (#32117)
test_router_restricted_cors_same_origin_reaches_child still named
ggml-org/tinygemma3-GGUF, which the router no longer finds since the test
models moved to ggml-org-dev, so the Server test job fails on master.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-12 02:01:19 +01:00
e1f7593c32 opencl: add decomposed FA for Adreno (#32046)
* opencl: add decomposed FA

Assisted-by: Claude Opus 5.5
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>

* opencl: build FA MQ kernel on first FA call

* so allows warmup to early build neccessary kernels

Assisted-by: Claude Opus 5.5

* opencl: handle inf for fully masked row

* opencl: refine condition for using fa decomp

* opencl: support dp4a for PV mm

Assisted-by: Codex

* opencl: add direct PV for FA decomp to avoid V transpose for small batch

Assisted-by: Codex

* opencl: enable direct pv by default

* opencl: expand X2E decomposed FA to smaller heads and batches

- Lower the default head dimension threshold from 256 to 128 on X2E.
- Allow DK64 with P8, no attention sinks, GQA >= 2 and n_kv >= max(1024, 2*n_q).
- Allow P8 with V transpose for 2-31 queries when direct PV is unavailable, DK >= 128 and n_kv >= 1024.

Assisted-by: Codex

* opencl: adjust format

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
2026-11-11 16:18:24 -08:00
Xuan-Son Nguyen 7bcbc04fbf ui: move js sandbox setting out of "developer" (#32054)
* ui: move js sandbox setting out of "developer"

* make js sandbox enabled by default
2026-11-11 23:42:34 +01:00
Iñigo Imaz dd00306bcf docs: update image link in README.md (#32038)
* Update image link in README.md

I guess the image in the main README is now this one https://github.com/ggml-org/llama-brand/blob/master/logo/svg/cover/llama-cpp/cover-llama-cpp-dark.svg

* Fix image link in README.md

* Update image alt text in README.md
2026-11-11 23:39:32 +01:00
Aleksander Grygier 1d81992784 ui: keep downloads in Discover during search and fix their progress with a draft (#31467)
* ui : keep downloads in Discover sidebar during search, sort base-org repos first

- Show the sticky Downloading section regardless of an active search query
- Order search results so repos published by their base model's org lead

Assisted-by: Claude (Opus 5.5 Medium)

* server : throttle download progress per file

Files of a download run in parallel threads, but progress updates were
throttled with one shared timer. A small draft sidecar reports far more
often than the main weights, so it took nearly every slot and the main
file's updates, including the first one carrying its total size, were
dropped until the draft finished. The UI then showed the draft's progress
first and restarted from 0% for the model.

Throttle each file on its own and always pass a file's first update.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: list a downloading repo once and match base orgs case-insensitively

A search hit that is downloading already shows under Downloading, so the
results leave it out. Base model orgs compare without case, as the hub
treats them.

Assisted-by: Claude (Opus 5.5 Medium)

* server : tell clients a download stopped on request

When a client paused or cancelled a download, every other client following
/models/sse got a bare download_failed and could not tell it from an error,
so the web UI showed "Download failed" for a pause made in the Llama app.
A stop asked for through unload or delete now fails with {"stopped": true}.

Deleting a paused download, which has no instance left, now sends
model_remove as deleting a running one does, so other clients drop its row.

Assisted-by: Claude (Opus 5.5 Medium)

* ui : follow downloads other clients start, pause or cancel

A download another client started (e.g. the Llama app) already showed up
from the feed; it now also picks up the sidecars the list reports for it.
A stop another client asked for reads as a pause instead of a failure, and
a removal clears the download row even when the list does not carry it yet.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-11 23:26:39 +01:00
Aleksander Grygier bd717e08f3 ui: take the logo and favicons from the llama-brand package (#32039)
* ui : take the logo and favicons from llama-brand

The web UI's logo mark and favicons now come from llama-brand, added as a
dev dependency from GitHub and pinned to a commit, so the UI draws the
same icon as the README cover instead of its own copy of an older one.
Logo.svelte inlines the brand's icon and fills it with currentColor, and
the PWA asset generator starts from the brand's web favicons, which carry
their own margin and optical centring, so the UI's colouring and padding
script, its test and its logo.svg go.

Assisted-by: Claude (Opus 5.5 Medium)

* ui : follow llama-brand's master instead of a pinned commit

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-11 23:25:35 +01:00
Xuan-Son NguyenandPascal 2cbe0ec429 server: fix --cors-origins handling + protect against DNS rebinding (#32098)
* server: fix --cors-origins localhost handling

* server: validate Host on loopback connections against DNS rebinding

A page whose own name resolves to 127.0.0.1 reaches the server with
that name in Host and a matching Origin, so the same-origin shortcut
let it through. On a loopback connection, Host must now name loopback
or the host of an allowed --cors-origins entry. The LAN tests reach
the server through a real interface instead of a forged Host.

* fix python type check

---------

Co-authored-by: Pascal <admin@serveurperso.com>
2026-11-11 21:17:46 +01:00
Xuan-Son Nguyen 664f031f49 ci: move test models to ggml-org-dev (#32096)
* ci: move test models to ggml-org-dev

* fix test spec
2026-11-11 21:17:16 +01:00
CC 4180722eeb metal : prevent out-of-bounds reads in SOLVE_TRI kernel (#32103) 2026-11-11 22:02:06 +02:00
Ruben Ortlam 055c3c26ef vulkan: support batched tensor copy API (#31418)
* vulkan: support batched tensor copy API

* allocate staging buffer only for copies that actually need it
2026-11-11 21:56:39 +02:00
Alessandro de Oliveira Faria (A.K.A.CABELO)andAlessandro de Oliveira Faria 2656b71a19 vendor : update LibreSSL to 4.4.1 [no ci] (#32106)
Co-authored-by: Alessandro de Oliveira Faria (A.K.A.CABELO) <cabelo@multicortex.ai>
2026-11-11 20:42:10 +01:00
Georgi Gerganov cb44fdb189 llama : reuse graph result instances across scheduler reserves (#32089)
Allocate gf_res_reserve and both gf_res_prev entries once in the llama_context constructor. Add llm_graph_result::set_max_nodes() and reset the existing instances in sched_reserve() instead of recreating them. get_gf_res_prev() now only selects the arena for the current batch.

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-11 19:58:39 +02:00
Aldehir Rojas cd2db800db jinja : support chained comparisons and argument unpacking (#32072) 2026-11-11 11:53:04 -06:00
Xuan-Son Nguyen f7b886fe6f chat: allow TTS model to provide custom text normalizer via jinja (#32053)
* chat: allow TTS model to provide custom text normalizer via jinja

* rm tts_norm_is_reserved_name
2026-11-11 17:34:55 +01:00
Sigbjørn Skjæret 956591f457 ci : remove dead snakes (#32084) 2026-11-11 15:13:14 +01:00
Georgi Gerganov ccd360f060 llama : remove non-fused GDN and ubatch-dependent graph nodes (#32078)
- remove build_delta_net_chunking and build_delta_net_autoregressive
- always use the fused gated delta net
- make graph_max_nodes independent of the ubatch size
- drop the fused GDN probes and fused node tags

Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
2026-11-11 15:16:16 +02:00
Asahi-PrvandAsahi-Prv b326dd6752 devops : add missing MKL runtime DLLs to the Windows SYCL release package (#32005)
ggml-sycl.dll built with oneAPI 2026.2 imports mkl_intel_thread.3.dll and libimalloc.dll, which are not part of the release package. LoadLibrary fails and the SYCL backend is silently skipped, llama.cpp falls back to CPU.

Adding both DLLs fixes the load: measured on Arc B570 with LFM2.5-8B-A1B Q4_K_M, pp4096 goes from 404 t/s (CPU fallback) to 2232 t/s (SYCL) with the extracted b12244 package.

Assisted-by: GLM (z-ai/glm-5.3-flash)

Co-authored-by: Asahi-Prv <Asahi-Prv@users.noreply.github.com>
2026-11-11 14:14:43 +01:00
Alessandro de Oliveira Faria (A.K.A.CABELO) b8abe41a50 vendor: update cpp-httplib to 0.66.0 (#32049) 2026-11-11 20:07:30 +08:00
Tarek DakhranandClaude Opus 5.5 2e325dcd57 backend : re-instantiate CUDA graphs when the cluster dims of a kernel change (#31284)
* CUDA: re-instantiate graphs when the cluster dims of a kernel change

* CUDA: keep the cluster dims with the graph instance, check them only on Hopper+

Assisted-by: Claude Opus 5.5

* CUDA: use CUDA_CHECK and a helper struct for the graph cluster dims

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* CUDA: add a TODO for the graph cluster dims test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 14:00:45 +02:00
Oliver Simons 1648d64117 test-fp8.cpp: use only the public ggml API (#32083) 2026-11-11 13:33:12 +02:00
fox5611 432433637b ROCm/HIP: fixed BLAS workspace instead of rocBLAS allocating one per call (#31838)
* ROCm/HIP: fixed BLAS workspace instead of rocBLAS allocating one per call

Assisted-by: Claude Opus 5

* ROCm/HIP: corrected by only setting the workspace on ROCm 7.0 or newer
2026-11-11 12:32:39 +01:00