The result-checking and test debug paths in ggml-vulkan.cpp call ggml_graph_compute_with_ctx() to compute a CPU reference graph, but that symbol is defined in ggml-cpu, which ggml-vulkan does not link. Enabling -DGGML_VULKAN_CHECK_RESULTS=ON (or -DGGML_VULKAN_RUN_TESTS=ON) therefore fails to link with an unresolved external (e.g. LNK2019 on MSVC, undefined reference on GCC/Clang). This regressed after ggml-cpu was split into its own library. Link ggml-cpu under those two options so the debug builds link again.
Signed-off-by: Wyatt Caldwell <218154709+Detensable@users.noreply.github.com>
* server : improve message span logic
* cont : cast size_t to int32_t in comparisons
* server : create checkpoints before every user msg
* chat : remove \n in gemma4 delimiters
* chat : merge msg delimiter structs into one
* cont : reword comment
* cont : initialize tokens in delimiter
* cont : add server_tokens::get_raw_tokens() for mtmd
* cont : move message finding to server_tokens and skip mtmd tokens
* cont : update cohere2moe parser
* cont : increase min-step to 8192 and always produce a chkpt for last user message
* server: real-time model load progress tracking via /models/sse
* update docs
* server: move model download to child process
* rm unused
* fix most problems
* clean up
* nit fixes
* fix test case
* do not detact() thread
* shorter MODEL_DOWNLOAD_TIMEOUT in test
* throttle
* ui: model status and load progress via /models/sse feed
* ui: centralize SSE wire-format delimiters into shared constants for the chat and /models/sse parsers
* ui: type /models/sse event names as a ServerModelsSseEventType enum
Address review from allozaur
line_start -1 normalized to n+1, so append inserted at lines.begin() + n + 1,
one past end() -> heap-buffer-overflow in vector::_M_range_insert.
Normalize -1 to n (insert at end()), restrict -1 to append mode and reject it
for replace/delete instead of silently clobbering the last line. Parenthesize
the insert offset so empty-file append computes the position as int first,
avoiding a transient begin() - 1 on a null vector data pointer.
* common/peg : refactor until gbnf grammar into an ac automaton
* cont : add a test with multiple strings
* cont : pad state with 0s so rules line up
* cont : clean up comments
* cont : use set everywhere
* cont : inline state num string padding
* cont : add a ref to PR
* cont : fix regression in server-tools.cpp
* arg: try fixing test-args-parser randomly fails
* return ref
* try triggering the workflow
* exception wrapper
* wip
* test
* test 2
* arg: guard win32 utf8 argv override
make_utf8_argv rebuilds argv from GetCommandLineW to fix utf8 handling of
non ascii arguments on windows. the override runs unconditionally inside
common_params_parse, so it also clobbers a programmatic argv passed by a
caller. test-arg-parser builds a synthetic argv but then sees the real
process command line instead, the model argument is never parsed, and the
assert that expects success aborts via fastfail (0xC0000409). this shows up
as a random failure in the openvino windows workflow.
only override argv when its length matches the caller argc, so the utf8
repair still applies to real binaries while a programmatic argv stays intact.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* server: avoid forwarding auth headers in CORS proxy
* format
* fix test
* fix e2e test
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
GLM-5.2 ships the DSA "lightning indexer" on only a subset of layers (the
"full" layers; others omit it), but the GLM_DSA loader created the five
indexer tensors on every layer as required, so loading any GLM-5.2 GGUF
failed with e.g. `missing tensor 'blk.3.indexer.k_norm.weight'`.
GLM_DSA's graph is llama_model_deepseek2::graph (plain MLA) and does not use
the indexer tensors (indexer runtime not yet implemented), so they are
loaded-but-unused. Marking them TENSOR_NOT_REQUIRED lets layers without an
indexer load as nullptr and the model runs as full MLA attention.
DeepSeek-V3.2 (uniform indexer on all layers) is unaffected.
Use std::partial_sort to order only the requested top-n tokens instead
of the full vocabulary
logprobs sort: vocab=128000 n_top=0 iters=100
full sort: 8555.6 us/op
partial sort: 704.3 us/op
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Absorb get_slot_by_id logic into get_available_slot so slot selection
is handled by a single function call. When a specific slot id is
requested, the LCP similarity check still runs to enable proper
prompt cache updates.
Assisted-by: pi:llama.cpp/Qwen3.6-27B
* ggml-cpu: support K tails in Power10 MMA Q8/Q4 matmul
This patch removes the requirement that K be divisible by kc in the tinyBlas_Q0_PPC tiled matmul path. Process the final K panel using its actual depth and pass the reduced panel size through packing and kernel execution. This allows more workloads to use the MMA kernel and reduces fallback to mnpack.
* Apply suggestion from @taronaeo
Co-authored-by: Aaron Teo <taronaeo@gmail.com>
---------
Co-authored-by: Aaron Teo <taronaeo@gmail.com>
* cuda: add GGML_OP_COL2IM_1D, follow-up to the CPU op
* cuda: col2im_1d use fast_div_modulo for the index decomposition
* cuda: col2im_1d tighten supports_op, type match and contiguous dst
* server: add "X-Accel-Buffering": "no" header to streaming endpoints
This header tells Nginx (as a reverse proxy) to NOT buffer responses. (only affects streaming endpoints)
Without it, Nginx will break streaming with certain applications (notably the Pi coding harness).
* ui : add model selector storybook stories
Covers list, favorites, single-model, all status states
(loading/loaded/sleeping/failed/idle), and selection states.
* ui : improve model selector mobile UX with hover media queries
Use @media (hover:none) to show action buttons directly on touch
devices and color-code them by model status (amber=sleeping,
green=loaded, muted=idle). Status dots hidden on touch. Desktop
hover behavior unchanged.
Throw on grammar parse failure so the server returns HTTP 400
instead of silently dropping the constraint.
Add a regression test for the invalid-grammar response.
Fixes#24144
* rename GGML_SYCL_SUPPORT_LEVEL_ZERO to GGML_SYCL_SUPPORT_LEVEL_ZERO_API, and GGML_SYCL_ENABLE_LEVEL_ZERO to GGML_SYCL_USE_LEVEL_ZERO_API
* fix code format
* fix error when rebase
* ggml: Conditionally enable power11 backend based on compiler support
Guard POWER11 backend creation behind a compiler flag check for -mcpu=power11. This avoids build failures on current GCC/Clang toolchains while preserving forward compatibility once POWER11 support becomes available.
* Update CMakeLists.txt
ggml-cpu: Use -mcpu=power10 for P10 and P11
Reuse existing rope kernels with a function constant to toggle forward/backward
rotation, avoiding duplicate kernel code.
Assisted-by: pi:llama.cpp/Qwen3.6-27B
* metal : add f16 and bf16 support for concat operator
Extend the Metal backend concat operator to support f16 and bf16 tensor
types in addition to the existing f32 and i32 support.
- Template kernel_concat on type T with specializations for float, half,
bfloat, and int
- Add type-specific pipeline getter ggml_metal_library_get_pipeline_concat()
- Update device support check to allow f16 unconditionally and bf16 when
device supports bfloat16
- Update dispatch to select the correct kernel specialization by type
Assisted-by: pi:llama.cpp/Qwen3.6-27B
* metal : extend concat operator to support f16, bf16, i8, i16 and i64
Assisted-by: pi:llama.cpp/Qwen3.6-27B
* add dev2dev memcpy by SYCL API
* mv GGML_SYCL_DEV2DEV_MEMCPY to runntime table
* update the detect method for p2p comm
* fix the erro created during fix confilct
---------
Co-authored-by: Neo Zhang <NA>
* webui: export conversations as jsonl
each session is one jsonl file, a session header line followed by one line per message
exporting multiple conversations bundles them into a zip, one jsonl file each
* webui: import jsonl and zip conversation exports
parse the new jsonl session format and zip archives on import
keep supporting the legacy json format
* UI : fix SSE transport detection and routing through CORS proxy. Assisted-by: Antigravity
* ui : replace magic strings with constants in MCP transport handling
* Add interface is_model_splitted() to check the c-graph is splited or not
* Infer and propagate dynamic-dimension indices for all tensors in the GGML graph in api compute_model_outputs()
* Only do this for fallback sub graph
* Move dynamic dims compute in graph missmatch
* ggml-openvino: fix tensor data handling for PERMUTE/VIEW ops in split models
* ggml-openvino:add comments
* ggml-openvino: override VIEW op_case to 0 for split model inputs
* openvino backend: Handle unsupported VIEW shape-mismatch in OpenVINO backend
* Enable additional mul_mat tests and add tensor data saving function (#81)
* ggml-openvino: fix CONT/TRANSPOSE mapping and improve dynamic-dimension handling
* OpenVINO: add NORM/TANH support and rework SOFT_MAX translation
* ggml-openvino: extend VIEW handling
* Enable -fa off (#118)
* Enable --context-shift
* Fix llm param compute error for normal softmax not the softmax in attention
* OpenVINO backend: fix error for attention size compute in llm param
* use tensor->extra in infer_request i/o
* OpenVINO backend: refacter the compute_llm_params() func add get_attention_pattern_case to easy extand
* OpenVINO backend: clean unused code
* 1to1 match op update (#146)
* added translate_1to1_match_1_input function and updated gelu and tanh translations
* Remove unused translation function calls
---------
Co-authored-by: Mustafa Cavus <mustafacavus@intel.com>
* initial gemma4 support
* removed hardcoded names for kv cache slicing
* OpenVINO backend: Add new attention pattern for llm parameters compute
* flash attn Q shape static conversion
* Remove slice in permute translation when n_seq is 1
* return optional in extract_layer_from_name
* OpenVINO backend: refactor VIEW related operation (#148)
* OpenVINO backend: refactor VIEW related operation
* Enable VIEW handling in following ops
* OpenVINO backend does not support GGML_OP_NORM & GGML_OP_L2_NORM with VIEW input accuracy issue from OpenVINO
* OpenVINO backend: Add ops l2_norm & pad
* OpenVINO backend does not support CPY with non-contiguous data or mismatched types
* add op SSM_CONV GATED_DELTA_NET
* OpenVINO backend: fix error for bf16 in OV gpu plugin
* reverted static Q input shape for attention layer
* OpenVINO backend: remove hardcode name inp_tokens, which ignore some leaf case
* Disable remote tensor due to bug in ov gpu
* Disable n_token > 1 GATED_DELTA_NET on gpu
* OpenVINO backend: fix the view op dynamic handling issue in gemma4 & enable view + get_row
* OpenVINO backend: clean code
* OpenVINO backend: enable view + norm/rms_norm
* OpenVINO backend: concat op
* OpenVINO backend: argsort op
* OpenVINO backend: enable unary + view & GGML_UNARY_OP_SOFTPLUS
* Fix issue for test-backend-ops in TOPK_MOE, which compare VIEW ops result, VIEW node in OpenVINO no need compare, the whole graph result is correct
* OpenVINO backend: enable sum_rows
* OpenVINO backend: enable clamp
* OpenVINO backend: enable DIV
* OpenVINO backend: enable GGML_OP_MUL_MAT_ID
* OpenVINO backend: disable MUL_MAT_ID_FUSION case with large mem needed
* OpenVINO backend: Disable GGML_OP_ARGSORT, cause test_backend-ops failed
* OpenVINO backend: fix issue in mul_mat_id
* OpenVINO backend: Disable DIV with broadcast on GPU
* OpenVINO backend: update DIV
* use ov internal op GatedDeltaNet
* OpenVINO backend: enable llama erch test qwen3next
* OpenVINO backend: enable RMS_NORM + VIEW & remove op_case 2 for rope
* OpenVINO backend: fix error
* suggested changes, need review
* suggested changes, need review
* OpenVINO backend: clean unused code & fix build warning
* OpenVINO backend: enable minicpm3 for arch test
* Disable GDN op (#177)
* disable gated_delta_net
* update stateful_kv_size correctly in mismatch case
* OpenVINO backend: enable arch test for qwen3vl
* OpenVINO backend: enable cohere2 for arch test
* OpenVINO backend: enable t5 for arch test
* OpenVINO backend: enable jamba for arch test
* OpenVINO backend: remove warning for tmp
* OpenVINO backend: enable kimi-linear for arch test
* Remove unused
* Fix gpt-oss accuracy issue
* OpenVINO backend: enable arctic for arch test
* OpenVINO backend: enable grok for arch test
* Gemma4 initial npu support (#179)
* Initiall gemma4 npu support
* temp. fix for gemma4 accuracy bug on npu
* Remove hardcoded names for npu-fold handling
* revert static n tokens for cont translation as it is not needed
* removed unused variable
* ggml-openvino: add GGML_OPENVINO_ENABLE_CACHE env var to control decoder cache. Add environment variable GGML_OPENVINO_ENABLE_CACHE (default: YES). When set to NO, the decoder_cache is bypassed and models are rebuilt from the cgraph on every inference call in both dynamic and static compute paths. This is useful for debugging and verifying correctness without caching interference.
* Revert "Gemma4 initial npu support (#179)"
This reverts commit 0d29a9c4a52dc2c8aa52990f1a3854cfb01768ad.
* OpenVINO backend: disable debug log print
* Update TBB discovery. Delegated to OpenVINOs own config.
* OpenVINO backend: GGML_OPENVINO_ENABLE_CACHE YES -> 1
* OpenVINO backend: fallback FLASH_ATTN_EXT in gemma3n to CPU backend
* Add raw ov infer profiling metric
* Add OV raw infer time metric to static compute path
Co-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>
* Modify precision of static profiling
* update to OV 2026.2, add OV windows CI
* fix editorconfig-checks
* Initiall gemma4 npu support
* temp. fix for gemma4 accuracy bug on npu
* Remove hardcoded names for npu-fold handling
* revert static n tokens for cont translation as it is not needed
* removed unused variable
* test-llama-archs fix
* Fix gemma4 flash_attn fallback
* support im2col
* fix code style
* disable add_rope_sin_cos optimization
* stateless boradcast and rope optimizations
* Enable manual gqa attn by default for stateless gpu
* manual gqa: fixed static batch
* gemma4 llama-bench ctx update fix
* Update OV win CI
* stateful rope fusion temp. fix
* OpenVINO backend: Conslolidate supported ops
* Exclude unsupported GGML_OP_SUB cases
* Exclude unsupported TOPK_MOE cases
* OpenVINO Backend: MUL_MAT enhancements
* Update OV CI
* support f16 mask input for npu
* Make GGML_OPENVINO_* env vars usage uniform
Standardize all GGML_OPENVINO_* env flags:
positive integers >0 to enable. Unset, empty, =0, or non-numeric values to disable.
This fixes cases where text values or empty strings enabled features.
* OpenVINO backend: Enhance envvar handling
* more cleanup
* move ggml_openvino_env_flag to appropriate place
* OpenVINO backend: add REPEAT translator, Q5_1 weights, and GLU view-input fix
* ggml-openvino: fix -Werror=cast-qual in extract_q5_1_data
* Update openvino.Dockerfile
Use BuildKit cache mounts for faster Docker rebuilds.
Use apt instead of dpkg, remove unused .ddeb downloads, add DLLAMA_BUILD_TESTS=OFF.
* ggml-openvino: centralize env var access via *getenv_str/getenv_int helpers
Replace getenv and legacy flags with _str and _int helpers.Minor cleanup, doc updates.
* OpenVINO backend: Enable GGML_OP_ADD_ID
* Uptade openvino backend clamg-format
* clang-format
* Update OPENVINO.md (#211)
* OpenVINO backend: fix accuracy issue for op CONCAT with i64 precision
* Remove strict concurrency for gpu-openvino-low-perf
* Update openvino CI keynames; add ccache-clear
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Fix formatting
---------
Co-authored-by: Xuejun Zhai <Xuejun.Zhai@intel.com>
Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
Co-authored-by: Mustafa Cavus <mustafacavus@intel.com>
Co-authored-by: Xuejun <XuejunZhai@intel.com>
Co-authored-by: Wang Yang <yang4.wang@intel.com>
Co-authored-by: Ravi Panchumarthy <ravi.panchumarthy@intel.com>
Co-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com>
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* SYCL: fix a bug with async memcpy
* make mmid_row_mapping_host persistent
* comment on stream->wait
* Apply suggestion from @sanmai
* Apply suggestion from @sanmai
* Apply suggestion from @sanmai
This introduces an optional feature to allocate large GPU buffers (≥ 1GB)
using USM system allocations if supported by the device. It allows using
buffers from the system allocator then letting the system manage memory
migrations between host and device as necessary.
This feature is disabled by default and requires the GGML_SYCL_USM_SYSTEM
environment variable to enable. If USM system allocations are not supported
by the device or the system, we fallback to regular allocations.
This feature can allow VRAM overcommit. For example, the test below fails
on B580 due to lack of memory for allocation, but it passes when enabling
USM system allocations:
./examples/sycl/test.sh -m Qwen3.5-27B-Q3_K_M.gguf -lv 4
Signed-off-by: Francois Dugast <francois.dugast@intel.com>
* ui: add source toggle to mermaid and svg blocks
Add a toggle button next to copy and preview that switches a rendered
mermaid or svg block to its source code and back. The button is shared by
both block types and the rendered view stays the default.
The source view reuses the code block scroll container and the highlighted
code element captured at transform time, so it matches the app code blocks
without highlighting again.
Make tall diagrams scroll like text code blocks: safe centering keeps the
diagram centered when it fits and falls back to start alignment when it
overflows, so the top stays reachable instead of clipping above.
Keep the block header opaque and layered above the scrolled diagram, and
ignore header clicks in the zoom handler, so a button click never falls
through to the zoom dialog.
* ui: transparent diagram block header, address review from @allozaur
Wyatt Caldwell