* ggml : fix for the radix TOP_K thread timing bug
* make the key an ordered 64-bit pair
* a rewrite without shift-skip logic
* fix casts
* warp-contiguous layout
* reduce the scope removing vulkan+metal changes
* turn it off for Metal/ROCm, note about cub::DeviceTopK
* fix the hip build
* skip Vulkan as well
* popc lambda from fattn
* do block_cols math in int64_t
* make it allowlist bc SYCL looks non deterministic too
* ggml : make Metal radix Top-K tied selections deterministic
The radix top-k kernel gathered the selected indices with atomic
counters, so the order of the tied candidates at the threshold (and,
with more ties than remaining slots, the subset itself) depended on
the thread scheduling.
Replace the two atomic compaction loops with a deterministic scheme:
count the elements above and tied with the threshold in contiguous
per-thread chunks, rank them with a simdgroup scan and a serial scan
over the (now unused) radix histogram, and write each kept element
exactly once in column order. The output is now byte-identical
across runs.
This is also faster than the atomic version: the strided atomic
loops become contiguous passes, reducing top-k times by ~20-45% on
M5 Ultra, with tie-heavy shapes within a few percent.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* tests : enable deterministic top-k check on Metal, add radix-path tie cases
Metal now produces deterministic tied selections for the radix top-k
path, so the determinism self-check runs for the MTL backend as well.
Add eval cases covering the radix path (ncols > 2048) with more ties
at the threshold than remaining slots, and tie-heavy perf cases to
monitor the cost of the deterministic tie handling.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* ggml : use coalesced accesses in the Metal radix Top-K compaction
The deterministic compaction walked contiguous per-thread chunks with
scalar sequential loads, which the memory system of the M2 Ultra can
not overlap well: the radix Top-K times regressed by ~1.5-2.4x on the
large-ncols shapes.
Give each warp a contiguous chunk of the row instead and let the lanes
interleave within it with stride 32, so every access is fully
coalesced. The lanes write their kept elements at lane-major ranks,
which is deterministic, and only a tie overflow falls back to a
per-step cross-lane ranking to keep the lowest tied indices in column
order.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* tests : remove the extra top-k tie and perf cases
The radix-path eval and tie-heavy perf cases were added to validate and
benchmark the Metal deterministic tie handling. Keep only the enabling
of the determinism self-check for the MTL backend.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* metal: scan the radix bins and warp totals with a simdgroup
The radix top-k selected the threshold bucket with a serial scan of
the 256 bins by thread 0 in each of the 4 passes, and ranked the warps
of the compaction with two serial scans by thread 0. The first
simdgroup now does both: each lane sums 8 bins, an exclusive prefix
sum and a ballot find the lane that reaches the k-th value, and the
warp totals get their exclusive bases from simd_prefix_exclusive_sum.
* tests: enable the deterministic top-k check on ROCm
HIP has no cub::DeviceTopK, so every row goes through the radix or
bitonic path, which break ties in the same way in each run. The check
covers all tie cases on ROCm, including the 1 and 2 row ones that CUDA
skips.
* cuda: remove the redundant radix top-k counter reset
Address review from ORippler. top_k_radix_init already zeroes
greater_count for each chunk of rows and top_k_radix_select keeps it,
so the reset kernel before the gather is one launch with no effect.
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Pascal <admin@serveurperso.com>
Zamba2 and similar architectures use a state size of 64 in their
recurrent-scan step, which fell outside the GPU kernel's existing
fast paths (96, 128, 256). Every scan step was falling back to CPU,
bottlenecking generation to 6-7 tok/s against the 50-70 tok/s band
other architectures achieve on the same hardware.
Add a 4th template instantiation for d_state=64, following the
existing pattern; no change to the underlying math. Verified against
the existing CPU-reference correctness harness with 3 new test cases
built from Zamba2's real hyperparameters (28/28 passing).
Measured ~35% throughput improvement (11 -> 15.1 tok/s) on Zamba2-7B-
Instruct. Full parity with other architectures not yet reached - a
separate, unrelated fraction of scan calls (~28%) still falls back to
CPU for a cause not yet diagnosed.
Assisted-by: Claude Sonnet 5
Co-authored-by: Sargentmajord <331229907+sargentmajord@users.noreply.github.com>
* ggml : switch to F32 src1 in the CPU backend
Change the default computation for GGML_TYPE_F16 to use F32 for src1
data in matrix multiplications: vec_dot_type(F16) is now F32 and the dot
product goes through the new ggml_vec_dot_f16_f32 (x = F16 src0, y = F32
src1), so src1 is no longer quantized to F16 and worst-case inputs that
only fit in the F32 FP range are handled correctly.
- conv_2d/conv_3d: store im2col patches (the inner mul_mat src1) as F32
and pass separate src0/src1 types to the inner mul_mat; the previous
F16 patches made the inner mul_mat widen src1 into the conv work
buffer, corrupting the patches and crashing on large inputs
- conv_transpose_1d/2d (F16 kernel): keep src1 in F32 in the work buffer
and use ggml_vec_dot_f16_f32; update work buffer planning accordingly
- supports_op: restore MUL_MAT support for F16 src1, which is now
widened to F32
- ggml_vec_dot_f16_f32: add GGML_SIMD support by converting chunks of x
to F32 and reusing the F32 SIMD dot
- mul_mat/mul_mat_id: add TODOs for a more efficient F16 x F16 path that
can be used if ggml_prec_get_src(dst, 1) allows converting src1 to F16
MUL_MAT f16 x f32 perf on M5 Ultra (GFLOPS):
- m=2048, n=512, k=16 : 120.9 -> 101.7
- m=2048, n=512, k=64 : 150.5 -> 344.2
- m=4096, k=14336 GEMV: 63 -> 343-415
- m=4096, n=512, k=14336: 65.6 -> 770.2
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* cont : add some F16 x F32 vectorized paths
* cont : enable the fused F16 x F32 dot on AVX without FMA
The 256-bit fused path of ggml_vec_dot_f16_f32 required AVX2 and FMA,
so CPUs with AVX and F16C only fell back to the chunked conversion and
F16 decode was 4x slower than before. Enable it on __AVX__ && __F16C__
and accumulate with GGML_F32x8_FMA, which does mul + add without FMA.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* llama, mtmd: follow the reference precision of ggml_prec_set_acc calls
Requests that the reference implementation does not keep in F32 move
to GGML_PREC_BF16: the GLM4 family down and output projections, the
MiMoVL vision matmuls and the Qwen3-TTS speaker encoder conv1d. The
DeepSeek V4/V4.1 routers and the MiniMax-M3 indexer stay in F32 with
a link to the reference line that computes them in F32.
* ggml: handle GGML_PREC_BF16 the same way as GGML_PREC_F32 in backends
Backends that reject GGML_PREC_F32 on the src1 of MUL_MAT_ID now also
reject GGML_PREC_BF16 (CUDA, Hexagon, OpenCL, WebGPU, Vulkan), and
backends that accept GGML_PREC_F32 for flash attention now also accept
GGML_PREC_BF16 with the same F32 accumulation (CPU, SpaceMIT, ET).
Vulkan keeps the F32 transfer between devices for both. The CUDA
cuBLAS path already handles GGML_PREC_BF16 and is unchanged.
* tests: cover GGML_PREC_BF16 in the flash attention tests
The flash attention cases run with GGML_PREC_BF16 next to F32 and
DEFAULT, and var_to_str prints bf16 so these cases keep their own name.
* llama: request GGML_PREC_BF16 for the KQ multiplication
* cont : add TODO in clip.cpp
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* opencl: add kernel_gemv_noshuffle_q8_0_f32_mc
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: small-n MoE GEMV for q5_1, q4_k
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: adreno kernel k floor 256 for x2
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: small-n MoE GEMV for more quants
Assisted-by: Claude Opus 5.5
Co-authored-by: Shaofei Qi <shaoqi@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: rename function
* opencl: disable bin FA for small verify batch
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* hexagon: use absolute skel path
* hexagon: support non-ascii dir and long path in skel-abs path
* hex-libdir: add support for FASTRPC_MANAGE_DSP_LIBRARY_PATH and GGML_HEXAGON_LIBHTP_DIR on all platforms
* hex-libdir: keep extra fwd slash in abs path on Android and Linux
* hex-libpath: always start with absolute path then fallback to session ctl then env
* hex-libdir: add CDSP1,2,3 to the fallback, needed for the IQ10
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* hex-nhvx: apply nhvx updates to the latest master
* hex-nhvx: set N_HVX and N_THREADS based on the arch to avoid overallocation
* hex-build: add support for enabling HTP_DEBUG via build.py
* hex-trace: move tracing state/helpers into new htp-trace.h header
* hex-dcache: cleanup dcache #defines
* hex-hnvx: minimize the diff by reflowing the macros
* hex-ci: fix python type/linter warnings
* hex-nhvx: consitent logging
* mtmd: support MageVL-Cowork model
* mtmd: keep magevl rope_freqs at model level and check its shape
The rope factors are shared by every layer, so they live in clip_model
instead of a pointer copied into each clip_layer. The loader now checks
that v.rope_freqs is F32 with d_head elements and that d_head splits
into the 4:6:6 pairs the graph expects.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* moe-cache : compute part of the misses on the CPU
--moe-cache-cpu-share F computes a share F of the MoE cache misses on the CPU
from the host experts instead of uploading them. The graph builds the expert
FFN twice: on the device from the cache slots, where the misses left to the CPU
read zero slots, and on the CPU from the host experts, where the experts
computed on the device have id -1. The two results are added before the down
bias.
- ggml-cpu: MUL_MAT_ID skips negative ids and writes zero rows
- sched: the copy callback is called for all the inputs of a split, and before
the host sync when there are no events
- the slot lookup callback reads the inputs of the CPU part together with the
selected experts, so the CPU part does not wait for the device, and its
result is copied back asynchronously
- each token uses distinct zero slots, mul_mat_id does not support an expert
selected twice by a token
- LLAMA_MOE_CACHE_TRACE writes the selected experts of each lookup, for a
simulator of the cache
* scripts : add moe-cache-tune.py to pick the MoE cache arguments
The script wraps a llama-server command: it starts the server at two cache
sizes and three CPU shares, fits the upload and CPU speeds of the machine, and
replays the routing of the runs through a copy of the cache logic to predict
other sizes and shares. It prints the recommended --moe-cache-mib and
--moe-cache-cpu-share.
docs/moe-cache.md describes the MoE cache arguments and the script.
* add am17an as codeowner for MoE cache
* moe-cache : upload all the misses until the cache is full
With a high --moe-cache-cpu-share, few misses are uploaded, so an empty cache
fills slowly and more misses than needed go to the CPU. The CPU share now
starts once all the slots of a cache group hold an expert. moe-cache-tune.py
replays the same rule.
* add backend sync for async transfer queue
* moe-cache : let the scheduler copy the results of the CPU part
* Fix MSVC compilation
* remove sync change
* llama-graph : extract the CPU share condition into can_cpu_share
* llama-graph : make llm_graph_params::moe_cache const
The graph construction no longer updates the MoE cache directly: the CPU
parts of the layers are accumulated in the graph result and applied to
the cache in llama_context::process_ubatch, after the graph inputs are
set and before the graph is computed.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* scripts : replay the ClassLRU policy in moe-cache-tune.py
* moe-cache : pick CPU misses by CRF instead of random (#31853)
* moe-cache : remove the backend logic of the CPU part from the graph
The CPU part reads the host experts, so the scheduler places it on the
CPU without set_tensor_backend. The cache gives no CPU map for batches
that a device would offload. Graphs with a CPU part look up their own
slot map, so the graph does not register the part with the cache.
ggml-backend : the inputs of the next split that are already computed
are copied to host memory before the current split runs, so a CPU split
does not wait for the device split before it. This replaces the stash
of the cache, and the copy callback is again called for host weights
only.
* llama-graph : return the MoE cache slots and CPU ids in a struct
The scheduler prefetch no longer checks that the next split runs on
another backend: the inputs of a split are never on its own backend,
so the input loop already skips that case.
---------
Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
The kernel_group_norm_f32 kernel and its host dispatch ignored the
batch dimension (ne03) of the source tensor: only ngrp threadgroups
were dispatched and each normalized the flat range
[tgpig*gs, (tgpig+1)*gs) with gs = ne00*ne01*ceil(ne02/ngrp). For
tensors with ne03 > 1, all but the first ne00*ne01*ne02 elements were
left unwritten (exposed by the new GROUP_NORM test with
ne=[3,1,1,65537], num_groups=1).
Dispatch one threadgroup per (group, i03) pair (ngrp in x, ne03 in z,
same work distribution as the Vulkan backend) and compute per-batch
statistics, matching the CPU reference. Also clamp the group element
count for the last group when ne02 is not divisible by ngrp and use
int64_t indexing.
The kernel now only requires ggml_is_contiguous_rows(op->src[0])
instead of full contiguity: it addresses src0 and dst with their own
strides (nb03 and the dst strides are added to the kernel args - dst
can be a strided view of src0 with ggml_group_norm_inplace) and falls
back from the flat fast path to a strided path that decodes the linear
element index when rows or channels are padded. The flat/strided
selection is done on the host in
ggml_metal_library_get_pipeline_group_norm and compiled into one of
two specialized pipeline variants via a FC_group_norm_flat function
constant, so the dead path is eliminated at pipeline creation.
GROUP_NORM support is also gated on F32 src and dst in supports_op -
the kernel only implements f32 and an f16 input would silently be read
as f32.
Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)
* server: visit parent slots before child slots
A parent slot wakes its waiting child slots once it finishes the shared
prompt prefix, while slots are visited in id order. When a child task
lands on a lower slot id than its parent, the child is visited before
the parent wakes it and waits an extra batch to process its suffix.
Non-child slots are now visited before active child slots, so every
child woken by its parent joins the same batch.
* server: check systemone test events through test tags
The systemone tests matched free-form debug messages, including the
formatted f_keep value of the prompt cache. The server now emits
__TEST_TAG_WAIT_PARENT_SLOT__ when a child slot waits for its parent,
__TEST_TAG_PROMPT_CACHE_RESTORE__ on every prompt cache restore and
__TEST_TAG_PROMPT_CACHE_FULL_RESTORE__ when the whole cached prompt is
a prefix of the new one, compared on token counts.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
* Only run webgpu CI on my fork
* Add webgpu only workflow
* Add WebGPU group_norm kernel
* clean up
* resolve merge conflicts
* parameterize data type; clean up cmake changes
* add nullptr check
* clean up webkit experiments
* more cleanup
* more cleanup
* clang format
* Remove unused feature check for WebGPU
* clean up supports_op condition and update documentation
* add perf tests for group_norm
* clean up doc with script and undo change in test-backend-ops
* Do 2d dispatch of workgroups
* Do 2d dispatch of workgroups
* Unstage unrelated changes
* undo docs churn
* add new test case
* undo small change
* undo small change
* ggml-backend-meta : minimum number of compute views per buffer
Assisted-by: Claude Opus 5.5
* tests : revert the deepseek41 test model to 6 layers
Assisted-by: Claude Opus 5.5
* hexagon: fuse post-norm residual add into RMS_NORM_MUL
* hex-fusion: refactor RMS_NORM fusion and add dedicated RMS_NORM_MUL_ADD opcode
Improves performance by keeping things in HVX registers.
Also simplifies overall fusion logic and logging.
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* model: add Kimi-K3.1-Pro text model
K3 with the full-attention MLA layers replaced by DSA: on this checkpoint 24
of the 93 layers carry a lightning indexer (32 heads x 128, top_k 2048, no
rope). The KDA (linear-attention) layers and the rest of the K3 skeleton -
cross-layer residuals, latent MoE, situ, MLA output gate, full-rank KDA gate -
are unchanged, so this reuses the K3 graph and converter and adds the indexer.
Indexer:
- the selection is an additive mask over the whole cache, not a gather of
the top-k rows, so it reuses the existing sparse flash attention path;
off that path the mask stays correct but still costs a kq over the cache
- the indexer key LayerNorm takes its eps from the model's rms_norm_eps,
because dsa_config declares no eps of its own
Graph and cache:
- new arch with four indexer tensors per DSA layer, and a hybrid memory
that keeps a K-only MLA cache for those layers, recurrent state for the
KDA layers, and the indexer key cache
- the graph node budget shares the K3 entry
Conversion:
- KimiK31Model(KimiK3Model), picked by KimiK31ForConditionalGeneration
- dsa_config becomes attention.indexer.{head_count,key_length,top_k,types}
plus the indexer LayerNorm eps; the layer list is 1-indexed like
linear_attn_config's, and the converter asserts that it covers exactly
the full-attention layers
- the routed experts stay MXFP4 and are repacked as in K3, not dequantized
- the vision tower and mm_projector are skipped: text only for now
- the arch key follows the checkpoint's own model_type ("kimi_k31") and the
loader registers KimiK31ForConditionalGeneration; the release name carries
a -Pro suffix that nothing in the config or the tensor names uses
Quantisation: the indexer's q_b, k and proj are kept out of the quant
overrides, since the top-k selection is sensitive to their precision.
Verified on an 8-layer slice against the HF checkpoint: 38 tensors checked
exactly, the indexer metadata against the config, and the selection against
full attention - identical PPL at top_k beyond the context length, divergent
once top_k is forced below it. Arch test fixture and the full arch suite pass.
* converter: avoid clutter in the k3 base
* narrow comment to match model
* update metal fusion expectations
* restore
* sync with PR 31733 refactoring
* hexagon: Generalization of CONV_2D implementation for CONV_3D
- New op HTP_OP_CONV_3D is wired through htp-ops.h, main.c (shares op_conv_2d) and the op remap in htp-kparams.cpp.
- conv_3d runs on the existing HMX tiled (implicit GEMM) conv2d path. KD and KH are folded into one virtual kernel row (KD*KH), and output rows are counted as N*OD*OH.
- htp_kparams_solve_conv3d checks the conv_3d layout and calls the tiled solver htp_kparams_conv2d_tiled, which takes an is3d flag.
- The tiled kernel takes the sizes and strides that differ between conv_2d and conv_3d from a struct htp_conv_geom. Only the input stage unfolds depth.
- 256-entry DMA queue can no longer overflow
Assisted-by: Claude Code
* hexagon: modify checking to eliminate divide-by-zero hazards
* hexagon: conv2d tiled: compute the conv_3d unfold divisors on the host
The KH and OH fastdiv values used to unfold conv_3d kh into (kd, kh) and
oh into (od, oh) were built per op on the device, each with a 64-bit
divide. Pass them in the kernel params instead; they fit in the space the
window variant already reserves, so the struct size does not change.
Assisted-by: Claude Code
* hexagon : preserve fold scheduling in conv2d for small size
hexagon : support arbitrary positive conv2d strides
Support positive horizontal strides for WHCN and CWHN inputs. Preserve
load/shuffle paths for strides 1/2 and use gather for larger strides.
hexagon : improve conv2d tile size selection
Prefer the largest feasible output-channel block and derive spatial
chunk capacity from VTCM usage. Compare capacity and thread-aligned
candidates using per-thread work and chunk overhead.
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
* hex-conv: fix compiler warning
---------
Co-authored-by: Yiwei Shao <yiwei@aizip.ai>
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* tests : qualify isnan and signbit with std::
The unqualified calls rely on the C math.h names also being present in the
global namespace, which is not the case for the NetBSD 11.99.9/x86_64 gcc
14.4.0 build reported in the issue. Prefix the calls with std:: to select
the standard C++ overloads declared by <cmath>.
ref: https://github.com/ggml-org/ggml/issues/1681
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : qualify C math calls with std::
Follow-up to the isnan / signbit fix: make the remaining math calls in
test-quantize-fns.cpp consistent with the std:: isfinite / max / numeric_limits
already used in the same file.
The float variants are overloads in C++, not distinct names, so the mapping is
fabsf -> std::fabs, copysignf -> std::copysign, nextafterf -> std::nextafter -
not std::fabsf & friends, which do not exist in libstdc++ and would break the
gcc build. NAN and INFINITY are macros and are left unqualified.
Only behavioural difference: array_rmse() fed a double accumulator to sqrtf(),
now std::sqrt() keeps the computation in double. The result is more accurate
and differs from the previous value by at most 1.4e-7 relative, three orders of
magnitude below the tightest tolerance in this test.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
The self-hosted CPU runners are containers with 8 CPUs, but
std::thread::hardware_concurrency reports the cores of the host. The
table mode of test-llama-context sizes its pool from that count, so it
runs 4 models x 8 threads on an 8-CPU job and hangs.
Add TEST_ARG_HARDWARE_CONCURRENCY and TEST_ARG_PARALLEL_WORKERS as the
defaults for the auto-selected counts of test-llama-context and of
test-backend-ops, and set them to 8 and 4 for cpu-x64-high-perf.
Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
* moe-cache : LRFU/MQ-inspired hybrid cache
* make the clock advance by N tokens: use the real token count (ne[1] is always 1, ids have n_expert_used entries per token)
* sel is a RESHAPE node, the orig input is in src[0], so sel->src[0]->ne[1] is the count
* c for the class index, last_use for last use time
* explain the roles of these
* compute the key once
* use the key directly instead of re-reading
* show how the score is computed from the age
* remove the branch and clarify exp2 and log2 relationship
* reword the header comment and elaborate on unused slots and sorting
* not a copy, not just skip
* std::array maybe
* Revert "std::array maybe"
This reverts commit 7649b1f172db4941a854ad115bec0c7e062b65b9.
* half life it is
* add key and iteration helpers
* extract decayed_crf
* bump classes to 8, document the formula, display the cache size in token worth count
* assert the computed number of classes matches the half life
* opencl: use the multi-column q6_K vocab head GEMV for 2..4 tokens on X2E
Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5
* opencl: rename GGML_OPENCL_Q6K_MC3 to GGML_OPENCL_Q6_K_MC3
Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5
* opencl: rename GGML_OPENCL_Q40/Q41/Q4K_MC3 to GGML_OPENCL_Q4_0/Q4_1/Q4_K_MC3
GGML_OPENCL_Q4K_MC3_DQ and GGML_OPENCL_Q4K_MC3_LDS become GGML_OPENCL_Q4_K_MC3_DQ
and GGML_OPENCL_Q4_K_MC3_LDS.
Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5
* opencl: remove unnecessary debug probe
Co-authored-by: Shangqing Gu <shawngu@qti.qualcomm.com>
Assisted-by: Claude Opus 5.5
* opencl: clean up
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* hex-repack: move repacking/tiling into graph-optimize
* hex-tiling: simplify tiling rules and remove obsolete helpers
* hex-ops: use a hash for de-duplicating tensor descriptors
* hex-bufs: improve tensor hash
* hex-ops: accept empty (zero-elements) ops
* hex-bufs: enable hostbuf by default and reject tensors that do not fit into max_vmem
* hex-bufs: improve error message and update readme
* hex-bufs: make sure to sync flush opbatch in all sync set_tensor paths