Compare commits

...
Author SHA1 Message Date
LumpiastyandClaude Opus 5.5 af815458ca ggml-cpu: faster AVX2 NVFP4 x Q8_0 dot product (two blocks per iteration, vector UE4M3 scales)
The x86 AVX2 ggml_vec_dot_nvfp4_q8_0 did one block per iteration with four
scalar UE4M3 table lookups, two scalar FP16 conversions, four scalar muls and
broadcasts per 64 weights, plus two FMAs on one accumulator chain.

New AVX2 path, two blocks (128 weights) per iteration:
- E2M1 decode with 256-bit pshufb on the whole 32-byte qs, unpack + permute2x128
  into the q8_0 element order (low nibbles are elements 0-7 of a sub-block,
  high nibbles 8-15);
- the 8 sub-block sums reduced exactly in int16 (a sub-block sum is at most
  16*12*127) with two levels of hadd_epi16 and one madd, one cvt;
- the 8 UE4M3 scales decoded in one vector (bit manipulation, bit-exact with
  ggml_ue4m3_to_fp32 for all 256 bytes: x0.5, 0x7F -> 0, sign bit ignored,
  subnormals), the 4 q8_0 FP16 scales assembled in a GPR and converted with one
  vcvtph2ps; one mul and one FMA per iteration.
An odd trailing block uses the existing scalar loop. The AVX (non-AVX2) path
and the generic reference are unchanged.

tests: test-backend-ops perf cases at Inkling-2-Mini expert shapes (n_embd 2304,
gate_up 2304, down 1152, 128 experts / 8 used; n = 1, 8, 16, 32) for nvfp4,
mxfp4 and q4_0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-23 00:01:59 +01:00
LumpiastyandClaude Opus 5.5 27c23d20fe docs: FORK.md/runbook/AGENTS.md for the 2026-11-22 rebase onto 5bacb7c86 - carries, #32807 vs F13, --fit for Inkling, F14 refinement, phase 1 results, window plan
- AGENTS.md: one-line pointer to FORK.md and the runbook at the top.
- FORK.md: candidate status; commit map rows for the new commits; F12
  state after the rebase (#32735 merged and dropped; #32740 split down to
  the GGML contract, the 808fd1c4e carry kept, owner decision; #32777
  carried) and how to use --fit for Inkling, with the F13 interaction read
  (fit measures F13's compute buffer; the visit count is taken on the
  all-device probe, so the views do not hide the loop); F13 section on
  upstream #32807, the LLAMA_LOOP_FULL_COPY knob and the window A/B; F14
  refinement (system-prompt checkpoint, disk cache read, lifetime) and
  validation plan items 5-7.
- Runbook: a rebase is not done until fork-validate phases 1 and 2 pass;
  ssh access and the persistent rootfs; gdb persists; the shrinking-PR
  case for carries; the 2026-11-22 rebase record; the overlayfs lock_id
  issue; phase 1 results and the ordered list of what the window must run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 21:06:33 +01:00
LumpiastyandClaude Opus 5.5 70d39bdbca scripts: fork-validate.sh - scratch arenas on /dev/shm, lock_id failed on overlayfs
Since 2026-11-22 the pod's root filesystem persists and /tmp is on
overlayfs. There stat reports the overlay device (st_dev 0xb3) while
/proc/locks prints the device of the upper filesystem (00:84), so lock_id
never matched the scratch token.lock and arbiter/start failed with "A
serves but does not hold token.lock" although A held the lock. Production's
arena is on tmpfs (/dev/shm/llama-vram), where both agree. Put the scratch
arenas there too (/dev/shm/fork-validate.<suffix of the scratch dir>,
removed by finish()); logs and the rest of the scratch stay in /tmp.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:59:28 +01:00
IzentoandClaude Opus 5.5 6c895a491d upstream-carry #32777: fit : prioritize looped MoE layers in device memory (open upstream PR, temporary)
Temporary carry of the open upstream PR
https://github.com/ggml-org/llama.cpp/pull/32777 at head 01945dcb0 (one
commit), carried at the owner's request: loop-aware --fit placement for
inkling2_loop (llama_model_layer_visits ranks the looped layers first, so
their experts stay in device memory before the single-pass layers). Drop
this commit when the PR is merged upstream (FORK.md F12).

Applied cleanly on upstream 5bacb7c86 plus the fork; no conflict
resolution. The F13 interaction (views break the pointer-identity loop
detection) is fixed in a separate fork commit, not here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:41:11 +01:00
LumpiastyandClaude Opus 5.5 5f88124b38 llama: env knob LLAMA_LOOP_FULL_COPY=0 to skip upstream's inkling2_loop full-copy guard (#32807)
Upstream #32807 makes sched_copy_experts return false for inkling2_loop,
so the scheduler copies every expert of a host weight it shares across the
loop passes (the fix for #32755). With the F13 per-pass views each pass
has its own device copy and its own ids, so the partial copy (routing
readback below the F1 skip threshold, 32-63-token ubatches on
Inkling-2-Mini) is correct again and the guard only costs uploads.
LLAMA_LOOP_FULL_COPY=0 skips the guard for an A/B; the default keeps
upstream's behaviour. With LLAMA_LOOP_HOST_VIEWS=0 the knob brings #32755
back, do not combine them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:41:04 +01:00
LumpiastyandClaude Opus 5.5 e7e907b7c3 server: F14 - keep the batch break at the first user message (system-prompt checkpoint)
With LLAMA_USERSTART_MERGE on (the default), F14 skipped every user-message
batch break when the rest of the prompt fits in one ubatch. On a fresh
prompt that also dropped the break at the first user message, so no context
checkpoint was left at the end of the system prompt, and a new chat with the
same system prompt re-processed the system prompt from scratch (a
recurrent/hybrid state cannot be cut back without a checkpoint).

Keep that one break in the merged path: the batch ends right before the
first user message, and the next batch starts there and gets the checkpoint
(is_user_start, near-end rule). The batch only reaches the first user
message when the prompt is processed from before it: checkpoints past the
restore point are erased on reuse, so this is a fresh prompt or one that
diverged inside the system prompt. Follow-up turns start past it and keep
the merge (no break at the last user message). Boundaries below 8 tokens
(only a BOS before the first user message) are skipped.

Cost: on a fresh prompt the system prompt runs as its own batch again (the
upstream layout for turn 1), measured before as part of turn 1's
6.08 vs 3.99 s TTFT (FORK.md F14). The unmerged path is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:40:50 +01:00
LumpiastyandClaude Opus 5.5 61544a5841 docs: FORK.md/runbook after the f3397cded deploy - F14 measured (35B TTFT A/B, batch layout, edit cost, disk cache), pointer 16
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Opus 5.5 988dd620f5 docs: FORK.md F14, user-message batch-break merge (not deployed)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Opus 5.5 7e738d7478 server: merge user-message batch breaks when the rest of the prompt fits in one ubatch
Upstream (#22929, #24176, #25420) ends a prompt batch at the start of the last
user message (and of earlier ones on a fresh prompt) so that a context
checkpoint lands there. On a hybrid/recurrent model with MoE experts on the
host, that turns a typical chat turn (previous answer + new user message, under
one ubatch) into two partial ubatches, and each one of >= 32 tokens uploads the
host experts again.

When the rest of the prompt from the current batch start fits in one ubatch,
skip the user-start break. The near-end rule still checkpoints the batch start
and the end-4 rule the prompt end, so a later prompt that diverges inside the
last user message re-processes less than one ubatch more, in the same ubatch.
Prompts longer than one ubatch, and batches after a media chunk, keep the
upstream behaviour. LLAMA_USERSTART_MERGE=0 restores it everywhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Opus 5.5 bbee69f354 docs: FORK.md section 5, Inkling vs Qwen3.6-35B quality comparison
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Opus 5.5 87916b18f8 docs: FORK.md/runbook after the b13c41355 deploy - F13 measured (KLD bug-window test, fit and placement guide, quants, profile), Inkling -ot k=4 without the offload knob, nothink fix, pointer 15
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Opus 5.5 ef59f30d36 docs: FORK.md F13, per-pass views of host weights in inkling2_loop
What and why (#32755 and the 6.5 GiB of shared expert copies), the knob,
F1/F4/MoE-cache interactions, the extra upload cost, how to drop it, and the
pod validation plan (reference with the loop experts on the GPU, views on,
views off).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Opus 5.5 53d2828bb5 llama: inkling2_loop - per-pass views of host weights for the later loop passes
inkling2_loop runs the core layers (6..17 on Inkling-2-Mini) three times, and
post_arch_tensors makes every cache slot share the tensors of its physical
layer. With the experts in host memory (--n-cpu-moe, -ot exps=CPU) the
scheduler makes one device copy per (host tensor, backend) per graph, at the
first split that reads it, and keeps it until the last reader. So:

- the expert copies of all 12 loop layers are alive at once (about 6.5 GiB on
  UD-Q4_K_XL), which is most of the ~7 GiB Vulkan0 compute buffer;
- sched_copy_experts runs only for the first reader. When it reads the ids back
  (ubatches of 32-63 tokens: MUL_MAT_ID is offloaded from 32 tokens, the F1
  skip starts at 4*n_expert ids), it uploads only the experts that pass 0
  selected, and passes 1 and 2 read stale VRAM for the others (upstream issue
  #32755, "GGGG" output).

Add a post_load_tensors hook to llama_model (called after the weights are
loaded, and on the no_alloc path after the dummy buffers are set). In
inkling2_loop it replaces, for each slot of a later pass, every tensor in a
host buffer with a view of it (name "<name>.visit<pass>", buffer and data of
the original). The scheduler hashes tensors by pointer, so each pass gets its
own copy, freed after that pass, and the copy callback sees the ids of that
pass. Weights in device buffers and CPU extra buffers (CPU_REPACK, not host)
stay shared. The pointer-keyed weight maps (w_scales, prec_policy, hdmd.rot)
get entries for the views. The views live in a model-owned context outside
ctxs_bufs, so the F4 release/restore, tensor counts and the memory breakdown
do not see them; they point only at host buffers, which F4 never releases.

LLAMA_LOOP_HOST_VIEWS=0 keeps the shared tensors (A/B knob).

Not built or run yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Opus 5.5 1a19c3bbba docs: FORK.md/runbook after the 3621025e8 deploy - F12 measurements, Inkling ub1024 + ctx 65536 and the offload knob, nothink finding, pointer 14
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Opus 5.5 01812e8b02 docs: FORK.md F12 temporary upstream carries (#32735, #32740), Inkling config; runbook carry procedure
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:38:33 +01:00
Piotr WilkinandClaude Opus 5.5 66b3c3f72f upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan
Temporary carry of the open upstream PR
https://github.com/ggml-org/llama.cpp/pull/32740 at head 808fd1c4e,
its four commits (1fa93f902, d29f9d489, ff91e4f84, 808fd1c4e) squashed.
Inkling's per-query bias table is applied inside flash attention
instead of a dense [n_kv, n_tokens, n_head] bias tensor, which cuts
Inkling's compute buffers and long-context prefill time on the RX 580.
Drop this commit when the PR is merged upstream (FORK.md, "Temporary
upstream carries").

One conflict, ggml-vulkan.cpp use_mask_opt: kept the fork's GCN
`HSK >= 256 || HSV >= 256` clause and its comment (F2) and upstream's
`!use_sparse`, added the PR's `!bias &&`.

Original commit trailer: Assisted-by: Codex

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Sonnet d2143f1e47 docs: runbook/FORK.md fixes from the 2026-11-21 rebase, validate and deploy cycle
HF_UI_VERSION must use the tag after the '|' in .ui-stamp; bundle path
for agents told not to push; build backup and copy-based rollback;
status, pointers and last-run record for 3bf6b55b3; q35.py build path
note; disk-cache restore test wording; open questions.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Sonnet 5.5 372f14e0b2 docs: runbook and FORK.md status for the 2026-11-21 rebase onto e6d624b9a
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Sonnet 5.5 cf2790794d ggml-vulkan: gate upstream's tiled concat below one tile, honour the fast-path knob
Upstream #30591 added its own tiled-transpose concat pipelines, selected
for any dim-0 concat whose src1 is transposed. Behind the fork's fast path
(which already takes ne0 >= 32 and honours GGML_VK_CONCAT_FASTPATH) that
selector still caught the decode case (ne0 < 32), bringing back the
decode regression the TILE_DIM gate fixed (FORK.md F3), and
GGML_VK_CONCAT_FASTPATH=0 no longer gave the generic kernel.

Apply the same gate and the knob to upstream's selector.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Sonnet 86bf601db6 docs: runbook/FORK.md after the adc2aa3ec deploy
Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Opus 5.5 92abb2d89b docs: F11 mmproj incident and fix; runbook - production command line, build targets, detached scripts
FORK.md F11 records the rolled-back deploy (the GGML_ASSERT(!has_mtmd) abort
with the production mmproj), the fix, the new mmproj cases (15 PASS), and
that the earlier 35B numbers were measured with --no-mmproj. The 35B run with
production's exact command line (diskcache-window3.sh) is ready, not run, and
a precondition for deploying --cache-disk-path.

Runbook:
- tests must use production's exact model command line, mmproj included;
- the validated build has all three targets;
- the deploy mtime list includes libllama-server-impl.so and
  libllama-common.so (llama-server is a stub);
- deploy and rollback steps run as one setsid-wrapped script on the pod,
  never as a backgrounded cat && sh -n && setsid chain from kubectl exec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Opus 5.5 b3b7ad7973 scripts: fork-validate.sh --diskcache - the mmproj cases
Production runs the 35B and gemma with an mmproj, and the first deploy of the
disk prompt cache aborted on that path. Every case had used --no-mmproj. Now
--no-mmproj is in DC_MM, separate from DC_ARGS, and two cases run with the
0.8B's mmproj (--no-mmproj-offload, arbiter env, LLAMA_MMPROJ_ONDEMAND=1),
against their own reference:
- restore-mmproj: turn 1 is saved on SIGTERM and restored after the restart,
  prompt_n equal to the reference, output identical;
- mmproj-image: an image request (no restore, answer "red"), then the text
  chat (restored, identical), then an image again, then SIGTERM with media in
  the slot (no state copied, exit 0, the file count unchanged).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Opus 5.5 a188884487 server: disk prompt cache - text-only prompts with an mmproj loaded
The deploy of cb48bf224 with --cache-disk-path was rolled back: the first
request to the production 35B aborted in disk_cache_load().
server_tokens::get_tokens() has GGML_ASSERT(!has_mtmd), and has_mtmd is true
whenever an mmproj is loaded, even for a text-only prompt. Production's 35B
and gemma-26B always load one (-hf pulls it). disk_cache_collect() and
server_prompt_disk_cache::save() made the same call, so the shutdown save
would have aborted too. Every test had used --no-mmproj, which hid it.

disk_cache_text() takes the list from get_text_tokens(). A prompt is
text-only when that list is as long as size(), because media positions hold
LLAMA_TOKEN_NULL. So a text-only prompt is saved and restored with an mmproj
loaded, which is the main case (OpenWebUI chats on the 35B and gemma), and a
prompt with media is never saved or restored. When the slot itself holds
media, its reuse estimate is server_tokens::get_common_prefix(). These four
calls were the only get_tokens()/mtmd-unsafe accesses in the feature.

fork-validate --diskcache with the new mmproj cases, beside production:
15 PASS, 0 FAIL, GPU legs included. With the 0.8B's mmproj loaded, a text
chat is restored: prompt_n 58 of 596, output identical to the reference.
An image request is answered "red" with no restore, the text chat after it
is restored, and a SIGTERM with media in the slot copies no state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Opus 5.5 1505d6a52b docs: FORK.md F11 - 35B results, the two leg-c anomalies explained
The swapdown window: --diskcache --arbiter 29 PASS. The 35B, 18760 tokens:
83.4 s prefill, turn 2 after a restart 3.0 s with identical output. Warm
save: token free in 328 ms, exit 1952 ms, peak RSS +430 MiB. Cold save:
peak +432 MiB.

Leg c, output != reference: c sent turn 2 again, so the server rolled back
to the checkpoint at the prompt end and processed 4 tokens, while the
reference processed 27 from turn 1. These are different reuse points. A
controlled same-reuse-point check on the 0.8B (X never restarts vs Z after a
restore, a cold save and a restore) is bit-identical, top-3 logprobs
included, on CPU and GPU. The 35B version is ready as diskcache-window2.sh
and has not run.

Leg c, +3055 MiB and token free in 1881 ms: the driver's fake waiter (flock
-s file sleep) leaked the lock into sleep, so the SIGTERM landed in an
ordinary evict_kv release (the full KV shadow, F4). Not the cache; the
driver is fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Opus 5.5 3d862b1357 docs: FORK.md F11 disk prompt cache, the CPU_REPACK release fix; runbook notes
F11 in the per-feature format: design, invariants, the measured costs (CPU
only) and llama-swap's stop behaviour (it waits for the exit, SIGKILL at
30 s). Also: the recommended config (not applied), the commit map entries, the
F4 incident note for the CPU_REPACK crash, F7's shutdown time with the cache,
the F9 log lines, and the section 7 gaps (GPU leg, 35B and --arbiter not run
yet). Runbook: --diskcache in phase 1, the pod window script, the deploy
notes (every deploy empties the cache, /root free space, a slower stop), a
post-deploy check, and the cache dir.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Opus 5.5 fe6c0f70a6 scripts: fork-validate.sh --diskcache, the disk prompt cache
Runs beside production. The model is the small hybrid (Qwen3.5-0.8B), CPU only
(-dev none -ngl 0, no VRAM), with chat requests and greedy decoding. A
reference server that never restarts runs turn 1 and turn 2. The candidate
saves turn 1 on SIGTERM and restarts. Turn 2 must then be restored from disk
with the same prompt_n as the reference (only the suffix processed) and the
same answer. Cases:
- restore plain, with the arbiter env warm, and forced cold by a fake waiter
  on want.lock (the save from a released state). Each case records the exit
  time and SIGTERM-to-token-free, also once without the cache;
- no restore for another conversation, or with another n_ctx, KV type or
  model path (other key);
- a truncated file is removed at startup and a byte-flipped one at load,
  never used;
- the size limit evicts the oldest file;
- a write error (RLIMIT_FSIZE 1 MiB, SIGXFSZ ignored: EFBIG) is logged, the
  exit stays 0 with no file left, and the next start serves;
- the same restore on the GPU (-ngl 99, arbiter env, warm and cold) when
  >= 1.5 GiB of VRAM is free, else SKIP.
start_server gains SRV_WRAP, a command prefix that execs the server.

First run (2026-10-25, beside production): 11 PASS, 0 FAIL; the GPU leg
SKIPped (1219 MiB free). The first run found the CPU_REPACK release crash
(exit 139 in the cold case).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:33 +01:00
LumpiastyandClaude Opus 5.5 f1f1f32f3f server: disk prompt cache, save slots on shutdown and restore after a restart
llama-swap SIGTERMs a model when the owner switches models (only one large
model fits in RAM), and switching back re-prefills the whole conversation:
about 150 s for a 30k-token chat on the 35B. With --cache-disk-path DIR,
each slot with a prompt is saved to DIR on shutdown. After the restart, a
request whose prompt starts with a saved prompt is restored from the file,
so only the suffix is processed. Off by default.

Save, in server_context::start_loop() after the loop returns:
- disk_cache_collect() copies the state of each slot with >= 256 text tokens
  to host memory (llama_state_seq_get_data_ext, flags NONE) while the device
  buffers still exist. A cold server reads its released state through
  llama_context_restore_memory_host(), with no device use and no extra copy.
- vram_share_exit() then frees the VRAM and releases the GPU token as before,
  so only the D2H copy delays the token release, never the disk IO.
- disk_cache_flush() writes the files. A slot is skipped if its KV does not
  cover its token list (seq_pos_max + 1 != n_tokens: the last batch was not
  decoded) or if the same token list is already on disk.

File (server_prompt_disk_cache, server-task.cpp):
- A 48-byte header (magic, version, key hash, sizes, payload hash), then a
  payload of 8-byte padded sections: tokens, checkpoint table, state, the
  checkpoint states. The hash is FNV-1a over the 64-bit words of the payload.
- Plain fwrite to <name>.tmp, fdatasync, POSIX_FADV_DONTNEED, rename. The read
  is a plain fread plus DONTNEED. There is no mmap: upstream #28092 livelocked
  the kernel on AMD + Vulkan by DMA-ing into a MAP_SHARED file.
- The index is built at startup from the headers. A bad magic, a size that
  does not match the header, or an inconsistent table means the file is
  removed. A hash mismatch at load also removes it, and nothing is restored.
  Other keys and other versions are kept. A .tmp older than 1 h is removed.
- --cache-disk-size N MiB (default 4096, 0 = no limit): before a write, the
  least recently used *.lpcd files of every model in DIR are removed until the
  new file fits; a restore updates the file mtime. Entries the new prompt
  extends are dropped after the write and do not count against the limit.
  A write also needs its size + 1 GiB free on the file system. A write error
  or a full disk is logged; the file is not saved and the exit stays clean.

Key: the build commit, the model path + size + mtime, LoRA and control
vectors, n_ctx, n_ctx_seq, n_seq_max, n_ubatch, kv_unified, swa_full, the
K/V types, the flash-attn type, and the RoPE/YaRN parameters.

Restore, in get_available_slot() after the RAM cache:
- Only while warm (holding the GPU token), with cache_prompt on and no media
  tokens. The file must reuse at least 256 more tokens than the slot holds.
- Reusable means one of two things. Either the saved token list is an exact
  prefix of the new prompt, or (for a recurrent/SWA model) a saved checkpoint
  at or below the common prefix can be used. A plain-KV model can be cut back
  to any prefix. Exact prefix alone is not enough for OpenWebUI on Qwen: the
  template rewrites the previous assistant turn (the reasoning is dropped), so
  the saved list is almost never a prefix of the next turn. The checkpoints
  are exact snapshots, and update_slots() rolls back to one exactly as it does
  for the RAM cache. They are restored with id_task -1, so one that fails to
  load means a re-prefill, not an abort.
- The slot's current prompt goes to the RAM cache first. A set_data failure
  clears the slot and removes the file.

Not supported with speculative decoding (warning, disabled): the draft state
is not saved. Only the slots are saved, not the RAM cache entries.

Measured on the pod, CPU only, Qwen3.5-0.8B (hybrid, 6 of 24 layers attention):
- 573-token chat: 64.5 MiB file with 2 checkpoints, copy 18 ms, write 171 ms,
  exit 314 ms (125 ms without the cache). After the restart: read 166 ms,
  restore 3 ms, turn 2 prompt_n 58 of 596 in 462 ms against a 2913 ms full
  prefill. The answer is identical to a server that never restarted.
- 12k tokens: 200.7 MiB, copy 110 ms, write 350 ms, exit 641 ms, peak RSS
  +162 MiB during the exit save. After the restart: read 307 ms, restore 18 ms,
  turn 2 prompt_n 35 of 12188 in 1.2 s against a 72 s prefill.
- The /root PVC writes at about 0.66 GB/s with fdatasync. /tmp (the node
  overlay) stalled once: 5.8 s for 64.5 MiB.
llama-swap v239 waits for the old process to exit before it starts the next
model, and SIGKILLs a process that is still there after 30 s. So the exit
save adds to the model switch time: about 1 s per GB on the PVC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 aa46b82321 llama: read a released KV/recurrent state from its host shadow
After llama_context_release_device(ctx, true) the KV cache and the recurrent
state live only in host shadows, and their tensors have no buffer, so
llama_state_seq_get_data cannot read them. The disk prompt cache must save
a cold server's slots at exit, and a shutting-down server must not restore
to the device (it may not hold the GPU token, F7).

llama_context_restore_memory_host() makes the state readable without the
device. restore_device_buffers(to_host = true) wraps each shadow as a
CPU_Mapped buffer (ggml_backend_cpu_buffer_from_ptr) and points the tensors
into it: there is no copy and no allocation. The first version copied into a
CPU buffer instead, which needs the full KV allocation a second time, about
2.7 GB for the 35B at ctx 131072; the node has had an OOM-controller kill
before. The buffer base is rounded down to the 32-byte alignment that
cpu_buffer_from_ptr asserts (a large std::vector is only 16-byte aligned);
those leading bytes are never accessed. With to_host the shadows are kept,
because the buffers point into them.

It sets device_discarded, so restore_device() aborts afterwards. Exit only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 724a5d6373 llama: do not release CPU extra buffers (CPU_REPACK) on device release
A model with weights on the CPU device (-dev none / -ngl 0) segfaulted in
llama_model::release_device_weights() the first time the VRAM arbiter put
it to sleep: CPU_REPACK buffers are not host buffers, so load_tensors marked
them releasable, and capturing the shadow called their NULL get_tensor.
Reproduced with the production build 2fc9f9c35 (Qwen3.5-0.8B, -dev none
-ngl 0, LLAMA_SLEEP_VRAM_ONLY=1 LLAMA_SLEEP_EVICT_KV=1, force a yield with a
waiter on want.lock): exit 139 at the release, before any SIGTERM. The
faulting call (rip 0) returned into release_device_weights.

Only buffers on a non-CPU device are releasable now. These buffers are in RAM
already, so there is nothing to free. Production is not affected: every
arbiter model there has its weights on the GPU, and a releasable CPU_REPACK
buffer would have crashed it. Found by the --diskcache check, which runs
its arbiter cases CPU only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 80777eb831 docs: FORK.md and runbook after the 2fc9f9c35 deploy
Feedback from the 2026-10-25 deploy, which used only these docs:
- status: production is 2fc9f9c35; F10 deployed and verified on a 35B
  GPU load; --moe-exact and median --perf have run on the GPU; last
  full fork-validate run (37 PASS) and its timings; the first-candidate
  -arm VRAM spike seen twice is an open question.
- runbook: a no-rebase redeploy path; --arbiter beside production only
  if 1536 MiB VRAM is free, else in the swapdown window; full --moecache
  model path; page-cache eviction before and after phase 2; deploy
  steps reordered so the memory check follows stopping llama-swap;
  in-place rebuild and system_fingerprint check; mtime check through the
  versioned symlinks; rollback pointer numbering and one format; disk
  space; nothink alias and a PPM image helper for post-deploy requests;
  power1_cap via sysfs; restart and OOM checks; harmless UI warning.
- fork-validate.sh header: note the --arbiter VRAM requirement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 e5762e0ec2 docs: FORK.md F10 page-cache drop, -lm dio dead end; runbook note
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 554f1f33f4 llama: drop the model file's page cache after a copying load
With -lm none the GGUF is read() into our own buffers, but the kernel keeps
the file in page cache afterwards, charged to the pod's cgroup: about 20 GB
after a 35B load. On 2026-10-24 the Talos OOM controller killed the pod right
after a load under exactly that pressure. Those pages are pure waste once the
data is copied.

llama_file::drop_cache() wraps posix_fadvise(POSIX_FADV_DONTNEED) (Linux only,
errors logged at debug). load_all_data() drops each copied tensor's range right
after reading it, so the cache never grows past about one tensor, and drops the
whole file at the final cleanup to catch the metadata and partial pages at
tensor edges. Both only when the loader holds no mapping of the files
(mappings.empty(): not mmap, not mmap+mlock, no lazy tensors), so nothing that
is mapped is ever advised away. Split GGUFs are covered (every file in files).
clip.cpp always copies the mmproj via ifstream, so it drops the whole file
after loading it.

Measured on the pod, gemma-4-E2B UD-Q4_K_XL (2.5 GiB) + mmproj BF16 (0.9 GiB),
CPU only, -lm none, cold cache, same binary with the drops disabled as the
control (4 rounds, rotated order):
- file pages resident after load (mincore): 2431-2444 / 940 MiB -> 0 / 0
- peak model-file residency during load: 2.5 GiB -> 1.2-1.5 GiB (largest tensor)
- pod cgroup file after load: +3.37 GiB -> +0; host MemFree after load
  0.6 GB -> 2.5-2.8 GB (MemAvailable counts clean cache, so it cannot show this)
- load time: 5.87 s -> 6.82 s median (+0.95 s); a post-load-only drop is
  6.56 s (+0.68 s). The cost is synchronous page freeing that reclaim would
  otherwise do later; -lm dio (direct IO) costs the same (+0.9 s) and does not
  cover metadata or the mmproj.
- greedy output byte-identical; -lm mmap keeps the model file fully resident
  and mapped (2498 MiB, no refaults); -lzm on skips the drop (file mapped).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 d10d090798 docs: runbook - new fork-validate checks, gdb gone, scratch lost on restart
Phase 1 now covers the VRAM release/restore check and the handoff latency
A/B against --ref (which must be the production build); phase 2 adds
--moe-exact on the 35B and keeps its scratch dir. Explains median gating and
--perf-report, that a pod restart empties /tmp, that the manual checks are
now automated, and that gdb is not in the pod image (lost in the 2026-10-24
restart). States the commit trailer convention and what llama-swap does with
a child that exits vs one that wedges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 2edc33c430 docs(readme): short fork summary, FORK.md is the source of truth
The README fork section still said "the two code changes below are auto-on"
(the fork has three prefill changes plus the VRAM sharing and server fixes)
and recommended --n-cpu-moe 28 on the 35B, which FORK.md section 4 records as
worse at depth and unable to hold a long context. Replace it with a short
accurate summary that points to FORK.md, the runbook and the scripts. The
Laguna time profile, the iSWA KV note and Laguna's --parallel 1 saving were
only in the README and move to FORK.md sections 4 and 5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 c829418367 docs: FORK.md - review fixes, new checks, llama-swap restart behaviour
- status: no hardcoded fork-commit count (the list includes docs and
  scripts; count git log origin/master..HEAD); what is after the production
  commit; the commit-trailer convention of this private fork vs upstream's
  AGENTS.md rules
- commit map: the commits after the production commit
- F1: validated by the new --moe-exact check and the LLAMA_MOE_IDS_SKIP knob
- F3: every graph that builds concat(state, transpose(x), 0) (bailingmoe3,
  clef, kimi-linear, kimi-k3, glm5-next, mamba-base, plamo2, lfm-base,
  qwen4exp besides delta-net); only the delta-net path is measured
- F4/F6: the new v-vram-* and a-handoff-latency checks with first results
- F5: encode_mtmd_chunk was split out of process_mtmd_chunk, not renamed
- F6: the vram_cold ordering rule restated from the code (the measured
  failure was under the doorbell design; with the inline probe should_sleep
  cannot run inside the restore; keep it, and A/B handoff latency before any
  change) and why vram_acquire_for_load sets the flag early
- F7: what llama-swap does when a child dies, verified with a scratch
  llama-swap: "upstream exited unexpectedly", restart on the next request
- F9: new flags, phases, median gating, 2026-10-24 reference values, the
  /tmp and gdb loss on the pod restart, new grep targets and gaps
- section 7: stale comments fixed, what is not yet run on the GPU

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 384a7a1bec scripts: fork-validate.sh - --moe-exact, greedy identity for the readback skip
F1 (the routing-ids readback skip in sched_copy_experts) had no automated
check. --moe-exact MODEL reads expert_count / expert_used_count from the GGUF
header, computes the skip threshold ceil(4 * n_expert / n_expert_used) tokens
per ubatch (128 on the 35B), and runs one greedy request with a prompt of
about 2x that in a single ubatch, every expert on the host, so the prefill
takes the skip and each decode step takes the readback. The candidate's 64
tokens must equal the reference's byte for byte; with the LLAMA_MOE_IDS_SKIP
knob in the candidate, also the candidate with the skip forced off (=0), which
proves the skip exact even when the reference carries it, and =log must show
skips, all at >= 4 * n_expert ids. Without the knob skip-engaged SKIPs with
the batch-size argument. It needs the GPU to itself (same guard as --perf,
now shared as gpu_exclusive); FV_TEST_SHARED_GPU=1 bypasses the guard for
script tests against stub binaries only.

Also unset inherited LLAMA_MOE* variables.

Not run on the GPU yet (needs a swapdown window). Dry-run on the pod
against stub servers and the real 35B GGUF header: threshold 128, all-PASS
when outputs match, FAIL on a differing reference, SKIP without the knob,
FAIL for a dense model and while llama-swap runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 743bea264b scripts: fork-validate.sh - check that a cold server gives its VRAM back
Nothing checked that a release actually frees VRAM. --arbiter now takes a
baseline of mem_info_vram_used before any scratch server starts, then with A
warm and alone on the scratch arena makes it release through a fake waiter
on want.lock. v-vram-release requires the card-wide delta to drop by at
least 70% of what A added warm; v-vram-restore requires it back (and the
output equal to the reference) after the next request. Production's own
share is re-read at every point from the DRM fdinfo of every process the
script did not start (deduplicated by drm-client-id); if it moved by more
than 64 MiB the check SKIPs instead of blaming the candidate.

Run beside production on the pod (reb1024 vs itself): warm +546 MiB, cold
+4 MiB, restored +546 MiB, production steady at 5053 MiB; 16 PASS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 060146e129 scripts: fork-validate.sh - A/B ordinary handoff latency against --ref
--arbiter only reported the candidate's handoff latency, while the fork's
process rule (two production deploys lost) is to A/B the ordinary handoff,
not only the pathological cases. After the arbiter cases, a-handoff-latency
starts fresh A/B server pairs of each build on the scratch arena and runs
blocks ref, cand, cand, ref of 16 alternating handoffs each. It FAILs if the
candidate median exceeds the reference median by more than max(25%, 200 ms)
and reports both medians, maxima and per-block medians.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 2c38d72eba scripts: fork-validate.sh - gate --perf on medians, not means
--perf averaged the two per-arm llama-bench means of each build, so one
catastrophic sample (CPU contention from the ~60 other pods on the node) moved
the result, against the fork's own rule of medians, never means. llama-bench
now writes -o jsonl (every repetition sample) next to the markdown table, and
the gates use the median of all samples of a build pooled over its two arms
(4 per prefill cell, 6 for decode); the table shows min-max and n.

--perf-report TSV re-aggregates a saved <scratch>/perf.tsv without a GPU.

Checked offline on the pod: the 2026-09 A/B samples in results.txt give
pp2048 reference 250.24 (old mean) vs 247.76 (new median), candidate 280.15
vs 276.93; a synthetic 2026-10-24-like set with one 180 t/s candidate sample
read -9.3% (FAIL) the old way and -0.2% (PASS) with medians.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 da621da4f8 scripts: rx580-bench - decode at tg256 depth 4096, not tg32
The llama-bench sweep measured decode as tg32 at the default depth and the
README called it the continuity metric, while tg32 at n_ctx 32 is launch /
barrier bound on this card and is not a valid decode metric (it once read
-36% between builds at parity at tg256). lbsweep.sh now runs prefill
(-p $LB_P -n 0, depth 0) and decode (-p 0 -n 256 -d 4096, LB_N / LB_D) as
two invocations and labels depth cells (tg256@d4096). summarize.py marks
old depth-less tg rows as invalid-metric history. The README examples and
the reference values are updated to the 2026-10-24 production numbers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 67007a7be8 llama: env knob for the routing-ids readback skip
LLAMA_MOE_IDS_SKIP=0 forces the ids readback in sched_copy_experts (the
upstream path) and =log reports each skip on stderr, so validation can show
that the skip engages at prefill and that output is byte-identical with and
without it in one build. Read once (function-local statics); unset changes
nothing. Not set in production.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 fcfadfbbd2 server: fix stale VRAM-arbiter comments and the sleep log line
Comments only, plus one log string:
- the arbiter block comment said the KV cache is never evicted; with
  LLAMA_SLEEP_EVICT_KV it is (and the scheduler and scratch with it)
- vram_go_cold said vram_ensure_warm() rebuilds the encoder; only
  encode_mtmd_chunk() restores it, and only for an encode
- vram_ensure_warm and the mtmd init block named process_mtmd_chunk; the
  restore lives in encode_mtmd_chunk
- handle_sleeping_state said waking happens lazily before decode; it
  restores eagerly on the wake transition, before update_slots()
- the vram_cold ordering note now says why it still holds with the inline
  want-lock probe
- the sleep log line said "keeping KV cache" even when LLAMA_SLEEP_EVICT_KV
  evicts it; it now names what actually happens (the prefix
  "server entering sleeping state" that fork-validate.sh counts is kept)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 4ed4e58cf5 docs: FORK.md and rebase runbook for the RX 580 fork
FORK.md documents every fork feature (what and why with measured numbers,
where, how it works, invariants, past failures, rebase hazards, validation),
the dead ends measured on this hardware, the production configuration and
the measurement traps. docs/fork-rebase-runbook.md is the rebase, validation,
deploy, rollback and GPU-recovery procedure plus an operations reference.
The README fork section links both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 1e997ec43c scripts: add fork-validate.sh, post-rebase validation of the fork
Checks the op kernels, the concat fast path, the VRAM arbiter incl. every
shutdown path that has caused a production incident, and prefill/decode
against a reference build. Prints PASS/FAIL per check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 70f3d25045 ggml-vulkan: env knob for the concat transpose fast path
GGML_VK_CONCAT_FASTPATH=0 forces the generic path and =log reports each
fast-path dispatch, so validation can prove the path fires at prefill and
not at decode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 83cc51c9da server: release the GPU token promptly on shutdown
Two ways a shutting-down server kept the VRAM arbiter's token, stalling every
other model:

- SIGTERM with a request in flight: terminate() stopped only the task queue.
  The HTTP handler kept waiting for results the loop would never send, the main
  thread blocked joining it, and the token was released only in the destructor
  after that join - i.e. never, until the client went away. Terminate the
  result queue too (readers now get "server is shutting down": a 503, or an SSE
  error event), and let wait_until_no_sleep give up once stopping.

- SIGTERM while waiting for the token: the loop thread sat in a blocking flock
  inside the wake callback, holding mutex_tasks, so terminate() waited for the
  current holder to finish and then for a pointless restore of our weights.
  Poll the lock non-blocking and abort the wait on shutdown; the aborted wake
  stays cold, skips the restore, and the loop exits before update_slots.

Also release the token right after start_loop() returns rather than in the
destructor, and only after the device buffers are actually freed: the old
order unlocked before destroy(), so the next model could start uploading into
VRAM that was still occupied. Freeing at exit uses a new
llama_context_discard_device(), which drops weights, KV, MoE cache and scratch
without capturing host shadows first.

Off production: SIGTERM mid-generation or mid-wait now exits in ~0.2 s with
exit 0 and frees the token within 5-70 ms (previously never / after the
holder's whole generation). Handoff latency and greedy output are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 33f8b89f9b server: run the shutdown handler off the signal-interrupted thread
A SIGTERM that lands while the start_loop() thread is inside a sleeping
callback deadlocks the server. The callbacks run under
server_queue::mutex_tasks, and the kernel delivers the signal to that same
thread; the handler calls terminate(), which locks mutex_tasks again, and the
thread waits on itself forever.

With the VRAM arbiter that window is wide - a whole device release, or a
blocking wait for the GPU token - and llama-swap sends SIGTERM to evict
models, so it hit production: an evicted model wedged mid-release, never
released token.lock, and every other model queued behind it. SIGTERM could
not stop it either; only SIGKILL did.

Make the handler only write the signal number to a self-pipe, which is
async-signal-safe, and run shutdown_handler on a dedicated thread. terminate()
there simply waits until the loop thread drops mutex_tasks, after which the
loop exits normally. The thread is joined before the objects the handler uses
are destroyed. If pipe() fails the old direct call is kept.

Reproduced 6/6 off production (SIGTERM during release and during the token
wait); with the fix 8/8 clean exits, and handoffs, image requests and greedy
output are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 ea144c8a9e scripts: rx580-bench - use -lm none, document the MoE cache option
Upstream #28334 removed --mmap/--no-mmap/--mlock and llama-bench's -mmp, so the
harness flags no longer parse: the server would refuse to start and llama-bench
would silently produce no rows. Switch both harnesses to -lm none, the same
load mode, and note --moe-cache-mib as a server-only option worth measuring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5.5 1ee9747e18 llama: release/restore the MoE cache with the device weights
The MoE cache (--moe-cache-mib, #29887) keeps the most recently used host
experts in device banks. The on-demand VRAM path (release_device /
restore_device, used by the server VRAM arbiter and the on-demand mmproj path)
did not know about it, so its banks were never freed on handoff. Restoring them
by reallocation alone would also be wrong: the banks come back zeroed while the
slot maps still mark experts as cached, so the model would silently read zero
weights.

Free the banks on release and empty the cache instead of shadowing it: re-init
every LRU, re-point its slot-map pointers at the host slot maps and clear those
to -1. The banks are reallocated (which also re-inits their views) and cleared
on restore, before the scheduler is rebuilt, and the cache refills from the
host experts as before. Release runs in both modes, weights-only and evict_kv.

device_resident() now covers the cache too, for the early returns in
release/restore, the auto-restore in decode() and
llama_context_weights_resident().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5 9df58a9616 llama: skip the compute-buffer size check when the context is cold
release_device(evict_kv) drops the scheduler, so a context evicted by the VRAM
arbiter and then destroyed reaches the destructor with a null sched. The upstream
"compute buffer size matches expectation" loop calls
ggml_backend_sched_get_buffer_size() unconditionally, whose GGML_ASSERT(sched)
then aborts: the server exits 134 instead of 0 on every shutdown taken while
cold, which under llama-swap makes an ordinary stop look like a crashed child and
leaves a ggml backtrace in the log each time. It also skipped the rest of the
destructor, so the cold teardown path had never actually run to completion.

Guard at the call site rather than relaxing the assert - the assert is right, and
every other caller reserves the scheduler first. Same shape as the null guards in
synchronize(), memory_breakdown() and the released-buffer iteration.

The loop is a diagnostic size comparison, already reported by sched_reserve() at
load, so skipping it when cold loses nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5 aa1c2b4562 server: replace the VRAM doorbell with a want-lock
The doorbell was edge-triggered: a waiter wrote <arena>/doorbell/<pid> and the
holder's warden turned the inotify event into a queue flag. A ring carried no
notion of "still wanted", so it was lost whenever the holder was not already warm
and listening - during its model load (the watch does not exist yet, and the
kernel does not queue events for a watch that is not there) or inside
restore_device - and any ring that survived past the point its sender had been
satisfied caused a spurious release. Lost rings wedged the waiter permanently,
because flock(LOCK_EX) never times out and, with idle-sleep disabled, the holder
had no other reason to release. The wait blocks under mutex_tasks, so the whole
server stopped answering while /health still returned 200.

Express the request as kernel state instead. A waiter holds <arena>/want.lock
shared while it waits and drops it once it owns the token; the sleep decision
probes that lock non-blocking and releases the GPU while anyone is waiting. The
probe needs its own fd - flock treats two open file descriptions of one file
independently, so probing on the waiter's fd would convert our own lock rather
than conflict with it. Nothing can be missed, nothing goes stale, and a waiter
that dies is cleaned up by the kernel.

Probe from should_sleep() on the loop thread rather than from a warden thread.
Routing it through a flag is what made the first attempts fail: start_loop()
holds mutex_tasks from should_sleep() through the callbacks to the wait, so a
warden's request_yield() blocks on that mutex and is admitted only after the flag
has been consumed, latching a release for the next wake. Reading live state where
the decision is made has no edge to latch, and drops the warden's poll latency.

Two sleep-path bugs this exposed: wait_until_no_sleep() waited on !sleeping but
the loop clears req_stop_sleeping on the way in, so a loop that slept again
before the waiter ran stranded it forever - re-ask on every wake. And a task
queued after the waiter saw us awake could not wake us by itself, so sleep now
also breaks on a non-empty queue. Hold the GPU for 100 ms after a wake: the
request that woke us is not queued yet, and yielding at once only sends it round
again.

Measured on the RX 580 pod, two servers contending, 60 alternating handoffs:
0 stranded, median 0.309 s against the doorbell's 0.314 s.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5 e51074d080 scripts: add the RX 580 benchmark harness
Encodes the production config, fixed corpus slices, repeat/median discipline
and the noise floor, so the measurement method does not have to be
rediscovered each time. Runs the corpus prefill test through llama-server and
the llama-bench sweep as a controlled cross-check, with interleaved A/B.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:38:32 +01:00
LumpiastyandClaude Opus 5 c68da23dd8 ggml-vulkan: skip the concat transpose fast path below one tile
The tiled-transpose fast path in ggml_vk_concat is a large prefill win but a
loss at decode. On a hybrid model like Qwen3.6-35B (gated delta-net on 30 of
40 layers) build_conv_state emits a transposed concat per recurrent layer, so
the fast path runs 30 times per decoded token.

At one token the transposed source is a single column, so there is no
uncoalesced stride left to fix, but the path still costs two dispatches and an
unconditional ggml_vk_sync_buffers - a full pipeline barrier that serializes
the command stream. Measured -17% tg256 on RX 580; the barrier, not the
half-empty transpose dispatch, is nearly all of it.

Require the transposed source to be at least one TILE_DIM wide. That source is
qkv_mixed transposed, so ne[0] is the ubatch token count: prefill keeps the
fast path, decode and small speculative drafts take the generic one.

pp8192 +11% retained, tg256 back to parity, generation byte-identical.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:38:32 +01:00
Lumpiasty a546b6638c docs(readme): Polaris MoE profile - GCN mask_opt, the ids-readback win, tuning and dead ends 2026-11-22 20:38:32 +01:00
Lumpiasty 3f61190fde server: on-demand mmproj - free encoder VRAM on the text path
The vision encoder sits idle in VRAM on every text request. With
LLAMA_MMPROJ_ONDEMAND=1 the server keeps the encoder in RAM (released
after load) and brings it into VRAM just-in-time before mtmd_batch_encode,
releasing it again afterwards - so the encoder's VRAM is free for KV /
expert cache on the common text-only path.

When the encoder does not fit (that VRAM has been claimed), evict the LLM
backbone weights first: they are not needed while the encoder runs (the
decode that uses them happens afterwards), their host shadow is read-only
(cheap free, no D2H), and the KV / prompt cache is left untouched. Order
on the way back matters: free the encoder before re-uploading the weights
so the peak stays within VRAM.

Also wire the encoder release/restore into the VRAM arbiter: vram_go_cold
releases it; the just-in-time encode-path restore replaces the previous
restore in vram_ensure_warm.

Assisted-by: Claude
2026-11-22 20:38:32 +01:00
Lumpiasty f5c6f04b4e mtmd: release/restore the encoder weights from VRAM on demand
Add clip_release_device/clip_restore_device (and mtmd_release_device/
mtmd_restore_device wrappers over the vision + audio contexts) that free
the multimodal encoder's device weight buffer to a read-only host shadow
and rebuild it on demand, using the same shadow/free/reallocate pattern
as llama_model weights. No-op for a CPU-backed encoder. This lets the
server drop the ~hundreds-of-MiB vision encoder from VRAM when it is not
encoding an image.

Assisted-by: Claude
2026-11-22 20:38:32 +01:00
Lumpiasty 392787dee9 llama: free backend compute scratch on cold; guard memory_breakdown
On device release with KV eviction, also call ggml_backend_free_scratch()
on each backend so a cold model drops its Vulkan compute preallocations
(reallocated lazily on the next compute via restore).

Also guard llama_context::memory_breakdown() against a freed scheduler:
evict_kv release resets sched to null, so a cold model that is then torn
down (e.g. terminated by the process manager) hit GGML_ASSERT(sched) in
ggml_backend_sched_get_buffer_type. Skip the compute-buffer accounting
when sched is null.

Assisted-by: Claude
2026-11-22 20:38:32 +01:00
Lumpiasty 7992a094d0 ggml: add ggml_backend_free_scratch to drop Vulkan compute prealloc
Add an optional backend interface method free_scratch (with a public
ggml_backend_free_scratch wrapper) that frees transient/scratch device
memory a backend holds outside of any allocated buffer, keeping the
backend usable - the scratch is reallocated lazily on the next compute.

Implement it for the Vulkan backend (ggml_backend_vk_free_scratch): free
the prealloc_x/y/split_k/add_rms_partials and sync_staging device buffers
and reset their sizes, so an idle/cold model does not hold the vision or
matmul compute preallocations in VRAM. All other backends leave the hook
null (no-op).

Assisted-by: Claude
2026-11-22 20:38:32 +01:00
Lumpiasty b4a09407a0 llama: evict recurrent/SSM state on device release
The recurrent (SSM/conv) state of hybrid models (e.g. Qwen3.5) was left
resident when a model's device buffers were released for on-demand VRAM
sharing - llama_memory_recurrent::release_device_buffers() was a no-op
default. Implement it (and restore_device_buffers) with the same
capture-host-shadow / free / reallocate pattern as llama_kv_cache, so
llama_memory_hybrid now evicts both its attention KV and its recurrent
state. The state is read-write, so its shadow is recaptured on every
release.

Assisted-by: Claude
2026-11-22 20:38:32 +01:00
Lumpiasty 298da30d91 server: also free the compute-graph scheduler on KV eviction
Under LLAMA_SLEEP_EVICT_KV a cold model still held the scheduler's worst-case
compute buffer (hundreds of MiB, e.g. ~700 MiB for gemma-26B) plus the resident
experts. release_device(evict_kv) now also frees the sched (sched.reset() +
sched_need_reserve), and restore_device() rebuilds it via sched_reserve(), so a
fully-evicted model holds essentially no VRAM (gemma cold: 7147 -> 51 MiB).
Trades a heavier re-warm (weights+KV H2D + sched reserve) for the freed VRAM;
weights-only mode is unchanged (fast switch, keeps KV+compute).

Assisted-by: Claude
2026-11-22 20:38:31 +01:00
Lumpiasty e457c2fd63 server: restore weights/KV on wake, not at decode, so KV eviction is safe
With LLAMA_SLEEP_EVICT_KV the KV cache device buffers are freed when a model
goes cold. update_slots() touches the KV before the decode-time vram_ensure_warm
(e.g. SWA models create a checkpoint that reads the KV via ggml_backend_tensor_get),
so the KV must already be resident by then. Move the restore to the sleep-wake
handler (handle_sleeping_state(false)), which runs before update_slots, fixing a
GGML_ASSERT(buffer) crash on iSWA models (gemma) when KV eviction is enabled.

Assisted-by: Claude
2026-11-22 20:38:31 +01:00
Lumpiasty 9119ec6e89 llama: guard buffer iteration against released device buffers
release_device_weights()/release_device_buffers() leave a null buffer in
ctxs_bufs while the device memory is released for on-demand VRAM sharing.
The memory-breakdown / total_size / clear paths iterated these and called
ggml_backend_buffer_get_size()/get_type()/clear() on the null buffer, tripping
GGML_ASSERT(buffer) and aborting on shutdown of a cold model. Skip null buffers
in llama_model::memory_breakdown and llama_kv_cache::{memory_breakdown,
total_size,clear}.

Assisted-by: Claude
2026-11-22 20:38:31 +01:00
Lumpiasty b9c57d1c2e server: coordinate model load with the VRAM arbiter to avoid load-time OOM
Before uploading a model's weights, acquire the shared VRAM token (ring the
doorbell so any resident model releases first). Previously load uploaded weights
to VRAM before the arbiter was active, so loading a large model while another
(e.g. the warm 4B task model) held VRAM could exceed the budget and OOM.

- add vram_arena_open() (idempotent flock/doorbell setup) and
  vram_acquire_for_load(), called from load_model() before
  common_init_from_params().
- a coordinated load now stays warm holding the token and serves its first
  request without a re-warm (drop the init-time go_cold cold-start).

Assisted-by: Claude
2026-11-22 20:38:31 +01:00
Lumpiasty da03d4c1f2 server: keep the sleep-wake path active when the VRAM arbiter is enabled
The request handler only calls wait_until_no_sleep() (which wakes a server out
of its sleeping state) when sleep_idle_seconds >= 0. But the VRAM arbiter's
cross-process doorbell can put a server to sleep even when idle-sleep is
disabled, so without this a doorbell-slept server would never wake and requests
to it would hang until timeout.

Do not bypass the wake path when LLAMA_SLEEP_VRAM_ONLY is set, so the arbiter no
longer depends on --sleep-idle-seconds being configured.

Assisted-by: Claude
2026-11-22 20:38:31 +01:00
Lumpiasty 3b410b905c server: optionally evict the KV cache too (Phase 2 of VRAM sharing)
Extend on-demand device residency to the KV cache so that when a model's KV
plus another model would not fit in VRAM, the KV can also be evicted to a host
shadow (D2H on release, H2D on restore) instead of only the weights.

- llama_memory_i: add release_device_buffers()/restore_device_buffers()
  (default no-op). Implemented in llama_kv_cache (D2H shadow of the live
  ctxs_bufs, freed and reallocated like the weights); llama_memory_hybrid and
  llama_kv_cache_iswa delegate to their child caches.
- llama_context::release_device(evict_kv): also evict the memory's device
  buffers when requested; restore_device() rebuilds them. Public API
  llama_context_release_device gains an evict_kv flag.
- server: LLAMA_SLEEP_EVICT_KV=1 enables it. Off by default (weights-only),
  since the KV shadow adds a D2H/H2D copy of the live cache each cycle.

Validated on RX 580 (Vulkan), 4B @ 32k ctx: weights-only cold VRAM 1750 MB
(KV stays); weights+KV cold VRAM 726 MB (KV freed, ~1 GB reclaimed). KV
survives the round-trip: prompt cache reused after the cycle (prompt_n 4 vs
42), correct output.

Assisted-by: Claude
2026-11-22 20:38:31 +01:00
Lumpiasty cb489f3367 server: on-demand VRAM sharing to time-share one GPU between models
Add release/restore of a model's GPU weight buffers (keeping a host shadow
and the KV cache) so several always-loaded llama-server processes can
time-share a single GPU without reloading or losing the prompt cache.

- llama-model: release_device_weights()/restore_device_weights() capture a
  compact host shadow (stable iteration order, view-skipping) and free then
  realloc the device weight buffers; weights_resident() query.
- llama-context: release_device()/restore_device() wrappers; decode() auto-
  restores; public C API llama_context_release_device/restore_device.
- server: LLAMA_SLEEP_VRAM_ONLY makes idle-sleep release only the VRAM weights
  (not a full unload/reload). A cross-process flock token in LLAMA_VRAM_ARENA
  enforces "resident iff holds token"; an inotify doorbell forces the holder
  to release on contention. The warden thread only touches the task queue, so
  releases run on the loop thread and never race a decode.

Validated on RX 580 (Vulkan): two models share 8GB, never both resident,
correct output under contention, KV cache preserved (no re-prefill).

Assisted-by: Claude
2026-11-22 20:38:31 +01:00
Lumpiasty 5a0aca082a ggml-vulkan: tiled transpose fast-path for concat with transposed source
The generic concat shader reads a transposed source (nb[1]==type_size) with an
uncoalesced stride, which is catastrophically slow on discrete GPUs (~5.6ms for
a 16MB concat on the RX 580 vs ~130us of memory bandwidth), a ~9% prefill
hotspot on Qwen3.5-4B (delta-net state concat).

Add a fast path for concat along dim 0 where one source is stored transposed and
the other source + dst are contiguous along dim 0: copy the contiguous source
with copy.comp and transpose the other source into the matching dst sub-region
with the existing tiled copy_transpose shader (shared-memory 32x32 transpose,
coalesced read+write). Reuses pipeline_cpy_* / pipeline_cpy_transpose_* with
custom push constants + doffset, no new shader. Falls back to the generic path
otherwise (gated on type/shape/contiguity + 16-bit doffset bound).

Assisted-by: opencode
2026-11-22 20:38:31 +01:00
LumpiastyandClaude Opus 5 5c5af7b391 vulkan: record why flash-attn shmem staging stays off for GCN
Staging K/V through shared memory looks like an obvious win on GCN: without it
each rowgroup re-reads the whole K/V block from global memory, and with row_split
4 that pulls one 16KB block through a 16KB vector L1 four times.

Measured, it loses: -6.7% pp2048 at depth 16k and -7.4% at 32k on Polaris at head
size 128. The kvsh stride of D/4+1 dwords is 4 mod 32, which costs an 8-way LDS
bank conflict on wave64 - the +1 padding is tuned for warp32 - and the extra
shared memory eats occupancy this shader is already short of.

Comment only, no behaviour change. Leaving a note so the next person does not
spend a GPU on it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-11-22 20:38:31 +01:00
Lumpiasty 2b5e1962dd ggml-vulkan: enable flash-attn mask_opt for GCN large head sizes
mask_opt was disabled on AMD GCN for head sizes <= 256, but it is
beneficial there in high-context prefill: on fully-visible causal
blocks it skips the per-block mask load+add, and it skips fully-masked
blocks entirely. The attention op on GCN is compute-bound on the
softmax path (no matrix cores), so this cuts real work.

Verified lossless (perplexity bit-identical with it on vs off) and a
measured prefill win on Qwen3.5-35B-A3B (head_dim 256) on an RX 580:
pp2048 unchanged at short context, +8% @ 16k, +12% @ 32k, growing with
depth. Enable for GCN when HSK/HSV >= 256; the existing large-mask
conditions keep it off for decode.

Assisted-by: Claude
2026-11-22 20:38:31 +01:00
LumpiastyandClaude Opus 5.5 2ca30e5f35 llama: skip the routing-ids readback in sched_copy_experts when the batch uses every expert
Offloaded expert weights are uploaded per split, and to decide which experts to
upload the copy callback reads the routing ids back to the host. The ids are
produced on the same device we are about to upload to, so the readback forces a
full pipeline flush - 47 of them per eval on a 48-layer MoE.

Once the batch draws enough experts the readback stops telling us anything. At
2048 tokens x 10 experts over 256 experts every expert comes back used, so the
used set is all true and the copy is a single whole-tensor range anyway.

Skip the readback when ggml_nelements(ids) >= 4 * n_expert and mark every expert
used. Uploading an expert that no id selects cannot change the result, since
mul_mat_id only reads the rows the ids point at, so this stays exact. With the
MoE cache on, the all-used set still goes through moe_cache->copy_experts, which
copies the cached experts device-to-device from the host slot maps and uploads
the rest - still exact, and no readback is needed for it. Decode is unaffected:
n_ids there is the number of experts per token, far below the threshold. The
small-batch MoE cache slot path has its own readback and is not touched.

The st.ids/st.used caching is unchanged: a repeat call with the same ids tensor
reuses the all-true set, which is what the threshold would decide again.

Port of dbc449afd ("sched: skip the MoE routing-ids readback when the batch
uses every expert"), whose target block in ggml_backend_sched_compute_splits was
removed upstream (#29943) in favour of this copy callback.

Measured (original, before the port) on an RX 580 (Vulkan, Polaris) running
Laguna-S-2.1 118B IQ2_M with all experts host-resident: pp2048 +4.7% at depth
32k, +0.9% at 16k, tg32 +3%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 20:38:31 +01:00
Piotr Wilkin (ilintar) 5bacb7c868 llama : copy all experts of a host weight shared by several nodes (#32807)
* llama : copy all experts of a host weight shared by several nodes

The scheduler copies a host weight to the device once per graph, so a
partial expert copy made for the first MUL_MAT_ID is stale for the
others (e.g. the shared layers of inkling2_loop with --n-cpu-moe).

Assisted-by: Claude Opus 5.5

* llama : copy all experts of a host weight shared by several nodes

Assisted-by: Codex
2026-11-22 20:04:21 +01:00
Max S. d2bceda3de vocab: enable ignore_merges for inkling2 (#32799) 2026-11-22 19:52:33 +01:00
Adrien Gallouët 56eea9e4b0 server : add offset pagination to grep_search tool (#32792)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-22 19:00:05 +01:00
Pedro Cuenca e14a2d3a63 skill: Pay attention to tokenizer flags (#32819) 2026-11-22 18:59:17 +01:00
Aldehir Rojas 1014775c8a chat : implement token aware parsing (#29307) 2026-11-22 11:36:03 -06:00
Xuan-Son Nguyen 0b51bb2fed server: support /models/sse on single-model mode (#32398)
* server: support /models/sse on single-model mode

* nits

* nits
2026-11-22 17:52:56 +01:00
chessMan 8d60873dfb ggml/gguf : avoid creating invalid enum values from untrusted file data (#32641)
* ggml/gguf : avoid creating invalid enum values from untrusted file data

* removing the tests
2026-11-22 13:40:10 +02:00
gianni-corandJohannes Gäßler ad50351f50 ggml : add SIGMOID backward pass (#32126)
* ggml : add SIGMOID backward pass

Sigmoid is used in Qwen3.5 hybrid models and had no backward pass.
Bump RPC protocol patch version for the new op.

Signed-off-by: gianni-cor <gianfranco.cordella@tether.io>

* replace assert with ggml_assert

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* ggml : keep sigmoid backward checks in release builds

Assisted-by: Codex

* ggml : preserve existing operation IDs when adding SIGMOID_BACK

Assisted-by: Codex

---------

Signed-off-by: gianni-cor <gianfranco.cordella@tether.io>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
2026-11-22 13:33:13 +02:00
Michael 42f6a58fd1 chat : trigger lazy grammar on bare JSON tool calls (#32497)
Models like Llama 3.x emit tool calls as plain {"name": ...} JSON without a start marker, so with tool_choice=auto no grammar was produced at all. Use a pattern trigger on the opening of the call object instead.

Assisted-by: Claude Code
2026-11-22 12:19:04 +01:00
Alexey Kopytko 9c10c94e16 ggml : fix nonzero clear on WebGPU and non-UMA devices (#32727) 2026-11-22 20:10:42 +09:00
Eddie Zhou 8cb0d98ccd HIP: fix double halving of NVFP4 scales on CDNA3 (#32790)
Fixes #32789
2026-11-22 12:06:12 +01:00
Alexey Kopytko e300919cc7 webgpu : update GDN supports_op check (#32782) 2026-11-22 17:08:21 +09:00
Alexey Kopytko 5d28d64bf2 ggml-webgpu : Update to a recent version of Dawn (#32779)
Dawn v20260908.214631 doesn't compile with emsdk latest, that is 7.0.1.
2026-11-22 17:07:48 +09:00
shaofeiqi 1fb8a26960 opencl: add coopmat GEMM for Adreno 850 (Q4_0, Q5_0, Q8_0, Q4_K, Q5_K, Q6_K) (#32686)
* opencl: add Q4_0 coopmat GEMM for Adreno 850

Assisted-by: Claude Opus 5.5

* opencl: add Q5_0 coopmat GEMM for Adreno 850

* opencl: add Q8_0 coopmat GEMM for Adreno 850

* opencl: add Q4_K coopmat GEMM for Adreno 850

* opencl: add Q5_K coopmat GEMM for Adreno 850

* opencl: add Q6_K coopmat GEMM for Adreno 850
2026-11-22 00:01:14 -08:00
Revertron 8a97fa98ab vulkan : support non-contiguous src in SCALE (#32735)
Assisted-By: Opus 5.5
2026-11-22 07:28:46 +01:00
Aman Gupta 225a5345bb server : fit non-causal batches in one ubatch (#32713) 2026-11-22 11:25:13 +08:00
Alexey Kopytko b052049276 vulkan : fix memset for unaligned ranges for non-UMA devices (#32725)
See https://github.com/ggml-org/llama.cpp/pull/32574
2026-11-22 10:11:39 +09:00
Xuan-Son Nguyen cbb1c1c50b model: fix no_realloc for clef and vllm-decision2 (#32754) 2026-11-22 00:35:43 +01:00
Aldehir Rojas afe77e18fa chat : pass reasoning_effort = none if supported (#32763) 2026-11-21 15:27:25 -06:00
148 changed files with 11955 additions and 950 deletions

No files matched your search

+1 -1
View File
@@ -72,7 +72,7 @@ jobs:
- name: Fetch emdawnwebgpu
run: |
DAWN_TAG="v20260908.214631"
DAWN_TAG="v20261120.121236"
EMDAWN_PKG="emdawnwebgpu_pkg-${DAWN_TAG}.zip"
echo "Downloading ${EMDAWN_PKG}"
curl -L -o emdawn.zip \
+2
View File
@@ -1,5 +1,7 @@
# Instructions for llama.cpp
> **Private fork: read [FORK.md](FORK.md) and [docs/fork-rebase-runbook.md](docs/fork-rebase-runbook.md) before any work.**
> [!IMPORTANT]
>
> AI-generated code is allowed. What is **not** allowed is submitting code you do not understand. You are 100% responsible for every line, however it was produced.
+1894
View File
File diff suppressed because it is too large. Load diff
+26
View File
@@ -23,6 +23,32 @@ The main goal of `llama.cpp` is to enable LLM (and MLLM) inference with minimal
The `llama.cpp` project is built on top of the [ggml](https://github.com/ggml-org/ggml) library.
## This fork - RX 580 (Polaris / GCN) serving of large MoE models
A private fork maintained for one machine: an AMD RX 580 8 GB (RADV Vulkan) serving Qwen3.6-35B-A3B,
gemma-4-26B-A4B and smaller models under llama-swap, with the MoE experts in host RAM
(`--n-cpu-moe`). **[FORK.md](FORK.md) is the source of truth**: every fork change with its
invariants, rebase hazards and validation, the measured dead ends, the production configuration and
the measurement traps. The rebase and deploy procedure is
[docs/fork-rebase-runbook.md](docs/fork-rebase-runbook.md); `scripts/fork-validate.sh` validates a
build against a reference; `scripts/rx580-bench/` is the benchmark harness.
In short, the fork adds:
- three Polaris prefill wins, on without a flag and token-identical to mainline: the MoE routing-ids
readback skip in `sched_copy_experts`, flash-attention `mask_opt` for GCN at head size >= 256, and
a tiled-transpose fast path for the delta-net conv-state concat;
- opt-in VRAM time-sharing between several llama-server processes (`LLAMA_SLEEP_VRAM_ONLY`,
`LLAMA_SLEEP_EVICT_KV`, `LLAMA_MMPROJ_ONDEMAND`, `LLAMA_VRAM_ARENA`): a model releases everything it
holds in VRAM and gets it back without a restart, so its RAM prompt cache survives;
- fixes to the server's sleep, wake and shutdown paths and to cold teardown that the sharing exposed;
- validation-only env knobs (`GGML_VK_CONCAT_FASTPATH`, `LLAMA_MOE_IDS_SKIP`), unset in production.
Production settings for the 35B are in FORK.md section 5 (`-lm none -t 6 -ngl 99 --n-cpu-moe 40
-b 2048 -ub 2048 -fa 1 --parallel 1`, ctx 131072). Keep all experts on the host (`--n-cpu-moe 40`)
for long contexts: lower values are faster only at short context and run out of VRAM as the KV grows
(FORK.md section 4).
## Quick start
A few options to get `llama.cpp` installed on your machine:
+22 -5
View File
@@ -326,7 +326,7 @@ static bool parse_bool_value(const std::string & value) {
// common_models_handler
//
static std::string get_default_local_path(const std::string & url) {
std::string common_get_default_local_path(const std::string & url) {
auto f = string_split<std::string>(url, '#').front();
f = string_split<std::string>(f, '?').front();
return fs_get_cache_file(string_split<std::string>(f, '/').back()).string();
@@ -436,7 +436,7 @@ static std::vector<common_download_task> build_url_tasks(const common_params_mod
if (parts.size() == 1) {
common_download_task task;
task.url = parts[0];
task.local_path = model.path.empty() ? get_default_local_path(parts[0]) : model.path;
task.local_path = model.path.empty() ? common_get_default_local_path(parts[0]) : model.path;
task.opts = opts;
tasks.push_back(std::move(task));
return tasks;
@@ -454,7 +454,7 @@ static std::vector<common_download_task> build_url_tasks(const common_params_mod
task.url = part;
task.opts = opts;
std::string local = get_default_local_path(part);
std::string local = common_get_default_local_path(part);
if (!base_dir.empty()) {
auto pos = local.rfind('/');
std::string name = pos == std::string::npos ? local : local.substr(pos + 1);
@@ -487,7 +487,7 @@ void common_models_handler_apply(common_models_handler & handler, common_params
auto handle_url = [&](common_params_model & model) {
if (!model.url.empty()) {
if (model.path.empty()) {
model.path = get_default_local_path(model.url);
model.path = common_get_default_local_path(model.url);
}
}
};
@@ -498,7 +498,7 @@ void common_models_handler_apply(common_models_handler & handler, common_params
// optionally, if docker repo is set, resolve it
if (!params.model.docker_repo.empty()) {
params.model.url = common_docker_resolve_model(params.model.docker_repo);
params.model.path = get_default_local_path(params.model.url);
params.model.path = common_get_default_local_path(params.model.url);
}
// handle plain "url" tasks (non-hf)
@@ -1704,6 +1704,23 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.cache_ram_explicit = true;
}
).set_env("LLAMA_ARG_CACHE_RAM").set_examples({LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_CLI}));
add_opt(common_arg(
{"--cache-disk-path"}, "PATH",
"save the state of each slot with a prompt to this directory on shutdown, and restore it for a prompt that starts with it after a restart (default: disabled)",
[](common_params & params, const std::string & value) {
params.cache_disk_path = value;
}
).set_env("LLAMA_ARG_CACHE_DISK_PATH").set_examples({LLAMA_EXAMPLE_SERVER}));
add_opt(common_arg(
{"--cache-disk-size"}, "N",
string_format("size limit of the --cache-disk-path directory in MiB, the least recently used files are removed (default: %d, 0 - no limit)", params.cache_disk_mib),
[](common_params & params, int value) {
if (value < 0) {
throw std::invalid_argument("cache-disk-size must be non-negative");
}
params.cache_disk_mib = value;
}
).set_env("LLAMA_ARG_CACHE_DISK_SIZE").set_examples({LLAMA_EXAMPLE_SERVER}));
add_opt(common_arg(
{"-kvu", "--kv-unified"},
{"-no-kvu", "--no-kv-unified"},
+3
View File
@@ -149,6 +149,9 @@ common_models_handler common_models_handler_init(const common_params & params, l
// check if the model is a preset repo (i.e. has a preset file)
bool common_models_handler_is_preset_repo(const common_models_handler & handler);
// the cache path a model downloaded from a plain url is stored at
std::string common_get_default_local_path(const std::string & url);
// download and update params with the downloaded model path
void common_models_handler_apply(common_models_handler & handler, common_params & params, common_download_callback * callback = nullptr);
+24 -8
View File
@@ -69,9 +69,21 @@ common_chat_params peg_generator::generate_parser(const common_chat_template &
std::string trigger_marker = !autoparser.tools.format.section_start.empty() ? autoparser.tools.format.section_start :
autoparser.tools.format.per_call_start;
// bare JSON calls without a start marker (e.g. Llama 3.x): trigger the lazy grammar on the call's opening instead
std::string trigger_pattern;
{
const auto & fmt = autoparser.tools.format;
bool top_level_name = (fmt.function_field.empty() || fmt.function_field == "function") &&
fmt.name_field.find('.') == std::string::npos;
if (trigger_marker.empty() && fmt.mode == tool_format::JSON_NATIVE && !fmt.fun_name_is_key &&
!fmt.tools_array_wrapped && top_level_name) {
trigger_pattern = "\\{\\s*\"" + regex_escape(fmt.name_field) + "\"\\s*:";
}
}
bool has_response_format = !inputs.json_schema.empty() && inputs.json_schema.is_object();
bool include_grammar = has_response_format || (has_tools &&
((inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO && !trigger_marker.empty()) ||
((inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO && (!trigger_marker.empty() || !trigger_pattern.empty())) ||
inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
if (include_grammar) {
@@ -82,9 +94,11 @@ common_chat_params peg_generator::generate_parser(const common_chat_template &
// Set grammar triggers based on tool section markers (fall back to per-call markers)
if (data.grammar_lazy) {
data.grammar_triggers = {
{ COMMON_GRAMMAR_TRIGGER_TYPE_WORD, trigger_marker }
};
if (!trigger_marker.empty()) {
data.grammar_triggers = { { COMMON_GRAMMAR_TRIGGER_TYPE_WORD, trigger_marker } };
} else {
data.grammar_triggers = { { COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN, trigger_pattern } };
}
if (autoparser.tools.format.openai_wrapper_trigger) {
// model emits the OpenAI function wrapper, trigger on it
data.grammar_triggers.push_back({ COMMON_GRAMMAR_TRIGGER_TYPE_WORD, "{\"type\": \"function\"," });
@@ -359,7 +373,9 @@ common_peg_parser analyze_tools::build_tool_parser_tag_tagged(parser_build_conte
auto & p = ctx.p;
const auto & inputs = ctx.inputs;
auto until_suffix = p.rule("until-suffix", p.until(arguments.value_suffix));
// A raw string value needs a suffix to end on, without one it is parsed as JSON
bool raw_strings = !arguments.value_suffix.empty();
auto until_suffix = raw_strings ? p.rule("until-suffix", p.until(arguments.value_suffix)) : p.eps();
common_peg_parser tool_choice = p.choice();
@@ -375,9 +391,9 @@ common_peg_parser analyze_tools::build_tool_parser_tag_tagged(parser_build_conte
p.tool_arg(p.tool_arg_open(arguments.name_prefix + p.tool_arg_name(p.literal(param.name)) +
arguments.name_suffix) +
arguments.value_prefix +
(param.schema->may_be_string() ?
p.ac(p.tool_arg_string_value(until_suffix) +
p.tool_arg_close(p.literal(arguments.value_suffix)), arguments.value_suffix) :
(raw_strings && param.schema->may_be_string() ?
p.through(p.tool_arg_string_value(until_suffix),
p.tool_arg_close(p.literal(arguments.value_suffix))) :
(p.tool_arg_json_value(p.schema(
p.json(), "tool-" + std::to_string(tool_index) + "-arg-" + std::to_string(param_index) + "-schema", doc, *param.schema)) +
p.tool_arg_close(p.literal(arguments.value_suffix)))));
+1
View File
@@ -71,6 +71,7 @@ struct generation_params {
bool is_inference = true;
bool add_inference = false;
bool mark_input = true; // whether to mark input strings in the jinja context
common_peg_tokens specials;
bool has_continuation() const {
return continue_final_message != COMMON_CHAT_CONTINUATION_NONE && !continue_msg.empty();
+11
View File
@@ -71,6 +71,9 @@ constexpr size_t COMMON_CHAT_MAX_PERMUTE = 6;
class common_chat_peg_builder : public common_peg_parser_builder {
public:
common_chat_peg_builder() = default;
explicit common_chat_peg_builder(common_peg_tokens tokens) : common_peg_parser_builder(std::move(tokens)) {}
// Tag constants (from former common_chat_peg_base_builder)
static constexpr const char * REASONING_BLOCK = "reasoning-block";
static constexpr const char * REASONING = "reasoning";
@@ -200,6 +203,14 @@ inline common_peg_arena build_chat_peg_parser(
return builder.build();
}
inline common_peg_arena build_chat_peg_parser(
const common_peg_tokens & specials,
const std::function<common_peg_parser(common_chat_peg_builder & builder)> & fn) {
common_chat_peg_builder builder(specials);
builder.set_root(fn(builder));
return builder.build();
}
class tag_based_peg_mapper {
public:
std::map<std::string, std::string> tags;
+18 -2
View File
@@ -315,6 +315,7 @@ struct common_chat_templates {
bool add_bos;
bool add_eos;
bool has_explicit_template; // Model had builtin template or template overridden was specified.
common_peg_tokens specials;
std::unique_ptr<common_chat_template> template_default; // always set (defaults to chatml)
std::unique_ptr<common_chat_template> template_tool_use;
std::unique_ptr<common_chat_template> template_tts; // the prompt of a text to speak
@@ -802,6 +803,10 @@ void common_chat_templates_free(struct common_chat_templates * tmpls) {
delete tmpls;
}
void common_chat_templates_set_specials(struct common_chat_templates * tmpls, common_peg_tokens specials) {
tmpls->specials = std::move(specials);
}
bool common_chat_templates_was_explicit(const struct common_chat_templates * tmpls) {
return tmpls->has_explicit_template;
}
@@ -978,6 +983,7 @@ common_chat_templates_ptr common_chat_templates_init(const struct llama_model *
std::string token_eos = eos_token_override;
bool add_bos = false;
bool add_eos = false;
common_peg_tokens specials;
if (model) {
const auto * vocab = llama_model_get_vocab(model);
const auto get_token = [&](llama_token token, const char * name, const char * jinja_variable_name) {
@@ -997,11 +1003,13 @@ common_chat_templates_ptr common_chat_templates_init(const struct llama_model *
token_eos = get_token(llama_vocab_eos(vocab), "EOS", "eos_token");
add_bos = llama_vocab_get_add_bos(vocab);
add_eos = llama_vocab_get_add_eos(vocab);
specials = common_peg_tokens(vocab);
}
common_chat_templates_ptr tmpls(new common_chat_templates());
tmpls->has_explicit_template = has_explicit_template;
tmpls->add_bos = add_bos;
tmpls->add_eos = add_eos;
tmpls->specials = std::move(specials);
try {
tmpls->template_default = std::make_unique<common_chat_template>(default_template_src, token_bos, token_eos);
} catch (const std::exception & e) {
@@ -1543,6 +1551,7 @@ static common_chat_params common_chat_templates_apply_jinja(const struct common_
params.add_generation_prompt = inputs.add_generation_prompt;
params.add_bos = tmpls->add_bos;
params.add_eos = tmpls->add_eos;
params.specials = tmpls->specials;
params.continue_final_message = inputs.continue_final_message;
if (params.continue_final_message != COMMON_CHAT_CONTINUATION_NONE) {
@@ -1825,6 +1834,7 @@ common_chat_input common_chat_input_tokenize(const llama_vocab * vocab, const st
}
if (input.text != text) {
// the pieces do not give back the same text, keep the text without tokens
LOG_WRN("%s: token pieces do not match the text, parsing without token ids\n", __func__);
return common_chat_input(text);
}
return input;
@@ -1935,11 +1945,17 @@ common_chat_session::common_chat_session(const common_chat_templates * tm
thinking_ends = std::move(applied.thinking_end_tags);
parser_params.format = applied.format;
parser_params.generation_prompt = vocab ? common_chat_input_tokenize(vocab, applied.generation_prompt)
: common_chat_input(applied.generation_prompt);
parser_params.debug = params.debug;
parser_params.parser = std::move(applied.parser);
if (vocab) {
parser_params.generation_prompt = common_chat_input_tokenize(vocab, applied.generation_prompt);
} else if (params.tokenize) {
parser_params.generation_prompt = params.tokenize(applied.generation_prompt);
} else {
parser_params.generation_prompt = common_chat_input(applied.generation_prompt);
}
delimiters = std::move(applied.message_delimiters);
if (vocab) {
+6
View File
@@ -372,6 +372,9 @@ common_chat_templates_ptr common_chat_templates_init(const struct llama_model *
const std::string & bos_token_override = "",
const std::string & eos_token_override = "");
// override special tokens for testing
void common_chat_templates_set_specials(struct common_chat_templates * tmpls, common_peg_tokens specials);
bool common_chat_templates_was_explicit(const struct common_chat_templates * tmpls);
std::string common_chat_templates_source(const struct common_chat_templates * tmpls, const std::string & variant = "");
@@ -397,6 +400,9 @@ common_chat_msg common_chat_peg_parse(const common_peg_arena & src_parser, const
struct common_chat_session_params {
bool echo = false; // include the assistant prefill in the output when continuing a message
bool debug = false; // enable debug output for the PEG parser
// tokenize the generation prompt when there is no vocab for testing
std::function<common_chat_input(const std::string &)> tokenize;
};
class common_chat_session {
+2
View File
@@ -663,6 +663,8 @@ struct common_params {
int32_t checkpoint_min_step = 8192; // minimum spacing between context checkpoints
int32_t cache_ram_mib = 8192; // -1 = no limit, 0 - disable, 1 = 1 MiB, etc.
bool cache_ram_explicit = false; // set by --cache-ram, decision models disable the cache otherwise
std::string cache_disk_path; // directory of the disk prompt cache, empty = disabled
int32_t cache_disk_mib = 4096; // size limit of cache_disk_path in MiB, 0 = no limit
// llama-connect params
bool server_connect = false;
+20 -6
View File
@@ -98,6 +98,9 @@ class ProgressBar : public common_download_callback {
std::string filename;
size_t len = 0;
// the caller's callback, if any, still receives every event
common_download_callback * next;
static void cleanup(const ProgressBar * line) {
lines.erase(line);
if (lines.empty()) {
@@ -110,9 +113,12 @@ class ProgressBar : public common_download_callback {
}
public:
ProgressBar() = default;
ProgressBar(common_download_callback * next) : next(next) {}
void on_start(const common_download_progress & p) override {
if (next) {
next->on_start(p);
}
filename = url_get_filename(p.url);
for (size_t i = 0; i < filename.size(); ++i) {
@@ -126,12 +132,22 @@ public:
}
}
void on_done(const common_download_progress &, bool) override {
void on_done(const common_download_progress & p, bool ok) override {
if (next) {
next->on_done(p, ok);
}
std::lock_guard<std::mutex> lock(mutex);
cleanup(this);
}
bool is_cancelled() const override {
return next && next->is_cancelled();
}
void on_update(const common_download_progress & p) override {
if (next) {
next->on_update(p);
}
if (!p.total || !is_output_a_tty()) {
return;
}
@@ -519,11 +535,9 @@ int common_download_file_single(const std::string & url,
const std::string & lock_path,
const std::string & alt_path) {
if (!opts.offline) {
ProgressBar tty_cb;
ProgressBar tty_cb(opts.callback);
common_download_opts online_opts = opts;
if (!online_opts.callback) {
online_opts.callback = &tty_cb;
}
online_opts.callback = &tty_cb;
return common_download_file_single_online(url, path, online_opts, skip_etag, lock_path, alt_path);
}
+145 -17
View File
@@ -7,6 +7,7 @@
#include "../src/llama-ext.h"
#include <algorithm>
#include <array>
#include <cassert>
#include <stdexcept>
@@ -64,7 +65,8 @@ static std::vector<llama_device_memory_data> common_get_device_memory_data_impl(
uint32_t & hp_ngl_skip,
uint32_t & hp_n_ctx_train,
uint32_t & hp_n_expert,
ggml_log_level log_level) {
ggml_log_level log_level,
std::vector<uint32_t> * hp_layer_visits = nullptr) {
common_fit_log_scope log_scope(log_level);
llama_model_params mparams_copy = *mparams;
@@ -157,6 +159,9 @@ static std::vector<llama_device_memory_data> common_get_device_memory_data_impl(
hp_ngl_skip = mparams->load_mtp ? 0 : llama_model_n_layer_nextn(model);
hp_n_ctx_train = llama_model_n_ctx_train(model);
hp_n_expert = llama_model_n_expert(model);
if (hp_layer_visits != nullptr) {
*hp_layer_visits = llama_model_layer_visits(model);
}
common_memory_breakdown_print(ctx);
@@ -277,6 +282,7 @@ static void common_params_fit_impl(
uint32_t hp_nsk = 0; // number of skipped NextN layers, counted by the model loader but not by hp_ngl
uint32_t hp_nct = 0; // hparams.n_ctx_train
uint32_t hp_nex = 0; // hparams.n_expert
std::vector<uint32_t> hp_layer_visits;
// with non-unified kv, we need to take into account n_streams
// for example, if memory can hold more than model's trained context size, we must extend the n_ctx to hold enough n_streams
@@ -348,7 +354,7 @@ static void common_params_fit_impl(
// step 1: get data for default parameters and check whether any changes are necessary in the first place
LOG_TRC("%s: getting device memory data for initial parameters:\n", __func__);
dmds_t dmds_full = common_get_device_memory_data_impl(paths_model, mparams, cparams, devs, hp_ngl, hp_nsk, hp_nct, hp_nex, log_level);
dmds_t dmds_full = common_get_device_memory_data_impl(paths_model, mparams, cparams, devs, hp_ngl, hp_nsk, hp_nct, hp_nex, log_level, &hp_layer_visits);
// saturate instead of overflowing, this also preserves the UINT32_MAX sentinel of n_ctx_min:
const uint32_t n_ctx_max = (uint32_t) std::min<uint64_t>(uint64_t(hp_nct) * n_streams, UINT32_MAX);
@@ -631,6 +637,42 @@ static void common_params_fit_impl(
const size_t ntbo = llama_max_tensor_buft_overrides();
const bool has_looped_layers = std::any_of(
hp_layer_visits.begin(), hp_layer_visits.end(),
[](uint32_t v) { return v != 1; });
// For looped architectures (e.g. inkling2_loop), logical layers share physical weights across passes
// and extra KV slots have visits == 0. Rank physical layers by descending visit count so hot looped
// MoE layers stay in device memory before single-pass prelude/coda layers.
auto get_device_phys_layers = [&](const uint32_t dev_il0, const uint32_t dev_il1) -> std::vector<uint32_t> {
std::vector<uint32_t> phys;
const uint32_t limit = std::min(dev_il1, (uint32_t) hp_layer_visits.size());
for (uint32_t il = dev_il0; il < limit; ++il) {
if (hp_layer_visits[il] > 0) {
phys.push_back(il);
}
}
std::stable_sort(phys.begin(), phys.end(), [&](uint32_t a, uint32_t b) {
return hp_layer_visits[a] > hp_layer_visits[b];
});
return phys;
};
auto device_layer_slice = [&](const std::vector<ngl_t> & cfg, const size_t target_id) -> std::pair<uint32_t, uint32_t> {
uint32_t total_ngl = 0;
for (size_t id = 0; id < nd; ++id) {
total_ngl += cfg[id].n_layer;
}
uint32_t cur_il0 = hp_ngl + 1 - total_ngl;
for (size_t id = 0; id < target_id; ++id) {
cur_il0 += cfg[id].n_layer;
}
const uint32_t dev_n_tf = (target_id == nd - 1 && cfg[target_id].n_layer > 0)
? cfg[target_id].n_layer - 1
: cfg[target_id].n_layer;
return { cur_il0, cur_il0 + dev_n_tf };
};
// utility function to set n_gpu_layers and tensor_split
auto set_ngl_tensor_split_tbo = [&](
const std::vector<ngl_t> & ngl_per_device,
@@ -663,22 +705,45 @@ static void common_params_fit_impl(
mparams.tensor_split = tensor_split;
size_t itbo = 0;
for (size_t id = 0; id < nd; id++) {
il0 += ngl_per_device[id].n_full();
for (uint32_t il = il0; il < il0 + ngl_per_device[id].n_part; il++) {
if (itbo + 1 >= ntbo) {
tensor_buft_overrides[itbo].pattern = nullptr;
tensor_buft_overrides[itbo].buft = nullptr;
itbo++;
mparams.tensor_buft_overrides = tensor_buft_overrides;
throw common_params_fit_exception("llama_max_tensor_buft_overrides() == "
+ std::to_string(ntbo) + " is insufficient for model");
}
tensor_buft_overrides[itbo].pattern = get_overflow_pattern(il, il == il0 ? ngl_per_device[id].overflow_type : LAYER_FRACTION_MOE);
tensor_buft_overrides[itbo].buft = il == il0 ? overflow_bufts[id] : ggml_backend_cpu_buffer_type();
auto push_tbo = [&](const char * pattern, ggml_backend_buffer_type_t buft) {
if (itbo + 1 >= ntbo) {
tensor_buft_overrides[itbo].pattern = nullptr;
tensor_buft_overrides[itbo].buft = nullptr;
itbo++;
mparams.tensor_buft_overrides = tensor_buft_overrides;
throw common_params_fit_exception("llama_max_tensor_buft_overrides() == "
+ std::to_string(ntbo) + " is insufficient for model");
}
il0 += ngl_per_device[id].n_part;
tensor_buft_overrides[itbo].pattern = pattern;
tensor_buft_overrides[itbo].buft = buft;
itbo++;
};
for (size_t id = 0; id < nd; id++) {
if (!has_looped_layers) {
il0 += ngl_per_device[id].n_full();
for (uint32_t il = il0; il < il0 + ngl_per_device[id].n_part; il++) {
push_tbo(
get_overflow_pattern(il, il == il0 ? ngl_per_device[id].overflow_type : LAYER_FRACTION_MOE),
il == il0 ? overflow_bufts[id] : ggml_backend_cpu_buffer_type());
}
il0 += ngl_per_device[id].n_part;
continue;
}
const uint32_t dev_n_tf = (id == nd - 1 && ngl_per_device[id].n_layer > 0)
? ngl_per_device[id].n_layer - 1
: ngl_per_device[id].n_layer;
const std::vector<uint32_t> phys = get_device_phys_layers(il0, il0 + dev_n_tf);
const uint32_t n_part_tf = std::min(dev_n_tf, ngl_per_device[id].n_part);
const uint32_t n_full_tf = dev_n_tf - n_part_tf;
for (size_t i = n_full_tf; i < phys.size(); ++i) {
const bool is_first = i == n_full_tf;
push_tbo(
get_overflow_pattern(phys[i], is_first ? ngl_per_device[id].overflow_type : LAYER_FRACTION_MOE),
is_first ? overflow_bufts[id] : ggml_backend_cpu_buffer_type());
}
il0 += ngl_per_device[id].n_layer;
}
tensor_buft_overrides[itbo].pattern = nullptr;
tensor_buft_overrides[itbo].buft = nullptr;
@@ -715,6 +780,7 @@ static void common_params_fit_impl(
};
int64_t global_surplus_cpu_moe = 0;
bool keep_looped_moe_in_step3 = false;
if (hp_nex > 0) {
const static std::string pattern_moe_all = "blk\\.\\d+\\.ffn_(up|down|gate_up|gate)_(ch|)exps"; // matches all MoE tensors
ggml_backend_buffer_type_t cpu_buft = ggml_backend_cpu_buffer_type();
@@ -740,6 +806,43 @@ static void common_params_fit_impl(
__func__, -global_surplus_cpu_moe/MiB);
}
// For looped MoE models (e.g. inkling2_loop), CPU-offloading repeated core layers can inflate the
// compute buffer enough that keeping looped experts on GPU uses less total VRAM than offloading all MoEs.
if (has_looped_layers) {
size_t itbo = 0;
bool overrides_fit = true;
for (uint32_t il = 0; il < hp_layer_visits.size(); ++il) {
if (hp_layer_visits[il] == 1) {
if (itbo + 1 >= ntbo) {
overrides_fit = false;
break;
}
tensor_buft_overrides[itbo++] = {get_overflow_pattern(il, LAYER_FRACTION_MOE), cpu_buft};
}
}
if (overrides_fit && itbo > 0) {
tensor_buft_overrides[itbo] = {nullptr, nullptr};
mparams->tensor_buft_overrides = tensor_buft_overrides;
LOG_TRC("%s: getting device memory data with only non-looped MoE tensors moved to system memory:\n", __func__);
dmds_t dmds_looped_gpu = common_get_device_memory_data_impl(
paths_model, mparams, cparams, devs, hp_ngl, hp_nsk, hp_nct, hp_nex, log_level);
add_extra_memory(dmds_looped_gpu);
int64_t surplus_looped_gpu = 0;
for (size_t id = 0; id < nd; id++) {
surplus_looped_gpu += dmds_looped_gpu[id].free;
surplus_looped_gpu -= int64_t(dmds_looped_gpu[id].mb.total()) + margins[id];
}
if (surplus_looped_gpu > 0 && surplus_looped_gpu >= global_surplus_cpu_moe) {
LOG_TRC("%s: keeping looped MoE weights in device memory yields a larger surplus of %" PRId64 " MiB\n",
__func__, surplus_looped_gpu/MiB);
global_surplus_cpu_moe = surplus_looped_gpu;
keep_looped_moe_in_step3 = true;
}
}
}
// reset
tensor_buft_overrides[0] = {nullptr, nullptr};
mparams->tensor_buft_overrides = tensor_buft_overrides;
@@ -784,6 +887,17 @@ static void common_params_fit_impl(
ngl_per_device_high[id].n_layer = n_unassigned;
if (hp_nex > 0) {
ngl_per_device_high[id].n_part = size_t(id) < nd - 1 ? ngl_per_device_high[id].n_layer : ngl_per_device_high[id].n_layer - 1;
if (keep_looped_moe_in_step3) {
const auto [dev_il0, dev_il1] = device_layer_slice(ngl_per_device_high, id);
uint32_t n_hot = 0;
const uint32_t limit = std::min(dev_il1, (uint32_t) hp_layer_visits.size());
for (uint32_t il = dev_il0; il < limit; ++il) {
if (hp_layer_visits[il] > 1) {
n_hot++;
}
}
ngl_per_device_high[id].n_part -= std::min(ngl_per_device_high[id].n_part, n_hot);
}
}
if (ngl_per_device_high[id].n_layer > 0) {
std::vector<int64_t> mem_high = get_memory_for_layers(__func__, ngl_per_device_high, overflow_bufts);
@@ -858,6 +972,12 @@ static void common_params_fit_impl(
ngl_per_device_high[jd].n_part = 0;
}
size_t id_dense_start_high = nd - 1;
if (has_looped_layers && id == id_dense_start && ngl_per_device_high[id].n_layer == ngl_per_device[id].n_layer) {
const auto [dev_il0, dev_il1] = device_layer_slice(ngl_per_device_high, id);
const uint32_t n_phys_dev = (uint32_t) get_device_phys_layers(dev_il0, dev_il1).size();
const uint32_t dev_n_tf = dev_il1 - dev_il0;
ngl_per_device_high[id].n_part = dev_n_tf - std::min(dev_n_tf, n_phys_dev);
}
std::vector<int64_t> mem_high = get_memory_for_layers(__func__, ngl_per_device_high, overflow_bufts);
if (mem_high[id] > targets[id]) {
@@ -909,7 +1029,15 @@ static void common_params_fit_impl(
}
// try to fit at least part of one more layer
if (ngl_per_device[id_dense_start].n_layer > (id < nd - 1 ? 0 : 1)) {
bool can_fit_partial = ngl_per_device[id_dense_start].n_layer > (id < nd - 1 ? 0 : 1);
if (can_fit_partial && has_looped_layers && id == id_dense_start) {
const auto [dev_il0, dev_il1] = device_layer_slice(ngl_per_device, id);
const uint32_t n_phys_dev = (uint32_t) get_device_phys_layers(dev_il0, dev_il1).size();
const uint32_t dev_n_tf = dev_il1 - dev_il0;
const uint32_t n_full_tf = dev_n_tf - std::min(dev_n_tf, ngl_per_device[id].n_part);
can_fit_partial = n_full_tf < n_phys_dev;
}
if (can_fit_partial) {
std::vector<ngl_t> ngl_per_device_test = ngl_per_device;
size_t id_dense_start_test = id_dense_start;
ngl_per_device_test[id_dense_start_test].n_layer--;
+5 -2
View File
@@ -593,7 +593,7 @@ caps caps_get(jinja::program & prog) {
if (result.supports_reasoning_effort) {
JJ_DEBUG("%s\n", ">>> Running capability check: reasoning effort levels");
static const std::vector<std::string> candidates = { "minimal", "low", "medium", "high", "xhigh", "max" };
static const std::vector<std::string> candidates = { "none", "minimal", "low", "medium", "high", "xhigh", "max" };
// renders a single user turn with the given effort, nothing when the template raises
const auto render = [&](const std::string & effort) {
@@ -631,7 +631,10 @@ caps caps_get(jinja::program & prog) {
for (const auto & level : candidates) {
const auto rendered = render(level);
if (!rendered) {
rejected.push_back(level);
// "none" also turns thinking off, so the server does not answer it with an error
if (level != "none") {
rejected.push_back(level);
}
continue;
}
if (rendered == unknown) {
+2 -2
View File
@@ -676,8 +676,8 @@ private:
const auto & node = trie.nodes[idx];
std::string rejects;
auto first = true;
for (const auto & [cpt, child] : node.children) {
std::string c = common_unicode_cpt_to_utf8(cpt);
for (const auto & [sym, child] : node.children) {
std::string c = common_unicode_cpt_to_utf8(sym.value);
rejects += c;
if (first) {
first = false;
+29 -26
View File
@@ -24,7 +24,9 @@ struct depth_guard {
// Gemma 4 value parsers
void gemma4_value_rules(common_chat_peg_builder & p) {
// The tags are what gemma4_to_json() in the mapper reads, the rules may wrap them freely.
p.rule("gemma4-string", p.tag(gemma4::STRING, p.literal("<|\"|>") + p.tag(gemma4::STRING_CONTENT, p.until("<|\"|>")) + p.literal("<|\"|>")));
auto string_delim = p.token("<|\"|>");
p.rule("gemma4-string", p.tag(gemma4::STRING, string_delim + p.tag(gemma4::STRING_CONTENT, p.until(string_delim)) + string_delim));
p.rule("gemma4-bool", p.tag(gemma4::BOOL, p.json_bool()));
@@ -34,7 +36,7 @@ void gemma4_value_rules(common_chat_peg_builder & p) {
// Keys are bare in the template, but Gemma 4 occasionally wraps them in quotes
p.rule("gemma4-dict-key", p.tag(gemma4::DICT_KEY, p.choice({
p.literal("<|\"|>") + p.tag(gemma4::DICT_KEY_NAME, p.until("<|\"|>")) + p.literal("<|\"|>"),
string_delim + p.tag(gemma4::DICT_KEY_NAME, p.until(string_delim)) + string_delim,
p.literal("\"") + p.tag(gemma4::DICT_KEY_NAME, p.chars("[^\"]", 1, -1)) + p.literal("\""),
p.literal("'") + p.tag(gemma4::DICT_KEY_NAME, p.chars("[^']", 1, -1)) + p.literal("'"),
p.tag(gemma4::DICT_KEY_NAME, p.chars("[^:}]", 1, -1)),
@@ -144,7 +146,7 @@ class gemma4_schema_parser {
alts |= name();
}
// Gemma 4 may emit keys in its string delimiters or quotes
alts |= p_.literal("<|\"|>") + name() + p_.literal("<|\"|>");
alts |= p_.token("<|\"|>") + name() + p_.token("<|\"|>");
if (key.find('\'') == std::string::npos) {
alts |= p_.literal("'") + name() + p_.literal("'");
}
@@ -331,18 +333,23 @@ common_chat_params common_chat_params_init_gemma4(const common_chat_template &
auto include_grammar = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
auto extract_reasoning = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
data.parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
auto start = p.rule("start", p.optional(p.literal("<|turn>model\n")));
data.parser = build_chat_peg_parser(inputs.specials, [&](common_chat_peg_builder & p) {
auto start = p.rule("start", p.optional(p.token("<|turn>") + p.literal("model\n")));
auto channel_start = p.token("<|channel>");
auto channel_end = p.token("<channel|>");
auto think_start = channel_start + p.literal("thought");
auto tool_call_start = p.token("<|tool_call>");
// the model may start a tool call before closing the thought channel
auto think_end = channel_end | p.peek(tool_call_start);
if (extract_reasoning) {
// the model may start a tool call before closing the thought channel
p.rule("thought", p.literal("<|channel>thought") + p.space() + p.reasoning(p.until_one_of({"<channel|>", "<|tool_call>"})) + (p.literal("<channel|>") | p.peek(p.literal("<|tool_call>"))));
p.rule("thought", think_start + p.space() + p.reasoning(p.until(channel_end | tool_call_start)) + think_end);
} else {
p.rule("thought", p.content(p.literal("<|channel>thought") + p.space() + p.until_one_of({"<channel|>", "<|tool_call>"}) + (p.literal("<channel|>") | p.peek(p.literal("<|tool_call>")))));
p.rule("thought", p.content(think_start + p.space() + p.until(channel_end | tool_call_start) + think_end));
}
auto consume_empty_channels = p.gbnf(p.zero_or_more(p.literal("<|channel>") + p.negate(p.literal("thought"))), "");
auto thought = (p.peek(p.literal("<|channel>")) + consume_empty_channels + p.ref("thought")) | p.negate(p.literal("<|channel>"));
auto consume_empty_channels = p.gbnf(p.zero_or_more(channel_start + p.negate(p.literal("thought"))), "");
auto thought = (p.peek(channel_start) + consume_empty_channels + p.ref("thought")) | p.negate(channel_start);
if (has_response_format) {
auto response_format = p.literal("```json") <<
@@ -378,39 +385,35 @@ common_chat_params common_chat_params_init_gemma4(const common_chat_template &
})));
});
auto tool_call = p.trigger_rule("tool-call", p.repeat(
"<|tool_call>call:" + tool_choice + "<tool_call|>",
/* min = */ inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED ? 1 : 0,
/* max = */ inputs.parallel_tool_calls ? -1 : 1
));
auto tool_call_rest = p.literal("call:") + tool_choice + p.token("<tool_call|>");
auto tool_call = p.rule("tool-call", tool_call_start + tool_call_rest);
auto more = inputs.parallel_tool_calls ? p.zero_or_more(tool_call) : p.eps();
auto tool_calls = p.trigger_rule("tool-calls", tool_call_start, tool_call_rest + more);
if (inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED) {
return start + thought + tool_call;
return start + thought + tool_calls;
}
auto scan_to_toolcall = p.rule("scan-to-toolcall", p.until("<|tool_call>"));
auto content = p.rule("content", p.content(p.until_one_of({"<|channel>", "<channel|>", "<|tool_call>"})));
auto scan_to_toolcall = p.rule("scan-to-toolcall", p.until(tool_call_start));
auto content = p.rule("content", p.content(p.until(channel_start | channel_end | tool_call_start)));
auto message = p.rule("message", thought + content);
return start + p.zero_or_more(message) + scan_to_toolcall + tool_call;
return start + p.zero_or_more(message) + scan_to_toolcall + p.optional(tool_calls);
}
// Gemma 4 may emit an extra <|channel>thought\n<channel|> at the end of the content. It may
// also emit a single trailing <channel|> token. Consume all complete reasoning blocks and
// then stop at the first unmatched <channel|> token.
auto content = p.rule("content", p.content(p.until_one_of({"<|channel>", "<channel|>"})));
auto content = p.rule("content", p.content(p.until(channel_start | channel_end)));
auto message = p.rule("message", thought + content);
return start + p.one_or_more(message);
});
if (include_grammar) {
data.grammar_lazy = !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
data.grammar_lazy = false;
data.grammar = build_grammar([&](const common_grammar_builder & builder) {
data.parser.build_grammar(builder, data.grammar_lazy);
data.parser.build_grammar(builder, !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED)));
});
data.grammar_triggers = {
{ COMMON_GRAMMAR_TRIGGER_TYPE_WORD, "<|tool_call>" },
};
}
return data;
+31 -30
View File
@@ -65,17 +65,18 @@ common_chat_params common_chat_params_init_gpt_oss(const common_chat_template &
auto include_grammar = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
auto extract_reasoning = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
data.parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
auto start = p.rule("start", p.literal("<|start|>assistant"));
auto end = p.rule("end", p.literal("<|end|>"));
auto content = p.rule("message-content", p.until("<|end|>"));
auto channel = p.literal("<|channel|>") + (p.literal("commentary") | p.literal("analysis"));
data.parser = build_chat_peg_parser(inputs.specials, [&](common_chat_peg_builder & p) {
auto start_assistant = p.token("<|start|>") + p.literal("assistant");
auto start = p.rule("start", start_assistant);
auto end = p.rule("end", p.token("<|end|>"));
auto content = p.rule("message-content", p.until(p.token("<|end|>")));
auto channel = p.token("<|channel|>") + (p.literal("commentary") | p.literal("analysis"));
auto constrain_type = p.chars("[A-Za-z0-9_-]", 1, -1);
auto constraint = p.optional(p.space() + p.optional(p.literal("<|constrain|>")) + constrain_type);
auto constraint = p.optional(p.chars("[ ]", 1, -1) + p.optional(p.token("<|constrain|>")) + constrain_type);
// Occasionally, gpt-oss-20b will prefix channels with this commentary
auto stray_commentary = p.optional(p.literal("<|channel|>commentary") + p.optional(p.literal(" to=assistant")));
auto start_analysis = stray_commentary + p.literal("<|channel|>analysis<|message|>");
auto stray_commentary = p.optional(p.token("<|channel|>") + p.literal("commentary") + p.optional(p.literal(" to=assistant")));
auto start_analysis = stray_commentary + p.token("<|channel|>") + p.literal("analysis") + p.token("<|message|>");
if (extract_reasoning) {
p.rule("analysis", start_analysis + p.reasoning(content) + end);
@@ -84,8 +85,8 @@ common_chat_params common_chat_params_init_gpt_oss(const common_chat_template &
}
auto analysis = p.ref("analysis");
auto preamble = p.rule("preamble", p.literal("<|channel|>commentary<|message|>") + p.content(content) + end);
auto final_msg = p.rule("final", stray_commentary + p.literal("<|channel|>final") + constraint + p.literal("<|message|>") + p.content(content));
auto preamble = p.rule("preamble", p.token("<|channel|>") + p.literal("commentary") + p.token("<|message|>") + p.content(content) + end);
auto final_msg = p.rule("final", stray_commentary + p.token("<|channel|>") + p.literal("final") + constraint + p.token("<|message|>") + p.content(content));
// Consume any unsolicited tool calls, e.g. builtin functions
auto unsolicited = p.rule("unsolicited", p.atomic(p.optional(channel) + p.literal(" to=") + content + end));
@@ -94,58 +95,58 @@ common_chat_params common_chat_params_init_gpt_oss(const common_chat_template &
if (has_response_format) {
auto response_format = p.rule("response-format",
p.literal("<|channel|>final") + constraint + p.literal("<|message|>") +
p.token("<|channel|>") + p.literal("final") + constraint + p.token("<|message|>") +
p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema)));
return p.zero_or_more(start + analysis) + start + response_format;
}
if (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE) {
auto tool_choice = p.choice();
auto in_role = p.choice();
auto in_channel = p.choice();
foreach_function(inputs.tools, [&](size_t tool_index, const json & tool) {
const auto & function = tool.at("function");
std::string name = function.at("name");
const auto params = common_chat_tool_parameters(function);
auto func_name = p.literal(" to=functions.") + p.tool_name(p.literal(name));
auto func_name = p.tool_name(p.literal(name));
auto args = p.tool_args(p.schema(p.json(), "tool-" + std::to_string(tool_index) + "-schema", params));
// recipient in role header
// <|start|>assistant to=functions.NAME<|channel|>(commentary|analysis)[constraint]<|message|>ARGS
auto tool_in_role = p.tool(p.tool_open(func_name + channel + constraint + p.literal("<|message|>")) + args);
in_role |= p.rule("tool-" + std::to_string(tool_index) + "-in-role",
p.atomic(func_name + channel + constraint + p.token("<|message|>")) + args);
// recipient in channel header
// <|channel|>(commentary|analysis) to=functions.NAME[constraint]<|message|>ARGS
auto tool_in_channel = p.tool(p.tool_open(channel + func_name + constraint + p.literal("<|message|>")) + args);
tool_choice |= p.rule("tool-" + std::to_string(tool_index), tool_in_role | tool_in_channel);
// <|start|>assistant<|channel|>(commentary|analysis) to=functions.NAME[constraint]<|message|>ARGS
in_channel |= p.rule("tool-" + std::to_string(tool_index) + "-in-channel",
p.atomic(func_name + constraint + p.token("<|message|>")) + args);
});
auto tool_call = p.trigger_rule("tool-call", tool_choice);
// The recipient prefix is the trigger, so the grammar constrains the name that follows it
auto to_functions = p.tool_open(p.literal(" to=functions."));
auto tool_call = p.choice({
p.trigger_rule("tool-call-in-role", start_assistant + to_functions, in_role),
p.trigger_rule("tool-call-in-channel", start_assistant + channel + to_functions, in_channel)
});
if (inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED) {
return p.zero_or_more(start + any) + start + tool_call;
return p.zero_or_more(start + any) + tool_call;
}
return p.zero_or_more(start + any) + start + (tool_call | final_msg);
return p.zero_or_more(start + any) + (tool_call | (start + final_msg));
}
return p.zero_or_more(start + any) + start + (final_msg | unsolicited);
});
if (include_grammar) {
data.grammar_lazy = !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
// The grammar scans for the triggers itself, so the sampler runs it from the first token
data.grammar_lazy = false;
data.grammar = build_grammar([&](const common_grammar_builder & builder) {
data.parser.build_grammar(builder, data.grammar_lazy);
data.parser.build_grammar(builder, !has_response_format && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO);
});
data.grammar_triggers = {
{ COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN, "^\\s+to$" },
{ COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN, "^<\\|channel\\|>(?:commentary|analysis)\\s+to=functions$" },
{ COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN, "<\\|start\\|>assistant(\\s+to)" },
{ COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN, "<\\|start\\|>assistant(<\\|channel\\|>(?:commentary|analysis)\\s+to)" }
};
}
return data;
+5 -2
View File
@@ -99,8 +99,11 @@ common_chat_params common_chat_params_init_harmony_xml(const common_chat_templat
auto arg_close = p.tool_arg_close(p.gbnf(
p.literal(PARAM_CLOSE) | (p.literal("\n") + p.peek(p.optional(p.literal("\n")) + next_open)),
gbnf_format_literal("\n") + " " + gbnf_format_literal(PARAM_CLOSE.substr(1)) + "?"));
auto arg_string = p.rule("rfl-arg-string",
p.ac(p.tool_arg_string_value(p.until_one_of(string_ends)) + arg_close, PARAM_CLOSE));
auto string_value = p.tool_arg_string_value(p.until_one_of(string_ends));
auto arg_string = p.rule("rfl-arg-string", p.through(string_value, p.tool_arg_close(p.literal(PARAM_CLOSE))));
// the parser also accepts a value with no closing tag, the grammar still requires the closing tag
auto missing_close = string_value + p.tool_arg_close(p.literal("\n") + p.peek(p.optional(p.literal("\n")) + next_open));
arg_string = p.gbnf(arg_string | missing_close, "rfl-arg-string");
auto arg_sep = p.optional(p.literal("\n"));
std::string call_close_gbnf = gbnf_format_literal("<|end|>");
+2 -2
View File
@@ -91,7 +91,7 @@ common_chat_params common_chat_params_init_k2_horizon(const common_chat_template
}
auto think_body = p.until_one_of(data.thinking_end_tags);
auto think_block = [&](const common_peg_parser & body) {
return p.optional(THINK_START + p.space() + p.ac(body + think_end, data.thinking_end_tags));
return p.optional(THINK_START + p.space() + p.through(body, think_end));
};
auto reasoning = extract_reasoning ? think_block(p.reasoning(think_body)) : p.eps();
@@ -110,7 +110,7 @@ common_chat_params common_chat_params_init_k2_horizon(const common_chat_template
tool_choice = p.standard_json_tools(CALL_START, CALL_END, inputs.tools, false, true);
} else {
auto arg_close = p.tool_arg_close(p.literal(ARG_VAL_END));
auto arg_string = p.rule("xml-arg-string", p.ac(p.tool_arg_string_value(p.until(ARG_VAL_END)) + arg_close, ARG_VAL_END));
auto arg_string = p.rule("xml-arg-string", p.through(p.tool_arg_string_value(p.until(ARG_VAL_END)), arg_close));
// The models leave out <ifm|arg_type> even when asked for xml_typed
auto arg_type = call_format == "xml_typed" ? p.optional(ARG_TYPE + p.until(ARG_TYPE_END) + ARG_TYPE_END + p.space()) : p.eps();
+2 -2
View File
@@ -63,8 +63,8 @@ common_chat_params common_chat_params_init_minicpm5(const common_chat_template &
// CDATA lets a value carry characters that would otherwise close the tag (e.g.
// </param>); capture the inner text only, excluding the CDATA markers.
auto string_value = p.choice({
p.literal("<![CDATA[") + p.ac(p.tool_arg_string_value(p.until("]]>")) + p.literal("]]>"), "]]>") + p.tool_arg_close(p.literal("</param>")),
p.negate(p.literal("<![CDATA[")) + p.ac(p.tool_arg_string_value(p.until("</param>")) + p.tool_arg_close(p.literal("</param>")), "</param>")
p.literal("<![CDATA[") + p.through(p.tool_arg_string_value(p.until("]]>")), "]]>") + p.tool_arg_close(p.literal("</param>")),
p.negate(p.literal("<![CDATA[")) + p.through(p.tool_arg_string_value(p.until("</param>")), p.tool_arg_close(p.literal("</param>")))
});
auto tool_choice = p.choice();
+3 -3
View File
@@ -64,8 +64,8 @@ common_chat_params common_chat_params_init_minimax_m3(const common_chat_template
if (extract_reasoning) {
auto block = inputs.enable_thinking
? p.literal(THINK_START) + p.space() +
p.ac(p.reasoning(p.until(THINK_END)) + p.literal(THINK_END), THINK_END)
: p.literal(THINK_START) + p.ac(p.until(THINK_END) + p.literal(THINK_END), THINK_END);
p.through(p.reasoning(p.until(THINK_END)), THINK_END)
: p.literal(THINK_START) + p.through(p.until(THINK_END), THINK_END);
// A turn without reasoning is prefixed with a bare </mm:think>, written either by the
// generation prompt (thinking_mode = "disabled") or by the model itself.
@@ -113,7 +113,7 @@ common_chat_params common_chat_params_init_minimax_m3(const common_chat_template
// A string accepts anything, so a union with a string alternative is a string
if (schema.may_be_string()) {
return p.ac(p.tool_arg_string_value(p.until(close)) + close_tag, close);
return p.through(p.tool_arg_string_value(p.until(close)), close_tag);
}
if (schema.kind() == common_chat_schema::KIND_ANY_OF) {
+25 -29
View File
@@ -17,7 +17,7 @@ common_chat_params common_chat_params_init_muse_glimmer(const common_chat_templa
data.supports_thinking = true;
data.preserved_tokens = {
"<|start|>", "<|message|>", "<|eom|>", "<|eot|>",
"<|start|>", "<|message|>", "<|eom|>",
// ATEM tool-call markup emitted on " to=<tool>" turns.
"<atem:function_calls>", "<atem:invoke", "<atem:parameter", "</atem:parameter>",
"</atem:invoke>", "</atem:function_calls>",
@@ -48,37 +48,38 @@ common_chat_params common_chat_params_init_muse_glimmer(const common_chat_templa
// Constrained grammar whenever tools are offered or a response format is requested.
auto include_grammar = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
data.parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
auto start = p.rule("start", p.literal("<|start|>assistant"));
data.parser = build_chat_peg_parser(inputs.specials, [&](common_chat_peg_builder & p) {
auto start_token = p.token("<|start|>");
auto message = p.token("<|message|>");
auto eom = p.token("<|eom|>");
auto start = p.rule("start", start_token + p.literal("assistant"));
if (!extract_reasoning && !include_grammar) {
return start + p.content(p.rest());
}
if (extract_reasoning) {
p.rule("analysis", p.literal(" to=self<|message|>") + p.reasoning(p.until("<|eom|>")) + p.literal("<|eom|>"));
p.rule("analysis", p.literal(" to=self") + message + p.reasoning(p.until(eom)) + eom);
} else {
p.rule("analysis", p.literal(" to=self<|message|>") + p.content(p.until("<|eom|>")) + p.literal("<|eom|>"));
p.rule("analysis", p.literal(" to=self") + message + p.content(p.until(eom)) + eom);
}
auto analysis = p.ref("analysis");
auto recipient = p.optional(p.literal(" to=user"));
auto final_msg = p.rule("final", recipient + p.literal("<|message|>") +
p.content(p.until_one_of({ "<|eot|>", "<|eom|>" })));
auto final_msg = p.rule("final", recipient + message + p.content(p.until(eom)));
if (has_response_format) {
auto response_json = p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema));
auto response_format = p.rule("response-format",
recipient + p.literal("<|message|>") +
recipient + message +
((p.literal("```json") + p.space() + response_json + p.space() + p.literal("```")) | response_json));
return p.zero_or_more(start + analysis) + start + response_format;
}
if (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE) {
auto string_value = p.ac(
p.tool_arg_string_value(p.until("</atem:parameter>")) + p.tool_arg_close(p.literal("</atem:parameter>")),
"</atem:parameter>");
auto string_value = p.through(p.tool_arg_string_value(p.until("</atem:parameter>")),
p.tool_arg_close(p.literal("</atem:parameter>")));
auto tool_choice = p.choice();
foreach_function(inputs.tools, [&](size_t tool_index, const json & tool) {
@@ -106,40 +107,35 @@ common_chat_params common_chat_params_init_muse_glimmer(const common_chat_templa
args = p.zero_or_more(p.choice(arg_rules) + p.space());
}
auto tool_parser = p.tool(
p.tool_open(p.literal(" to=") + p.until("<|message|>") +
p.literal("<|message|><atem:function_calls>") + p.space() +
p.literal("<atem:invoke name=\"") + p.tool_name(p.literal(name)) + p.literal("\">") + p.space())
auto open = p.tool_open(start_token + p.literal("assistant to=") + p.tool_name(p.literal(name)) + message);
auto body = p.rule("tool-" + std::to_string(tool_index),
p.literal("<atem:function_calls>") + p.space() + p.literal("<atem:invoke name=\"" + name + "\">") + p.space()
<< p.tool_args(args)
<< p.tool_close(p.literal("</atem:invoke>") + p.space() + p.literal("</atem:function_calls>")));
tool_choice |= p.rule("tool-" + std::to_string(tool_index), tool_parser);
tool_choice |= open + body;
});
auto tool_calls = inputs.parallel_tool_calls
? p.trigger_rule("tool-call", tool_choice + p.zero_or_more(p.literal("<|eom|>") + start + tool_choice))
: p.trigger_rule("tool-call", tool_choice);
auto more = inputs.parallel_tool_calls ? p.zero_or_more(eom + tool_choice) : p.eps();
auto tool_calls = p.rule("tool-calls", tool_choice + more);
if (inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED) {
return p.zero_or_more(start + analysis) + start + tool_calls;
return p.zero_or_more(start + analysis) + tool_calls;
}
auto trailing_calls = p.optional(p.literal("<|eom|>") + start + tool_calls);
return p.zero_or_more(start + analysis) + start + (tool_calls | (final_msg + trailing_calls));
auto trailing_calls = p.optional(eom + tool_calls);
return p.zero_or_more(start + analysis) + (tool_calls | (start + final_msg + trailing_calls));
}
return p.zero_or_more(start + analysis) + start + final_msg;
});
if (include_grammar) {
data.grammar_lazy = !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
// Every message starts with a recipient header, so the grammar is not lazy.
// The model cannot generate a recipient that is not self, user, or a tool.
data.grammar_lazy = false;
data.grammar = build_grammar([&](const common_grammar_builder & builder) {
data.parser.build_grammar(builder, data.grammar_lazy);
data.parser.build_grammar(builder);
});
data.grammar_triggers = {
{ COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN,
"(?:^|<\\|start\\|>assistant)( to=(?!self<\\|message\\|>)(?!user<\\|message\\|>)[^<]*?<\\|message\\|>)" },
};
}
return data;
+2 -3
View File
@@ -86,9 +86,8 @@ common_chat_params common_chat_params_init_muse_spark(const common_chat_template
}
if (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE) {
auto string_value = p.ac(
p.tool_arg_string_value(p.until("</atem:parameter>")) + p.tool_arg_close(p.literal("</atem:parameter>")),
"</atem:parameter>");
auto string_value = p.through(p.tool_arg_string_value(p.until("</atem:parameter>")),
p.tool_arg_close(p.literal("</atem:parameter>")));
auto tool_choice = p.choice();
foreach_function(inputs.tools, [&](size_t tool_index, const json & tool) {
+31 -48
View File
@@ -59,27 +59,16 @@ common_chat_params common_chat_params_init_qwen3_coder(const common_chat_templat
data.prompt.append(data.generation_prompt);
}
std::vector<std::string> tool_call_starts = { "<tool_call>" };
if (is_qwen3_coder) {
// Match complete <function=name> opener for Qwen3-Coder models that occasionally omit the
// starting <tool_call>. The model may hallucinate a tool name, but it is preferable over
// constraining on <function which may occur in valid content generation, e.g. #include <functional>
foreach_function(inputs.tools, [&](size_t, const json & tool) {
const std::string name = tool.at("function").at("name");
tool_call_starts.push_back("<function=" + name + ">");
});
}
data.parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
data.parser = build_chat_peg_parser(inputs.specials, [&](common_chat_peg_builder & p) {
auto generation_prompt = p.literal(GEN_PREFIX);
// The template strips the newline before </think>, so leave it out of the reasoning
auto think_body = p.until_one_of({ "\n</think>", "</think>", "<tool_call>" });
auto think_close = p.literal("\n</think>") | p.literal("</think>") | p.peek(p.literal("<tool_call>"));
auto think_end = (p.literal("\n") + p.token("</think>")) | p.token("</think>");
auto think_body = p.until(think_end | p.token("<tool_call>"));
auto think_close = think_end | p.peek(p.token("<tool_call>"));
auto reasoning_extracted = p.optional("<think>" + p.space() + p.reasoning(think_body) + think_close);
auto reasoning_skipped = p.optional("<think>" + p.space() + think_body + think_close);
auto reasoning_extracted = p.optional(p.token("<think>") + p.space() + p.reasoning(think_body) + think_close);
auto reasoning_skipped = p.optional(p.token("<think>") + p.space() + think_body + think_close);
auto reasoning = p.eps();
if (supports_reasoning && extract_reasoning) {
@@ -99,7 +88,13 @@ common_chat_params common_chat_params_init_qwen3_coder(const common_chat_templat
if (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE) {
auto arg_close = p.tool_arg_close(p.literal("\n</parameter>\n"));
auto arg_string = p.rule("xml-arg-string",
p.ac(p.tool_arg_string_value(p.until("\n</parameter>\n")) + arg_close, "\n</parameter>\n"));
p.through(p.tool_arg_string_value(p.until("\n</parameter>\n")), arg_close));
auto min_calls = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED ? 1 : 0;
auto more = inputs.parallel_tool_calls ? p.zero_or_more(p.ref("tool-call")) : p.eps();
auto starts = p.token("<tool_call>");
auto tool_calls = p.choice();
auto tool_choice = p.choice();
foreach_function(inputs.tools, [&](size_t tool_index, const json & tool) {
@@ -155,35 +150,29 @@ common_chat_params common_chat_params_init_qwen3_coder(const common_chat_templat
args = args + p.zero_or_more(p.choice(optional_args));
}
auto func = p.tool(p.tool_open("<function=" + p.tool_name(p.literal(name)) + ">\n") +
p.tool_args(args) +
p.tool_close(p.literal("</function>\n")));
auto opener = p.tool_open("<function=" + p.tool_name(p.literal(name)) + ">");
auto body = p.literal("\n") + p.tool_args(args) + p.tool_close(p.literal("</function>\n"));
tool_choice |= p.rule("tool-" + std::to_string(tool_index), func);
tool_choice |= p.rule("tool-" + std::to_string(tool_index), p.tool(opener + body));
if (is_qwen3_coder) {
// Qwen3-Coder models sometimes omit <tool_call>, so the full <function=name> opener is also a trigger.
// Do not trigger on <function alone, it occurs in normal content, e.g. #include <functional>.
tool_calls |= p.trigger_rule("tool-calls-" + std::to_string(tool_index), opener, body + p.token("</tool_call>") + p.space() + more);
starts |= opener;
}
});
auto min_calls = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED ? 1 : 0;
auto tool_call_body = tool_choice + p.token("</tool_call>") + p.space();
p.rule("tool-call", p.token("<tool_call>") + p.literal("\n") + tool_call_body);
auto tool_call_body = tool_choice + "</tool_call>" + p.space();
auto tool_call = p.rule("tool-call", "<tool_call>\n" + tool_call_body);
// Qwen3-Coder models may occasionally omit the <tool_call> token.
auto tool_call_first = is_qwen3_coder ?
p.rule("tool-call-first", p.optional(p.literal("<tool_call>\n")) + tool_call_body) :
tool_call;
auto calls = inputs.parallel_tool_calls ? tool_call_first + p.zero_or_more(tool_call) : tool_call_first;
auto tool_calls = p.trigger_rule("tool-call-root", p.repeat(calls, min_calls, 1));
tool_calls |= p.trigger_rule("tool-calls", p.token("<tool_call>"), p.literal("\n") + tool_call_body + more);
// The templates separate content from the tool calls with one or two newlines,
// stop the content before them
std::vector<std::string> content_ends;
for (const auto & start : tool_call_starts) {
content_ends.insert(content_ends.end(), { "\n\n" + start, "\n" + start, start });
}
auto content_end = (p.literal("\n\n") + starts) | (p.literal("\n") + starts) | starts;
return generation_prompt +
(reasoning << p.content(p.until_one_of(content_ends)) << tool_calls);
return generation_prompt + (reasoning << p.content(p.until(content_end)) << p.repeat(tool_calls, min_calls, 1));
}
// Content only parser
@@ -191,17 +180,11 @@ common_chat_params common_chat_params_init_qwen3_coder(const common_chat_templat
});
if (include_grammar) {
data.grammar_lazy = !has_response_format && has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO;
// The grammar scans for the triggers itself, so the sampler runs it from the first token
data.grammar_lazy = false;
data.grammar = build_grammar([&](const common_grammar_builder & builder) {
data.parser.build_grammar(builder, data.grammar_lazy);
data.parser.build_grammar(builder, !has_response_format && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO);
});
if (data.grammar_lazy) {
for (const auto & start : tool_call_starts) {
data.grammar_triggers.push_back({ COMMON_GRAMMAR_TRIGGER_TYPE_WORD, start });
}
}
}
return data;
+544 -116
View File
@@ -208,6 +208,89 @@ void common_peg_ast_arena::visit(const common_peg_parse_result & result, const c
}
}
common_peg_tokens::common_peg_tokens(const llama_vocab * vocab) {
const llama_token n_tokens = llama_vocab_n_tokens(vocab);
for (llama_token id = 0; id < n_tokens; ++id) {
const auto attr = llama_vocab_get_attr(vocab, id);
if (attr & (LLAMA_TOKEN_ATTR_CONTROL | LLAMA_TOKEN_ATTR_USER_DEFINED)) {
std::string text = llama_vocab_get_text(vocab, id);
ids.emplace(text, id);
tokens.emplace(id, common_peg_token{ std::move(text), attr });
}
}
}
llama_token common_peg_tokens::token_id(const std::string & text) const {
auto it = ids.find(text);
return it != ids.end() ? it->second : LLAMA_TOKEN_NULL;
}
// Insert every delimiter the parser matches into the trie, starting from each node in from. Returns the end nodes.
// A choice adds the delimiters of each branch. A sequence joins every combination of its children.
static std::set<size_t> insert_delimiter(common_trie & trie, const common_peg_arena & arena, common_peg_parser_id id,
std::set<size_t> from = { 0 }) {
return std::visit([&](const auto & p) -> std::set<size_t> {
using T = std::decay_t<decltype(p)>;
if constexpr (std::is_same_v<T, common_peg_tag_parser> || std::is_same_v<T, common_peg_atomic_parser>) {
return insert_delimiter(trie, arena, p.child, std::move(from));
} else if constexpr (std::is_same_v<T, common_peg_literal_parser>) {
std::set<size_t> ends;
for (size_t node : from) {
size_t pos = 0;
while (pos < p.literal.size()) {
auto result = common_parse_utf8_codepoint(p.literal, pos);
if (result.status != utf8_parse_result::SUCCESS) {
break;
}
node = trie.extend(node, common_trie::symbol::codepoint(result.codepoint));
pos += result.bytes_consumed;
}
ends.insert(node);
}
return ends;
} else if constexpr (std::is_same_v<T, common_peg_token_parser>) {
std::set<size_t> ends;
for (size_t node : from) {
ends.insert(trie.extend(node, common_trie::symbol::token(p.token)));
}
return ends;
} else if constexpr (std::is_same_v<T, common_peg_choice_parser>) {
std::set<size_t> ends;
for (auto child : p.children) {
ends.merge(insert_delimiter(trie, arena, child, from));
}
return ends;
} else if constexpr (std::is_same_v<T, common_peg_sequence_parser>) {
for (auto child : p.children) {
from = insert_delimiter(trie, arena, child, std::move(from));
}
return from;
} else {
throw std::invalid_argument("a delimiter may only be a sequence of literals, tokens, and choices of them, possibly wrapped in tags and atomics");
}
}, arena.get(id));
}
// Build the matcher of an until parser, each pattern valued by the branch it belongs to
static common_trie build_until_matcher(const common_peg_arena & arena,
const std::vector<common_peg_until_branch> & branches,
bool include) {
common_trie matcher;
for (size_t b = 0; b < branches.size(); b++) {
for (size_t end : insert_delimiter(matcher, arena, branches[b].delimiter)) {
if (end == 0) {
throw std::invalid_argument("delimiter must not match the empty string");
}
// nothing follows a delimiter without include, so the first branch to claim it wins
auto owner = matcher.mark(end, (int32_t) b);
if (owner != (int32_t) b && include) {
throw std::invalid_argument("until branches " + std::to_string(owner) + " and " + std::to_string(b) + " must not share a delimiter");
}
}
}
return matcher;
}
struct parser_executor;
common_peg_parser_id common_peg_arena::add_parser(common_peg_parser_variant parser) {
@@ -298,6 +381,49 @@ struct parser_executor {
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, pos);
}
common_peg_parse_result operator()(const common_peg_token_parser & p) {
if (start_pos >= ctx.input.size()) {
if (!ctx.is_lenient()) {
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_FAIL, start_pos);
}
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_NEED_MORE_INPUT, start_pos);
}
if (start_pos >= ctx.tokens.size() || ctx.tokens[start_pos] != p.token) {
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_FAIL, start_pos);
}
// Skip the rest of the piece, up to the next token or the end of the input
auto pos = start_pos + 1;
while (pos < ctx.input.size() && ctx.tokens[pos] == LLAMA_TOKEN_NULL) {
++pos;
}
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, pos);
}
common_peg_parse_result operator()(const common_peg_token_not_parser & p) {
if (start_pos >= ctx.input.size()) {
if (!ctx.is_lenient()) {
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_FAIL, start_pos);
}
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_NEED_MORE_INPUT, start_pos);
}
if (start_pos >= ctx.tokens.size() || ctx.tokens[start_pos] == LLAMA_TOKEN_NULL ||
std::find(p.tokens.begin(), p.tokens.end(), ctx.tokens[start_pos]) != p.tokens.end()) {
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_FAIL, start_pos);
}
// Skip the rest of the piece, up to the next token or the end of the input
auto pos = start_pos + 1;
while (pos < ctx.input.size() && ctx.tokens[pos] == LLAMA_TOKEN_NULL) {
++pos;
}
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, pos);
}
common_peg_parse_result operator()(const common_peg_sequence_parser & p) {
if (ctx.is_debug()) {
LOG_DBG("%sSEQ start at %zu '%s' (%zu children)\n", debug_indent().c_str(), start_pos,
@@ -689,7 +815,11 @@ struct parser_executor {
}
common_peg_parse_result operator()(const common_peg_until_parser & p) const {
common_trie matcher(p.delimiters);
if (p.include) {
return parse_through(p);
}
const auto & matcher = p.matcher;
// Scan input and check for delimiters
size_t pos = start_pos;
@@ -697,24 +827,24 @@ struct parser_executor {
std::vector<common_peg_invalid_utf8> invalid_utf8;
while (pos < ctx.input.size()) {
auto utf8_result = common_parse_utf8_codepoint(ctx.input, pos);
auto step = matcher.next(ctx.input, ctx.tokens, pos);
if (utf8_result.status == utf8_parse_result::INCOMPLETE && ctx.is_lenient()) {
if (step.status == utf8_parse_result::INCOMPLETE && ctx.is_lenient()) {
// The rest of the sequence may still arrive, return what we have so far
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_NEED_MORE_INPUT, start_pos, last_valid_pos, {}, std::move(invalid_utf8));
}
if (utf8_result.status != utf8_parse_result::SUCCESS) {
if (step.status != utf8_parse_result::SUCCESS) {
// Malformed UTF-8, or a sequence truncated by the end of a complete input.
// A delimiter cannot start inside bytes that fail to decode, so consume them and move on
invalid_utf8.push_back({pos, utf8_result.bytes_consumed});
pos += utf8_result.bytes_consumed;
invalid_utf8.push_back({pos, step.bytes_consumed});
pos += step.bytes_consumed;
last_valid_pos = pos;
continue;
}
// Check if a delimiter starts at this position
auto match = matcher.check_at(ctx.input, pos);
auto match = matcher.check_at(ctx.input, ctx.tokens, pos);
if (match == common_trie::COMPLETE_MATCH) {
// Found a complete delimiter, return everything before it
@@ -727,7 +857,7 @@ struct parser_executor {
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, pos, {}, std::move(invalid_utf8));
}
pos += utf8_result.bytes_consumed;
pos += step.bytes_consumed;
last_valid_pos = pos;
}
@@ -738,6 +868,48 @@ struct parser_executor {
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, last_valid_pos, {}, std::move(invalid_utf8));
}
// Parse content, then the first branch whose delimiter and rest follow it, like C (D1 R1 | D2 R2 | ...)
common_peg_parse_result parse_through(const common_peg_until_parser & p) const {
auto content = arena.parse(p.content, ctx, start_pos);
if (content.fail() || content.need_more_input()) {
return content;
}
auto pos = content.end;
for (const auto & branch : p.branches) {
auto nodes = content.nodes;
auto invalid_utf8 = content.invalid_utf8;
auto end = pos;
bool matched = true;
for (auto id : { branch.delimiter, branch.rest }) {
if (id == COMMON_PEG_INVALID_PARSER_ID) {
continue;
}
auto result = arena.parse(id, ctx, end);
if (result.fail()) {
matched = false;
break;
}
nodes.insert(nodes.end(), result.nodes.begin(), result.nodes.end());
invalid_utf8.insert(invalid_utf8.end(), result.invalid_utf8.begin(), result.invalid_utf8.end());
end = result.end;
if (result.need_more_input()) {
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_NEED_MORE_INPUT, start_pos, end, std::move(nodes), std::move(invalid_utf8));
}
}
if (matched) {
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, end, std::move(nodes), std::move(invalid_utf8));
}
}
if (p.optional && pos == ctx.input.size()) {
auto type = ctx.is_lenient() ? COMMON_PEG_PARSE_RESULT_NEED_MORE_INPUT : COMMON_PEG_PARSE_RESULT_SUCCESS;
return common_peg_parse_result(type, start_pos, pos, std::move(content.nodes), std::move(content.invalid_utf8));
}
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_FAIL, start_pos, pos);
}
common_peg_parse_result operator()(const common_peg_schema_parser & p) {
return arena.parse(p.child, ctx, start_pos);
}
@@ -823,10 +995,6 @@ struct parser_executor {
common_peg_parse_result operator()(const common_peg_gbnf_parser & p) {
return arena.parse(p.child, ctx, start_pos);
}
common_peg_parse_result operator()(const common_peg_ac_parser & p) {
return arena.parse(p.child, ctx, start_pos);
}
};
common_peg_parse_result common_peg_arena::parse(common_peg_parse_context & ctx, size_t start) const {
@@ -899,19 +1067,33 @@ void common_peg_arena::resolve_refs() {
std::is_same_v<T, common_peg_not_parser> ||
std::is_same_v<T, common_peg_tag_parser> ||
std::is_same_v<T, common_peg_atomic_parser> ||
std::is_same_v<T, common_peg_gbnf_parser> ||
std::is_same_v<T, common_peg_ac_parser>) {
std::is_same_v<T, common_peg_gbnf_parser>) {
p.child = resolve_ref(p.child);
} else if constexpr (std::is_same_v<T, common_peg_until_parser>) {
if (p.content != COMMON_PEG_INVALID_PARSER_ID) {
p.content = resolve_ref(p.content);
}
for (auto & branch : p.branches) {
branch.delimiter = resolve_ref(branch.delimiter);
if (branch.rest != COMMON_PEG_INVALID_PARSER_ID) {
branch.rest = resolve_ref(branch.rest);
}
}
} else if constexpr (std::is_same_v<T, common_peg_rule_parser>) {
p.child = resolve_ref(p.child);
if (p.start != COMMON_PEG_INVALID_PARSER_ID) {
p.start = resolve_ref(p.start);
p.rest = resolve_ref(p.rest);
}
} else if constexpr (std::is_same_v<T, common_peg_schema_parser>) {
p.child = resolve_ref(p.child);
} else if constexpr (std::is_same_v<T, common_peg_epsilon_parser> ||
std::is_same_v<T, common_peg_start_parser> ||
std::is_same_v<T, common_peg_end_parser> ||
std::is_same_v<T, common_peg_ref_parser> ||
std::is_same_v<T, common_peg_until_parser> ||
std::is_same_v<T, common_peg_literal_parser> ||
std::is_same_v<T, common_peg_token_parser> ||
std::is_same_v<T, common_peg_token_not_parser> ||
std::is_same_v<T, common_peg_string_parser> ||
std::is_same_v<T, common_peg_chars_parser> ||
std::is_same_v<T, common_peg_any_parser> ||
@@ -955,6 +1137,14 @@ std::string common_peg_arena::dump_impl(common_peg_parser_id
return "End";
} else if constexpr (std::is_same_v<T, common_peg_literal_parser>) {
return "Literal(" + p.literal + ")";
} else if constexpr (std::is_same_v<T, common_peg_token_parser>) {
return "Token(" + p.piece + ", " + std::to_string(p.token) + ")";
} else if constexpr (std::is_same_v<T, common_peg_token_not_parser>) {
std::vector<std::string> parts;
for (auto token : p.tokens) {
parts.push_back(std::to_string(token));
}
return "TokenNot(" + string_join(parts, ", ") + ")";
} else if constexpr (std::is_same_v<T, common_peg_sequence_parser>) {
std::vector<std::string> parts;
for (const auto & child : p.children) {
@@ -981,8 +1171,6 @@ std::string common_peg_arena::dump_impl(common_peg_parser_id
return "Atomic(" + dump_impl(p.child, visited) + ")";
} else if constexpr (std::is_same_v<T, common_peg_gbnf_parser>) {
return "Gbnf(" + p.grammar + ", " + dump_impl(p.child, visited) + ")";
} else if constexpr (std::is_same_v<T, common_peg_ac_parser>) {
return "Ac(" + string_join(p.delimiters, " | ") + ", " + dump_impl(p.child, visited) + ")";
} else if constexpr (std::is_same_v<T, common_peg_any_parser>) {
return "Any";
} else if constexpr (std::is_same_v<T, common_peg_space_parser>) {
@@ -995,7 +1183,18 @@ std::string common_peg_arena::dump_impl(common_peg_parser_id
} else if constexpr (std::is_same_v<T, common_peg_string_parser>) {
return "String(" + std::string(1, p.delimiter) + ")";
} else if constexpr (std::is_same_v<T, common_peg_until_parser>) {
return "Until(" + string_join(p.delimiters, " | ") + ")";
std::vector<std::string> branches;
for (const auto & branch : p.branches) {
auto text = dump_impl(branch.delimiter, visited);
if (branch.rest != COMMON_PEG_INVALID_PARSER_ID) {
text += " " + dump_impl(branch.rest, visited);
}
branches.push_back(std::move(text));
}
if (p.include) {
return "Until(" + dump_impl(p.content, visited) + ", " + string_join(branches, " | ") + (p.optional ? ", optional" : "") + ")";
}
return "Until(" + string_join(branches, " | ") + ")";
} else if constexpr (std::is_same_v<T, common_peg_schema_parser>) {
return "Schema(" + dump_impl(p.child, visited) + ", " + (p.node ? common_chat_schema::kind_name(p.node->kind()) : "null") + ")";
} else if constexpr (std::is_same_v<T, common_peg_rule_parser>) {
@@ -1089,6 +1288,67 @@ common_peg_parser operator|(const std::string & str, const common_peg_parser & p
common_peg_parser_builder::common_peg_parser_builder() {}
common_peg_parser_builder::common_peg_parser_builder(common_peg_tokens tokens) {
arena_.tokens_ = std::move(tokens);
}
common_peg_parser common_peg_parser_builder::until_one_of(const std::vector<std::string> & delimiters) {
common_peg_until_parser p;
for (const auto & d : delimiters) {
p.branches.push_back({ literal(d).id() });
}
p.matcher = build_until_matcher(arena_, p.branches, p.include);
return add(std::move(p));
}
common_peg_parser common_peg_parser_builder::until(const common_peg_parser & delimiter) {
common_peg_until_parser p;
p.branches = { { delimiter.id() } };
p.matcher = build_until_matcher(arena_, p.branches, p.include);
return add(std::move(p));
}
common_peg_parser common_peg_parser_builder::through(const common_peg_parser & content, const std::vector<common_peg_until_branch> & branches, bool optional) {
if (branches.empty()) {
throw std::runtime_error("through requires at least one branch");
}
common_peg_until_parser p;
p.branches = branches;
p.include = true;
p.optional = optional;
p.content = content.id();
p.matcher = build_until_matcher(arena_, p.branches, p.include);
return add(std::move(p));
}
common_peg_parser common_peg_parser_builder::token(const std::string & piece) {
auto token = arena_.tokens_.token_id(piece);
if (token == LLAMA_TOKEN_NULL) {
return literal(piece);
}
return add(common_peg_token_parser{token, piece});
}
common_peg_parser common_peg_parser_builder::token(const std::vector<std::string> & pieces, bool negate) {
if (!negate) {
std::vector<common_peg_parser> alts;
for (const auto & piece : pieces) {
alts.push_back(token(piece));
}
return choice(alts);
}
common_peg_token_not_parser p;
for (const auto & piece : pieces) {
auto token = arena_.tokens_.token_id(piece);
if (token == LLAMA_TOKEN_NULL) {
throw std::invalid_argument("token is not registered with the builder: " + piece);
}
p.tokens.push_back(token);
}
return add(p);
}
common_peg_parser common_peg_parser_builder::sequence(const std::vector<common_peg_parser_id> & parsers) {
// Flatten nested sequences
std::vector<common_peg_parser_id> flattened;
@@ -1195,6 +1455,17 @@ common_peg_parser common_peg_parser_builder::rule(const std::string & name, cons
return ref(clean_name);
}
common_peg_parser common_peg_parser_builder::trigger_rule(const std::string & name, const common_peg_parser & start, const common_peg_parser & rest) {
common_trie scratch;
if (insert_delimiter(scratch, arena_, start.id()).count(0)) {
throw std::invalid_argument("trigger start must not match the empty string");
}
auto clean_name = gbnf_format_rule_name(name);
auto rule_id = arena_.add_parser(common_peg_rule_parser{clean_name, (start + rest).id(), true, start.id(), rest.id()});
arena_.add_rule(clean_name, rule_id);
return ref(clean_name);
}
void common_peg_parser_builder::set_root(const common_peg_parser & p) {
arena_.set_root(p.id());
}
@@ -1392,13 +1663,6 @@ common_peg_parser common_peg_parser_builder::json_member(const std::string & key
});
}
common_peg_parser common_peg_parser_builder::ac(const common_peg_parser & p, const std::vector<std::string> & delimiters) {
if (delimiters.empty()) {
throw std::runtime_error("ac parser requires at least one delimiter");
}
return add(common_peg_ac_parser{p, delimiters});
}
static std::string gbnf_escape_char_class(uint32_t c) {
if (c == '-' || c == ']' || c == '[' || c == '\\') {
return "\\" + std::string(1, (char) c);
@@ -1457,15 +1721,63 @@ static std::string gbnf_char_class(const std::vector<uint32_t> & chars, bool neg
return s + "]";
}
// Token set such as <[1,5-9]>, with consecutive ids collapsed into ranges
static std::string gbnf_token_set(const std::vector<uint32_t> & tokens, bool negate) {
std::string s = negate ? "!<[" : "<[";
for (size_t i = 0; i < tokens.size();) {
size_t j = i;
while (j + 1 < tokens.size() && tokens[j + 1] == tokens[j] + 1) {
j++;
}
s += (i == 0 ? "" : ",") + std::to_string(tokens[i]);
if (j > i) {
s += "-" + std::to_string(tokens[j]);
}
i = j + 1;
}
return s + "]>";
}
// Symbols on the transitions out of an automaton state, codepoints and tokens kept apart
struct gbnf_ac_symbols {
std::vector<uint32_t> chars;
std::vector<uint32_t> tokens;
void add(const common_trie::symbol & sym) {
(sym.is_token() ? tokens : chars).push_back(sym.value);
}
// One alternative for the codepoints and one for the tokens, each followed by next if given
std::vector<std::string> alternatives(const std::string & next = "") const {
std::vector<std::string> alts;
const std::string tail = next.empty() ? "" : " " + next;
if (!chars.empty()) {
alts.push_back(gbnf_char_class(chars, false) + tail);
}
if (!tokens.empty()) {
alts.push_back(gbnf_token_set(tokens, false) + tail);
}
return alts;
}
};
static std::string gbnf_ac_grammar(
const common_grammar_builder & builder,
const std::string & prefix,
const std::vector<std::string> & strings,
const std::function<std::string(const std::vector<uint32_t> &,
const std::map<size_t, std::vector<uint32_t>> &,
const std::vector<uint32_t> &,
const common_grammar_builder & builder,
const std::string & prefix,
const common_trie & trie,
const std::set<uint32_t> & opaque,
const std::function<std::string(const gbnf_ac_symbols &,
const std::map<size_t, gbnf_ac_symbols> &,
const std::map<size_t, gbnf_ac_symbols> &,
const std::string &,
const std::function<std::string(size_t)> &)> & build_rule) {
common_aho_corasick ac(strings);
common_aho_corasick ac(trie);
// With only tokens in the alphabet, the grammar can step a whole token at a time
bool token_only = true;
for (const auto & sym : ac.alphabet) {
token_only = token_only && sym.is_token();
}
auto state_name = [&](size_t s) -> std::string {
if (s == 0) {
@@ -1481,27 +1793,43 @@ static std::string gbnf_ac_grammar(
continue; // match states
}
std::map<size_t, std::vector<uint32_t>> buckets;
std::vector<uint32_t> completing; // chars that complete a delimiter
std::vector<uint32_t> specific; // chars with an explicit transition
for (uint32_t c : ac.alphabet) {
size_t d = ac.next(q, c);
std::map<size_t, gbnf_ac_symbols> buckets;
std::map<size_t, gbnf_ac_symbols> completing_to; // completing symbols by the match state they reach
gbnf_ac_symbols completing; // symbols that complete a delimiter
gbnf_ac_symbols specific; // symbols with an explicit transition
for (const auto & sym : ac.alphabet) {
size_t d = ac.next(q, sym);
if (ac.is_terminal(d)) {
completing.push_back(c);
specific.push_back(c);
completing.add(sym);
completing_to[d].add(sym);
specific.add(sym);
} else if (d != 0) {
buckets[d].push_back(c); // specific non-root destination
specific.push_back(c);
buckets[d].add(sym); // specific non-root destination
specific.add(sym);
}
}
builder.add_rule(state_name(q), build_rule(completing, buckets, specific, state_name));
}
// every other symbol goes back to the start
std::string fallback;
if (token_only) {
fallback = gbnf_token_set(specific.tokens, true);
} else {
fallback = specific.chars.empty() ? "." : gbnf_char_class(specific.chars, true);
// An empty delimiter makes the start state terminal. Emit an entry rule
// that matches the empty string so the returned reference stays valid.
if (ac.is_terminal(0)) {
builder.add_rule(prefix, "|");
// Char rules never match a token named by a token rule, so allow the other named tokens by name
std::vector<uint32_t> others;
for (auto token : opaque) {
if (std::find(specific.tokens.begin(), specific.tokens.end(), token) == specific.tokens.end()) {
others.push_back(token);
}
}
if (!others.empty()) {
fallback = "(" + fallback + " | " + gbnf_token_set(others, false) + ")";
}
}
fallback += " " + state_name(0);
builder.add_rule(state_name(q), build_rule(completing, completing_to, buckets, fallback, state_name));
}
return state_name(0);
@@ -1514,52 +1842,104 @@ static std::string gbnf_ac_grammar(
// ref: https://github.com/ggml-org/llama.cpp/pull/24839
static std::string gbnf_excluding_grammar(const common_grammar_builder & builder,
const std::string & prefix,
const std::vector<std::string> & strings) {
return gbnf_ac_grammar(builder, prefix, strings,
[](const std::vector<uint32_t> & /*completing*/,
const std::map<size_t, std::vector<uint32_t>> & buckets,
const std::vector<uint32_t> & specific,
const common_trie & trie,
const std::set<uint32_t> & opaque) {
return gbnf_ac_grammar(builder, prefix, trie, opaque,
[](const gbnf_ac_symbols & /*completing*/,
const std::map<size_t, gbnf_ac_symbols> & /*completing_to*/,
const std::map<size_t, gbnf_ac_symbols> & buckets,
const std::string & fallback,
const std::function<std::string(size_t)> & state_name) {
// every state is accepting and completing chars get no
// alternative, so a forbidden string can never be matched
// every state accepts and completing symbols get no alternative, so a forbidden string never matches
std::string rhs = "|";
for (const auto & [d, chars] : buckets) {
rhs += " " + gbnf_char_class(chars, false) + " " + state_name(d) + " |";
for (const auto & [d, symbols] : buckets) {
for (const auto & alt : symbols.alternatives(state_name(d))) {
rhs += " " + alt + " |";
}
}
rhs += " " + gbnf_char_class(specific, true) + " " + state_name(0);
rhs += " " + fallback;
return rhs;
});
}
// GBNF grammar matching everything up to and including the first occurrence of
// any string in `strings`. Emits the Aho-Corasick automaton DFA and returns
// the start state rule name.
static std::string gbnf_including_grammar(const common_grammar_builder & builder,
const std::string & prefix,
const std::vector<std::string> & strings) {
return gbnf_ac_grammar(builder, prefix, strings,
[](const std::vector<uint32_t> & completing,
const std::map<size_t, std::vector<uint32_t>> & buckets,
const std::vector<uint32_t> & specific,
const std::function<std::string(size_t)> & state_name) {
// GBNF grammar matching up to the first pattern in the trie, then the rest rule of that pattern's value.
// For the pattern "ab" with rest "x":
//
// prefix ::= [a] prefix-01 | [^a] prefix
// prefix-01 ::= [b] rest | [a] prefix-01 | [^ab] prefix
// rest ::= "x"
static std::string gbnf_including_grammar(const common_grammar_builder & builder,
const std::string & prefix,
const common_trie & trie,
const std::set<uint32_t> & opaque,
const std::vector<std::string> & rests,
bool optional) {
common_aho_corasick ac(trie);
// every pattern ending at a match state, the longest and the ones reached by suffix links
auto rests_at = [&](size_t s, bool & terminal) {
std::vector<std::string> alts;
terminal = false;
for (; s != 0; s = ac.fail[s]) {
auto pattern = ac.t.nodes[s].pattern;
if (pattern < 0) {
continue;
}
if (rests[pattern].empty()) {
terminal = true;
} else if (std::find(alts.begin(), alts.end(), rests[pattern]) == alts.end()) {
alts.push_back(rests[pattern]);
}
}
return alts;
};
return gbnf_ac_grammar(builder, prefix, trie, opaque,
[&](const gbnf_ac_symbols & /*completing*/,
const std::map<size_t, gbnf_ac_symbols> & completing_to,
const std::map<size_t, gbnf_ac_symbols> & buckets,
const std::string & fallback,
const std::function<std::string(size_t)> & state_name) {
std::vector<std::string> alts;
if (!completing.empty()) {
alts.push_back(gbnf_char_class(completing, false)); // terminate on match
gbnf_ac_symbols terminating; // symbols that complete a pattern with no rest
for (const auto & [d, symbols] : completing_to) {
bool terminal = false;
auto rest = rests_at(d, terminal);
if (!rest.empty()) {
for (auto & alt : symbols.alternatives(rest.size() == 1 ? rest[0] : "(" + string_join(rest, " | ") + ")")) {
alts.push_back(std::move(alt));
}
}
if (terminal) {
terminating.chars.insert(terminating.chars.end(), symbols.chars.begin(), symbols.chars.end());
terminating.tokens.insert(terminating.tokens.end(), symbols.tokens.begin(), symbols.tokens.end());
}
}
for (const auto & [d, chars] : buckets) {
alts.push_back(gbnf_char_class(chars, false) + " " + state_name(d));
for (auto & alt : terminating.alternatives()) {
alts.push_back(std::move(alt));
}
// every other character keeps scanning from the start state
alts.push_back(gbnf_char_class(specific, true) + " " + state_name(0));
return string_join(alts, " | ");
for (const auto & [d, symbols] : buckets) {
for (auto & alt : symbols.alternatives(state_name(d))) {
alts.push_back(std::move(alt));
}
}
// every other symbol keeps scanning from the start state
alts.push_back(fallback);
return (optional ? "| " : "") + string_join(alts, " | ");
});
}
static std::set<std::string> collect_reachable_rules(
// The rules and the tokens named by token rules that the grammar of a parser reaches
struct common_peg_reachable {
std::set<std::string> rules;
std::set<uint32_t> tokens;
};
static void collect_reachable(
const common_peg_arena & arena,
const common_peg_parser_id & rule
const common_peg_parser_id & root,
common_peg_reachable & reachable
) {
std::set<std::string> reachable;
std::set<std::string> visited;
std::function<void(common_peg_parser_id)> visit = [&](common_peg_parser_id id) {
@@ -1571,13 +1951,27 @@ static std::set<std::string> collect_reachable_rules(
if constexpr (std::is_same_v<T, common_peg_epsilon_parser> ||
std::is_same_v<T, common_peg_start_parser> ||
std::is_same_v<T, common_peg_end_parser> ||
std::is_same_v<T, common_peg_until_parser> ||
std::is_same_v<T, common_peg_literal_parser> ||
std::is_same_v<T, common_peg_chars_parser> ||
std::is_same_v<T, common_peg_space_parser> ||
std::is_same_v<T, common_peg_any_parser> ||
std::is_same_v<T, common_peg_string_parser>) {
// These parsers do not have any children
} else if constexpr (std::is_same_v<T, common_peg_token_parser>) {
reachable.tokens.insert(p.token);
} else if constexpr (std::is_same_v<T, common_peg_token_not_parser>) {
reachable.tokens.insert(p.tokens.begin(), p.tokens.end());
} else if constexpr (std::is_same_v<T, common_peg_until_parser>) {
// The delimiters name their tokens in the grammar
if (p.content != COMMON_PEG_INVALID_PARSER_ID) {
visit(p.content);
}
for (const auto & branch : p.branches) {
visit(branch.delimiter);
if (branch.rest != COMMON_PEG_INVALID_PARSER_ID) {
visit(branch.rest);
}
}
} else if constexpr (std::is_same_v<T, common_peg_sequence_parser>) {
for (auto child : p.children) {
visit(child);
@@ -1592,13 +1986,12 @@ static std::set<std::string> collect_reachable_rules(
std::is_same_v<T, common_peg_tag_parser> ||
std::is_same_v<T, common_peg_atomic_parser> ||
std::is_same_v<T, common_peg_gbnf_parser> ||
std::is_same_v<T, common_peg_ac_parser> ||
std::is_same_v<T, common_peg_schema_parser>) {
visit(p.child);
} else if constexpr (std::is_same_v<T, common_peg_rule_parser>) {
if (visited.find(p.name) == visited.end()) {
visited.insert(p.name);
reachable.insert(p.name);
reachable.rules.insert(p.name);
visit(p.child);
}
} else if constexpr (std::is_same_v<T, common_peg_ref_parser>) {
@@ -1611,12 +2004,13 @@ static std::set<std::string> collect_reachable_rules(
}, parser);
};
visit(rule);
return reachable;
visit(root);
}
// GBNF generation implementation
void common_peg_arena::build_grammar(const common_grammar_builder & builder, bool lazy) const {
common_peg_reachable reachable;
// A raw string value is parsed by the child rather than constrained by the schema
auto schema_delegates = [](const common_peg_schema_parser & s) -> bool {
return !s.node || (s.raw && s.node->may_be_string());
@@ -1655,6 +2049,10 @@ void common_peg_arena::build_grammar(const common_grammar_builder & builder, boo
return "";
} else if constexpr (std::is_same_v<T, common_peg_literal_parser>) {
return gbnf_format_literal(p.literal);
} else if constexpr (std::is_same_v<T, common_peg_token_parser>) {
return "<[" + std::to_string(p.token) + "]>";
} else if constexpr (std::is_same_v<T, common_peg_token_not_parser>) {
return gbnf_token_set(std::vector<uint32_t>(p.tokens.begin(), p.tokens.end()), true);
} else if constexpr (std::is_same_v<T, common_peg_sequence_parser>) {
std::string s;
for (const auto & child : p.children) {
@@ -1746,10 +2144,19 @@ void common_peg_arena::build_grammar(const common_grammar_builder & builder, boo
const std::string delim(1, p.delimiter);
return R"(( [^)" + delim + R"(\\] | "\\" ( [)" + delim + R"(\\/ bfnrt] | "u" [0-9a-fA-F]{4} ) )*)";
} else if constexpr (std::is_same_v<T, common_peg_until_parser>) {
if (p.delimiters.empty()) {
return ".*";
auto prefix = "until-" + std::to_string(id);
if (!p.include) {
if (p.branches.empty()) {
return ".*";
}
return gbnf_excluding_grammar(builder, prefix, p.matcher, reachable.tokens);
}
return gbnf_excluding_grammar(builder, "until-" + std::to_string(id), p.delimiters);
std::vector<std::string> branch_rests;
for (size_t b = 0; b < p.branches.size(); b++) {
auto rest = p.branches[b].rest;
branch_rests.push_back(rest == COMMON_PEG_INVALID_PARSER_ID ? "" : builder.add_rule(prefix + "-rest-" + std::to_string(b), to_gbnf(rest)));
}
return gbnf_including_grammar(builder, prefix, p.matcher, reachable.tokens, branch_rests, p.optional);
} else if constexpr (std::is_same_v<T, common_peg_schema_parser>) {
if (schema_delegates(p)) {
return to_gbnf(p.child);
@@ -1766,38 +2173,52 @@ void common_peg_arena::build_grammar(const common_grammar_builder & builder, boo
return to_gbnf(p.child);
} else if constexpr (std::is_same_v<T, common_peg_gbnf_parser>) {
return p.grammar;
} else if constexpr (std::is_same_v<T, common_peg_ac_parser>) {
return gbnf_including_grammar(builder, "ac-" + std::to_string(id), p.delimiters);
} else {
static_assert(is_always_false_v<T>);
}
}, parser);
};
// Collect reachable rules
std::set<std::string> reachable_rules;
if (lazy) {
// Collect rules reachable from trigger rules
for (const auto & [name, id] : rules_) {
const auto & parser = parsers_.at(id);
if (auto rule = std::get_if<common_peg_rule_parser>(&parser)) {
if (rule->trigger) {
// Mark trigger as reachable and visit it
reachable_rules.insert(name);
auto add_rules = collect_reachable_rules(*this, id);
reachable_rules.insert(add_rules.begin(), add_rules.end());
}
// Trigger rules, sorted for a predictable order
std::vector<const common_peg_rule_parser *> trigger_rules;
for (const auto & [name, id] : rules_) {
if (auto rule = std::get_if<common_peg_rule_parser>(&parsers_.at(id))) {
if (rule->trigger) {
trigger_rules.push_back(rule);
}
}
}
std::sort(trigger_rules.begin(), trigger_rules.end(), [](const auto * a, const auto * b) { return a->name < b->name; });
// A lazy grammar scans for the starts when the trigger rules were built from a start and rest
bool scans = false;
for (const auto * rule : trigger_rules) {
scans = scans || rule->start != COMMON_PEG_INVALID_PARSER_ID;
}
if (scans) {
for (const auto * rule : trigger_rules) {
if (rule->start == COMMON_PEG_INVALID_PARSER_ID) {
throw std::runtime_error("trigger rule without a start in a scanning grammar: " + rule->name);
}
}
}
if (lazy && scans) {
for (const auto * rule : trigger_rules) {
collect_reachable(*this, rule->start, reachable);
collect_reachable(*this, rule->rest, reachable);
}
} else if (lazy) {
for (const auto * rule : trigger_rules) {
collect_reachable(*this, rules_.at(rule->name), reachable);
}
} else {
// Collect rules reachable from root
reachable_rules = collect_reachable_rules(*this, root_);
collect_reachable(*this, root_, reachable);
}
// Create GBNF rules for all reachable rules
for (const auto & [name, rule_id] : rules_) {
if (reachable_rules.find(name) == reachable_rules.end()) {
if (reachable.rules.find(name) == reachable.rules.end()) {
continue;
}
@@ -1807,20 +2228,27 @@ void common_peg_arena::build_grammar(const common_grammar_builder & builder, boo
}
}
if (lazy) {
// Generate root rule from trigger rules only
std::vector<std::string> trigger_names;
for (const auto & [name, rule_id] : rules_) {
const auto & parser = parsers_.at(rule_id);
if (auto rule = std::get_if<common_peg_rule_parser>(&parser)) {
if (rule->trigger) {
trigger_names.push_back(rule->name);
if (lazy && scans) {
// The root scans for a trigger start and then constrains its rest
common_trie starts;
std::vector<std::string> rests;
for (const auto * rule : trigger_rules) {
auto value = (int32_t) rests.size();
rests.push_back(builder.add_rule(rule->name + "-rest", to_gbnf(rule->rest)));
for (size_t end : insert_delimiter(starts, *this, rule->start)) {
auto owner = starts.mark(end, value);
if (owner != value) {
throw std::runtime_error("trigger rules " + trigger_rules[owner]->name + " and " + rule->name + " must not share a start");
}
}
}
// Sort for predictable order
std::sort(trigger_names.begin(), trigger_names.end());
builder.add_rule("root", gbnf_including_grammar(builder, "trigger", starts, reachable.tokens, rests, /* optional = */ true));
} else if (lazy) {
// Generate root rule from trigger rules only
std::vector<std::string> trigger_names;
for (const auto * rule : trigger_rules) {
trigger_names.push_back(rule->name);
}
builder.add_rule("root", string_join(trigger_names, " | "));
} else if (root_ != COMMON_PEG_INVALID_PARSER_ID) {
builder.add_rule("root", to_gbnf(root_));
+77 -17
View File
@@ -3,6 +3,7 @@
#include "json-schema.h"
#include "json.h"
#include "llama.h"
#include "trie.h"
#include <memory>
#include <set>
@@ -188,6 +189,22 @@ struct common_peg_recursion_error : std::invalid_argument {
using std::invalid_argument::invalid_argument;
};
struct common_peg_token {
std::string text;
llama_token_attr attr;
};
struct common_peg_tokens {
std::unordered_map<llama_token, common_peg_token> tokens;
std::unordered_map<std::string, llama_token> ids;
common_peg_tokens() = default;
explicit common_peg_tokens(const llama_vocab * vocab);
// LLAMA_TOKEN_NULL if text is not a special token
llama_token token_id(const std::string & text) const;
};
struct common_peg_parse_context {
std::string input; // [h, e, l, l, o, _, w, o, r, l, d]
std::vector<llama_token> tokens; // [id, -1, -1, -1, -1, id, -1, -1, -1, -1, -1]
@@ -226,6 +243,16 @@ struct common_peg_literal_parser {
std::string literal;
};
struct common_peg_token_parser {
llama_token token;
std::string piece;
};
// Matches one token that is not one of the tokens
struct common_peg_token_not_parser {
std::vector<llama_token> tokens;
};
struct common_peg_sequence_parser {
std::vector<common_peg_parser_id> children;
};
@@ -270,8 +297,18 @@ struct common_peg_string_parser {
char delimiter;
};
// A delimiter an until parser stops at, and the parser that follows it when the delimiter is consumed
struct common_peg_until_branch {
common_peg_parser_id delimiter;
common_peg_parser_id rest = COMMON_PEG_INVALID_PARSER_ID;
};
struct common_peg_until_parser {
std::vector<std::string> delimiters;
std::vector<common_peg_until_branch> branches;
bool include = false; // consume the first delimiter and parse the rest of its branch
bool optional = false; // with include, the input may end before any delimiter
common_peg_parser_id content = COMMON_PEG_INVALID_PARSER_ID; // with include, parsed up to the delimiter
common_trie matcher; // each pattern valued by its branch
};
struct common_peg_schema_parser {
@@ -288,6 +325,9 @@ struct common_peg_rule_parser {
std::string name;
common_peg_parser_id child;
bool trigger;
// set when the rule has a start and rest, the lazy grammar scans for the start
common_peg_parser_id start = COMMON_PEG_INVALID_PARSER_ID;
common_peg_parser_id rest = COMMON_PEG_INVALID_PARSER_ID;
};
struct common_peg_ref_parser {
@@ -308,17 +348,14 @@ struct common_peg_gbnf_parser {
std::string grammar;
};
struct common_peg_ac_parser {
common_peg_parser_id child;
std::vector<std::string> delimiters;
};
// Variant holding all parser types
using common_peg_parser_variant = std::variant<
common_peg_epsilon_parser,
common_peg_start_parser,
common_peg_end_parser,
common_peg_literal_parser,
common_peg_token_parser,
common_peg_token_not_parser,
common_peg_sequence_parser,
common_peg_choice_parser,
common_peg_repetition_parser,
@@ -334,14 +371,14 @@ using common_peg_parser_variant = std::variant<
common_peg_ref_parser,
common_peg_atomic_parser,
common_peg_tag_parser,
common_peg_gbnf_parser,
common_peg_ac_parser
common_peg_gbnf_parser
>;
class common_peg_arena {
std::vector<common_peg_parser_variant> parsers_;
std::unordered_map<std::string, common_peg_parser_id> rules_;
common_peg_parser_id root_ = COMMON_PEG_INVALID_PARSER_ID;
common_peg_tokens tokens_;
public:
const common_peg_parser_variant & get(common_peg_parser_id id) const { return parsers_.at(id); }
@@ -356,6 +393,8 @@ class common_peg_arena {
common_peg_parser_id root() const { return root_; }
void set_root(common_peg_parser_id id) { root_ = id; }
const common_peg_tokens & tokens() const { return tokens_; }
common_peg_parse_result parse(common_peg_parse_context & ctx, size_t start = 0) const;
common_peg_parse_result parse(common_peg_parser_id id, common_peg_parse_context & ctx, size_t start) const;
@@ -384,6 +423,7 @@ class common_peg_parser_builder {
public:
common_peg_parser_builder();
explicit common_peg_parser_builder(common_peg_tokens tokens);
// Match nothing, always succeed.
// S -> ε
@@ -401,6 +441,16 @@ class common_peg_parser_builder {
// S -> "hello"
common_peg_parser literal(const std::string & literal) { return add(common_peg_literal_parser{literal}); }
// Matches a token. Falls back to a literal if the token is not registered with the builder.
// S -> <[token-id]>
common_peg_parser token(const std::string & piece);
// Matches one of the tokens. With negate, matches one token that is not in the set.
// A negated set may only name tokens registered with the builder.
// S -> <[id1]> | <[id2]> | ... or !<[id1,id2,...]>
common_peg_parser token(const std::vector<std::string> & pieces, bool negate = false);
common_peg_parser token(std::initializer_list<std::string> pieces, bool negate = false) { return token(std::vector<std::string>(pieces), negate); }
// Matches a sequence of parsers in order, all must succeed.
// S -> A B C
common_peg_parser sequence() { return add(common_peg_sequence_parser{}); }
@@ -457,16 +507,29 @@ class common_peg_parser_builder {
// Matches all characters until a delimiter is found (delimiter not consumed).
// Invalid UTF-8 is consumed and recorded on the AST nodes.
// S -> (!delim .)*
common_peg_parser until(const std::string & delimiter) { return add(common_peg_until_parser{{delimiter}}); }
common_peg_parser until(const std::string & delimiter) { return until_one_of({delimiter}); }
// Matches all characters until one of the delimiters in the list is found (delimiter not consumed).
// S -> (!delim .)*
common_peg_parser until_one_of(const std::vector<std::string> & delimiters) { return add(common_peg_until_parser{delimiters}); }
common_peg_parser until_one_of(const std::vector<std::string> & delimiters);
// Matches all characters until the delimiter is found (delimiter not consumed).
// The delimiter may only be a sequence of literals, tokens, and choices of them. Tags and atomics may wrap them.
// S -> (!delim .)*
common_peg_parser until(const common_peg_parser & delimiter);
// Matches everything
// S -> .*
common_peg_parser rest() { return until_one_of({}); }
// Matches content, then the first delimiter, then the rest of the branch of that delimiter.
// Branches must not share a delimiter. With optional, the input may end before any delimiter.
// S -> C (D1 R1 | D2 R2 | ...)
common_peg_parser through(const common_peg_parser & content, const std::vector<common_peg_until_branch> & branches, bool optional = false);
common_peg_parser through(const common_peg_parser & content, std::initializer_list<common_peg_until_branch> branches, bool optional = false) { return through(content, std::vector<common_peg_until_branch>(branches), optional); }
common_peg_parser through(const common_peg_parser & content, const common_peg_parser & delimiter) { return through(content, { { delimiter } }); }
common_peg_parser through(const common_peg_parser & content, const std::string & delimiter) { return through(content, literal(delimiter)); }
// Matches between min and max repetitions of a parser (inclusive).
// S -> A{m,n}
// Use -1 for max to represent unbounded repetition (equivalent to {m,})
@@ -537,6 +600,10 @@ class common_peg_parser_builder {
common_peg_parser trigger_rule(const std::string & name, const common_peg_parser & p) { return rule(name, p, true); }
common_peg_parser trigger_rule(const std::string & name, const std::function<common_peg_parser()> & builder) { return rule(name, builder, true); }
// Creates a trigger rule that matches start followed by rest. A lazy grammar scans for the start of any such rule, then only allows its rest.
// The start has the same limits as an until() delimiter.
common_peg_parser trigger_rule(const std::string & name, const common_peg_parser & start, const common_peg_parser & rest);
// Creates an atomic parser. Atomic parsers do not create an AST node if
// the child results in a partial parse, i.e. NEEDS_MORE_INPUT. This is
// intended for situations where partial output is undesirable.
@@ -550,13 +617,6 @@ class common_peg_parser_builder {
// the child's grammar. Parsing delegates entirely to the child.
common_peg_parser gbnf(const common_peg_parser & p, const std::string & grammar) { return add(common_peg_gbnf_parser{p, grammar}); }
// Wraps a child parser but emits a GBNF grammar built from the Aho-Corasick
// automaton of `delimiters`, matching everything up to and including the
// first delimiter. Parsing delegates entirely to the child, which is
// responsible for consuming the delimiter (e.g. until(D) + literal(D)).
common_peg_parser ac(const common_peg_parser & p, const std::vector<std::string> & delimiters);
common_peg_parser ac(const common_peg_parser & p, const std::string & delimiter) { return ac(p, std::vector<std::string>{delimiter}); }
void set_root(const common_peg_parser & p);
common_peg_arena build();
+6 -2
View File
@@ -31,14 +31,18 @@ struct token_matcher {
static common_trie build_trie(const std::vector<llama_tokens> & seqs) {
common_trie t;
for (const auto & seq : seqs) {
t.insert(std::vector<uint32_t>(seq.begin(), seq.end()));
std::vector<common_trie::symbol> symbols;
for (llama_token token : seq) {
symbols.push_back(common_trie::symbol::token(token));
}
t.insert(symbols);
}
return t;
}
// returns the index into seqs of the longest sequence ending at this token, or -1
int32_t advance(llama_token token) {
state = ac.next(state, (uint32_t) token);
state = ac.next(state, common_trie::symbol::token(token));
const int32_t p = ac.match_pattern(state);
if (p >= 0) {
state = 0;
+54 -20
View File
@@ -4,26 +4,46 @@
#include <deque>
common_trie::step common_trie::next(std::string_view sv, const std::vector<int32_t> & token_map, size_t pos) const {
if (!tokens.empty() && pos < token_map.size()) {
const int32_t id = token_map[pos];
if (id >= 0 && tokens.find(id) != tokens.end()) {
size_t end = pos + 1;
while (end < token_map.size() && token_map[end] < 0) {
++end;
}
return { utf8_parse_result::SUCCESS, symbol::token(id), end - pos };
}
}
auto result = common_parse_utf8_codepoint(sv, pos);
return { result.status, symbol::codepoint(result.codepoint), result.bytes_consumed };
}
common_trie::match_result common_trie::check_at(std::string_view sv, size_t start_pos) const {
return check_at(sv, {}, start_pos);
}
common_trie::match_result common_trie::check_at(std::string_view sv, const std::vector<int32_t> & token_map, size_t start_pos) const {
size_t current = 0; // Start at root
size_t pos = start_pos;
// LOG_DBG("%s: checking at pos %zu, sv='%s'\n", __func__, start_pos, std::string(sv).c_str());
while (pos < sv.size()) {
auto result = common_parse_utf8_codepoint(sv, pos);
if (result.status != utf8_parse_result::SUCCESS) {
auto step = next(sv, token_map, pos);
if (step.status != utf8_parse_result::SUCCESS) {
break;
}
auto it = nodes[current].children.find(result.codepoint);
auto it = nodes[current].children.find(step.symbol);
if (it == nodes[current].children.end()) {
// Can't continue matching
return match_result{match_result::NO_MATCH};
}
current = it->second;
pos += result.bytes_consumed;
pos += step.bytes_consumed;
// Check if we've matched a complete word
if (nodes[current].pattern >= 0) {
@@ -42,7 +62,7 @@ common_trie::match_result common_trie::check_at(std::string_view sv, size_t star
}
int32_t common_trie::insert(const std::string & word) {
std::vector<uint32_t> symbols;
std::vector<symbol> symbols;
size_t pos = 0;
while (pos < word.length()) {
auto result = common_parse_utf8_codepoint(word, pos);
@@ -50,28 +70,42 @@ int32_t common_trie::insert(const std::string & word) {
break;
}
symbols.push_back(result.codepoint);
symbols.push_back(symbol::codepoint(result.codepoint));
pos += result.bytes_consumed;
}
return insert(symbols);
}
int32_t common_trie::insert(const std::vector<uint32_t> & symbols) {
int32_t common_trie::insert(const std::vector<symbol> & symbols) {
size_t current = 0;
for (uint32_t ch : symbols) {
auto it = nodes[current].children.find(ch);
if (it == nodes[current].children.end()) {
size_t child = create_node();
nodes[current].children[ch] = child;
current = child;
} else {
current = it->second;
}
for (symbol ch : symbols) {
current = extend(current, ch);
}
if (nodes[current].pattern < 0) {
nodes[current].pattern = n_patterns++;
int32_t pattern = mark(current, n_patterns);
if (pattern == n_patterns) {
n_patterns++;
}
return nodes[current].pattern;
return pattern;
}
size_t common_trie::extend(size_t node, symbol ch) {
if (ch.is_token()) {
tokens.insert((int32_t) ch.value);
}
auto it = nodes[node].children.find(ch);
if (it != nodes[node].children.end()) {
return it->second;
}
size_t child = create_node();
nodes[node].children[ch] = child;
return child;
}
int32_t common_trie::mark(size_t node, int32_t value) {
if (nodes[node].pattern < 0) {
nodes[node].pattern = value;
}
return nodes[node].pattern;
}
common_aho_corasick::common_aho_corasick(common_trie trie) : t(std::move(trie)) {
@@ -113,7 +147,7 @@ common_aho_corasick::common_aho_corasick(common_trie trie) : t(std::move(trie))
}
}
size_t common_aho_corasick::next(size_t state, uint32_t ch) const {
size_t common_aho_corasick::next(size_t state, common_trie::symbol ch) const {
const auto & nodes = t.nodes;
while (state && nodes[state].children.find(ch) == nodes[state].children.end()) {
state = fail[state];
+51 -10
View File
@@ -1,5 +1,7 @@
#pragma once
#include "unicode.h"
#include <cstdint>
#include <map>
#include <set>
@@ -10,13 +12,42 @@
// Trie for matching multiple literals.
// This is used in common_peg_until_parser and to build a GBNF exclusion grammar
struct common_trie {
// A symbol is either a UTF-8 codepoint or a token
struct symbol {
enum kind_type : uint8_t { CODEPOINT, TOKEN };
kind_type kind = CODEPOINT;
uint32_t value = 0;
static symbol codepoint(uint32_t cpt) { return { CODEPOINT, cpt }; }
static symbol token(int32_t id) { return { TOKEN, (uint32_t) id }; }
bool is_token() const { return kind == TOKEN; }
bool operator<(const symbol & other) const {
return kind != other.kind ? kind < other.kind : value < other.value;
}
bool operator==(const symbol & other) const {
return kind == other.kind && value == other.value;
}
};
struct step {
enum utf8_parse_result::status status;
common_trie::symbol symbol;
size_t bytes_consumed;
};
struct node {
std::map<uint32_t, size_t> children; // Use uint32_t to store Unicode codepoints
std::map<symbol, size_t> children;
int32_t pattern = -1; // index of the pattern ending at this node, -1 if none
};
std::vector<node> nodes;
// every token id used by a pattern, the input is split so these match as single symbols
std::set<int32_t> tokens;
common_trie() {
create_node(); // root node
}
@@ -29,15 +60,25 @@ struct common_trie {
enum match_result { NO_MATCH, PARTIAL_MATCH, COMPLETE_MATCH };
// Read the symbol at pos. It is a token if a pattern uses the token at pos, else a UTF-8 codepoint.
step next(std::string_view sv, const std::vector<int32_t> & token_map, size_t pos) const;
// Check if a delimiter starts at the given position
match_result check_at(std::string_view sv, size_t start_pos) const;
match_result check_at(std::string_view sv, const std::vector<int32_t> & token_map, size_t start_pos) const;
// Insert a word as a sequence of Unicode codepoints, returns its pattern index
int32_t insert(const std::string & word);
// Insert a raw symbol sequence, returns its pattern index (insertion order,
// duplicates keep the first index)
int32_t insert(const std::vector<uint32_t> & symbols);
// Insert a symbol sequence, returns its pattern index (insertion order, duplicates keep the first index)
int32_t insert(const std::vector<symbol> & symbols);
// Child of node on a symbol, created if missing
size_t extend(size_t node, symbol ch);
// Mark node as the end of a pattern with a value from the caller. Returns the value already there, or value if the node had none.
// A trie must take its values from mark or from insert, not both.
int32_t mark(size_t node, int32_t value);
private:
int32_t n_patterns = 0;
@@ -51,11 +92,11 @@ struct common_trie {
// Aho-Corasick automaton
struct common_aho_corasick {
common_trie t;
std::vector<size_t> fail; // failure links
std::vector<size_t> order; // states in BFS order
std::vector<int32_t> match; // longest pattern ending at each state (directly or via a suffix link), -1 if none
std::set<uint32_t> alphabet; // every character with a transition
common_trie t;
std::vector<size_t> fail; // failure links
std::vector<size_t> order; // states in BFS order
std::vector<int32_t> match; // longest pattern ending at each state (directly or via a suffix link), -1 if none
std::set<common_trie::symbol> alphabet; // every symbol with a transition
common_aho_corasick(common_trie trie);
@@ -69,5 +110,5 @@ struct common_aho_corasick {
int32_t match_pattern(size_t s) const { return match[s]; }
// follow failure links until a transition on `ch` exists.
size_t next(size_t state, uint32_t ch) const;
size_t next(size_t state, common_trie::symbol ch) const;
};
+131
View File
@@ -67,6 +67,8 @@ For a more complete example, see `test_example_native()` in
- **`end()`** - Matches the end of input (anchor `$`)
- **`literal(string)`** - Matches an exact literal string
- **`any()`** - Matches any single character (`.`)
- **`token(piece)`** - Matches a special token, or the literal `piece` if it is not registered with the builder (see [Special Tokens](#special-tokens))
- **`token(pieces, negate)`** - Matches one of the tokens, or with `negate` one token that is not one of them
### Combinators
@@ -89,6 +91,8 @@ For a more complete example, see `test_example_native()` in
- **`space()`** - Matches zero or more whitespace characters (space, tab, newline)
- **`until(delimiter)`** - Matches characters until delimiter is found (delimiter not consumed)
- **`until_one_of(delimiters)`** - Matches characters until any delimiter in the list is found
- **`until(parser)`** - Matches characters until a delimiter built from literals and tokens is found (delimiter not consumed, see [until(parser)](#untilparser))
- **`through(content, branches, optional)`** - Matches content, then the first delimiter found and the rest of its branch (delimiter consumed). The grammar scans for the delimiters instead of following content
- **`rest()`** - Matches everything remaining (`.*`)
### JSON Parsers
@@ -108,6 +112,7 @@ For a more complete example, see `test_example_native()` in
- **`ref(name)`** - Creates a lightweight reference to a named rule (for recursive grammars)
- **`rule(name, p, trigger)`** - Creates a named rule and returns a reference
- **`trigger_rule(name, p)`** - Creates a trigger rule (entry point for lazy grammar generation)
- **`trigger_rule(name, start, rest)`** - Creates a trigger rule matching `start` then `rest`, gating the lazy grammar on the start
- **`schema(p, name, schema, raw)`** - Wraps parser with JSON schema metadata for grammar generation
- **`schema(p, name, doc, node, raw)`** - Same, for a node of a `common_chat_schema_document` built earlier, e.g. one tool parameter
@@ -116,6 +121,110 @@ For a more complete example, see `test_example_native()` in
- **`atomic(p)`** - Prevents AST node creation for partial parses
- **`tag(tag, p)`** - Creates AST nodes with semantic tags (multiple nodes can share tags)
## Special Tokens
Matching a special token by its text alone is ambiguous: a model may emit
`<|end|>` as a single control token, or spell it out as plain text, such as
when quoting a chat template. The parser can tell them apart when the builder
knows the special tokens of the vocabulary, and the input carries the ids of
the tokens that produced it.
A `common_peg_tokens` table constructed from a `llama_vocab` registers every
control and user-defined token. Pass it to the builder:
```cpp
common_peg_tokens specials(vocab);
auto parser = build_chat_peg_parser(specials, [&](common_chat_peg_builder & p) {
return p.content(p.until(p.token("<|end|>"))) + p.token("<|end|>");
});
```
Chat templates initialized with a model build this table from its vocabulary,
and format handlers receive it as `inputs.specials`.
### token()
`token(piece)` matches only where the input holds the id of that token, and
consumes the whole piece. The same characters without the id do not match.
If `piece` is not registered with the builder, it falls back to
`literal(piece)`.
`token(pieces)` matches any of the tokens, and `token(pieces, true)` matches one
token that is not in the set. Every piece of a negated set must be registered,
otherwise it throws `std::invalid_argument`.
In the grammar, a token emits `<[id]>` and a negated set emits `!<[...]>`:
```cpp
p.token({ "<a>", "<b>" }) + p.token({ "<a>", "<b>" }, true)
```
```
root ::= (<[1]> | <[2]>) !<[1-2]>
```
### until(parser)
`until(delimiter)` with a parser instead of a string stops at a delimiter that
may contain tokens, or several literals joined in a sequence:
```cpp
auto end = p.token("<|end|>");
auto final = p.token("<|channel|>") + p.literal("final");
p.reasoning(p.until(end | final))
```
The delimiter may only be built from:
- `literal()` and `token()`
- `sequence()` and `choice()` of the above
- `tag()` and `atomic()` wrapping any of the above
Any other parser throws `std::invalid_argument`. A choice inside a sequence
expands to every combination, so `"a" + ("b" | "d")` stops at both `ab` and
`ad`.
The grammar scans for the delimiters the same way as `until_one_of()`:
```cpp
p.until((p.literal("a") + p.literal("b")) | p.literal("cd"))
```
```
root ::= until-5
until-5 ::= | [a] until-5-01 | [c] until-5-03 | [^ac] until-5
until-5-01 ::= | [a] until-5-01 | [c] until-5-03 | [^abc] until-5
until-5-03 ::= | [a] until-5-01 | [c] until-5-03 | [^acd] until-5
```
A token never matches a character rule such as `.` or `[^a]`, so the scan lets
the other tokens named elsewhere in the grammar through by name. Here `<b>` is
token 2:
```cpp
p.until(p.token("<a>") + p.literal("x")) + p.token("<a>") + p.literal("x") + p.token("<b>")
```
```
root ::= until-3 <[1]> "x" <[2]>
until-3 ::= | <[1]> until-3-01 | (. | <[2]>) until-3
until-3-01 ::= | <[1]> until-3-01 | ([^x] | <[2]>) until-3
```
When the delimiters are made of tokens alone, the scan consumes whole tokens
with a negated token set instead:
```cpp
p.until(p.token("<a>")) + p.token("<a>")
```
```
root ::= until-1 <[1]>
until-1 ::= | !<[1]> until-1
```
## GBNF Grammar Generation
The PEG parser also acts as a convenient DSL for generating GBNF grammars, with
@@ -143,6 +252,28 @@ are emitted in the grammar. All trigger rules are added as alternations in the
root rule. It is still necessary to define trigger patterns, as the parser has
no interaction with the grammar sampling.
With `trigger_rule(name, start, rest)`, the grammar instead scans for a start and
then constrains the rest. It runs from the first token, so `grammar_lazy` is false
and no trigger patterns are needed. For example:
```cpp
auto ab = p.literal("ab");
auto cd = p.literal("cd");
return p.until(ab | cd) + (p.trigger_rule("one", ab, p.literal("x")) | p.trigger_rule("two", cd, p.literal("y")));
```
```
root ::= trigger
trigger ::= | [a] trigger-01 | [c] trigger-03 | [^ac] trigger
trigger-01 ::= | [b] one-rest | [a] trigger-01 | [c] trigger-03 | [^abc] trigger
trigger-03 ::= | [d] two-rest | [a] trigger-01 | [c] trigger-03 | [^acd] trigger
one-rest ::= "x"
two-rest ::= "y"
```
A start may only be a sequence of literals, tokens, and choices of them, possibly
wrapped in tags and atomics.
### JSON Schema
The `schema(p, name, schema, raw)` parser will use the `json-schema-to-grammar`
+780
View File
@@ -0,0 +1,780 @@
# Fork rebase and deploy runbook (RX 580 fork)
Step-by-step procedure for moving the fork onto a new upstream and putting it into production, as it
has evolved over the 2026-09-09, 2026-10-09, 2026-10-24 and 2026-11-21 rebases. Read [FORK.md](../FORK.md) first:
it lists every fork feature with its invariants and rebase hazards, and this runbook refers to it.
Ground rules:
- **Never build on the workstation.** All builds run on the RX 580 pod. Locally, rely on careful reading.
- **Never resolve a conflict with `-X theirs` or `-X ours`.** Resolve every hunk by hand.
- **"Applied cleanly" is not "correct".** Re-read every clean pick against the new upstream.
- **Never `pkill -f` on the pod.** It matches the `kubectl exec` shell and kills it. Kill by explicit PID.
- **Never leave the pod not serving.** Anything that stops llama-swap must restart it from a trap.
- Keep the previous production branch untouched as the safety net. Do not push for the owner; the owner
reviews and pushes.
Contents: 1. Survey; 2. Trial rebase; 3. The real rebase; 4. Checks without a compiler; 5. Build on
the pod; 6. Validate; 7. Deploy; 8. Post-deploy checks; 9. Rollback; 10. Recovering a hung GPU;
11. Operations reference.
**Redeploying a new tip on the same upstream base (no rebase)**, e.g. new fork commits on the
production branch (2026-10-25: `6c480f7e7` -> `2fc9f9c35`, same branch, same base):
- Skip sections 1-3. Of section 4 only the ASCII check (on `git diff <prod>..<tip>`) and a careful
read of that diff apply; `<prod>` is `git -C /root/llama.cpp rev-parse HEAD` on the pod. There is
nothing to range-diff.
- Do sections 5-9 as written: build the tip in a scratch worktree, validate it with `--ref` = the
production build, then deploy.
- The branch name stays the same, so deploy step 2's `git checkout -B <same branch> <tip>` moves the
production branch on the pod to the tip. After that the rollback pointer file is the only record
of the old production commit on the pod, so write it first (deploy step 1).
---
## 1. Survey upstream
Local remotes: `origin` is upstream (github.com/ggerganov/llama.cpp), `gitea` is the fork
(git@gitea.lumpiasty.xyz:Lumpiasty/llama.cpp.git, SSH push from the workstation). On the pod,
`origin` is the same gitea repository over HTTPS (fetch only).
```sh
git fetch origin
OLD_BASE=$(git merge-base origin/master HEAD) # the old upstream base (still an ancestor of the new master)
OLD_TIP=$(git rev-parse HEAD)
git log --oneline "$OLD_BASE"..HEAD # the fork commits, oldest last
git log --oneline "$OLD_BASE"..origin/master | wc -l # size of the upstream delta
```
Then look at upstream changes in the areas FORK.md lists as hazards:
```sh
git log --oneline "$OLD_BASE"..origin/master -- \
src/llama-context.cpp src/llama-context.h src/llama-model.cpp src/llama-model.h \
src/llama-kv-cache.cpp src/llama-kv-cache.h src/llama-kv-cache-iswa.cpp \
src/llama-memory.h src/llama-memory-recurrent.cpp src/llama-memory-hybrid.cpp \
src/llama-moe-cache.cpp src/llama-moe-cache.h src/models/delta-net-base.cpp \
ggml/src/ggml-vulkan/ ggml/src/ggml-backend.cpp ggml/src/ggml-backend-impl.h ggml/include/ggml-backend.h \
tools/server/server-queue.cpp tools/server/server-queue.h tools/server/server-context.cpp tools/server/server.cpp \
tools/mtmd/clip.cpp tools/mtmd/mtmd.cpp common/arg.cpp tools/llama-bench/
```
**Temporary upstream carries (FORK.md F12)** come first: `git log --oneline --grep '^upstream-carry'
"$OLD_BASE"..HEAD`. For each: merged into the new upstream -> do not cherry-pick it (check that the
merged version is the carried head or newer); still open -> re-fetch the PR head and re-carry it if
it moved; closed -> ask the owner. A PR head can also shrink: on 2026-11-22 #32740 dropped the Vulkan part the fork needs, so the old
head stayed carried (FORK.md F12). Compare `git diff --stat` of the old and new head before re-carrying.
To carry a PR (2026-11-21: #32735, #32740; 2026-11-22: #32777):
```sh
git fetch https://github.com/ggml-org/llama.cpp pull/<N>/head:pr-<N>
git cherry-pick $(git merge-base pr-<N> HEAD)..pr-<N> # the PR's commits, oldest first; resolve by hand
# squash them into one commit with the PR author, subject "upstream-carry #<N>: <PR title>", and a
# body with the PR URL, the carried head and any conflict resolution (git commit-tree on the final tree)
diff <(git diff $(git merge-base pr-<N> HEAD) pr-<N> | grep '^[+-]') <(git diff HEAD~1 HEAD | grep '^[+-]')
```
The last diff shows only the lines you changed while resolving conflicts (for #32740: none, the F2
line differs only in context). Validate a carry like any compiled change (sections 5-6), plus the
PR's own tests.
For each fork commit decide: still needed, needs a semantic merge, or obsolete. Examples of past
decisions:
- 2026-09-09: dropped the weight-pinning chain, the expert prefetch and fractional `-ncmoe` (all measured
losses or no win on this card, FORK.md section 4).
- 2026-10-09: the routing-ids readback skip had nothing to apply to (upstream #29943 deleted its block
in `ggml-backend.cpp`); it was re-authored in `llama_context::sched_copy_experts`. Upstream's new MoE
cache (#29887) needed a new fork commit so the VRAM arbiter releases and invalidates it.
- Also check CLI changes that break production config or the scripts (upstream #28334 removed
`--no-mmap`/`--mmap`/`--mlock` and llama-bench `-mmp`; the default `--port` changed to 9931).
## 2. Trial rebase in a scratch worktree
Measure the conflicts before doing the real thing:
```sh
git worktree add /tmp/trial-rebase origin/master
cd /tmp/trial-rebase
git cherry-pick "$OLD_BASE".."$OLD_TIP" # stop at each conflict, note file + hunk, then:
git cherry-pick --abort # or resolve roughly to see the next one
cd - && git worktree remove --force /tmp/trial-rebase
```
Write down, per commit: clean / conflict (which file, which hunk, what both sides did). That list
becomes the brief for the real rebase.
## 3. The real rebase
```sh
NEW=fable5/vram-ondemand-rebase-$(date +%Y%m%d)
git switch -c "$NEW" origin/master
git cherry-pick <first-fork-commit> # one at a time, in the original order
```
**A rebase is not done until `fork-validate.sh` phases 1 and 2 (section 6) pass on a pod build of
the new branch.** A rebased branch that is only picked, checked by reading, or built is unfinished
work, whatever the cherry-picks looked like. Report it as unfinished until both phases pass.
Rules:
- Cherry-pick in the original order, keep authors and messages. New commits you author get the
fork's trailer (`Co-Authored-By: Claude ...` for agent-written commits, FORK.md status block),
unless the owner asks for something else.
- Resolve every conflict by hand. Union both sides where both added code; keep upstream's new terms.
- Fold each fix into the commit it belongs to so every commit builds on its own:
`git commit --fixup=<sha>` then `GIT_SEQUENCE_EDITOR=: git rebase -i --autosquash origin/master`.
- Leave the previous production branch untouched.
**What `-X theirs` silently broke in a trial (2026-10-09).** It "applied" every commit and produced
broken code:
- resurrected the scheduler readback block upstream had deleted from `ggml-backend.cpp`;
- dropped `!use_sparse` from the GCN `use_mask_opt` condition;
- dropped upstream's `copy_experts_info` (compile error);
- dropped upstream's `is_empty()` in the recurrent memory;
- dropped upstream's MoE-cache memory breakdown block;
- left a static/extern mismatch with the new `ggml-vulkan-common.h` declarations (compile error).
The correct resolutions in that rebase: keep upstream's `n_embd` width check plus our decode
auto-restore; `copy_experts_info` plus our `kv_device_evicted`; `is_empty()` plus our `dev_shadows`;
upstream's `if (moe_cache)` breakdown block outside our `if (sched)` guard; make `ggml_vk_concat` and
`ggml_backend_vk_free` non-static to match the new header while our helpers stay static.
**Clean picks that were wrong (2026-10-24).**
- Concat fast path (3df719e1e): applied cleanly, did not compile. Upstream `733f57a99` renamed
`pipeline_cpy_f32_f32`/`pipeline_cpy_f16_f16` to `pipeline_cpy_32`/`pipeline_cpy_16` (bitwise copies,
same push constants).
- MoE cache release/restore (7ac2014a8): applied cleanly, did not compile (upstream `33d6135cb` changed
LRU to SLRU, `g.lru` -> `g.slru`) and had a runtime bug: `moe_cache_slru::init()` resets only the
probation segment, so re-initialising a used cache corrupts its lists. `invalidate()` now assigns a
fresh `moe_cache_slru()` first.
- VRAM sharing core (481be8b28): besides its one conflict, upstream's new `load_tensors_hadamard()`
pushes two more entries into `pimpl->ctxs_bufs`. Without matching `weight_release.emplace_back()`
calls, `release_device_weights()` reads past the end of `weight_release`. Nothing conflicted.
- Conflicts that round: `ctx_buf_maps.emplace_back` replaced upstream by `ctx_bufs.emplace_back(ctx,
std::move(buf_ptrs))` (keep upstream's line, our release-slot block before it); `llama-context.h`
(keep upstream's `n_reallocs_same/diff` beside our `kv_device_evicted`); `recv_with_timeout`
(upstream `4e201388c` uses a fixed deadline: keep our order "check `!running`, return null", then
`wait_until(lock, deadline)`).
**What the 2026-11-21 rebase (upstream `e6d624b9a`, 757 commits past `3f03b1772`) changed** (nothing
built or run locally; every item below is by reading only and needs the pod build):
- Upstream refactors that moved fork code (hand ports, not conflicts of one hunk):
- `clip.cpp`/`clip.h` are gone (#31865). The encoder release/restore (F5) is now
`mtmd_model_release_device`/`mtmd_model_restore_device` on `mtmd_model` (`mtmd-model.cpp`), called
from `mtmd_release_device`/`mtmd_restore_device` under `compute_mutex` after syncing the shared
scheduler; the page-cache drop of F10 moved to `mtmd_model_loader::load_tensors`.
- `server_prompt_cache` moved to `server-slot-sched.{h,cpp}` and `get_available_slot()` was replaced
by `server_slot_sched::select()`. The disk prompt cache (F11) lives there now
(`server_prompt_disk_cache`), is called from the task dispatch after `update_prompt_cache()`, and
uses `common_state_data` and `shared_ptr<const common_prompt_checkpoint>`. Only prompts made with
the server lora config are saved, and a restored prompt gets that config in `prompt.lora`.
- Upstream's `llama_memory_bufs` replaced the raw `ctxs_bufs` vectors of the KV cache and the recurrent
memory: the null-buffer guard moved into `llama_memory_bufs::memory_breakdown()`, and the release
code indexes `ctxs_bufs.bufs[i]`. Those two did not compile after a clean pick.
- The MoE cache is now a ClassLRU (`moe_cache_lru`, `g.lru`) with `no_alloc` as a constructor
argument and `cpu_share`. `invalidate()` rebuilds `g.lru`, refills every slot map with
`g.lru.empty` (not -1), clears the CPU maps and `cpu_experts`.
- `server.cpp` was split into `server_run_instance` and friends: the self-pipe shutdown thread
(F7) is started in `server_run_instance` only, just before `server_install_signal_handlers()`.
- `server_queue` got `n_inflight` (#32133). `should_sleep` checks it first, so a response that is still
open also blocks the want-lock yield. This is a decision, not an accident: check it with `--arbiter`.
- Upstream added its own tiled-transpose concat (#30591) with no size gate. The fork path still runs
first (`ne0 >= 32`); the gate and the `GGML_VK_CONCAT_FASTPATH` knob were added to upstream's selector
(`ggml_vk_concat_is_transposed`), otherwise decode would take upstream's tiled path again (F3).
- Memory types with no release override (stay in VRAM when cold, not used in production):
`llama_kv_cache_dsa`, `_dsa_iswa`, `_dsv4`, `_msa`, `llama_memory_hybrid_iswa`. The KV rotation
tensors (`attn_rot_bufs`, #31073) are small and are not released either.
**What the 2026-11-22 rebase (upstream `5bacb7c86`, 19 commits past `e6d624b9a`) changed.** Branch
`fable5/vram-ondemand-rebase-20261122`. The 64 picks (65 fork commits minus the #32735 carry) all
applied cleanly; the range-diff against the old branch is identical except for that dropped carry and
the new commits at the end. Reading checks:
- #32735 merged as `8a97fa98a`, patch identical to the carried head `0ea4ffd00`: carry dropped.
- #32740 is still open but was split: its head `8c6f0c0ca` is now only the GGML contract (op API,
CPU reference, test-backend-ops cases); the Vulkan kernel, the `kv_rows` support and the Inkling
graph use were removed from it and are in no open PR (Metal is #32808). Carrying that head would
drop Inkling's in-kernel bias (back to the 15.8 GiB compute buffer at ctx 65536, FORK.md F12). The
validated carry of head `808fd1c4e` is kept as is (FORK.md F12 has the owner decision).
- #32807 (`5bacb7c86`) adds an `LLM_ARCH_INKLING2_LOOP` guard to `sched_copy_experts` (whole-tensor
copies), upstream's fix for #32755. It sits beside F1 in the same function; no conflict. It does
not remove the held per-pass copies, so F13 stays (FORK.md F13, "Upstream #32807").
- #32713 (non-causal prompts fit in one ubatch) touches `update_slots` near the F14 loop but not
the user-start break; F14 unchanged by it.
- #32799 (`ignore_merges` for inkling2) changes Inkling's tokenization: Inkling prompts cached by
the previous build no longer match. The disk cache key holds the build commit, so nothing stale is
restored.
- New commits: the F14 first-user-message break (FORK.md F14), the `LLAMA_LOOP_FULL_COPY` knob
(F13) and the carry of #32777 (F12).
Open questions from the 2026-11-21 rebase:
- Is the fork's own concat fast path (F3) now redundant with upstream's tiled concat (#30591)?
Benchmark `GGML_VK_CONCAT_FASTPATH=0` against the default on pp and decode before deciding.
- Does `n_inflight` blocking the want-lock yield delay handoffs? Validation says no: handoff median
candidate 692 ms vs reference 1068 ms (`a-handoff-latency`, 2026-11-21).
**Semantic re-checks to do for every rebase, clean or not** (details in FORK.md):
- F1: `sched_copy_experts` and its `st` struct unchanged in meaning.
- F2: `use_mask_opt` still contains upstream's terms plus our `>= 256` (and `!bias &&` from #32740, carried as F12 or merged upstream).
- F3: `build_conv_state` still emits `ggml_concat(conv_states, ggml_transpose(qkv_mixed), 0)` with
`qkv_mixed` contiguous `[ch, n_seq_tokens, n_seqs]`; pipeline names; `types_ok` still excludes
quantized types.
- F4: every `ctxs_bufs` push in `src/llama-model.cpp` has a `weight_release` push
(`git grep -n "ctxs_bufs.emplace_back\|weight_release" src/llama-model.cpp`); list memory types
(`git grep -n "public llama_memory_i" src/`) and note any without a release override; MoE-cache
internals; `sched_reserve()` still goes through `sched_new()` and re-registers callbacks;
`free_scratch` still the last `ggml_backend_i` member; Vulkan prealloc fields match `ggml_vk_cleanup`.
- F6/F7: `server_queue::start_loop` still evaluates `should_sleep()` only from the idle branch, never
inside `yield_to_queue()`; `handle_sleeping_state`, `server_res_generator`, `load_model` ordering;
signal setup in `server.cpp`.
- F8: new iterators over `ctxs_bufs` or users of `sched` on teardown paths.
- Scripts: flag spellings used by `scripts/rx580-bench/*` and `scripts/fork-validate.sh`, and the log
lines `fork-validate.sh` greps for (FORK.md F9).
- Production config: every flag in `/root/config.yaml` still exists in `common/arg.cpp`.
## 4. Checks without a compiler
Before handing the branch over for a build:
```sh
git diff origin/master..HEAD | grep -n '^[+-]\(<<<<<<<\|=======\|>>>>>>>\)' # no conflict markers
git diff origin/master..HEAD | grep -nP '^\+.*[^\x00-\x7F]' # ASCII only in added lines
git range-diff "$OLD_BASE".."$OLD_TIP" origin/master..HEAD # patch vs patch
```
- The range-diff must show only the edits you made on purpose. In 2026-10-24 the whole patch was
identical to the old one except for 4 deliberate edits.
- Brace balance per changed file: the change in `{`/`}` balance from new base to new tip should equal
the change from old base to old tip.
- Every identifier in added lines must exist on the new master or be defined by the fork. This catches
a renamed function but not a renamed member (like `lru` -> `slru`); re-read member accesses by hand.
- Write down every spot where you would normally lean on the compiler and hand that list to whoever
builds.
Then the owner pushes the branch to gitea, and the pod fetches it from there (section 5).
**If you were told not to push**, do not stop at an unbuilt branch. Unless the task says otherwise, a
rebase task continues through the build and validation (sections 5 and 6) and reports the results;
the owner pushes afterwards. Get the branch onto the pod with a git bundle:
```sh
# workstation; <base> is a commit the pod already has, e.g. production's HEAD
git bundle create /tmp/x.bundle <base>..<branch>
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
kubectl -n llama exec -i deploy/supervisord -- sh -c 'cat > /root/x.bundle' < /tmp/x.bundle
kubectl -n llama exec deploy/supervisord -- sh -c 'cd /root/llama.cpp && git fetch /root/x.bundle <branch>:<branch>'
```
The branch is then local in `/root/llama.cpp`: in section 5 skip the `git fetch origin` and use
`git worktree add "$W" <branch>`. Delete `/root/x.bundle` afterwards (disk is tight). Deploy step 2
likewise uses the local branch instead of fetching from origin.
## 5. Build on the pod, in a scratch worktree
Production keeps serving during this step.
```sh
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
kubectl -n llama exec -i deploy/supervisord -- sh -c '
set -e
B=fable5/vram-ondemand-rebase-YYYYMMDD
W=/root/arms/rebYYYYMMDD
git -C /root/llama.cpp fetch origin "$B"
git -C /root/llama.cpp worktree add "$W" FETCH_HEAD
cd "$W"
export HF_UI_VERSION=b11906 # production's: cut -d'|' -f2 /root/llama.cpp/build/tools/ui/.ui-stamp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DGGML_VULKAN=1 -DGGML_NATIVE=ON -DLLAMA_BUILD_UI=OFF
nice -n 19 cmake --build build --target llama-server llama-bench test-backend-ops -j 8
'
```
- Build only the targets you need; the full `all` target fails on the UI assets (needs npm/network).
If a tree has a stale `tools/ui/dist` with a stub `index.html`, remove it (it breaks llama-ui-embed).
- `nice -n 19 -j 8` leaves the serving models CPU. A full build takes about 3.5 minutes.
- Report compile errors verbatim against the list from step 4.
- Do not overlap a build with a large benchmark (host OOM). Disk is tight: `/root` had about 13 GB
free (97% full) on 2026-10-25, so check `df -h /root` first. Each build tree is about 700 MB;
remove scratch worktrees you no longer need with `git -C /root/llama.cpp worktree remove <dir>`.
- `-DLLAMA_BUILD_UI=OFF` does NOT make the UI asset step harmless. The `llama-ui-assets` step still
runs, and when the build number has no HF tag it falls back to `latest`. On 2026-11-21 `latest`
broke the build (missing `bundle[hash].js`, `.css` and `workbox[hash].js`). Set
`HF_UI_VERSION=<stamp>` for both `cmake -B build` and `cmake --build`, where the stamp is
production's current one in `build/tools/ui/.ui-stamp` (b11906 on 2026-11-21). The file holds
`ggml-org/llama-ui|b11906` and CMake requires `^[A-Za-z0-9._-]+$`, so take only the tag with
`cut -d'|' -f2 build/tools/ui/.ui-stamp`; never pass `$(cat ...)`. The validated
worktree's UI assets may then differ from the deployed ones (the code is identical).
- Long runs: launch detached (`setsid nohup sh X > X.log 2>&1 </dev/null &`) and poll, rather than
holding a `kubectl exec` open for hours.
- The validated build must contain all three targets: `llama-server`, `llama-bench` and
`test-backend-ops` (`--perf` and `--ops` run the last two from `--build`). A build script that
makes only `llama-server` (as the 2026-10-25 `diskcache-build.sh` first did) is not a validated
build.
- Detach only one script. Upload it first (`kubectl exec -i ... sh -c 'cat > /root/arms/X.sh' <
X.sh`), then start it alone: `setsid nohup sh /root/arms/X.sh > X.log 2>&1 </dev/null &`. Do not
background a chain such as `cat > X && sh -n X && setsid ... &` inside the `kubectl exec` shell. The
chain is a child of that session, and it can die with the session before `setsid` runs.
## 6. Validate with `scripts/fork-validate.sh`
Copy the script from the candidate tree (it is in the repo) and run it in two phases. The header of
the script describes each check.
Rule: any test of a production model must use production's exact command line: the `cmd` of its
`/root/config.yaml` entry with the macros expanded, including the mmproj that `-hf` pulls (production
never passes `--no-mmproj` to the 35B or gemma), and supervisord's `environment=`. Change only
`--port`, the arena and scratch paths. The 2026-10-25 deploy of the disk prompt cache aborted on the
first request: every test had used `--no-mmproj`, which hid a `GGML_ASSERT(!has_mtmd)`, and it was
rolled back (FORK.md F11).
```sh
# phase 1, beside production
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build --ops --concat
```
Add `--arbiter` to phase 1 only if the card has >= 1536 MiB free (8192 MiB minus the bytes in
`cat /sys/class/drm/card*/device/mem_info_vram_used`); below that it FAILs at once. With gemma-26B
warm in production only about 1.2 GiB is free and nothing frees it without downtime; then run
`--arbiter` in the phase-2 swapdown window instead (about 2 minutes more downtime), as below.
The same applies to `--concat`'s `candidate-timing` gate: with the card nearly full (2026-11-22: 956
MiB free beside a warm Inkling) every concat arm ran about 2x slower and the fast path showed no
gain (off/cand 0.9x, FAIL); in the swapdown window the same build passed with 5.1x. A phase-1
`candidate-timing` FAIL with little free VRAM is not a verdict: rerun `--concat` in phase 2.
**Page cache before and after phase 2.** The `--ref` build may lack F10 (FORK.md), and `--perf`
loads the 35B many times: during a reference arm on 2026-10-25 MemAvailable fell to 27 GB. Run the
eviction snippet of deploy step 3 before phase 2 and again after it (swapdown does not), and check
`/proc/meminfo`.
```sh
# phase 2, with llama-swap stopped: run it under the swapdown wrapper, whose trap restarts llama-swap
cat > /root/arms/fv-perf.sh <<'EOF'
HUB=/root/.cache/huggingface/hub
Q35=$HUB/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
G26=$HUB/models--SC117--gemma-4-26B-A4B-it-qat-heretic-GGUF/snapshots/e6759b29958180a998f62f991e4c741fe9b4d452/gemma-4-26B-A4B-it-qat-heretic-UD-Q4_K_XL.gguf
bash /root/arms/rebYYYYMMDD/scripts/fork-validate.sh \
--build /root/arms/rebYYYYMMDD/build --ref /root/llama.cpp/build \
--arbiter --perf --moe-exact "$Q35" --moecache "$G26" --keep
EOF
setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/fv-perf.log 2>&1 </dev/null &
```
- `G26` is the production SC117 gemma-26B, the `refs/main` snapshot (`e6759b29...` on 2026-10-25;
an older snapshot `82c751632...` is also on disk). If it moves, read the snapshot name from
`$HUB/models--SC117--gemma-4-26B-A4B-it-qat-heretic-GGUF/refs/main`.
- `/root/bench/moe/swapdown.sh <script>` (pod-only helper, not in the repo) stops llama-swap, waits for
an idle GPU, runs the script, and restarts llama-swap from an EXIT/INT/TERM/HUP trap. It prints
`SWAPDOWN_DONE` at the end. Afterwards confirm llama-swap is RUNNING and `/v1/models` returns 200.
- The `--concat` check needs a build with the `GGML_VK_CONCAT_FASTPATH` knob (in the fork since
e085c347d, so a rebased candidate has it; older candidates need `--concat-build DIR`).
- `--moecache` needs a cache that actually hits (default 2048 MiB; at 256 MiB gemma-26B had 0 hits,
which proves nothing).
- `--arbiter` also checks that a cold server really frees its VRAM (`v-vram-release`,
`v-vram-restore`; SKIPs if production's VRAM moved during the measurement, then rerun) and A/Bs
ordinary handoff latency against `--ref` (`a-handoff-latency`). `--ref` must be the build currently
in production, so both comparisons are against what serves today.
- Phase 2 downtime, measured 2026-10-25: about 32 minutes in all (`--arbiter` about 2 min, `--perf`
about 28 min, `--moecache` under 30 s, `--moe-exact` 78 s). `--moe-exact` proves the routing-ids
readback skip exact (F1): the candidate's greedy output equals the reference's and, with the
`LLAMA_MOE_IDS_SKIP` knob (any build of this branch), the candidate's with the skip forced off.
- `--perf` gates on medians of all repetition samples (since 2512290f9); the samples are in
`<scratch>/perf.tsv`, and `fork-validate.sh --perf-report <file>` re-aggregates them offline. A pod
restart empties `/tmp`, so copy a scratch dir you want to keep to `/root/arms/` (the 2026-10-24
scratch dirs were lost that way).
- Overlayfs (since the persistent rootfs, 2026-11-22): `/tmp` is on overlayfs, where `stat`
reports another device than `/proc/locks` prints, so a scratch arena under `/tmp` made every
token check fail (`arbiter/start: A serves but does not hold token.lock` while A held the lock).
`fork-validate.sh` from `70d39bdbc` on keeps its arenas on `/dev/shm`. Use the script of that
commit or later, also for a `--ref` run of an older build.
- Phase 1 of the 2026-11-22 rebase (candidate build `/root/arms/reb20261122/build`, code
`6c895a491`, scripts `70d39bdbc`, vs production `f3397cded`, beside production with 3579 MiB
free; logs `/root/arms/reb20261122-p1.log`, `-p1b.log`): `--ops --concat --diskcache` 23 PASS
(ops 5/5 with FLASH_ATTN_EXT 5693/5693, concat off/cand 5.3x, diskcache 15/15 including the GPU
legs), `--arbiter` 16 PASS on the rerun with the fixed script (handoff median cand 789.5 / ref
768 ms; the first run's single FAIL was the overlayfs mismatch above). F12: test-backend-ops
SCALE and FLASH_ATTN_EXT OK, `test-flash-attn-indexed-bias` 16 Vulkan0 cases, max error 1.3e-4.
Still to run in the window: phase 2 (`--arbiter --perf --moe-exact --moecache`, the runbook
command with this build path) and the Inkling/F14 items below.
- What the 2026-11-22 window must run, in this order (GPU to itself under swapdown; about 75-90
minutes in all):
1. Phase 2 as written above with `W=/root/arms/reb20261122` (about 32 minutes).
2. F13 vs upstream's #32807 (FORK.md F13, "Window A/B to run": memory, `kld48.sh`, speed with
`LLAMA_LOOP_HOST_VIEWS=0` and `LLAMA_LOOP_FULL_COPY=0`).
3. `--fit` vs the k=4 `-ot` for Inkling (FORK.md F12, "How to use it for Inkling"). Smoke run
beside production (3.5 GiB free, so not a placement result; `/root/arms/reb20261122-fit-smoke.log`):
fit's loop path runs on Inkling; the probing took 27.8 s, which every Inkling cold start would
pay with `--fit` in the config (freeze the printed `-ot` instead); the projected memory is not
monotonic in `n_layer` (5173 MiB at 44, 6736 at 43) and fit still reported success with -3116
MiB free. Check both with the full card.
4. F14 refinement (FORK.md F14, validation plan items 5-7): the new-chat checkpoint, its disk-cache
variant and the M-only TTFT table, with the 35B think entry's exact command line.
5. The Inkling post-deploy items of section 8 step 8 against a scratch server of the candidate:
#32799 changed Inkling's tokenization, so judge the answer text.
- F14 run (2026-11-22, candidate `f3397cded` = `166ec22e2` (docs on `b13c41355`) + F14, server-only change; one
swapdown window, `/root/arms/usw-window.sh`, log `/root/arms/usw-window.log`, about 15 minutes):
the 35B TTFT A/B with and without `LLAMA_USERSTART_MERGE=0` (FORK.md F14, M U U M, production 35B
think command line with the mmproj), `--diskcache` 15 PASS, 0 FAIL, 0 SKIP, and the
`diskcache-window3.sh` 35B legs (all passed). The full `--perf`/`--arbiter` set was not rerun: the
change touches only the server prompt-batch loop. Deploy downtime 33 s (incremental in-place
build).
- Last full run (2026-11-22, candidate `b13c41355` = `3621025e8` + F13 vs reference `3621025e8`, one
swapdown window, `--ops --concat --diskcache --arbiter --perf --moe-exact --moecache`, log
`/root/arms/overnight/fv.log`): 52 PASS, 0 FAIL, 0 SKIP in 41 minutes. Perf medians ref -> cand:
pp2048 286.61 -> 284.94 (-0.6%), d16384 207.93 -> 207.96, d32768 161.93 -> 162.12, tg256@d4096
14.11 -> 14.47; handoff median ref 715 / cand 706.5 ms; peak VRAM ref 5478 / cand 7771 MiB (the
first-arm spike of FORK.md F9, this time in a candidate arm). With the card free, phase 1 and 2
can run as one invocation like this.
- Previous full run (2026-11-22, candidate `3621025e8` = `3bf6b55b3` + the F12 carries, vs reference
`3bf6b55b3`; logs `/root/arms/carry1122-p1.log`, `-p2.log`): phase 1 `--ops --concat --diskcache`
20 PASS, 1 FAIL (`concat/candidate-timing` beside a full card, see above), 1 SKIP (diskcache GPU
leg); phase 2 `--concat --arbiter --diskcache --perf --moe-exact --moecache` 47 PASS, 0 FAIL, 0
SKIP. Perf medians ref -> cand: pp2048 286.97 -> 286.32, d16384 208.31 -> 208.21, d32768 162.25 ->
162.37; handoff median ref 696 / cand 700 ms; peak VRAM ref 7771 / cand 5502 MiB. Plus SCALE
15/15 and `test-flash-attn-indexed-bias` (FORK.md F12). Window about 30 minutes.
- Earlier full run (2026-11-21, candidate `3bf6b55b3` vs reference `adc2aa3ec`): phase 1 39 PASS, phase 2
9 PASS plus the 35B disk-cache legs (`q35.py`, mmproj loaded). Perf medians ref -> cand: pp2048
259.39 -> 286.61 (+10.5%), d16384 199.26 -> 207.63 (+4.2%), d32768 156.61 -> 157.51 (+0.6%),
tg256@d4096 9.17 -> 8.85 t/s (noisy, not gated). Handoff median: candidate 692 ms, reference
1068 ms. Downtime: about 42 minutes of validation and about 7 minutes of deploy.
- Earlier full run (2026-10-25, candidate `2fc9f9c35` vs reference `6c480f7e7`, logs
`/root/arms/dep-2fc9f9c35-fv1.log` and `-fv2.log`): phase 1 `--ops --concat` 8 PASS, phase 2
`--arbiter --perf --moe-exact --moecache` 29 PASS; 37 PASS, 0 FAIL. Perf medians ref/cand: pp2048
274.53/273.82, d16384 200.52/200.37, d32768 157.60/157.57, tg256@d4096 16.82/16.66 t/s. Handoff
median ref 693.5 / cand 678 ms. `--moe-exact`: 39 skips logged, identical to the reference and to
the skip forced off. Peak VRAM per arm ref 5476, cand 7062, cand 5767, ref 5476 MiB: the ~1.6 GiB
spike in the first candidate arm has now been seen twice (open question, FORK.md F9).
- Any arbiter change: `a-handoff-latency` is the ordinary-handoff A/B that two lost deploys showed
is needed. For a change aimed at latency itself, also run more blocks by hand; 25%/200 ms is a
regression gate, not a measurement.
- Disk prompt cache (FORK.md F11, from `fable5/disk-prompt-cache`): add `--diskcache` to phase 1. It
is CPU only and runs beside production (about 2 minutes). Its GPU leg SKIPs below 1536 MiB of free
VRAM; then run `--diskcache` again in the phase-2 window. A 35B measurement driver is on the pod:
`/root/arms/diskcache-test/q35.py [CHARS]` (scratch port 8095, scratch arena and cache dir, the
production 35B flags, a pan-tadeusz conversation; about 6 minutes). The 2026-10-25 window script
`/root/arms/diskcache-window.sh` (page-cache evictions, `--diskcache --arbiter`, `q35.py`) runs
under swapdown (run 2026-10-25: 29 PASS). `/root/arms/diskcache-window3.sh` runs `q35.py`
with production's exact 35B command line (mmproj loaded): save/restore, cold save, the
same-reuse-point identity and an image request (FORK.md F11). Run 2026-11-21 on adc2aa3ec: all
legs passed (18.8k-token chat, prefill 84.6 s -> turn 2 4.7 s after the restart, identical
output; save 557 MB in 1.0 s, token free 0.34 s, peak RSS +430 MB; cold save identity; image
answered and not saved; log `/root/arms/diskcache-test/q35-mm.log`). It supersedes
`diskcache-window2.sh` (`cmp.py` runs with `--no-mmproj`).
`q35.py` and the window scripts live only on the pod (not in the repo) and hard-code a build path.
Before each use, edit them to point at the candidate build (`/root/arms/rebYYYYMMDD/build`), not
production's `/root/llama.cpp/build`, or you validate the wrong binary.
On any FAIL: stop and report; the scratch dir (`/tmp/fork-validate.*`) is kept with all logs.
## 7. Deploy
Deploy exactly the validated commit. Downtime: 6 minutes for the 2026-10-24 redeploy. The deploy
rebuilds the production tree in place (step 6) from the same commit and with the same cmake flags as
the validated worktree build, so nothing is copied over; compare
`grep -E 'GGML_NATIVE|LLAMA_BUILD_UI|BUILD_SHARED_LIBS|CMAKE_BUILD_TYPE' build/CMakeCache.txt` in both
trees if in doubt.
Run the stop, build and start (steps 4-7) as one script on the pod, started with `setsid nohup sh
<script> > <log> 2>&1 </dev/null &`, never as a backgrounded `cat && sh -n && setsid` chain from the
`kubectl exec` shell (see section 9).
1. **Rollback pointer.** Record the current production commit in the next free pointer file, as the
full SHA alone on line 1 (older files mix full SHAs, short SHAs and extra text; see section 9):
```sh
ls /root/DEPLOY-ROLLBACK-commit-*.txt # last used: 16 (holds b13c41355)
git -C /root/llama.cpp rev-parse HEAD > /root/DEPLOY-ROLLBACK-commit-17.txt
git -C /root/llama.cpp status --short # must be clean
```
2. **Check out the candidate** in the production tree (fetch first; the branch must be on gitea, or already local after a bundle fetch, section 3):
```sh
cd /root/llama.cpp
git fetch origin fable5/vram-ondemand-rebase-YYYYMMDD
git checkout -B fable5/vram-ondemand-rebase-YYYYMMDD <validated-commit>
```
On a redeploy with the same branch name this moves the production branch to the new commit.
3. **Evict the page cache.** The pod is BestEffort QoS on a shared 62 GB node, so it is the first OOM
victim, and benchmarks, mmap loads and builds without F10 leave tens of GB of model files cached.
During the first 2026-10-24 deploy the Talos OOM controller killed the whole pod 17 s into
verification (cause likely, not proven, memory pressure). Evict while production still runs
(unprivileged and safe: `-lm none` servers hold their weights in anonymous memory, and pages
mapped by an mmap model are not dropped). The same snippet is used before and after validation
phase 2:
```sh
python3 - <<'EOF'
import glob, os
n = 0
for root in ('/root/.cache/huggingface/hub', '/root/models'):
for p in glob.glob(root + '/**/*.gguf', recursive=True):
fd = os.open(p, os.O_RDONLY)
try:
os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED)
n += 1
finally:
os.close(fd)
print('evicted', n, 'files')
EOF
```
F10 (FORK.md) drops the cache of every `-lm none` load by itself (verified in production on a 35B
GPU load: node page cache stayed at 4.2-5.3 GB instead of growing by about 20 GB), but files cached
by mmap loads, benchmarks or older builds still need this step. `/sys/fs/cgroup/memory.stat`
inside the container is the HOST root cgroup; the pod's own cgroup is under
`/sys/fs/cgroup/kubepods/besteffort/pod<uid>/`.
4. **Back up the current build, then stop llama-swap and reap orphans.** The deploy rebuilds
`/root/llama.cpp/build` in place, so copy it first (production keeps serving during the copy);
rollback (section 9) is then a copy, not a rebuild. Check `df -h /root` first (about 850 MB; the
2026-11-21 deploy made `/root/arms/build-adc2aa3ec`, 851 MB):
```sh
cp -a /root/llama.cpp/build /root/arms/build-<oldsha> # <oldsha>: the HEAD recorded in step 1
```
Delete the copy once the deploy is confirmed stable. Then:
```sh
supervisorctl -c /root/supervisord.conf stop llama-swap
ps -eo pid,ppid,etimes,args | grep '[l]lama-server'
kill -TERM <pid> <pid> # explicit PIDs only, never pkill -f
ps -eo pid,args | grep '[l]lama-server' # must be empty
```
`supervisorctl stop llama-swap` has left its llama-server children running as orphans (ppid 1),
one still generating for a client for a minute. They exit within about 1-2 s of a SIGTERM. Keep
this check even when it finds nothing (2026-10-25: no orphans).
5. **Check memory**, now that the running models are gone: `grep -E 'MemAvailable|MemFree'
/proc/meminfo`. Proceed only with MemAvailable >= 25 GB. In 2026-10-24 the eviction took node-wide
file cache from 15.6 to 4.9 GB and MemAvailable to 45.8 GB.
6. **Build in place**, now that nothing runs from `/root/llama.cpp/build/bin` (llama-swap's
`LD_LIBRARY_PATH` points there, so do not rebuild it under running servers):
```sh
cd /root/llama.cpp && HF_UI_VERSION=$(cut -d'|' -f2 build/tools/ui/.ui-stamp) \
cmake --build build --target llama-server llama-bench test-backend-ops -j 12
ls -lL build/bin/llama-server build/bin/libllama-server-impl.so build/bin/libllama-common.so \
build/bin/libllama.so build/bin/libmtmd.so build/bin/libggml.so build/bin/libggml-base.so \
build/bin/libggml-cpu.so build/bin/libggml-vulkan.so
```
All must have fresh mtimes. Use `-L` (follow the symlinks): `build/bin` keeps every old versioned
`libggml-*.so.0.x` and `libmtmd.so.0.x`, so a plain `ls` lists stale files beside the new ones.
`llama-server` is a 17 KB stub. The server code is in `libllama-server-impl.so`, the common code
(arguments, sampling) in `libllama-common.so`; a server change shows only in their mtimes.
7. **Start and probe.**
```sh
supervisorctl -c /root/supervisord.conf start llama-swap
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models # 200
```
Sample `/proc/meminfo` every few seconds during the first model loads. Confirm the running commit
from the first chat response (section 8): `system_fingerprint` is `b<build number>-<short sha>`
(the build number changes with each rebase; the sha is the deployed commit). The first 35B request after that deploy took 34 s including
the model load and the answer.
8. **Disk prompt cache** (FORK.md F11), only once `--cache-disk-path` is in `/root/config.yaml`:
- Every deploy empties the cache: the key holds the build commit, so the previous build's files are
never restored. They stay in the directory as "other files" until the size limit evicts them.
To free the space at once, `rm /root/prompt-cache/*.lpcd` while llama-swap is stopped (step 4).
- Check `/root` free space (`df -h /root`; 11-13 GB on 2026-10-25). The cap (`--cache-disk-size`)
plus a 1 GiB floor bounds the cache, but builds and model downloads share the PVC.
- Stopping llama-swap takes longer: each large model writes its slot state on SIGTERM, about 1 s
per GB of state, before it exits (FORK.md F7, F11).
## 8. Post-deploy checks
All through llama-swap on :8080. A request helper:
```sh
ask() { curl -s -m 300 http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
-d "{\"model\":\"$1\",\"messages\":[{\"role\":\"user\",\"content\":\"$2\"}],\"max_tokens\":32,\"temperature\":0}"; echo; }
holder() { ino=$(stat -c %i /dev/shm/llama-vram/token.lock); grep "FLOCK.* WRITE .*:$ino " /proc/locks; }
# image request body: a 64x64 solid red binary PPM, no image tools needed (like fork-validate.sh)
img() { python3 - "$1" <<'EOF'
import base64, json, sys
ppm = b'P6\n64 64\n255\n' + b'\xff\x00\x00' * (64 * 64)
url = 'data:image/x-portable-pixmap;base64,' + base64.b64encode(ppm).decode()
print(json.dumps({'model': sys.argv[1], 'max_tokens': 32, 'temperature': 0, 'messages': [{'role': 'user',
'content': [{'type': 'image_url', 'image_url': {'url': url}},
{'type': 'text', 'text': 'What is the main colour of this image? One word.'}]}]}))
EOF
}
# img MODEL | curl -s -m 300 http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d @-
```
Use the `-nothink` aliases with these `max_tokens: 32` helpers. A thinking alias spends the 32 tokens
on `reasoning_content` and returns an empty `content`; for one, raise `max_tokens` to >= 512.
1. `ask 'unsloth/Qwen3.6-35B-A3B-GGUF-nothink:UD-Q4_K_XL' 'Capital of France? One word.'` answers;
`holder` shows the 35B's PID (check with `ps -o args -p <pid>`).
2. Handoffs: alternate `Qwen3.5-4B-task:Q4_K_M` and the 35B two or three times. Each answer is
correct and `holder` shows the PID of the model that just answered. Expect a few seconds per switch
(2026-10-24: 3.1-5.6 s).
3. Multimodal: one image request to `SC117/gemma-4-26B-A4B-it-qat-heretic-GGUF-nothink:UD-Q4_K_XL`
with the `img` helper; the answer names red. This also forces an eviction in the llama-swap matrix: the
evicted process must exit by itself, logging `VRAM arbiter: loop stopped, device memory freed,
releasing the GPU token`, with no kill escalation.
4. Text to gemma and to the 4B after the eviction; `holder` correct each time.
With the disk prompt cache on (F11): send the 35B a chat of at least 256 tokens before step 3.
Its eviction must log `disk cache: saved the prompt of N tokens`. Then resend the same chat plus one
new user message to the 35B (earlier messages must stay byte-identical: editing an earlier message
changes the prefix and legitimately gives no restore): the log shows `disk cache: restored`, and `timings.prompt_n` is
only the new suffix.
5. Ports: the children bind the `--port ${PORT}` values llama-swap passes (not upstream's default 9931).
6. Log scan of the new llama-swap log lines (`curl -s http://127.0.0.1:8080/logs`) for
`GGML_ASSERT|Assertion|terminate called|Segmentation fault|ErrorDeviceLost|ErrorOutOfDeviceMemory|DEVICE_LOST`: none.
7. `dmesg` clean (no `ring ... timeout`, no `GPU_HUNG`). No pod restart and no OOM kill:
- Restarts, from the workstation: `kubectl get pods -n llama | grep supervisord`; RESTARTS of the
Running pod must equal the value you noted before the deploy. A stale `supervisord-...-pbmjq`
in state `Unknown` is also listed; it is an old leftover, ignore it.
- OOM kills, in the pod: `dmesg | grep -E 'OOM controller|Sending SIGKILL to cgroup'`. The
timestamps are seconds since boot; compare them with the first field of `/proc/uptime` to see
whether any falls in the deploy window (2026-10-25: the newest was about 19 h old).
8. Inkling (FORK.md section 5, F12): judge the answer text, not only HTTP 200. The first request
after a fresh load should be a prompt of about 150 tokens (upstream #32778 gives all-'?' output
there); then a long prompt (Pan Tadeusz, 8K and 14K tokens, upstream #32755). Send an image
and a tool request to both entries: since the nothink macro dropped `--reasoning-budget 0`
(FORK.md F12) both answer them; repeated `<|message_model|><|content_thinking|><|end_message|>`
from the nothink entry means a budget of 0 is back. Since F13 a scratch server with the
production command line plus `-lv 4` must log `post_load_tensors: N views of host weights`
(production's own log level hides libllama's info lines, so the llama-swap log never shows it),
and the entries no longer set
`GGML_OP_OFFLOAD_MIN_BATCH=64` (F13 measured the 32-63-token window correct without it). The
placement is `-ot` with the loop layers 6..6+k-1 left on the GPU (FORK.md F13 placement guide);
after a change of context or quant, check the fit again (the breakdown must leave room for the
mmproj, about 0.8 GiB). The memory breakdown table is printed only at `-lv 4`,
which production does not set, so it is not in the llama-swap log: read it from a scratch server
started with production's command line plus `-lv 4`. Drivers used on 2026-11-22 (pod only):
`/root/arms/carry1122-post.py` (through llama-swap) and `/root/arms/carry1122-ink-direct.py`
(scratch server, GPU to itself).
A wedged arbiter looks like this: requests hang while `/health` still returns 200, and `/proc/locks`
shows a process holding WRITE on `token.lock` that is not generating. llama-swap restarts a child
that exits (on the next request for its model, FORK.md F7) but never one that wedges. Inspect it with
`gdb -p <pid> -batch -ex 'thread apply all bt'` before killing it. gdb is not in
the pod image; it was lost in the 2026-10-24 pod restart, but the rootfs persists since 2026-11-22 (section 11), so
an apt install now stays. Check `command -v gdb` and install it first if needed.
## 9. Rollback
```sh
supervisorctl -c /root/supervisord.conf stop llama-swap
ps -eo pid,args | grep '[l]lama-server' # SIGTERM any orphan by PID
cd /root/llama.cpp
git checkout -B <previous-production-branch> "$(head -1 /root/DEPLOY-ROLLBACK-commit-N.txt | cut -d' ' -f1)"
HF_UI_VERSION=$(cut -d'|' -f2 build/tools/ui/.ui-stamp) cmake --build build --target llama-server llama-bench -j 12
supervisorctl -c /root/supervisord.conf start llama-swap
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models
```
This is the fallback when no build backup exists. With the deploy step 4 backup, roll back by copy
instead (stop, checkout the old commit, replace the build dir, start):
```sh
supervisorctl -c /root/supervisord.conf stop llama-swap
ps -eo pid,args | grep '[l]lama-server' # SIGTERM any orphan by PID
cd /root/llama.cpp
git checkout -B <previous-production-branch> <oldsha>
mv build /root/arms/build-failed-<newsha> # or rm -rf build; mind the disk
cp -a /root/arms/build-<oldsha> build
supervisorctl -c /root/supervisord.conf start llama-swap
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models
```
Once the new deploy is confirmed stable, delete the copy (`rm -rf /root/arms/build-<oldsha>`; the
disk is about 98% full) and any `build-failed-*` directory.
Then run the post-deploy checks again. Roll back on any failed check; report rather than debug in
production. Keep the candidate worktree for a later redeploy.
If you run the rollback (or the deploy steps 4-7) from a workstation through `kubectl exec`, put the
whole sequence in one script on the pod and start it as `setsid nohup sh /root/arms/rollback.sh >
/root/arms/rollback.log 2>&1 </dev/null &`. Do not background a `cat > X && sh -n X && setsid ...`
chain in the exec shell: it can die with the exec session halfway, leaving llama-swap stopped (section
5). A rollback that also undoes a config change (for example removing `--cache-disk-path` from
`/root/config.yaml`, F11) needs the config restored first, from its `/root/config.yaml.bak-*`.
Pointer files 2 to 16 exist; for example commit-8 holds `64543a435`, commit-9 `5751c977b`,
commit-10 `6c480f7e7`, commit-11 was used by the failed cb48bf224 deploy, commit-12 holds
`2fc9f9c35`, commit-13 holds `adc2aa3ec`, commit-14 holds `3bf6b55b3`, commit-15 holds `3621025e8`, and commit-16 holds `b13c41355`, the rollback target
for the current production `f3397cded` (next free: 17; build copy `/root/arms/build-b13c41355`). The formats vary
(commit-4 to 6 hold short SHAs, commit-6 has branch and date after the SHA, commit-9 a subject on
line 2), hence `head -1 | cut -d' ' -f1`. New pointers: the full SHA alone on line 1.
## 10. Recovering a hung GPU
Symptoms: a process that survives SIGKILL (stuck in the driver); `ring sdma1 timeout` or
`ring comp_... timeout` in the kernel log; after a reboot the supervisord pod sits in FailedMount with
`/dev/kfd is not a character device` / `/dev/dri is not a directory`, and
`talosctl get pcidevices` lists no VGA device (the RX 580 and its HDMI audio function are missing).
A warm reboot does not bring the card back (classic Polaris failed reset).
The node (`anapistula-delrosalae`, Talos, bare metal) has a NanoKVM at http://192.168.255.8.
Credentials: ask the owner. Procedure (owner's instructions, follow exactly):
1. Soft shutdown: **one short press** of the power button through the NanoKVM.
2. **Never use the reset button.** It does not cut GPU power and is useless here.
3. Wait until the power LED goes off.
4. Press the power button once more to boot.
5. Only if the LED is still on 10+ minutes after the short press: one **long press of 8 seconds** to
force power off, wait for the LED to go off, then a short press to boot. With the GPU off the bus the
soft shutdown has hung before (Talos services stopped, LED stayed on), and the long press was needed.
Talos access from the workstation:
```sh
export TALOSCONFIG=/home/user/Projects/klaster/talos/generated/talosconfig
nix run nixpkgs#talosctl -- -n anapistula-delrosalae get pcidevices
nix run nixpkgs#talosctl -- -n anapistula-delrosalae dmesg
```
Afterwards: pod Running, RX 580 listed, llama-swap RUNNING, `/v1/models` 200, and the 120 W cap active:
`cat /sys/class/drm/card*/device/hwmon/hwmon*/power1_cap` reads `120000000` (the
`power1_cap ... -> 120000000` line in `/tmp/fan-control.log` may be missing after a pod restart). The two 2026-10-09 hangs both came at the
first 4096-token prefill right after a decode phase, before the cap was fixed at 120 W; both a fork
and a clean-upstream build hung, so they were treated as hardware/power, not code.
## 11. Operations reference
Pod access:
```sh
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
kubectl -n llama exec -i deploy/supervisord -- sh -c '<command>'
kubectl get pods -n llama -l app=llama-swap -o name # the pod name changes across restarts
```
Alternative (since 2026-11-22): the pod runs sshd, so `ssh root@2001:470:61a3:400::7` works instead
of `kubectl exec` (`scp 'root@[2001:470:61a3:400::7]:/path' .` for files). The same rules apply:
detach long runs with `setsid nohup ... </dev/null &` and never `pkill -f`.
The pod's root filesystem now persists across pod restarts: packages installed with apt (gdb,
for example) stay installed. `/tmp` and `/run` are still wiped on every boot, so copy a scratch dir
you want to keep (fork-validate's `/tmp/fork-validate.*`) to `/root/arms/`.
Wiring: k8s service `llama:11434` -> pod port 8080 (selector `app=llama-swap`) -> llama-swap `:8080`;
a Caddy proxy fronts `llama.lumpiasty.xyz`. llama-swap must listen on `:8080`.
Files (all under `/root`, a persistent volume):
- `/root/llama.cpp`: production tree, built in `build/` (shared libraries, so the real code is in
`build/bin/libllama.so` and `libggml-*.so`). Scratch worktrees go in `/root/arms/`.
- `/root/supervisord.conf`: supervisord (PID 1). `[program:llama-swap]` runs
`/root/llama-swap -config /root/config.yaml -listen :8080 -watch-config`; its `environment=` sets
`PATH` and `LD_LIBRARY_PATH` to `/root/llama.cpp/build/bin` and the arbiter variables (FORK.md section 5).
- `/root/config.yaml`: llama-swap models, macros (`moe_offload_args`, `moe_offload_gemma`,
`common_args`), matrix/exclusivity sets, preload. Backups `/root/config.yaml.bak-*`. Edit only with
the owner's approval.
- `/root/fan-control.sh`: fan curve and the fixed 120 W cap (separate supervisord program).
- `/root/bench/`: benchmark results and pod-side helpers (`/root/bench/moe/swapdown.sh`).
- `/root/DEPLOY-ROLLBACK-commit-N.txt`: rollback pointers.
- `/root/prompt-cache/` (configured 2026-11-21, `--cache-disk-path /root/prompt-cache
--cache-disk-size 4096` in the `moe_offload_args` and `moe_offload_gemma` macros; backup
`/root/config.yaml.bak-diskcache-20261121`): disk prompt cache files `*.lpcd` of every model
with the flag (FORK.md F11). Production check: a 35B 4.6k-token chat was saved on eviction (278.6
MiB, copy 126 ms, write 495 ms), restored after the reload (read 416 ms, restore 33 ms) and turn 2
prefilled 32 tokens; a gemma image request works with the flags on and is not cached. Safe to empty while llama-swap is stopped.
- `/dev/shm/llama-vram/`: production arbiter arena (`token.lock`, `want.lock`; an inert `doorbell/`
directory from the old design may remain). Never point a test at it.
Reloads: a `config.yaml` change is picked up by `-watch-config`, but a running model keeps its old
arguments until it restarts (`supervisorctl -c /root/supervisord.conf restart llama-swap`). An
`environment=` change needs `supervisorctl -c /root/supervisord.conf reread && supervisorctl -c
/root/supervisord.conf update`.
Useful commands:
- VRAM in use: `cat /sys/class/drm/card*/device/mem_info_vram_used`.
- Token holder and waiters: `/proc/locks` filtered by the inode of `token.lock` / `want.lock`
(WRITE = holder on token.lock, READ = registered waiters on want.lock).
- 120 W cap active: `cat /sys/class/drm/card*/device/hwmon/hwmon*/power1_cap` = `120000000`.
- The pod has `curl`, `python3`, `flock`; no `jq`, and `gdb` only if installed with apt (it persists
since 2026-11-22; without it `fork-validate.sh` runs its SIGTERM-in-release case by polling the log).
Second test box (optional): `ssh user@gaming-pc`, the owner's NixOS desktop with an RX 6600 (RDNA2).
Pod-built binaries can run there through the host glibc loader without compiling. Numbers from it do
not transfer: about 2.25x the RX 580's prefill, 13% slower decode, and the GCN-only mask_opt change does
not apply. Do not change its system config.
Not a fork problem: OpenWebUI's stop button does not cancel generation (its Redis task-command listener
never handles the stop); llama-server and llama-swap both cancel correctly on client disconnect.
+5
View File
@@ -109,6 +109,11 @@ extern "C" {
GGML_API enum ggml_status ggml_backend_graph_compute (ggml_backend_t backend, struct ggml_cgraph * cgraph);
GGML_API enum ggml_status ggml_backend_graph_compute_async(ggml_backend_t backend, struct ggml_cgraph * cgraph);
// Free transient/scratch device memory the backend holds outside of any allocated buffer
// (compute preallocations, staging buffers). No-op if the backend does not implement it.
// The backend remains usable; scratch is reallocated lazily on the next compute.
GGML_API void ggml_backend_free_scratch(ggml_backend_t backend);
// NOTE: will be removed, use device version instead
GGML_API bool ggml_backend_supports_op(ggml_backend_t backend, const struct ggml_tensor * op);
GGML_API bool ggml_backend_supports_buft(ggml_backend_t backend, ggml_backend_buffer_type_t buft);
+2 -2
View File
@@ -8,10 +8,10 @@ extern "C" {
#define RPC_PROTO_MAJOR_VERSION 10
#define RPC_PROTO_MINOR_VERSION 0
#define RPC_PROTO_PATCH_VERSION 1
#define RPC_PROTO_PATCH_VERSION 2
#ifdef __cplusplus
static_assert(GGML_OP_COUNT == 100, "GGML_OP_COUNT has changed - update RPC_PROTO_PATCH_VERSION");
static_assert(GGML_OP_COUNT == 101, "GGML_OP_COUNT has changed - update RPC_PROTO_PATCH_VERSION");
#endif
#define GGML_RPC_MAX_SERVERS 16
+18
View File
@@ -433,6 +433,8 @@ extern "C" {
GGML_OP_SLEEP,
GGML_OP_SIGMOID_BACK,
GGML_OP_COUNT,
};
@@ -1038,6 +1040,13 @@ extern "C" {
struct ggml_tensor * a,
struct ggml_tensor * b);
// a - dy
// b - x
GGML_API struct ggml_tensor * ggml_sigmoid_back(
struct ggml_context * ctx,
struct ggml_tensor * a,
struct ggml_tensor * b);
// hardswish(x) = x * relu6(x + 3) / 6
GGML_API struct ggml_tensor * ggml_hardswish(
struct ggml_context * ctx,
@@ -2283,6 +2292,15 @@ extern "C" {
struct ggml_tensor * a,
struct ggml_tensor * sinks);
GGML_API bool ggml_flash_attn_ext_has_indexed_bias(const struct ggml_tensor * a);
// Add table[indices[key, query], head, query] to the mask before its F16 rounding.
// Queries from all streams are flattened in stream order. Indices must be in [0, table->ne[0]).
GGML_API void ggml_flash_attn_ext_add_indexed_bias(
struct ggml_tensor * a,
struct ggml_tensor * table,
struct ggml_tensor * indices);
// TODO: needs to be adapted to ggml_flash_attn_ext
GGML_API struct ggml_tensor * ggml_flash_attn_back(
struct ggml_context * ctx,
+1
View File
@@ -35,6 +35,7 @@ bool ggml_op_can_inplace(enum ggml_op op) {
case GGML_OP_ROPE:
case GGML_OP_ROPE_BACK:
case GGML_OP_SILU_BACK:
case GGML_OP_SIGMOID_BACK:
case GGML_OP_RMS_NORM:
case GGML_OP_RMS_NORM_BACK:
case GGML_OP_CLAMP:
+5
View File
@@ -162,6 +162,11 @@ extern "C" {
// (optional) sort/optimize the nodes in the graph
void (*graph_optimize) (ggml_backend_t backend, struct ggml_cgraph * cgraph, struct ggml_backend_graph_optimize_params * params);
// (optional) free transient/scratch device memory the backend holds outside of any buffer
// (e.g. compute preallocations and staging buffers). The backend stays usable; the scratch
// is reallocated lazily on the next compute. Used to shrink an idle model's device footprint.
void (*free_scratch) (ggml_backend_t backend);
};
struct ggml_backend {
+6 -1
View File
@@ -152,6 +152,10 @@ static ggml_backend_buffer_type_t ggml_backend_meta_device_get_buffer_type(ggml_
static ggml_backend_buffer_type_t ggml_backend_meta_device_get_host_buffer_type(ggml_backend_dev_t dev);
static bool ggml_backend_meta_device_supports_op(ggml_backend_dev_t dev, const ggml_tensor * op) {
if (op->op == GGML_OP_FLASH_ATTN_EXT && ggml_flash_attn_ext_has_indexed_bias(op)) {
return false;
}
GGML_ASSERT(ggml_backend_dev_is_meta(dev));
const ggml_backend_meta_device_context * meta_dev_ctx = (const ggml_backend_meta_device_context *) dev->context;
return std::all_of(meta_dev_ctx->simple_devs.begin(), meta_dev_ctx->simple_devs.end(),
@@ -971,7 +975,8 @@ static struct ggml_backend_meta_split_state ggml_backend_meta_get_split_state(
case GGML_OP_CONCAT: {
split_state = handle_concat(src_ss);
} break;
case GGML_OP_SILU_BACK: {
case GGML_OP_SILU_BACK:
case GGML_OP_SIGMOID_BACK: {
split_state = handle_generic(src_ss, /*scalar_only =*/ false);
} break;
case GGML_OP_NORM:
+9
View File
@@ -775,6 +775,15 @@ void ggml_backend_synchronize(ggml_backend_t backend) {
backend->iface.synchronize(backend);
}
void ggml_backend_free_scratch(ggml_backend_t backend) {
GGML_ASSERT(backend);
if (backend->iface.free_scratch == NULL) {
return;
}
backend->iface.free_scratch(backend);
}
ggml_backend_graph_plan_t ggml_backend_graph_plan_create(ggml_backend_t backend, struct ggml_cgraph * cgraph) {
GGML_ASSERT(backend);
GGML_ASSERT(backend->iface.graph_plan_create != NULL);
+2 -2
View File
@@ -2679,8 +2679,8 @@ static bool ggml_backend_cann_supports_op(ggml_backend_dev_t dev, const ggml_ten
return true;
case GGML_OP_FLASH_ATTN_EXT:
{
// K/V row lists (ggml_flash_attn_ext_rows) are not supported
if (op->src[5] != nullptr) {
// K/V row lists and indexed bias are not supported
if (op->src[5] != nullptr || ggml_flash_attn_ext_has_indexed_bias(op)) {
return false;
}
#ifdef ASCEND_310P
+53 -37
View File
@@ -1083,6 +1083,29 @@ void ggml_vec_dot_mxfp4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const vo
*s = sumf;
}
#if defined(__AVX2__)
// The 8 UE4M3 scales of two NVFP4 blocks as fp32, equal to GGML_CPU_UE4M3_TO_FP32 (x0.5, 0x7F -> 0, sign bit
// ignored, subnormals), in the lane order of the reduction in ggml_vec_dot_nvfp4_q8_0:
// [b0s0 b0s2 b1s0 b1s2 | b0s1 b0s3 b1s1 b1s3]
static inline __m256 ue4m3x8_to_fp32_half(const uint8_t * d0, const uint8_t * d1) {
uint32_t a, b;
memcpy(&a, d0, sizeof(a));
memcpy(&b, d1, sizeof(b));
const __m256i v = _mm256_set1_epi64x((int64_t) (((uint64_t) b << 32) | a));
const __m256i x = _mm256_shuffle_epi8(v, _mm256_setr_epi8(
0, -1, -1, -1, 2, -1, -1, -1, 4, -1, -1, -1, 6, -1, -1, -1,
1, -1, -1, -1, 3, -1, -1, -1, 5, -1, -1, -1, 7, -1, -1, -1));
const __m256i nan = _mm256_cmpeq_epi32(x, _mm256_set1_epi32(0x7F));
const __m256i xm = _mm256_and_si256(x, _mm256_set1_epi32(0x7F));
// normal: fp32 exponent e + 119 (bias 7, times 0.5), mantissa bits in place
const __m256 fn = _mm256_castsi256_ps(_mm256_add_epi32(_mm256_slli_epi32(xm, 20), _mm256_set1_epi32(119 << 23)));
// subnormal (e == 0): m * 2^-10
const __m256 fs = _mm256_mul_ps(_mm256_cvtepi32_ps(xm), _mm256_set1_ps(1.0f/1024.0f));
const __m256 f = _mm256_blendv_ps(fn, fs, _mm256_castsi256_ps(_mm256_cmpgt_epi32(_mm256_set1_epi32(8), xm)));
return _mm256_andnot_ps(_mm256_castsi256_ps(nan), f);
}
#endif
void ggml_vec_dot_nvfp4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc) {
assert(nrc == 1);
UNUSED(nrc);
@@ -1100,47 +1123,40 @@ void ggml_vec_dot_nvfp4_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const vo
#if defined(__AVX2__)
const __m128i values128 = _mm_loadu_si128((const __m128i*)kvalues_fp4);
const __m128i m4b = _mm_set1_epi8(0x0f);
const __m256i mone = _mm256_set1_epi16(1);
const __m256i values = _mm256_broadcastsi128_si256(_mm_loadu_si128((const __m128i *) kvalues_fp4));
const __m256i m4b = _mm256_set1_epi8(0x0f);
const __m256i mone = _mm256_set1_epi16(1);
// two blocks per iteration: 8 sub-block sums, 8 scales and one FMA
__m256 accum = _mm256_setzero_ps();
for(; ib < nb; ib++){
for (; ib + 1 < nb; ib += 2) {
__m256i p[4];
for (int k = 0; k < 2; ++k) {
// 8 bytes per sub-block: low nibbles are elements 0-7, high nibbles 8-15
const __m256i q = _mm256_loadu_si256((const __m256i *) x[ib + k].qs);
const __m256i lo = _mm256_shuffle_epi8(values, _mm256_and_si256(q, m4b));
const __m256i hi = _mm256_shuffle_epi8(values, _mm256_and_si256(_mm256_srli_epi16(q, 4), m4b));
const __m256i a = _mm256_unpacklo_epi64(lo, hi); // [sub0 | sub2]
const __m256i b = _mm256_unpackhi_epi64(lo, hi); // [sub1 | sub3]
p[2*k + 0] = mul_add_epi8(_mm256_permute2x128_si256(a, b, 0x20), _mm256_loadu_si256((const __m256i *) y[2*(ib + k) + 0].qs));
p[2*k + 1] = mul_add_epi8(_mm256_permute2x128_si256(a, b, 0x31), _mm256_loadu_si256((const __m256i *) y[2*(ib + k) + 1].qs));
}
// a sub-block sum is at most 16*12*127, so it fits in int16:
// lanes [b0s0 b0s2 b1s0 b1s2 | b0s1 b0s3 b1s1 b1s3]
const __m256i h = _mm256_hadd_epi16(_mm256_hadd_epi16(p[0], p[1]), _mm256_hadd_epi16(p[2], p[3]));
const __m256 sums = _mm256_cvtepi32_ps(_mm256_madd_epi16(h, mone));
const __m128i q4bits_01 = _mm_loadu_si128((const __m128i *)(x[ib].qs + 0));
const __m128i q4bits_23 = _mm_loadu_si128((const __m128i *)(x[ib].qs + 16));
#if defined(__F16C__)
// assembled in a GPR: cheaper than four inserts
const uint64_t dyb = (uint64_t) y[2*ib + 0].d | ((uint64_t) y[2*ib + 1].d << 16) |
((uint64_t) y[2*ib + 2].d << 32) | ((uint64_t) y[2*ib + 3].d << 48);
const __m128 dy = _mm_cvtph_ps(_mm_cvtsi64_si128((int64_t) dyb));
#else
const __m128 dy = _mm_setr_ps(GGML_CPU_FP16_TO_FP32(y[2*ib + 0].d), GGML_CPU_FP16_TO_FP32(y[2*ib + 1].d),
GGML_CPU_FP16_TO_FP32(y[2*ib + 2].d), GGML_CPU_FP16_TO_FP32(y[2*ib + 3].d));
#endif
const __m256i q8_01 = _mm256_loadu_si256((const __m256i *)y[2*ib + 0].qs);
const __m256i q8_23 = _mm256_loadu_si256((const __m256i *)y[2*ib + 1].qs);
const __m128i q4_01_lo = _mm_shuffle_epi8(values128, _mm_and_si128(q4bits_01, m4b));
const __m128i q4_01_hi = _mm_shuffle_epi8(values128, _mm_and_si128(_mm_srli_epi16(q4bits_01, 4), m4b));
const __m128i q4_23_lo = _mm_shuffle_epi8(values128, _mm_and_si128(q4bits_23, m4b));
const __m128i q4_23_hi = _mm_shuffle_epi8(values128, _mm_and_si128(_mm_srli_epi16(q4bits_23, 4), m4b));
//reordering
const __m256i q4_01 = MM256_SET_M128I(_mm_unpackhi_epi64(q4_01_lo,q4_01_hi), _mm_unpacklo_epi64(q4_01_lo,q4_01_hi));
const __m256i q4_23 = MM256_SET_M128I(_mm_unpackhi_epi64(q4_23_lo,q4_23_hi),_mm_unpacklo_epi64(q4_23_lo,q4_23_hi));
const __m256i p01 = mul_add_epi8(q4_01,q8_01);
const __m256i p_1 = _mm256_madd_epi16(p01, mone);
const __m256i p23 = mul_add_epi8(q4_23,q8_23);
const __m256i p_2 = _mm256_madd_epi16(p23, mone);
const float dy0 = GGML_CPU_FP16_TO_FP32(y[2*ib].d);
const float dy1 = GGML_CPU_FP16_TO_FP32(y[2*ib+1].d);
const float s0 = GGML_CPU_UE4M3_TO_FP32(x[ib].d[0]) * dy0;
const float s1 = GGML_CPU_UE4M3_TO_FP32(x[ib].d[1]) * dy0;
const float s2 = GGML_CPU_UE4M3_TO_FP32(x[ib].d[2]) * dy1;
const float s3 = GGML_CPU_UE4M3_TO_FP32(x[ib].d[3]) * dy1;
const __m256 scales01 = _mm256_set_m128(_mm_set1_ps(s1), _mm_set1_ps(s0));
const __m256 scales23 = _mm256_set_m128(_mm_set1_ps(s3), _mm_set1_ps(s2));
accum = _mm256_fmadd_ps(scales01, _mm256_cvtepi32_ps(p_1), accum);
accum = _mm256_fmadd_ps(scales23, _mm256_cvtepi32_ps(p_2), accum);
accum = _mm256_fmadd_ps(_mm256_mul_ps(ue4m3x8_to_fp32_half(x[ib].d, x[ib + 1].d), _mm256_set_m128(dy, dy)), sums, accum);
}
sumf = hsum_float_8(accum);
+5
View File
@@ -1884,6 +1884,10 @@ static void ggml_compute_forward(struct ggml_compute_params * params, struct ggm
{
ggml_compute_forward_silu_back(params, tensor);
} break;
case GGML_OP_SIGMOID_BACK:
{
ggml_compute_forward_sigmoid_back(params, tensor);
} break;
case GGML_OP_NORM:
{
ggml_compute_forward_norm(params, tensor);
@@ -2392,6 +2396,7 @@ static int ggml_get_n_tasks(struct ggml_tensor * node, int n_threads) {
}
break;
case GGML_OP_SILU_BACK:
case GGML_OP_SIGMOID_BACK:
case GGML_OP_MUL:
case GGML_OP_DIV:
case GGML_OP_NORM:
+4
View File
@@ -422,6 +422,10 @@ static ggml_backend_buffer_t ggml_backend_cpu_device_buffer_from_host_ptr(ggml_b
}
static bool ggml_backend_cpu_device_supports_op(ggml_backend_dev_t dev, const struct ggml_tensor * op) {
if (op->op == GGML_OP_FLASH_ATTN_EXT && ggml_flash_attn_ext_has_indexed_bias(op)) {
return false;
}
const struct ggml_tensor * src0 = op->src[0];
const struct ggml_tensor * src1 = op->src[1];
+112
View File
@@ -2898,6 +2898,118 @@ void ggml_compute_forward_silu_back(
}
}
// ggml_compute_forward_sigmoid_back
static void ggml_compute_forward_sigmoid_back_f32(
const ggml_compute_params * params,
ggml_tensor * dst) {
const ggml_tensor * grad = dst->src[0];
const ggml_tensor * src1 = dst->src[1];
GGML_ASSERT(ggml_is_contiguous_1(grad));
GGML_ASSERT(ggml_is_contiguous_1(src1));
GGML_ASSERT(ggml_is_contiguous_1(dst));
GGML_ASSERT(ggml_are_same_shape(src1, dst));
GGML_ASSERT(ggml_are_same_shape(src1, grad));
const int ith = params->ith;
const int nth = params->nth;
const int nc = src1->ne[0];
const int nr = ggml_nrows(src1);
// rows per thread
const int dr = (nr + nth - 1)/nth;
// row range for this thread
const int ir0 = dr*ith;
const int ir1 = MIN(ir0 + dr, nr);
for (int i1 = ir0; i1 < ir1; i1++) {
ggml_vec_sigmoid_backward_f32(nc,
(float *) ((char *) dst->data + i1*( dst->nb[1])),
(float *) ((char *) src1->data + i1*(src1->nb[1])),
(float *) ((char *) grad->data + i1*(grad->nb[1])));
#ifndef NDEBUG
for (int k = 0; k < nc; k++) {
const float x = ((float *) ((char *) dst->data + i1*( dst->nb[1])))[k];
GGML_UNUSED(x);
assert(!isnan(x));
assert(!isinf(x));
}
#endif // NDEBUG
}
}
static void ggml_compute_forward_sigmoid_back_f16(
const ggml_compute_params * params,
ggml_tensor * dst) {
const ggml_tensor * grad = dst->src[0];
const ggml_tensor * src1 = dst->src[1];
GGML_ASSERT(ggml_is_contiguous_1(grad));
GGML_ASSERT(ggml_is_contiguous_1(src1));
GGML_ASSERT(ggml_is_contiguous_1(dst));
GGML_ASSERT(ggml_are_same_shape(src1, dst));
GGML_ASSERT(ggml_are_same_shape(src1, grad));
const int ith = params->ith;
const int nth = params->nth;
const int nc = src1->ne[0];
const int nr = ggml_nrows(src1);
// rows per thread
const int dr = (nr + nth - 1)/nth;
// row range for this thread
const int ir0 = dr*ith;
const int ir1 = MIN(ir0 + dr, nr);
for (int i1 = ir0; i1 < ir1; i1++) {
ggml_vec_sigmoid_backward_f16(nc,
(ggml_fp16_t *) ((char *) dst->data + i1*( dst->nb[1])),
(ggml_fp16_t *) ((char *) src1->data + i1*(src1->nb[1])),
(ggml_fp16_t *) ((char *) grad->data + i1*(grad->nb[1])));
#ifndef NDEBUG
for (int k = 0; k < nc; k++) {
const float x = ((ggml_fp16_t *) ((char *) dst->data + i1*( dst->nb[1])))[k];
const float v = GGML_CPU_FP16_TO_FP32(x);
GGML_UNUSED(v);
assert(!isnan(v));
assert(!isinf(v));
}
#endif // NDEBUG
}
}
void ggml_compute_forward_sigmoid_back(
const ggml_compute_params * params,
ggml_tensor * dst) {
const ggml_tensor * src0 = dst->src[0];
switch (src0->type) {
case GGML_TYPE_F32:
{
ggml_compute_forward_sigmoid_back_f32(params, dst);
} break;
case GGML_TYPE_F16:
{
ggml_compute_forward_sigmoid_back_f16(params, dst);
} break;
default:
{
GGML_ABORT("fatal error");
}
}
}
// ggml_compute_forward_reglu
// row loop shared by the BF16 GLU ops
+1
View File
@@ -41,6 +41,7 @@ void ggml_compute_forward_repeat(const struct ggml_compute_params * params, stru
void ggml_compute_forward_repeat_back(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_concat(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_silu_back(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_sigmoid_back(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_norm(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_rms_norm(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_rms_norm_mul_fused(const struct ggml_compute_params * params, struct ggml_tensor * dst_rms_norm, struct ggml_tensor * dst_mul);
+23
View File
@@ -1536,6 +1536,29 @@ inline static void ggml_vec_silu_backward_f16(const int n, ggml_fp16_t * dx, con
}
}
inline static float ggml_sigmoid_backward_f32(float x, float dy) {
const float s = 1.0f/(1.0f + expf(-x));
return dy*s*(1.0f - s);
}
inline static ggml_fp16_t ggml_sigmoid_backward_f16(ggml_fp16_t x, ggml_fp16_t dy) {
const float v = GGML_CPU_FP16_TO_FP32(x);
const float s = 1.0f/(1.0f + expf(-v));
return GGML_CPU_FP32_TO_FP16(GGML_CPU_FP16_TO_FP32(dy)*s*(1.0f - s));
}
inline static void ggml_vec_sigmoid_backward_f32(const int n, float * dx, const float * x, const float * dy) {
for (int i = 0; i < n; ++i) {
dx[i] = ggml_sigmoid_backward_f32(x[i], dy[i]);
}
}
inline static void ggml_vec_sigmoid_backward_f16(const int n, ggml_fp16_t * dx, const ggml_fp16_t * x, const ggml_fp16_t * dy) {
for (int i = 0; i < n; ++i) {
dx[i] = ggml_sigmoid_backward_f16(x[i], dy[i]);
}
}
inline static void ggml_vec_reglu_f32 (const int n, float * y, const float * x, const float * g) {
for (int i = 0; i < n; ++i) {
y[i] = (x[i] > 0.f) ? x[i] * g[i] : 0.f;
+1 -1
View File
@@ -911,7 +911,7 @@ static __device__ __forceinline__ float ggml_cuda_ue4m3_to_fp32(uint8_t x) {
// but CDNA3 supports only e4m3_fnuz (no inf).
const uint32_t bits = x * (x != 0x7F && x != 0xFF); // Convert NaN to 0.0f to match CPU implementation.
const __hip_fp8_e4m3_fnuz xf = *reinterpret_cast<const __hip_fp8_e4m3_fnuz *>(&bits);
return static_cast<float>(xf) / 2;
return static_cast<float>(xf);
#else
#if defined(FP8_AVAILABLE) && !defined(GGML_USE_HIP)
const uint32_t bits = x * (x != 0x7F && x != 0xFF); // Convert NaN to 0.0f to match CPU implementation.
+26 -3
View File
@@ -18,6 +18,23 @@
// The macro on the following line shifts it by a factor of 2**3=8, as was needed to fix https://github.com/ggml-org/llama.cpp/issues/18606 .
#define FATTN_KQ_MAX_OFFSET (3.0f*0.6931f)
struct fattn_indexed_bias {
const float * table;
const int32_t * indices;
int64_t n_bins;
int64_t n_keys;
int64_t query_stride;
__device__ fattn_indexed_bias for_head(int64_t first_query, int head) const {
return {table + first_query*query_stride + head*n_bins, indices + first_query*n_keys, n_bins, n_keys, query_stride};
}
__device__ half add(half mask, int query, int key) const {
const int bin = indices[query*n_keys + key];
return uint32_t(bin) < n_bins ? __float2half(__half2float(mask) + table[query*query_stride + bin]) : mask;
}
};
typedef void (* fattn_kernel_t)(
const char * __restrict__ Q,
const char * __restrict__ K,
@@ -40,7 +57,8 @@ typedef void (* fattn_kernel_t)(
const int32_t nb11, const int32_t nb12, const int64_t nb13,
const int32_t nb21, const int32_t nb22, const int64_t nb23,
const int32_t ne31, const int32_t ne32, const int32_t ne33,
const int32_t nb31, const int64_t nb32, const int64_t nb33);
const int32_t nb31, const int64_t nb32, const int64_t nb33,
const fattn_indexed_bias bias);
typedef float (*vec_dot_KQ_t)(
const char * __restrict__ K_c, const void * __restrict__ Q_v, const int * __restrict__ Q_q8 , const void * __restrict__ Q_ds);
@@ -1170,7 +1188,7 @@ void launch_fattn(
// Optional optimization where the mask is scanned to determine whether part of the calculation can be skipped.
// Only worth the overhead if there is at lease one FATTN_KQ_STRIDE x FATTN_KQ_STRIDE square to be skipped or
// multiple sequences of possibly different lengths, or kv_rows lists that end with padding.
const bool scan_mask = !use_sparse && mask && mask->ne[2] == 1 && n_kv_cols % FATTN_KQ_STRIDE == 0 && (Q->ne[1] >= 1024 || Q->ne[3] > 1 || kv_rows);
const bool scan_mask = !dst->src[6] && !use_sparse && mask && mask->ne[2] == 1 && n_kv_cols % FATTN_KQ_STRIDE == 0 && (Q->ne[1] >= 1024 || Q->ne[3] > 1 || kv_rows);
if (scan_mask) {
const int64_t s31 = mask->nb[1] / sizeof(half2);
const int64_t s33 = mask->nb[3] / sizeof(half2);
@@ -1324,7 +1342,12 @@ void launch_fattn(
K->ne[0], n_kv, K->ne[2], K->ne[3], nb11, nb12, nb13,
nb21, nb22, nb23,
mask ? mask->ne[1] : 0, mask ? mask->ne[2] : 0, mask ? mask->ne[3] : 0,
mask ? mask->nb[1] : 0, mask ? mask->nb[2] : 0, mask ? mask->nb[3] : 0
mask ? mask->nb[1] : 0, mask ? mask->nb[2] : 0, mask ? mask->nb[3] : 0,
fattn_indexed_bias{dst->src[6] ? (const float *) dst->src[6]->data : nullptr,
dst->src[7] ? (const int32_t *) dst->src[7]->data : nullptr,
dst->src[6] ? dst->src[6]->ne[0] : 0,
dst->src[7] ? dst->src[7]->ne[0] : 0,
dst->src[6] ? dst->src[6]->ne[0]*dst->src[6]->ne[1] : 0}
);
CUDA_CHECK(cudaGetLastError());
+3 -1
View File
@@ -1960,7 +1960,9 @@ static __global__ void flash_attn_ext_f16(
const int32_t nb11, const int32_t nb12, const int64_t nb13,
const int32_t nb21, const int32_t nb22, const int64_t nb23,
const int32_t ne31, const int32_t ne32, const int32_t ne33,
const int32_t nb31, const int64_t nb32, const int64_t nb33) {
const int32_t nb31, const int64_t nb32, const int64_t nb33,
const fattn_indexed_bias bias) {
GGML_UNUSED(bias);
ggml_cuda_pdl_sync(); // TODO optimize placement
#if defined(FLASH_ATTN_AVAILABLE) && (defined(VOLTA_MMA_AVAILABLE) || defined(TURING_MMA_AVAILABLE) || defined(AMD_WMMA_AVAILABLE) || defined(AMD_MFMA_AVAILABLE))
const char * GGML_CUDA_RESTRICT Q = Q_ptr;
+61 -37
View File
@@ -374,9 +374,9 @@ static constexpr __device__ int ggml_cuda_fattn_tile_get_nbatch_K(const int DKQ,
}
// TODO: deduplicate with mma-f16
template<int warp_size, int nwarps, int I, int J, int J_padding, bool oob_check>
template<int warp_size, int nwarps, int I, int J, int J_padding, bool oob_check, bool use_rows>
static __device__ __forceinline__ void flash_attn_tile_load_tile(
const half2 * const __restrict__ KV, half2 * const __restrict__ tile_KV, const int stride_KV, const int i_sup) {
const half2 * const __restrict__ KV, half2 * const __restrict__ tile_KV, const int stride_KV, const int i_sup, const int * rows) {
constexpr int cpy_nb = ggml_cuda_get_max_cpy_bytes();
constexpr int cpy_ne = cpy_nb / 4;
@@ -400,6 +400,7 @@ static __device__ __forceinline__ void flash_attn_tile_load_tile(
const int i = i0 + threadIdx.y*stride_i + (stride_j == warp_size ? 0 : threadIdx.x / stride_j);
if (i0 + nwarps*stride_i <= I || i < I) {
const int row = use_rows && (!oob_check || i < i_sup) ? rows[i] : i;
#pragma unroll
for (int j0 = j0_start; j0 < j0_stop; j0 += stride_j) {
const int j = j0*cpy_ne + (stride_j == warp_size ? threadIdx.x : threadIdx.x % stride_j)*cpy_ne;
@@ -407,7 +408,7 @@ static __device__ __forceinline__ void flash_attn_tile_load_tile(
const __align__(16) half2 zero[cpy_ne] = {{0.0f, 0.0f}};
ggml_cuda_memcpy_1<cpy_nb>(
tile_KV + i*(J/2 + J_padding) + j,
!oob_check || i < i_sup ? KV + i*stride_KV + j : zero);
(!oob_check || i < i_sup) && (!use_rows || row >= 0) ? KV + int64_t(row)*stride_KV + j : zero);
}
}
}
@@ -424,9 +425,9 @@ static __device__ __forceinline__ void flash_attn_tile_load_tile(
ggml_cuda_unroll<7>{}(load);
}
template<int warp_size, int nwarps, int I, int J, int J_padding, bool oob_check>
template<int warp_size, int nwarps, int I, int J, int J_padding, bool oob_check, bool use_rows>
static __device__ __forceinline__ void flash_attn_tile_load_tile(
const half2 * const __restrict__ KV, float * const __restrict__ tile_KV, const int stride_KV, const int i_sup) {
const half2 * const __restrict__ KV, float * const __restrict__ tile_KV, const int stride_KV, const int i_sup, const int * rows) {
constexpr int cpy_nb = ggml_cuda_get_max_cpy_bytes();
constexpr int cpy_ne = cpy_nb / 4;
@@ -450,6 +451,7 @@ static __device__ __forceinline__ void flash_attn_tile_load_tile(
const int i = i0 + threadIdx.y*stride_i + (stride_j == warp_size ? 0 : threadIdx.x / stride_j);
if (i0 + nwarps*stride_i <= I || i < I) {
const int row = use_rows && (!oob_check || i < i_sup) ? rows[i] : i;
#pragma unroll
for (int j0 = j0_start; j0 < j0_stop; j0 += stride_j) {
const int j = j0*(cpy_ne/2) + (stride_j == warp_size ? threadIdx.x : threadIdx.x % stride_j)*(cpy_ne/2);
@@ -457,7 +459,7 @@ static __device__ __forceinline__ void flash_attn_tile_load_tile(
const half2 zero[cpy_ne/2] = {{0.0f, 0.0f}};
__align__(16) half2 tmp_h2[cpy_ne/2];
ggml_cuda_memcpy_1<sizeof(tmp_h2)>(
tmp_h2, !oob_check || i < i_sup ? KV + i*stride_KV + j : zero);
tmp_h2, (!oob_check || i < i_sup) && (!use_rows || row >= 0) ? KV + int64_t(row)*stride_KV + j : zero);
__align__(16) float2 tmp_f2[cpy_ne/2];
#pragma unroll
@@ -481,16 +483,15 @@ static __device__ __forceinline__ void flash_attn_tile_load_tile(
// Function that performs a single iteration in for the KQ matrix multiplication:
template <int warp_size, int nwarps, int ncols1, int ncols2, int DKQ, int nbatch_fa, int nbatch_K,
bool use_logit_softcap, bool oob_check, typename T_vec_dot>
bool use_logit_softcap, bool oob_check, bool use_rows, typename T_vec_dot>
static __device__ __forceinline__ void flash_attn_tile_iter_KQ(
T_vec_dot * const Q_tmp,
const half2 * const __restrict__ K_h2,
T_vec_dot * const KV_tmp,
const int stride_K2,
const int k_VKQ_0,
const int k_VKQ_sup,
const int k_KQ_0,
float * KQ_acc) {
float * KQ_acc, const int * rows) {
constexpr int cpy_nb = ggml_cuda_get_max_cpy_bytes();
constexpr int cpy_ne = cpy_nb / 4;
@@ -498,8 +499,8 @@ static __device__ __forceinline__ void flash_attn_tile_iter_KQ(
constexpr int cpw = ncols > nwarps ? ncols/nwarps : 1; // Q columns per warp
constexpr int np = nwarps > ncols ? nwarps/ncols : 1; // number of parallel warps per Q column
flash_attn_tile_load_tile<warp_size, nwarps, nbatch_fa, nbatch_K, cpy_ne, oob_check>
(K_h2 + int64_t(k_VKQ_0)*stride_K2 + k_KQ_0/2, KV_tmp, stride_K2, k_VKQ_sup);
flash_attn_tile_load_tile<warp_size, nwarps, nbatch_fa, nbatch_K, cpy_ne, oob_check, use_rows>
(K_h2 + k_KQ_0/2, KV_tmp, stride_K2, k_VKQ_sup, rows);
__syncthreads();
#ifdef FAST_FP16_AVAILABLE
@@ -556,7 +557,7 @@ static __device__ __forceinline__ void flash_attn_tile_iter_KQ(
// Function that performs a single iteration of the main loop over up to nbatch_fa tokens.
template <int warp_size, int nwarps, int ncols1, int ncols2, int DKQ, int DV, int nbatch_fa, int nbatch_K,
bool use_logit_softcap, bool oob_check, typename T_vec_dot, typename T_KQ, typename T_acc>
bool use_logit_softcap, bool oob_check, bool use_rows, bool indexed_bias, typename T_vec_dot, typename T_KQ, typename T_acc>
static __device__ __forceinline__ void flash_attn_tile_iter(
T_vec_dot * const Q_tmp,
const half2 * const __restrict__ K_h2,
@@ -575,7 +576,8 @@ static __device__ __forceinline__ void flash_attn_tile_iter(
T_acc * const VKQ,
const int k_VKQ_0,
const int k_VKQ_max,
const int col_Q_0) {
const int col_Q_0,
const fattn_indexed_bias bias, const int * rows) {
constexpr int cpy_nb = ggml_cuda_get_max_cpy_bytes();
constexpr int cpy_ne = cpy_nb / 4;
@@ -594,6 +596,9 @@ static __device__ __forceinline__ void flash_attn_tile_iter(
#endif // FAST_FP16_AVAILABLE
static_assert(cpw % KQ_cs == 0, "bad KQ_cs");
const int k_VKQ_sup = k_VKQ_max - k_VKQ_0; // k supremum, only smaller k values have valid KV data
const half2 * K_tile = use_rows ? K_h2 : K_h2 + int64_t(k_VKQ_0)*stride_K2;
const half2 * V_tile = use_rows ? V_h2 : V_h2 + int64_t(k_VKQ_0)*stride_V2;
const int * tile_rows = use_rows ? rows + k_VKQ_0 : nullptr;
float KQ_max_new[cpw];
#pragma unroll
@@ -607,13 +612,13 @@ static __device__ __forceinline__ void flash_attn_tile_iter(
constexpr int nbatch_K_last = DKQ % nbatch_K;
#pragma unroll
for (int k_KQ_0 = 0; k_KQ_0 < DKQ - nbatch_K_last; k_KQ_0 += nbatch_K) {
flash_attn_tile_iter_KQ<warp_size, nwarps, ncols1, ncols2, DKQ, nbatch_fa, nbatch_K, use_logit_softcap, oob_check>(
Q_tmp, K_h2, KV_tmp, stride_K2, k_VKQ_0, k_VKQ_sup, k_KQ_0, KQ_acc);
flash_attn_tile_iter_KQ<warp_size, nwarps, ncols1, ncols2, DKQ, nbatch_fa, nbatch_K, use_logit_softcap, oob_check, use_rows>(
Q_tmp, K_tile, KV_tmp, stride_K2, k_VKQ_sup, k_KQ_0, KQ_acc, tile_rows);
}
if (nbatch_K_last > 0) {
constexpr int k_KQ_0 = DKQ - nbatch_K_last;
flash_attn_tile_iter_KQ<warp_size, nwarps, ncols1, ncols2, DKQ, nbatch_fa, nbatch_K_last, use_logit_softcap, oob_check>(
Q_tmp, K_h2, KV_tmp, stride_K2, k_VKQ_0, k_VKQ_sup, k_KQ_0, KQ_acc);
flash_attn_tile_iter_KQ<warp_size, nwarps, ncols1, ncols2, DKQ, nbatch_fa, nbatch_K_last, use_logit_softcap, oob_check, use_rows>(
Q_tmp, K_tile, KV_tmp, stride_K2, k_VKQ_sup, k_KQ_0, KQ_acc, tile_rows);
}
// Apply logit softcap + mask, update KQ_max:
@@ -636,8 +641,14 @@ static __device__ __forceinline__ void flash_attn_tile_iter(
}
if (!oob_check || i_KQ < k_VKQ_sup) {
KQ_acc[(i_KQ_0/(np*warp_size))*cpw + jc0] += (ncols2 > 1 || mask) ?
slope*__half2float(mask[j*stride_mask + k_VKQ_0 + i_KQ]) : 0.0f;
const int k = k_VKQ_0 + i_KQ;
if (ncols2 > 1 || mask) {
half m = mask[int64_t(j)*stride_mask + k];
if constexpr (indexed_bias) {
m = bias.add(m, j, k);
}
KQ_acc[(i_KQ_0/(np*warp_size))*cpw + jc0] += slope*__half2float(m);
}
KQ_max_new[jc0] = fmaxf(KQ_max_new[jc0], KQ_acc[(i_KQ_0/(np*warp_size))*cpw + jc0] + FATTN_KQ_MAX_OFFSET);
}
@@ -718,8 +729,8 @@ static __device__ __forceinline__ void flash_attn_tile_iter(
static_assert(nbatch_V % np == 0, "bad nbatch_V");
#pragma unroll
for (int k0 = 0; k0 < nbatch_fa; k0 += nbatch_V) {
flash_attn_tile_load_tile<warp_size, nwarps, nbatch_V, DV, 0, oob_check>
(V_h2 + int64_t(k_VKQ_0 + k0)*stride_V2, KV_tmp, stride_V2, k_VKQ_sup - k0);
flash_attn_tile_load_tile<warp_size, nwarps, nbatch_V, DV, 0, oob_check, use_rows>
(use_rows ? V_tile : V_tile + int64_t(k0)*stride_V2, KV_tmp, stride_V2, k_VKQ_sup - k0, use_rows ? tile_rows + k0 : nullptr);
__syncthreads();
#ifdef FAST_FP16_AVAILABLE
@@ -788,7 +799,7 @@ static __device__ __forceinline__ void flash_attn_tile_iter(
}
}
template<int DKQ, int DV, int ncols1, int ncols2, bool use_logit_softcap> // D == head size
template<int DKQ, int DV, int ncols1, int ncols2, bool use_logit_softcap, bool indexed_bias, bool use_rows> // D == head size
__launch_bounds__(ggml_cuda_fattn_tile_get_nthreads(DKQ, DV, ncols1*ncols2), ggml_cuda_fattn_tile_get_occupancy(DKQ, DV, ncols1*ncols2))
static __global__ void flash_attn_tile(
const char * Q_ptr,
@@ -812,7 +823,8 @@ static __global__ void flash_attn_tile(
const int32_t nb11, const int32_t nb12, const int64_t nb13,
const int32_t nb21, const int32_t nb22, const int64_t nb23,
const int32_t ne31, const int32_t ne32, const int32_t ne33,
const int32_t nb31, const int64_t nb32, const int64_t nb33) {
const int32_t nb31, const int64_t nb32, const int64_t nb33,
const fattn_indexed_bias bias) {
#ifdef FLASH_ATTN_AVAILABLE
const char * GGML_CUDA_RESTRICT Q = Q_ptr;
const char * GGML_CUDA_RESTRICT K = K_ptr;
@@ -822,7 +834,6 @@ static __global__ void flash_attn_tile(
const int * GGML_CUDA_RESTRICT KV_max = KV_max_ptr;
float * GGML_CUDA_RESTRICT dst = dst_ptr;
float2 * GGML_CUDA_RESTRICT dst_meta = dst_meta_ptr;
GGML_UNUSED(KV_rows_ptr);
// Skip unused kernel variants for faster compilation:
@@ -853,7 +864,9 @@ static __global__ void flash_attn_tile(
const int col_Q_0 = blockIdx.x * ncols1; // Index of the first Q column for this CUDA block to work on.
const int sequence = blockIdx.z / (ne02/ncols2);
const int * rows = use_rows ? KV_rows_ptr + int64_t(sequence)*ne11 : nullptr;
const int head0 = blockIdx.z*ncols2 - sequence*ne02; // == blockIdx.z % (ne02/ncols2)
const auto head_bias = indexed_bias ? bias.for_head(int64_t(sequence)*ne01.z, head0) : bias;
const int gqa_ratio = ne02 / ne12; // With grouped query attention there are > 1 Q matrices per K, V matrix.
const float * Q_f = (const float *) (Q + nb03*sequence + nb02* head0);
const half2 * K_h2 = (const half2 *) (K + nb13*sequence + nb12*(head0 / gqa_ratio));
@@ -959,24 +972,24 @@ static __global__ void flash_attn_tile(
int k_VKQ_0 = blockIdx.y*nbatch_fa;
while (k_VKQ_0 <= k_VKQ_max - nbatch_fa) {
constexpr bool oob_check = false;
flash_attn_tile_iter<warp_size, nwarps, ncols1, ncols2, DKQ, DV, nbatch_fa, nbatch_K, use_logit_softcap, oob_check>
flash_attn_tile_iter<warp_size, nwarps, ncols1, ncols2, DKQ, DV, nbatch_fa, nbatch_K, use_logit_softcap, oob_check, use_rows, indexed_bias>
(Q_tmp, K_h2, V_h2, maskh, ne01, logit_softcap, slope, KQ, KV_tmp,
stride_K2, stride_V2, stride_mask, KQ_max, KQ_sum, VKQ, k_VKQ_0, k_VKQ_max, col_Q_0);
stride_K2, stride_V2, stride_mask, KQ_max, KQ_sum, VKQ, k_VKQ_0, k_VKQ_max, col_Q_0, head_bias, rows);
k_VKQ_0 += gridDim.y*nbatch_fa;
}
if (k_VKQ_0 < k_VKQ_max) {
constexpr bool oob_check = true;
flash_attn_tile_iter<warp_size, nwarps, ncols1, ncols2, DKQ, DV, nbatch_fa, nbatch_K, use_logit_softcap, oob_check>
flash_attn_tile_iter<warp_size, nwarps, ncols1, ncols2, DKQ, DV, nbatch_fa, nbatch_K, use_logit_softcap, oob_check, use_rows, indexed_bias>
(Q_tmp, K_h2, V_h2, maskh, ne01, logit_softcap, slope, KQ, KV_tmp,
stride_K2, stride_V2, stride_mask, KQ_max, KQ_sum, VKQ, k_VKQ_0, k_VKQ_max, col_Q_0);
stride_K2, stride_V2, stride_mask, KQ_max, KQ_sum, VKQ, k_VKQ_0, k_VKQ_max, col_Q_0, head_bias, rows);
}
} else {
// Branch without out-of-bounds checks.
for (int k_VKQ_0 = blockIdx.y*nbatch_fa; k_VKQ_0 < k_VKQ_max; k_VKQ_0 += gridDim.y*nbatch_fa) {
constexpr bool oob_check = false;
flash_attn_tile_iter<warp_size, nwarps, ncols1, ncols2, DKQ, DV, nbatch_fa, nbatch_K, use_logit_softcap, oob_check>
flash_attn_tile_iter<warp_size, nwarps, ncols1, ncols2, DKQ, DV, nbatch_fa, nbatch_K, use_logit_softcap, oob_check, use_rows, indexed_bias>
(Q_tmp, K_h2, V_h2, maskh, ne01, logit_softcap, slope, KQ, KV_tmp,
stride_K2, stride_V2, stride_mask, KQ_max, KQ_sum, VKQ, k_VKQ_0, k_VKQ_max, col_Q_0);
stride_K2, stride_V2, stride_mask, KQ_max, KQ_sum, VKQ, k_VKQ_0, k_VKQ_max, col_Q_0, head_bias, rows);
}
}
@@ -1147,7 +1160,7 @@ static __global__ void flash_attn_tile(
#endif // FLASH_ATTN_AVAILABLE
}
template <int DKQ, int DV, int ncols2, bool use_logit_softcap>
template <int DKQ, int DV, int ncols2, bool use_logit_softcap, bool indexed_bias = false, bool use_rows = false>
static void launch_fattn_tile_switch_ncols1(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
const ggml_tensor * Q = dst->src[0];
@@ -1163,7 +1176,7 @@ static void launch_fattn_tile_switch_ncols1(ggml_backend_cuda_context & ctx, ggm
constexpr int cols_per_block = 64;
const int nwarps = ggml_cuda_fattn_tile_get_nthreads (DKQ, DV, cols_per_block, cc) / warp_size;
const int nbatch_fa = ggml_cuda_fattn_tile_get_nbatch_fa(DKQ, DV, cols_per_block, cc);
fattn_kernel_t fattn_kernel = flash_attn_tile<DKQ, DV, cols_per_block/ncols2, ncols2, use_logit_softcap>;
fattn_kernel_t fattn_kernel = flash_attn_tile<DKQ, DV, cols_per_block/ncols2, ncols2, use_logit_softcap, indexed_bias, use_rows>;
launch_fattn<DV, cols_per_block/ncols2, ncols2>
(ctx, dst, fattn_kernel, nwarps, nbytes_shared, nbatch_fa, true, true, false, false, warp_size);
return;
@@ -1179,7 +1192,7 @@ static void launch_fattn_tile_switch_ncols1(ggml_backend_cuda_context & ctx, ggm
constexpr int cols_per_block = 32;
const int nwarps = ggml_cuda_fattn_tile_get_nthreads (DKQ, DV, cols_per_block, cc) / warp_size;
const int nbatch_fa = ggml_cuda_fattn_tile_get_nbatch_fa(DKQ, DV, cols_per_block, cc);
fattn_kernel_t fattn_kernel = flash_attn_tile<DKQ, DV, cols_per_block/ncols2, ncols2, use_logit_softcap>;
fattn_kernel_t fattn_kernel = flash_attn_tile<DKQ, DV, cols_per_block/ncols2, ncols2, use_logit_softcap, indexed_bias, use_rows>;
launch_fattn<DV, cols_per_block/ncols2, ncols2>
(ctx, dst, fattn_kernel, nwarps, nbytes_shared, nbatch_fa, true, true, false, false, warp_size);
return;
@@ -1191,7 +1204,7 @@ static void launch_fattn_tile_switch_ncols1(ggml_backend_cuda_context & ctx, ggm
constexpr int cols_per_block = 16;
const int nwarps = ggml_cuda_fattn_tile_get_nthreads (DKQ, DV, cols_per_block, cc) / warp_size;
const int nbatch_fa = ggml_cuda_fattn_tile_get_nbatch_fa(DKQ, DV, cols_per_block, cc);
fattn_kernel_t fattn_kernel = flash_attn_tile<DKQ, DV, cols_per_block/ncols2, ncols2, use_logit_softcap>;
fattn_kernel_t fattn_kernel = flash_attn_tile<DKQ, DV, cols_per_block/ncols2, ncols2, use_logit_softcap, indexed_bias, use_rows>;
launch_fattn<DV, cols_per_block/ncols2, ncols2>
(ctx, dst, fattn_kernel, nwarps, nbytes_shared, nbatch_fa, true, true, false, false, warp_size);
return;
@@ -1203,7 +1216,7 @@ static void launch_fattn_tile_switch_ncols1(ggml_backend_cuda_context & ctx, ggm
constexpr int cols_per_block = 8;
const int nwarps = ggml_cuda_fattn_tile_get_nthreads (DKQ, DV, cols_per_block, cc) / warp_size;
const int nbatch_fa = ggml_cuda_fattn_tile_get_nbatch_fa(DKQ, DV, cols_per_block, cc);
fattn_kernel_t fattn_kernel = flash_attn_tile<DKQ, DV, cols_per_block/ncols2, ncols2, use_logit_softcap>;
fattn_kernel_t fattn_kernel = flash_attn_tile<DKQ, DV, cols_per_block/ncols2, ncols2, use_logit_softcap, indexed_bias, use_rows>;
launch_fattn<DV, cols_per_block/ncols2, ncols2>
(ctx, dst, fattn_kernel, nwarps, nbytes_shared, nbatch_fa, true, true, false, false, warp_size);
return;
@@ -1215,7 +1228,7 @@ static void launch_fattn_tile_switch_ncols1(ggml_backend_cuda_context & ctx, ggm
constexpr int cols_per_block = 4;
const int nwarps = ggml_cuda_fattn_tile_get_nthreads (DKQ, DV, cols_per_block, cc) / warp_size;
const int nbatch_fa = ggml_cuda_fattn_tile_get_nbatch_fa(DKQ, DV, cols_per_block, cc);
fattn_kernel_t fattn_kernel = flash_attn_tile<DKQ, DV, cols_per_block/ncols2, ncols2, use_logit_softcap>;
fattn_kernel_t fattn_kernel = flash_attn_tile<DKQ, DV, cols_per_block/ncols2, ncols2, use_logit_softcap, indexed_bias, use_rows>;
launch_fattn<DV, cols_per_block/ncols2, ncols2>
(ctx, dst, fattn_kernel, nwarps, nbytes_shared, nbatch_fa, true, true, false, false, warp_size);
return;
@@ -1226,7 +1239,7 @@ static void launch_fattn_tile_switch_ncols1(ggml_backend_cuda_context & ctx, ggm
constexpr int cols_per_block = 2;
const int nwarps = ggml_cuda_fattn_tile_get_nthreads (DKQ, DV, cols_per_block, cc) / warp_size;
const int nbatch_fa = ggml_cuda_fattn_tile_get_nbatch_fa(DKQ, DV, cols_per_block, cc);
fattn_kernel_t fattn_kernel = flash_attn_tile<DKQ, DV, cols_per_block/ncols2, ncols2, use_logit_softcap>;
fattn_kernel_t fattn_kernel = flash_attn_tile<DKQ, DV, cols_per_block/ncols2, ncols2, use_logit_softcap, indexed_bias, use_rows>;
launch_fattn<DV, cols_per_block/ncols2, ncols2>
(ctx, dst, fattn_kernel, nwarps, nbytes_shared, nbatch_fa, true, true, false, false, warp_size);
return;
@@ -1255,6 +1268,17 @@ static void launch_fattn_tile_switch_ncols2(ggml_backend_cuda_context & ctx, ggm
// a mask with a head dimension is not shared by the Q heads of a K/V head
const bool use_gqa_opt = mask && mask->ne[2] == 1 && max_bias == 0.0f && Q->ne[1] <= gqa_limit && K->ne[1] % FATTN_KQ_STRIDE == 0;
if constexpr (DKQ == 64 || DKQ == 128) {
if (dst->src[6]) {
if (dst->src[5]) {
launch_fattn_tile_switch_ncols1<DKQ, DV, 1, use_logit_softcap, true, true>(ctx, dst);
} else {
launch_fattn_tile_switch_ncols1<DKQ, DV, 1, use_logit_softcap, true, false>(ctx, dst);
}
return;
}
}
if constexpr (DKQ == 320) {
// This branch is only used for Mistral Small 4 which has a GQA ratio of 32.
// On AMD, simply use that GQA ratio with 32 columns / block since we always have enough SRAM.
+35 -16
View File
@@ -16,7 +16,7 @@ static constexpr __device__ int ggml_cuda_fattn_vec_get_nthreads_device() {
#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wpass-failed"
#endif // __clang__
template<int D, int ncols, ggml_type type_K, ggml_type type_V, bool use_logit_softcap> // D == head size
template<int D, int ncols, ggml_type type_K, ggml_type type_V, bool use_logit_softcap, bool indexed_bias = false, bool use_rows = false> // D == head size
__launch_bounds__(ggml_cuda_fattn_vec_get_nthreads_device(), 1)
static __global__ void flash_attn_ext_vec(
const char * Q_ptr,
@@ -40,7 +40,8 @@ static __global__ void flash_attn_ext_vec(
const int32_t nb11, const int32_t nb12, const int64_t nb13,
const int32_t nb21, const int32_t nb22, const int64_t nb23,
const int32_t ne31, const int32_t ne32, const int32_t ne33,
const int32_t nb31, const int64_t nb32, const int64_t nb33) {
const int32_t nb31, const int64_t nb32, const int64_t nb33,
const fattn_indexed_bias bias) {
ggml_cuda_pdl_lc();
#ifdef FLASH_ATTN_AVAILABLE
const char * GGML_CUDA_RESTRICT Q = Q_ptr;
@@ -51,7 +52,6 @@ static __global__ void flash_attn_ext_vec(
const int * GGML_CUDA_RESTRICT KV_max = KV_max_ptr;
float * GGML_CUDA_RESTRICT dst = dst_ptr;
float2 * GGML_CUDA_RESTRICT dst_meta = dst_meta_ptr;
GGML_UNUSED(KV_rows_ptr);
// Skip unused kernel variants for faster compilation:
if (use_logit_softcap && !(D == 128 || D == 256)) {
@@ -107,6 +107,7 @@ static __global__ void flash_attn_ext_vec(
const int sequence = blockIdx.z / ne02;
const int head = blockIdx.z - sequence*ne02;
const auto head_bias = indexed_bias ? bias.for_head(int64_t(sequence)*ne01.z, head) : bias;
const int gqa_ratio = ne02 / ne12; // With grouped query attention there are > 1 Q matrices per K, V matrix.
Q += nb03*sequence + nb02* head + nb01*ic0;
K += nb13*sequence + nb12*(head / gqa_ratio);
@@ -271,12 +272,15 @@ static __global__ void flash_attn_ext_vec(
}
const int k_VKQ_max = KV_max ? KV_max[sequence*gridDim.x + blockIdx.x] : ne11;
K += blockIdx.y*nthreads * nb11;
V += blockIdx.y*nthreads * nb21;
const int * rows = use_rows ? KV_rows_ptr + int64_t(sequence)*ne11 : nullptr;
if constexpr (!use_rows) {
K += blockIdx.y*nthreads * nb11;
V += blockIdx.y*nthreads * nb21;
}
maskh += blockIdx.y*nthreads;
for (int k_VKQ_0 = blockIdx.y*nthreads; k_VKQ_0 < k_VKQ_max; k_VKQ_0 += gridDim.y*nthreads,
// Increment pointers after each loop:
K += gridDim.y*nthreads*nb11, V += gridDim.y*nthreads*nb21, maskh += gridDim.y*nthreads) {
K += use_rows ? 0 : gridDim.y*nthreads*nb11, V += use_rows ? 0 : gridDim.y*nthreads*nb21, maskh += gridDim.y*nthreads) {
// Calculate KQ tile and keep track of new maximum KQ values:
float KQ_reg[ncols]; // KQ in registers.
@@ -290,22 +294,24 @@ static __global__ void flash_attn_ext_vec(
#pragma unroll
for (int i_KQ_0 = 0; i_KQ_0 < nthreads_KQ; ++i_KQ_0) {
const int i_KQ = threadIdx.y*WARP_SIZE + (nthreads_KQ == WARP_SIZE ? 0 : (threadIdx.x & ~(nthreads_KQ-1))) + i_KQ_0;
const int row = use_rows ? rows[k_VKQ_0 + i_KQ] : i_KQ;
#pragma unroll
for (int j = 0; j < ncols; ++j) {
float sum = vec_dot_KQ(K + i_KQ*nb11, Q_reg[j], Q_i32[j], Q_ds[j]);
float sum = vec_dot_KQ(K + (use_rows ? int64_t(max(row, 0))*nb11 : i_KQ*nb11), Q_reg[j], Q_i32[j], Q_ds[j]);
sum = warp_reduce_sum<nthreads_KQ>(sum);
if (use_logit_softcap) {
sum = logit_softcap*tanhf(sum);
}
if (mask && (ncols == 1 || ic0 + j < int(ne01.z))) {
sum += slope*__half2float(maskh[j*ne11 + i_KQ]);
if constexpr (!indexed_bias) {
if (mask && (ncols == 1 || ic0 + j < int(ne01.z))) {
sum += slope*__half2float(maskh[j*ne11 + i_KQ]);
}
KQ_max_new[j] = fmaxf(KQ_max_new[j], sum + FATTN_KQ_MAX_OFFSET);
}
KQ_max_new[j] = fmaxf(KQ_max_new[j], sum + FATTN_KQ_MAX_OFFSET);
if ((nthreads_KQ == WARP_SIZE ? threadIdx.x : threadIdx.x % nthreads_KQ) == uint32_t(i_KQ_0)) {
KQ_reg[j] = sum;
}
@@ -314,8 +320,14 @@ static __global__ void flash_attn_ext_vec(
#pragma unroll
for (int j = 0; j < ncols; ++j) {
if constexpr (indexed_bias) {
const int k = k_VKQ_0 + tid;
const half m = use_rows && rows[k] < 0 ? half(-INFINITY) : head_bias.add(maskh[j*ne11 + tid], ic0 + j, k);
KQ_reg[j] += slope*__half2float(m);
KQ_max_new[j] = fmaxf(KQ_max[j], KQ_reg[j] + FATTN_KQ_MAX_OFFSET);
}
#pragma unroll
for (int offset = nthreads_KQ; offset < WARP_SIZE; offset <<= 1) {
for (int offset = indexed_bias ? 1 : nthreads_KQ; offset < WARP_SIZE; offset <<= 1) {
KQ_max_new[j] = fmaxf(KQ_max_new[j], ggml_cuda_shfl_xor(KQ_max_new[j], offset, WARP_SIZE));
}
const float KQ_max_scale = expf(KQ_max[j] - KQ_max_new[j]);
@@ -345,6 +357,7 @@ static __global__ void flash_attn_ext_vec(
#pragma unroll
for (int k0 = 0; k0 < WARP_SIZE; k0 += V_cols_per_iter) {
const int k = threadIdx.y*WARP_SIZE + k0 + (nthreads_V == WARP_SIZE ? 0 : threadIdx.x / nthreads_V);
const char * V_row = V + (use_rows ? int64_t(max(rows[k_VKQ_0 + k], 0))*nb21 : k*nb21);
#ifdef V_DOT2_F32_F16_AVAILABLE
half2 KQ_k[ncols];
@@ -357,14 +370,14 @@ static __global__ void flash_attn_ext_vec(
half2 tmp[V_rows_per_thread/2];
if constexpr (type_V == GGML_TYPE_BF16) {
float2 tmp_f[V_rows_per_thread/2];
dequantize_V(V + k*nb21, tmp_f,
dequantize_V(V_row, tmp_f,
2*i_VKQ_0 + (nthreads_V == WARP_SIZE ? threadIdx.x : threadIdx.x % nthreads_V)*V_rows_per_thread);
#pragma unroll
for (int i_VKQ_1 = 0; i_VKQ_1 < V_rows_per_thread/2; ++i_VKQ_1) {
tmp[i_VKQ_1] = __float22half2_rn(tmp_f[i_VKQ_1]);
}
} else {
dequantize_V(V + k*nb21, tmp,
dequantize_V(V_row, tmp,
2*i_VKQ_0 + (nthreads_V == WARP_SIZE ? threadIdx.x : threadIdx.x % nthreads_V)*V_rows_per_thread);
}
#pragma unroll
@@ -384,7 +397,7 @@ static __global__ void flash_attn_ext_vec(
#pragma unroll
for (int i_VKQ_0 = 0; i_VKQ_0 < D/2; i_VKQ_0 += nthreads_V*V_rows_per_thread/2) {
float2 tmp[V_rows_per_thread/2];
dequantize_V(V + k*nb21, tmp,
dequantize_V(V_row, tmp,
2*i_VKQ_0 + (nthreads_V == WARP_SIZE ? threadIdx.x : threadIdx.x % nthreads_V)*V_rows_per_thread);
#pragma unroll
for (int i_VKQ_1 = 0; i_VKQ_1 < V_rows_per_thread/2; ++i_VKQ_1) {
@@ -545,7 +558,7 @@ static __global__ void flash_attn_ext_vec(
nb11, nb12, nb13,
nb21, nb22, nb23,
ne31, ne32, ne33,
nb31, nb32, nb33);
nb31, nb32, nb33, bias);
NO_DEVICE_CODE;
#endif // FLASH_ATTN_AVAILABLE
}
@@ -560,6 +573,12 @@ void ggml_cuda_flash_attn_ext_vec_case_impl(ggml_backend_cuda_context & ctx, ggm
const int nthreads = ggml_cuda_fattn_vec_get_nthreads_host(cc);
const int nwarps = nthreads / WARP_SIZE;
fattn_kernel_t fattn_kernel = flash_attn_ext_vec<D, cols_per_block, type_K, type_V, use_logit_softcap>;
if constexpr (cols_per_block == 1 && (D == 64 || D == 128) && !use_logit_softcap) {
if (dst->src[6]) {
fattn_kernel = dst->src[5] ? flash_attn_ext_vec<D, cols_per_block, type_K, type_V, use_logit_softcap, true, true> :
flash_attn_ext_vec<D, cols_per_block, type_K, type_V, use_logit_softcap, true, false>;
}
}
const bool need_f16_K = type_K == GGML_TYPE_F16;
const bool need_f16_V = type_V == GGML_TYPE_F16;
constexpr size_t nbytes_shared = 0;
+12
View File
@@ -669,6 +669,18 @@ static best_fattn_kernel ggml_cuda_get_best_fattn_kernel(const int device, const
return BEST_FATTN_KERNEL_NONE;
}
if (ggml_flash_attn_ext_has_indexed_bias(dst)) {
const float softcap = ggml_get_op_params_f32(dst, 2);
if (Q->ne[0] != 128 && !(Q->ne[0] == 64 && softcap == 0.0f)) {
return BEST_FATTN_KERNEL_NONE;
}
if (softcap == 0.0f && Q->ne[1] == 1 && n_kv % FATTN_KQ_STRIDE == 0 &&
ggml_cuda_get_fattn_vec_case(Q->ne[0], K->type, V->type)) {
return BEST_FATTN_KERNEL_VEC;
}
return BEST_FATTN_KERNEL_TILE;
}
if (mask && mask->ne[2] != 1) {
// the kv_rows mask is compacted per sequence, it has no head dimension
if (kv_rows) {
+1
View File
@@ -1277,6 +1277,7 @@ static bool ggml_backend_et_device_supports_op(ggml_backend_dev_t dev, const ggm
if (op->type == GGML_TYPE_F32 && op->src[0] && op->src[0]->type == GGML_TYPE_F32 && op->src[1] &&
(op->src[1]->type == GGML_TYPE_F32 || op->src[1]->type == GGML_TYPE_F16) && op->src[2] &&
(op->src[2]->type == GGML_TYPE_F32 || op->src[2]->type == GGML_TYPE_F16) && op->src[4] == nullptr && op->src[5] == nullptr &&
!ggml_flash_attn_ext_has_indexed_bias(op) &&
ggml_is_contiguous_rows(op) && ggml_is_contiguous_rows(op->src[0])) {
float max_bias = 0.0f;
float logit_softcap = 0.0f;
+2 -2
View File
@@ -355,8 +355,8 @@ static bool htp_kparams_solve_flash_attn(
return false;
}
// K/V row lists (ggml_flash_attn_ext_rows) are not supported
if (op->src[5] != nullptr) {
// K/V row lists and indexed bias are not supported
if (op->src[5] != nullptr || ggml_flash_attn_ext_has_indexed_bias(op)) {
return false;
}
+3
View File
@@ -1744,6 +1744,9 @@ bool ggml_metal_device_supports_op(ggml_metal_device_t dev, const struct ggml_te
case GGML_OP_ROLL:
return ggml_is_contiguous(op->src[0]);
case GGML_OP_FLASH_ATTN_EXT:
if (ggml_flash_attn_ext_has_indexed_bias(op)) {
return false;
}
// K/V row lists (ggml_flash_attn_ext_rows): one list of cache rows per q slice
if (op->src[5] != NULL) {
const struct ggml_tensor * kv_rows = op->src[5];
+7
View File
@@ -227,6 +227,13 @@ set(GGML_OPENCL_KERNELS
gemm_coopk_q4_0_q8_1_dp4a
gemm_coopk_q4_k_q8_1_dp4a
gemm_coopk_q6_k_q8_1_dp4a
pack_b_f16_coopmat
gemm_noshuffle_q4_0_f32_coopmat
gemm_noshuffle_q5_0_f32_coopmat
gemm_noshuffle_q8_0_f32_coopmat
gemm_noshuffle_q4_k_f32_coopmat
gemm_noshuffle_q5_k_f32_coopmat
gemm_noshuffle_q6_k_f32_coopmat
quant_a_q8_1
gemv_moe_q4_k_f32_ns
gemm_moe_q5_k_f32_ns
+221 -6
View File
@@ -1495,6 +1495,13 @@ struct ggml_backend_opencl_context {
cl_kernel kernel_gemm_noshuffle_iq4_nl_q8_1_dp4a = nullptr; // dp4a (int8) dense IQ4_NL prefill GEMM
cl_kernel kernel_gemm_noshuffle_q4_0_q8_1_dp4a = nullptr; // dp4a (int8) dense q4_0 prefill GEMM
cl_kernel kernel_gemm_noshuffle_q4_0_q8_1_dp4a_act_vload4 = nullptr;
cl_kernel kernel_pack_b_f16_coopmat = nullptr;
cl_kernel kernel_gemm_noshuffle_q4_0_f32_coopmat = nullptr;
cl_kernel kernel_gemm_noshuffle_q5_0_f32_coopmat = nullptr;
cl_kernel kernel_gemm_noshuffle_q8_0_f32_coopmat = nullptr;
cl_kernel kernel_gemm_noshuffle_q4_k_f32_coopmat = nullptr;
cl_kernel kernel_gemm_noshuffle_q5_k_f32_coopmat = nullptr;
cl_kernel kernel_gemm_noshuffle_q6_k_f32_coopmat = nullptr;
cl_kernel kernel_gemm_coopk_q4_0_q8_1_dp4a_c4 = nullptr;
cl_kernel kernel_gemm_coopk_q4_0_q8_1_dp4a_c2 = nullptr;
int nsg_coopk_dp4a_q4_0 = 8;
@@ -1979,6 +1986,137 @@ static void load_cl_kernels_gemm_coopk(ggml_backend_opencl_context * backend_ctx
}
}
static void load_cl_kernels_coopmat(ggml_backend_opencl_context * backend_ctx) {
if (backend_ctx->gpu_family != GPU_FAMILY::ADRENO || !strstr(backend_ctx->device_name.c_str(), "850")) {
return;
}
cl_int err;
const std::string opts = "-cl-std=CL" + std::to_string(backend_ctx->opencl_c_version.major) + "." +
std::to_string(backend_ctx->opencl_c_version.minor) +
(backend_ctx->adreno_use_large_buffer ? " -qcom-enable-large-buffer" : "");
// pack_b_f16_coopmat
{
#ifdef GGML_OPENCL_EMBED_KERNELS
const std::string kernel_src {
#include "pack_b_f16_coopmat.cl.h"
};
#else
const std::string kernel_src = read_file("pack_b_f16_coopmat.cl");
#endif
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
if (!prog) {
return;
}
CL_CHECK((backend_ctx->kernel_pack_b_f16_coopmat = clCreateKernel(prog, "kernel_pack_b_f16_coopmat", &err), err));
CL_CHECK(clReleaseProgram(prog));
GGML_LOG_CONT(".");
}
// gemm_noshuffle_q4_0_f32_coopmat
{
#ifdef GGML_OPENCL_EMBED_KERNELS
const std::string kernel_src {
#include "gemm_noshuffle_q4_0_f32_coopmat.cl.h"
};
#else
const std::string kernel_src = read_file("gemm_noshuffle_q4_0_f32_coopmat.cl");
#endif
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
if (prog) {
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q4_0_f32_coopmat = clCreateKernel(prog, "kernel_gemm_noshuffle_q4_0_f32_coopmat", &err), err));
CL_CHECK(clReleaseProgram(prog));
GGML_LOG_CONT(".");
}
}
// gemm_noshuffle_q5_0_f32_coopmat
{
#ifdef GGML_OPENCL_EMBED_KERNELS
const std::string kernel_src {
#include "gemm_noshuffle_q5_0_f32_coopmat.cl.h"
};
#else
const std::string kernel_src = read_file("gemm_noshuffle_q5_0_f32_coopmat.cl");
#endif
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
if (prog) {
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q5_0_f32_coopmat = clCreateKernel(prog, "kernel_gemm_noshuffle_q5_0_f32_coopmat", &err), err));
CL_CHECK(clReleaseProgram(prog));
GGML_LOG_CONT(".");
}
}
// gemm_noshuffle_q8_0_f32_coopmat
{
#ifdef GGML_OPENCL_EMBED_KERNELS
const std::string kernel_src {
#include "gemm_noshuffle_q8_0_f32_coopmat.cl.h"
};
#else
const std::string kernel_src = read_file("gemm_noshuffle_q8_0_f32_coopmat.cl");
#endif
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
if (prog) {
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q8_0_f32_coopmat = clCreateKernel(prog, "kernel_gemm_noshuffle_q8_0_f32_coopmat", &err), err));
CL_CHECK(clReleaseProgram(prog));
GGML_LOG_CONT(".");
}
}
// gemm_noshuffle_q4_k_f32_coopmat
{
#ifdef GGML_OPENCL_EMBED_KERNELS
const std::string kernel_src {
#include "gemm_noshuffle_q4_k_f32_coopmat.cl.h"
};
#else
const std::string kernel_src = read_file("gemm_noshuffle_q4_k_f32_coopmat.cl");
#endif
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
if (prog) {
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q4_k_f32_coopmat = clCreateKernel(prog, "kernel_gemm_noshuffle_q4_k_f32_coopmat", &err), err));
CL_CHECK(clReleaseProgram(prog));
GGML_LOG_CONT(".");
}
}
// gemm_noshuffle_q5_k_f32_coopmat
{
#ifdef GGML_OPENCL_EMBED_KERNELS
const std::string kernel_src {
#include "gemm_noshuffle_q5_k_f32_coopmat.cl.h"
};
#else
const std::string kernel_src = read_file("gemm_noshuffle_q5_k_f32_coopmat.cl");
#endif
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
if (prog) {
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q5_k_f32_coopmat = clCreateKernel(prog, "kernel_gemm_noshuffle_q5_k_f32_coopmat", &err), err));
CL_CHECK(clReleaseProgram(prog));
GGML_LOG_CONT(".");
}
}
// gemm_noshuffle_q6_k_f32_coopmat
{
#ifdef GGML_OPENCL_EMBED_KERNELS
const std::string kernel_src {
#include "gemm_noshuffle_q6_k_f32_coopmat.cl.h"
};
#else
const std::string kernel_src = read_file("gemm_noshuffle_q6_k_f32_coopmat.cl");
#endif
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
if (prog) {
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q6_k_f32_coopmat = clCreateKernel(prog, "kernel_gemm_noshuffle_q6_k_f32_coopmat", &err), err));
CL_CHECK(clReleaseProgram(prog));
GGML_LOG_CONT(".");
}
}
}
#endif
static std::string ggml_opencl_make_compile_opts(const ggml_backend_opencl_context * backend_ctx) {
@@ -7813,6 +7951,7 @@ static void load_cl_kernels(ggml_backend_opencl_context *backend_ctx) {
}
}
load_cl_kernels_gemm_coopk(backend_ctx);
load_cl_kernels_coopmat(backend_ctx);
#endif // GGML_OPENCL_USE_ADRENO_KERNELS
GGML_LOG_CONT("\n");
backend_ctx->kernels_loaded = true;
@@ -11739,7 +11878,8 @@ inline bool use_q4_0_bin_kernels(const ggml_backend_opencl_context *backend_ctx,
#ifdef GGML_OPENCL_USE_ADRENO_KERNELS
if (!backend_ctx->kernel_gemv_noshuffle_q4_0_f32_32b_trans ||
!(backend_ctx->kernel_gemm_noshuffle_q4_0_f32_32b_trans_ila_a8_bin ||
backend_ctx->kernel_gemm_noshuffle_q4_0_f32_32b_trans_ila_a7_bin)) {
backend_ctx->kernel_gemm_noshuffle_q4_0_f32_32b_trans_ila_a7_bin ||
backend_ctx->kernel_gemm_noshuffle_q4_0_f32_coopmat)) {
return false;
}
@@ -11893,7 +12033,8 @@ inline bool use_q6_k_bin_kernels(const ggml_backend_opencl_context *backend_ctx,
#ifdef GGML_OPENCL_USE_ADRENO_KERNELS
if (!backend_ctx->kernel_gemv_noshuffle_q6_k_f32_32b_trans ||
!(backend_ctx->kernel_gemm_noshuffle_q6_k_f32_32b_trans_ila_a8_bin ||
backend_ctx->kernel_gemm_noshuffle_q6_k_f32_32b_trans_ila_a7_bin)) {
backend_ctx->kernel_gemm_noshuffle_q6_k_f32_32b_trans_ila_a7_bin ||
backend_ctx->kernel_gemm_noshuffle_q6_k_f32_coopmat)) {
return false;
}
// bin kernels require ne2 == 1 and ne3 == 1 for weights
@@ -11911,7 +12052,8 @@ inline bool use_q4_k_bin_kernels(const ggml_backend_opencl_context *backend_ctx,
#ifdef GGML_OPENCL_USE_ADRENO_KERNELS
if (!backend_ctx->kernel_gemv_noshuffle_q4_k_f32_32b_trans ||
!(backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a8_bin ||
backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a7_bin)) {
backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a7_bin ||
backend_ctx->kernel_gemm_noshuffle_q4_k_f32_coopmat)) {
return false;
}
// bin kernels require ne2 == 1 and ne3 == 1 for weights
@@ -11929,7 +12071,8 @@ inline bool use_q5_k_bin_kernels(const ggml_backend_opencl_context *backend_ctx,
#ifdef GGML_OPENCL_USE_ADRENO_KERNELS
if (!backend_ctx->kernel_gemv_noshuffle_q5_k_f32_32b_trans ||
!(backend_ctx->kernel_gemm_noshuffle_q5_k_f32_32b_trans_ila_a8_bin ||
backend_ctx->kernel_gemm_noshuffle_q5_k_f32_32b_trans_ila_a7_bin)) {
backend_ctx->kernel_gemm_noshuffle_q5_k_f32_32b_trans_ila_a7_bin ||
backend_ctx->kernel_gemm_noshuffle_q5_k_f32_coopmat)) {
return false;
}
return (tensor->ne[0] % 256 == 0) && (tensor->ne[1] % 64 == 0);
@@ -12426,8 +12569,8 @@ static bool ggml_opencl_supports_op(ggml_backend_dev_t dev, const struct ggml_te
case GGML_OP_MEAN:
return op->src[0]->type == GGML_TYPE_F32;
case GGML_OP_FLASH_ATTN_EXT: {
// K/V row lists (ggml_flash_attn_ext_rows) are not supported
if (op->src[5] != nullptr) {
// K/V row lists and indexed bias are not supported
if (op->src[5] != nullptr || ggml_flash_attn_ext_has_indexed_bias(op)) {
return false;
}
#ifdef GGML_OPENCL_USE_ADRENO_KERNELS
@@ -25619,6 +25762,48 @@ static void ggml_cl_mul_mat_q2_0_f32_adreno(ggml_backend_t backend, const ggml_t
CL_CHECK(clReleaseMemObject(b_sub_buf));
}
static bool ggml_cl_mul_mat_coopmat(ggml_backend_opencl_context * backend_ctx, cl_kernel kernel,
std::initializer_list<cl_mem> weights,
const ggml_tensor * src1, ggml_tensor * dst, int K, int M, int N) {
if (kernel == nullptr || M % 64 != 0 || N < 2) {
return false;
}
ggml_tensor_extra_cl * extra1 = (ggml_tensor_extra_cl *)src1->extra;
ggml_tensor_extra_cl * extrad = (ggml_tensor_extra_cl *)dst->extra;
cl_ulong offset1 = extra1->offset + src1->view_offs;
cl_ulong offsetd = extrad->offset + dst->view_offs;
const int n_tiles = CEIL_DIV(N, 64);
backend_ctx->prealloc_act_trans.allocate(backend_ctx->context, (size_t)n_tiles * 64 * K * sizeof(cl_half));
cl_kernel pk = backend_ctx->kernel_pack_b_f16_coopmat;
CL_CHECK(clSetKernelArg(pk, 0, sizeof(cl_mem), &extra1->data_device));
CL_CHECK(clSetKernelArg(pk, 1, sizeof(cl_ulong), &offset1));
CL_CHECK(clSetKernelArg(pk, 2, sizeof(cl_mem), &backend_ctx->prealloc_act_trans.buffer));
CL_CHECK(clSetKernelArg(pk, 3, sizeof(int), &K));
CL_CHECK(clSetKernelArg(pk, 4, sizeof(int), &N));
size_t p_local[2] = { 64, 4 };
size_t p_global[2] = { (size_t)n_tiles * 64, (size_t)CEIL_DIV(K, 64) * 4 };
backend_ctx->enqueue_ndrange_kernel(pk, 2, p_global, p_local, dst);
cl_uint ka = 0;
for (cl_mem w : weights) {
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(cl_mem), &w));
}
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(cl_mem), &backend_ctx->prealloc_act_trans.buffer));
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(cl_mem), &extrad->data_device));
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(cl_ulong), &offsetd));
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(int), &K));
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(int), &M));
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(int), &N));
size_t c_local[3] = { 64, 1, 4 };
size_t c_global[3] = { 64, (size_t)(M / 64), (size_t)CEIL_DIV(n_tiles, 4) * 4 };
backend_ctx->enqueue_ndrange_kernel(kernel, 3, c_global, c_local, dst);
return true;
}
static void ggml_cl_mul_mat_q4_0_f32_adreno_ila(ggml_backend_t backend, const ggml_tensor * src0,
const ggml_tensor * src1, ggml_tensor * dst) {
GGML_ASSERT(src0);
@@ -25934,6 +26119,11 @@ static void ggml_cl_mul_mat_q4_0_f32_adreno(ggml_backend_t backend, const ggml_t
const bool use_bin = use_q4_0_bin_kernels(backend_ctx, src0);
if (use_bin && ggml_cl_mul_mat_coopmat(backend_ctx, backend_ctx->kernel_gemm_noshuffle_q4_0_f32_coopmat,
{ extra0_q4_0->q, extra0_q4_0->d }, src1, dst, K, M, N)) {
return;
}
if (use_bin) {
if (use_q40_mc3) {
static bool warned = false;
@@ -26498,6 +26688,11 @@ static void ggml_cl_mul_mat_q5_0_f32_adreno(ggml_backend_t backend, const ggml_t
int N = ne1;
int K = ne00;
if (ggml_cl_mul_mat_coopmat(backend_ctx, backend_ctx->kernel_gemm_noshuffle_q5_0_f32_coopmat,
{ extra0_q5_0->qs, extra0_q5_0->qh, extra0_q5_0->d }, src1, dst, K, M, N)) {
return;
}
if (ne1 == 1) {
cl_mem qs_img = nullptr;
cl_mem b_sub_buf = nullptr;
@@ -27340,6 +27535,11 @@ static void ggml_cl_mul_mat_q8_0_f32_adreno(ggml_backend_t backend, const ggml_t
int N = ne1;
int K = ne00;
if (ggml_cl_mul_mat_coopmat(backend_ctx, backend_ctx->kernel_gemm_noshuffle_q8_0_f32_coopmat,
{ extra0_q8_0->q, extra0_q8_0->d }, src1, dst, K, M, N)) {
return;
}
// Multi-column verify GEMV for ne1 2..4, reuses the ne1 == 1 image setup.
// Default on for X2E, GGML_OPENCL_Q8_0_MC=0/1 forces it.
static const char * q8_mc_env = getenv("GGML_OPENCL_Q8_0_MC");
@@ -28321,6 +28521,11 @@ static void ggml_cl_mul_mat_q4_k_f32_adreno(ggml_backend_t backend, const ggml_t
const bool use_bin = use_q4_k_bin_kernels(backend_ctx, src0);
if (use_bin && ggml_cl_mul_mat_coopmat(backend_ctx, backend_ctx->kernel_gemm_noshuffle_q4_k_f32_coopmat,
{ extra0_q4_k->q, extra0_q4_k->s, extra0_q4_k->d, extra0_q4_k->dm }, src1, dst, K, M, N)) {
return;
}
if (use_bin) {
if (use_mc3) {
static bool warned = false;
@@ -29231,6 +29436,11 @@ static void ggml_cl_mul_mat_q6_K_f32_adreno(ggml_backend_t backend, const ggml_t
const bool use_bin = use_q6_k_bin_kernels(backend_ctx, src0);
if (use_bin && ggml_cl_mul_mat_coopmat(backend_ctx, backend_ctx->kernel_gemm_noshuffle_q6_k_f32_coopmat,
{ extra0_q6_K->ql, extra0_q6_K->qh, extra0_q6_K->s, extra0_q6_K->d }, src1, dst, ne00, ne01, ne1)) {
return;
}
if (use_bin) {
if (use_q6k_mc3 || use_q6k_tiled_mc) {
static bool warned = false;
@@ -29971,6 +30181,11 @@ static void ggml_cl_mul_mat_q5_K_f32_adreno(ggml_backend_t backend, const ggml_t
const bool use_bin = use_q5_k_bin_kernels(backend_ctx, src0);
if (use_bin && ggml_cl_mul_mat_coopmat(backend_ctx, backend_ctx->kernel_gemm_noshuffle_q5_k_f32_coopmat,
{ extra0_q5_k->q, extra0_q5_k->qh, extra0_q5_k->s, extra0_q5_k->d, extra0_q5_k->dm }, src1, dst, K, M, N)) {
return;
}
if (use_bin) {
if (use_q5k_mc3) {
static bool warned = false;
@@ -0,0 +1,145 @@
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
#pragma OPENCL EXTENSION cl_qcom_kernel_cooperative_matrix : enable
// Q4_0 x F32 GEMM with cl_qcom_kernel_cooperative_matrix: one 64x64 f16 C tile per wave,
// A dequantized in registers, B packed by kernel_pack_b_f16_coopmat.
typedef half mat_a_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 16, CLK_COOPERATIVE_MATRIX_A_QCOM)));
typedef half mat_b_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 16, 64, CLK_COOPERATIVE_MATRIX_B_QCOM)));
typedef half mat_c_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 64, CLK_COOPERATIVE_MATRIX_ACCUMULATOR_QCOM)));
typedef half half1024 __attribute__((ext_vector_type(1024)));
typedef half half4096 __attribute__((ext_vector_type(4096)));
typedef union {
half1024 v;
half16 h;
} a_row_t;
typedef union {
half4096 v;
struct {
half16 h0;
half16 h1;
half16 h2;
half16 h3;
} s;
} c_row_t;
inline half8 dequant8(uint v, half d) {
uint4 t;
t.s0 = ( v & 0x000F) | ((v << 12) & 0x000F0000);
t.s1 = ((v >> 8) & 0x000F) | ((v << 4) & 0x000F0000);
t.s2 = ((v >> 16) & 0x000F) | ((v >> 4) & 0x000F0000);
t.s3 = ((v >> 24) & 0x000F) | ((v >> 12) & 0x000F0000);
t |= (uint4)(0x64006400);
return (as_half8(t) - (half8)(1032.0h)) * d;
}
inline void store_c16(global float * p, half16 h, int j0, int nv, int M) {
float16 f = convert_float16(h);
if (j0 + 0 < nv) p[(ulong)(j0 + 0) * M] = f.s0;
if (j0 + 1 < nv) p[(ulong)(j0 + 1) * M] = f.s1;
if (j0 + 2 < nv) p[(ulong)(j0 + 2) * M] = f.s2;
if (j0 + 3 < nv) p[(ulong)(j0 + 3) * M] = f.s3;
if (j0 + 4 < nv) p[(ulong)(j0 + 4) * M] = f.s4;
if (j0 + 5 < nv) p[(ulong)(j0 + 5) * M] = f.s5;
if (j0 + 6 < nv) p[(ulong)(j0 + 6) * M] = f.s6;
if (j0 + 7 < nv) p[(ulong)(j0 + 7) * M] = f.s7;
if (j0 + 8 < nv) p[(ulong)(j0 + 8) * M] = f.s8;
if (j0 + 9 < nv) p[(ulong)(j0 + 9) * M] = f.s9;
if (j0 + 10 < nv) p[(ulong)(j0 + 10) * M] = f.sa;
if (j0 + 11 < nv) p[(ulong)(j0 + 11) * M] = f.sb;
if (j0 + 12 < nv) p[(ulong)(j0 + 12) * M] = f.sc;
if (j0 + 13 < nv) p[(ulong)(j0 + 13) * M] = f.sd;
if (j0 + 14 < nv) p[(ulong)(j0 + 14) * M] = f.se;
if (j0 + 15 < nv) p[(ulong)(j0 + 15) * M] = f.sf;
}
#define LOAD_Q(qv, dd, o) { \
qv.s0 = qp[(4 * (o) + 0) * M]; \
qv.s1 = qp[(4 * (o) + 1) * M]; \
qv.s2 = qp[(4 * (o) + 2) * M]; \
qv.s3 = qp[(4 * (o) + 3) * M]; \
dd = dp[(o) * M]; }
#define MMA_K32(qv, dd, b) { \
a_row_t ua0; \
a_row_t ua1; \
ua0.h = (half16)(dequant8(qv.s0, dd), dequant8(qv.s1, dd)); \
ua1.h = (half16)(dequant8(qv.s2, dd), dequant8(qv.s3, dd)); \
mat_a_t a0 = __builtin_astype(ua0.v, mat_a_t); \
mat_a_t a1 = __builtin_astype(ua1.v, mat_a_t); \
mat_b_t b0 = qcom_coop_mat_load((b), CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
mat_b_t b1 = qcom_coop_mat_load((b) + 1024, CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
acc = qcom_coop_mat_mulAdd(a0, b0, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); \
acc = qcom_coop_mat_mulAdd(a1, b1, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); }
kernel void kernel_gemm_noshuffle_q4_0_f32_coopmat(
global const uint * src0_q, // A: q4_0 quants, [K/8][M] u32
global const half * src0_d, // A: q4_0 scales, [K/32][M] f16
global const half * src1, // B: packed f16 tiles
global char * dst, // C: [N][M] f32
ulong offsetd,
int K,
int M,
int N
) {
const int lane = get_local_id(0);
const int row = get_global_id(1) * 64;
const int tn = get_global_id(2);
const int n_tiles = (N + 63) / 64;
if (tn >= n_tiles) {
return;
}
global const uint * qp = src0_q + row + lane;
global const half * dp = src0_d + row + lane;
global const half * bp = src1 + (ulong)tn * (K / 16) * 1024;
mat_c_t acc = qcom_coop_mat_init(0.0);
{
const int odd = (K / 32) & 1;
const int ob = odd ? 0 : 1;
uint4 qa;
uint4 qb;
half da;
half db;
LOAD_Q(qa, da, 0)
LOAD_Q(qb, db, ob)
da = odd ? 0.0h : da;
MMA_K32(qa, da, bp)
MMA_K32(qb, db, bp + ob * 2048)
qp += (ob + 1) * 4 * M;
dp += (ob + 1) * M;
bp += (ob + 1) * 2048;
}
#pragma unroll 1
for (int k0 = (K / 32) & 1 ? 32 : 64; k0 < K; k0 += 64) {
uint4 qa;
uint4 qb;
half da;
half db;
LOAD_Q(qa, da, 0)
LOAD_Q(qb, db, 1)
MMA_K32(qa, da, bp)
MMA_K32(qb, db, bp + 2048)
qp += 8 * M;
dp += 2 * M;
bp += 4096;
}
c_row_t uc;
uc.v = __builtin_astype(acc, half4096);
global float * p = (global float *)(dst + offsetd) + (ulong)tn * 64 * M + row + lane;
const int nv = N - tn * 64;
store_c16(p, uc.s.h0, 0, nv, M);
store_c16(p, uc.s.h1, 16, nv, M);
store_c16(p, uc.s.h2, 32, nv, M);
store_c16(p, uc.s.h3, 48, nv, M);
}
@@ -0,0 +1,148 @@
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
#pragma OPENCL EXTENSION cl_qcom_kernel_cooperative_matrix : enable
// Q4_K x F32 GEMM with cl_qcom_kernel_cooperative_matrix: one 64x64 f16 C tile per wave,
// A dequantized in registers, B packed by kernel_pack_b_f16_coopmat.
#define K_SCALE_SIZE 12
typedef half mat_a_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 16, CLK_COOPERATIVE_MATRIX_A_QCOM)));
typedef half mat_b_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 16, 64, CLK_COOPERATIVE_MATRIX_B_QCOM)));
typedef half mat_c_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 64, CLK_COOPERATIVE_MATRIX_ACCUMULATOR_QCOM)));
typedef half half1024 __attribute__((ext_vector_type(1024)));
typedef half half4096 __attribute__((ext_vector_type(4096)));
typedef union {
half1024 v;
half16 h;
} a_row_t;
typedef union {
half4096 v;
struct {
half16 h0;
half16 h1;
half16 h2;
half16 h3;
} s;
} c_row_t;
inline half8 dequant8(uint v, half scale, half nminv) {
uint4 t;
t.s0 = ( v & 0x000F) | ((v << 12) & 0x000F0000);
t.s1 = ((v >> 8) & 0x000F) | ((v << 4) & 0x000F0000);
t.s2 = ((v >> 16) & 0x000F) | ((v >> 4) & 0x000F0000);
t.s3 = ((v >> 24) & 0x000F) | ((v >> 12) & 0x000F0000);
t |= (uint4)(0x64006400);
return (as_half8(t) - (half8)(1024.0h)) * scale + nminv;
}
inline void store_c16(global float * p, half16 h, int j0, int nv, int M) {
float16 f = convert_float16(h);
if (j0 + 0 < nv) p[(ulong)(j0 + 0) * M] = f.s0;
if (j0 + 1 < nv) p[(ulong)(j0 + 1) * M] = f.s1;
if (j0 + 2 < nv) p[(ulong)(j0 + 2) * M] = f.s2;
if (j0 + 3 < nv) p[(ulong)(j0 + 3) * M] = f.s3;
if (j0 + 4 < nv) p[(ulong)(j0 + 4) * M] = f.s4;
if (j0 + 5 < nv) p[(ulong)(j0 + 5) * M] = f.s5;
if (j0 + 6 < nv) p[(ulong)(j0 + 6) * M] = f.s6;
if (j0 + 7 < nv) p[(ulong)(j0 + 7) * M] = f.s7;
if (j0 + 8 < nv) p[(ulong)(j0 + 8) * M] = f.s8;
if (j0 + 9 < nv) p[(ulong)(j0 + 9) * M] = f.s9;
if (j0 + 10 < nv) p[(ulong)(j0 + 10) * M] = f.sa;
if (j0 + 11 < nv) p[(ulong)(j0 + 11) * M] = f.sb;
if (j0 + 12 < nv) p[(ulong)(j0 + 12) * M] = f.sc;
if (j0 + 13 < nv) p[(ulong)(j0 + 13) * M] = f.sd;
if (j0 + 14 < nv) p[(ulong)(j0 + 14) * M] = f.se;
if (j0 + 15 < nv) p[(ulong)(j0 + 15) * M] = f.sf;
}
#define MMA_K32(q0, q1, q2, q3, sc, nm, b) { \
a_row_t ua0; \
a_row_t ua1; \
ua0.h = (half16)(dequant8(q0, sc, nm), dequant8(q1, sc, nm)); \
ua1.h = (half16)(dequant8(q2, sc, nm), dequant8(q3, sc, nm)); \
mat_a_t a0 = __builtin_astype(ua0.v, mat_a_t); \
mat_a_t a1 = __builtin_astype(ua1.v, mat_a_t); \
mat_b_t b0 = qcom_coop_mat_load((b), CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
mat_b_t b1 = qcom_coop_mat_load((b) + 1024, CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
acc = qcom_coop_mat_mulAdd(a0, b0, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); \
acc = qcom_coop_mat_mulAdd(a1, b1, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); }
#define STEP64(sa, na, sb_, nb_) { \
uint8 qv; \
qv.s0 = qp[0]; \
qv.s1 = qp[M]; \
qv.s2 = qp[2 * M]; \
qv.s3 = qp[3 * M]; \
qv.s4 = qp[4 * M]; \
qv.s5 = qp[5 * M]; \
qv.s6 = qp[6 * M]; \
qv.s7 = qp[7 * M]; \
qp += 8 * M; \
MMA_K32(qv.s0, qv.s1, qv.s2, qv.s3, sa, na, bp) \
MMA_K32(qv.s4, qv.s5, qv.s6, qv.s7, sb_, nb_, bp + 2048) \
bp += 4096; }
kernel void kernel_gemm_noshuffle_q4_k_f32_coopmat(
global const uint * src0_q, // A: q4_K quants, [K/8][M] u32
global const uchar * src0_s, // A: 6-bit scale/min codes, [K/256*12][M] u8
global const half * src0_d, // A: per-superblock scale, [K/256][M] f16
global const half * src0_dm, // A: per-superblock min, [K/256][M] f16
global const half * src1, // B: packed f16 tiles
global char * dst, // C: [N][M] f32
ulong offsetd,
int K,
int M,
int N
) {
const int lane = get_local_id(0);
const int row = get_global_id(1) * 64;
const int tn = get_global_id(2);
const int n_tiles = (N + 63) / 64;
if (tn >= n_tiles) {
return;
}
const int lrow = row + lane;
global const uint * qp = src0_q + lrow;
global const half * bp = src1 + (ulong)tn * (K / 16) * 1024;
mat_c_t acc = qcom_coop_mat_init(0.0);
#pragma unroll 1
for (int k0 = 0; k0 < K; k0 += 128) {
const int sb = k0 >> 8;
const half dd = src0_d [(ulong)sb * M + lrow];
const half dmm = src0_dm[(ulong)sb * M + lrow];
global const uchar * sp = src0_s + (ulong)(sb * K_SCALE_SIZE) * M + lrow;
const uchar4 c0 = (uchar4)(sp[0], sp[M], sp[2 * M], sp[3 * M]);
const uchar4 c1 = (uchar4)(sp[4 * M], sp[5 * M], sp[6 * M], sp[7 * M]);
const uchar4 c2 = (uchar4)(sp[8 * M], sp[9 * M], sp[10 * M], sp[11 * M]);
const uchar4 sl = c0 & (uchar)0x3F;
const uchar4 ml = c1 & (uchar)0x3F;
const uchar4 sh = (c2 & (uchar)0x0F) | ((c0 >> (uchar)2) & (uchar)0x30);
const uchar4 mh = (c2 >> (uchar)4) | ((c1 >> (uchar)2) & (uchar)0x30);
const int hi = (k0 >> 7) & 1;
const half4 S = dd * convert_half4(hi ? sh : sl);
const half4 Nm = -(dmm * convert_half4(hi ? mh : ml));
STEP64(S.s0, Nm.s0, S.s1, Nm.s1)
STEP64(S.s2, Nm.s2, S.s3, Nm.s3)
}
c_row_t uc;
uc.v = __builtin_astype(acc, half4096);
global float * p = (global float *)(dst + offsetd) + (ulong)tn * 64 * M + row + lane;
const int nv = N - tn * 64;
store_c16(p, uc.s.h0, 0, nv, M);
store_c16(p, uc.s.h1, 16, nv, M);
store_c16(p, uc.s.h2, 32, nv, M);
store_c16(p, uc.s.h3, 48, nv, M);
}
@@ -0,0 +1,157 @@
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
#pragma OPENCL EXTENSION cl_qcom_kernel_cooperative_matrix : enable
// Q5_0 x F32 GEMM with cl_qcom_kernel_cooperative_matrix: one 64x64 f16 C tile per wave,
// A dequantized in registers, B packed by kernel_pack_b_f16_coopmat.
typedef half mat_a_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 16, CLK_COOPERATIVE_MATRIX_A_QCOM)));
typedef half mat_b_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 16, 64, CLK_COOPERATIVE_MATRIX_B_QCOM)));
typedef half mat_c_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 64, CLK_COOPERATIVE_MATRIX_ACCUMULATOR_QCOM)));
typedef half half1024 __attribute__((ext_vector_type(1024)));
typedef half half4096 __attribute__((ext_vector_type(4096)));
typedef union {
half1024 v;
half16 h;
} a_row_t;
typedef union {
half4096 v;
struct {
half16 h0;
half16 h1;
half16 h2;
half16 h3;
} s;
} c_row_t;
inline half8 dequant8(uint v, uint h, half d) {
uint4 t;
t.s0 = ( v & 0x000F) | ((v << 12) & 0x000F0000) | ((h << 4) & 0x0010) | ((h << 19) & 0x00100000);
t.s1 = ((v >> 8) & 0x000F) | ((v << 4) & 0x000F0000) | ((h << 2) & 0x0010) | ((h << 17) & 0x00100000);
t.s2 = ((v >> 16) & 0x000F) | ((v >> 4) & 0x000F0000) | ( h & 0x0010) | ((h << 15) & 0x00100000);
t.s3 = ((v >> 24) & 0x000F) | ((v >> 12) & 0x000F0000) | ((h >> 2) & 0x0010) | ((h << 13) & 0x00100000);
t |= (uint4)(0x64006400);
return (as_half8(t) - (half8)(1040.0h)) * d;
}
inline void store_c16(global float * p, half16 h, int j0, int nv, int M) {
float16 f = convert_float16(h);
if (j0 + 0 < nv) p[(ulong)(j0 + 0) * M] = f.s0;
if (j0 + 1 < nv) p[(ulong)(j0 + 1) * M] = f.s1;
if (j0 + 2 < nv) p[(ulong)(j0 + 2) * M] = f.s2;
if (j0 + 3 < nv) p[(ulong)(j0 + 3) * M] = f.s3;
if (j0 + 4 < nv) p[(ulong)(j0 + 4) * M] = f.s4;
if (j0 + 5 < nv) p[(ulong)(j0 + 5) * M] = f.s5;
if (j0 + 6 < nv) p[(ulong)(j0 + 6) * M] = f.s6;
if (j0 + 7 < nv) p[(ulong)(j0 + 7) * M] = f.s7;
if (j0 + 8 < nv) p[(ulong)(j0 + 8) * M] = f.s8;
if (j0 + 9 < nv) p[(ulong)(j0 + 9) * M] = f.s9;
if (j0 + 10 < nv) p[(ulong)(j0 + 10) * M] = f.sa;
if (j0 + 11 < nv) p[(ulong)(j0 + 11) * M] = f.sb;
if (j0 + 12 < nv) p[(ulong)(j0 + 12) * M] = f.sc;
if (j0 + 13 < nv) p[(ulong)(j0 + 13) * M] = f.sd;
if (j0 + 14 < nv) p[(ulong)(j0 + 14) * M] = f.se;
if (j0 + 15 < nv) p[(ulong)(j0 + 15) * M] = f.sf;
}
#define LOAD_Q(qv, hv, dd, o) { \
qv.s0 = (uint)qp[(8 * (o) + 0) * M] | ((uint)qp[(8 * (o) + 1) * M] << 16); \
qv.s1 = (uint)qp[(8 * (o) + 2) * M] | ((uint)qp[(8 * (o) + 3) * M] << 16); \
qv.s2 = (uint)qp[(8 * (o) + 4) * M] | ((uint)qp[(8 * (o) + 5) * M] << 16); \
qv.s3 = (uint)qp[(8 * (o) + 6) * M] | ((uint)qp[(8 * (o) + 7) * M] << 16); \
hv.s0 = hp[(4 * (o) + 0) * M]; \
hv.s1 = hp[(4 * (o) + 1) * M]; \
hv.s2 = hp[(4 * (o) + 2) * M]; \
hv.s3 = hp[(4 * (o) + 3) * M]; \
dd = dp[(o) * M]; }
#define MMA_K32(qv, hv, dd, b) { \
a_row_t ua0; \
a_row_t ua1; \
ua0.h = (half16)(dequant8(qv.s0, hv.s0, dd), dequant8(qv.s1, hv.s1, dd)); \
ua1.h = (half16)(dequant8(qv.s2, hv.s2, dd), dequant8(qv.s3, hv.s3, dd)); \
mat_a_t a0 = __builtin_astype(ua0.v, mat_a_t); \
mat_a_t a1 = __builtin_astype(ua1.v, mat_a_t); \
mat_b_t b0 = qcom_coop_mat_load((b), CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
mat_b_t b1 = qcom_coop_mat_load((b) + 1024, CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
acc = qcom_coop_mat_mulAdd(a0, b0, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); \
acc = qcom_coop_mat_mulAdd(a1, b1, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); }
kernel void kernel_gemm_noshuffle_q5_0_f32_coopmat(
global const ushort * src0_q, // A: q5_0 quants, [K/4][M] u16
global const uchar * src0_qh, // A: q5_0 high bits, [K/8][M] u8
global const half * src0_d, // A: q5_0 scales, [K/32][M] f16
global const half * src1, // B: packed f16 tiles
global char * dst, // C: [N][M] f32
ulong offsetd,
int K,
int M,
int N
) {
const int lane = get_local_id(0);
const int row = get_global_id(1) * 64;
const int tn = get_global_id(2);
const int n_tiles = (N + 63) / 64;
if (tn >= n_tiles) {
return;
}
global const ushort * qp = src0_q + row + lane;
global const uchar * hp = src0_qh + row + lane;
global const half * dp = src0_d + row + lane;
global const half * bp = src1 + (ulong)tn * (K / 16) * 1024;
mat_c_t acc = qcom_coop_mat_init(0.0);
{
const int odd = (K / 32) & 1;
const int ob = odd ? 0 : 1;
uint4 qa;
uint4 qb;
uint4 ha;
uint4 hb;
half da;
half db;
LOAD_Q(qa, ha, da, 0)
LOAD_Q(qb, hb, db, ob)
da = odd ? 0.0h : da;
MMA_K32(qa, ha, da, bp)
MMA_K32(qb, hb, db, bp + ob * 2048)
qp += (ob + 1) * 8 * M;
hp += (ob + 1) * 4 * M;
dp += (ob + 1) * M;
bp += (ob + 1) * 2048;
}
#pragma unroll 1
for (int k0 = (K / 32) & 1 ? 32 : 64; k0 < K; k0 += 64) {
uint4 qa;
uint4 qb;
uint4 ha;
uint4 hb;
half da;
half db;
LOAD_Q(qa, ha, da, 0)
LOAD_Q(qb, hb, db, 1)
MMA_K32(qa, ha, da, bp)
MMA_K32(qb, hb, db, bp + 2048)
qp += 16 * M;
hp += 8 * M;
dp += 2 * M;
bp += 4096;
}
c_row_t uc;
uc.v = __builtin_astype(acc, half4096);
global float * p = (global float *)(dst + offsetd) + (ulong)tn * 64 * M + row + lane;
const int nv = N - tn * 64;
store_c16(p, uc.s.h0, 0, nv, M);
store_c16(p, uc.s.h1, 16, nv, M);
store_c16(p, uc.s.h2, 32, nv, M);
store_c16(p, uc.s.h3, 48, nv, M);
}
@@ -0,0 +1,164 @@
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
#pragma OPENCL EXTENSION cl_qcom_kernel_cooperative_matrix : enable
// Q5_K x F32 GEMM with cl_qcom_kernel_cooperative_matrix: one 64x64 f16 C tile per wave,
// A dequantized in registers, B packed by kernel_pack_b_f16_coopmat.
#define K_SCALE_SIZE 12
typedef half mat_a_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 16, CLK_COOPERATIVE_MATRIX_A_QCOM)));
typedef half mat_b_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 16, 64, CLK_COOPERATIVE_MATRIX_B_QCOM)));
typedef half mat_c_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 64, CLK_COOPERATIVE_MATRIX_ACCUMULATOR_QCOM)));
typedef half half1024 __attribute__((ext_vector_type(1024)));
typedef half half4096 __attribute__((ext_vector_type(4096)));
typedef union {
half1024 v;
half16 h;
} a_row_t;
typedef union {
half4096 v;
struct {
half16 h0;
half16 h1;
half16 h2;
half16 h3;
} s;
} c_row_t;
inline half8 dequant8(uint v, uint h, half scale, half nminv) {
uint4 t;
t.s0 = ( v & 0x000F) | ((v << 12) & 0x000F0000);
t.s1 = ((v >> 8) & 0x000F) | ((v << 4) & 0x000F0000);
t.s2 = ((v >> 16) & 0x000F) | ((v >> 4) & 0x000F0000);
t.s3 = ((v >> 24) & 0x000F) | ((v >> 12) & 0x000F0000);
t.s0 |= ((h & 0x01u) << 4) | ((h & 0x02u) << 19);
t.s1 |= ((h & 0x04u) << 2) | ((h & 0x08u) << 17);
t.s2 |= ( h & 0x10u ) | ((h & 0x20u) << 15);
t.s3 |= ((h & 0x40u) >> 2) | ((h & 0x80u) << 13);
t |= (uint4)(0x64006400);
return (as_half8(t) - (half8)(1024.0h)) * scale + nminv;
}
inline void store_c16(global float * p, half16 h, int j0, int nv, int M) {
float16 f = convert_float16(h);
if (j0 + 0 < nv) p[(ulong)(j0 + 0) * M] = f.s0;
if (j0 + 1 < nv) p[(ulong)(j0 + 1) * M] = f.s1;
if (j0 + 2 < nv) p[(ulong)(j0 + 2) * M] = f.s2;
if (j0 + 3 < nv) p[(ulong)(j0 + 3) * M] = f.s3;
if (j0 + 4 < nv) p[(ulong)(j0 + 4) * M] = f.s4;
if (j0 + 5 < nv) p[(ulong)(j0 + 5) * M] = f.s5;
if (j0 + 6 < nv) p[(ulong)(j0 + 6) * M] = f.s6;
if (j0 + 7 < nv) p[(ulong)(j0 + 7) * M] = f.s7;
if (j0 + 8 < nv) p[(ulong)(j0 + 8) * M] = f.s8;
if (j0 + 9 < nv) p[(ulong)(j0 + 9) * M] = f.s9;
if (j0 + 10 < nv) p[(ulong)(j0 + 10) * M] = f.sa;
if (j0 + 11 < nv) p[(ulong)(j0 + 11) * M] = f.sb;
if (j0 + 12 < nv) p[(ulong)(j0 + 12) * M] = f.sc;
if (j0 + 13 < nv) p[(ulong)(j0 + 13) * M] = f.sd;
if (j0 + 14 < nv) p[(ulong)(j0 + 14) * M] = f.se;
if (j0 + 15 < nv) p[(ulong)(j0 + 15) * M] = f.sf;
}
#define MMA_K32(q0, q1, q2, q3, h0_, h1_, h2_, h3_, sc, nm, b) { \
a_row_t ua0; \
a_row_t ua1; \
ua0.h = (half16)(dequant8(q0, h0_, sc, nm), dequant8(q1, h1_, sc, nm)); \
ua1.h = (half16)(dequant8(q2, h2_, sc, nm), dequant8(q3, h3_, sc, nm)); \
mat_a_t a0 = __builtin_astype(ua0.v, mat_a_t); \
mat_a_t a1 = __builtin_astype(ua1.v, mat_a_t); \
mat_b_t b0 = qcom_coop_mat_load((b), CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
mat_b_t b1 = qcom_coop_mat_load((b) + 1024, CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
acc = qcom_coop_mat_mulAdd(a0, b0, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); \
acc = qcom_coop_mat_mulAdd(a1, b1, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); }
#define STEP64(sa, na, sb_, nb_) { \
uint8 qv; \
qv.s0 = qp[0]; \
qv.s1 = qp[M]; \
qv.s2 = qp[2 * M]; \
qv.s3 = qp[3 * M]; \
qv.s4 = qp[4 * M]; \
qv.s5 = qp[5 * M]; \
qv.s6 = qp[6 * M]; \
qv.s7 = qp[7 * M]; \
qp += 8 * M; \
uint8 hv; \
hv.s0 = qhp[0]; \
hv.s1 = qhp[M]; \
hv.s2 = qhp[2 * M]; \
hv.s3 = qhp[3 * M]; \
hv.s4 = qhp[4 * M]; \
hv.s5 = qhp[5 * M]; \
hv.s6 = qhp[6 * M]; \
hv.s7 = qhp[7 * M]; \
qhp += 8 * M; \
MMA_K32(qv.s0, qv.s1, qv.s2, qv.s3, hv.s0, hv.s1, hv.s2, hv.s3, sa, na, bp) \
MMA_K32(qv.s4, qv.s5, qv.s6, qv.s7, hv.s4, hv.s5, hv.s6, hv.s7, sb_, nb_, bp + 2048) \
bp += 4096; }
kernel void kernel_gemm_noshuffle_q5_k_f32_coopmat(
global const uint * src0_q, // A: q5_K low nibbles, [K/8][M] u32
global const uchar * src0_qh, // A: q5_K high bits, [K/8][M] u8
global const uchar * src0_s, // A: 6-bit scale/min codes, [K/256*12][M] u8
global const half * src0_d, // A: per-superblock scale, [K/256][M] f16
global const half * src0_dm, // A: per-superblock min, [K/256][M] f16
global const half * src1, // B: packed f16 tiles
global char * dst, // C: [N][M] f32
ulong offsetd,
int K,
int M,
int N
) {
const int lane = get_local_id(0);
const int row = get_global_id(1) * 64;
const int tn = get_global_id(2);
const int n_tiles = (N + 63) / 64;
if (tn >= n_tiles) {
return;
}
const int lrow = row + lane;
global const uint * qp = src0_q + lrow;
global const uchar * qhp = src0_qh + lrow;
global const half * bp = src1 + (ulong)tn * (K / 16) * 1024;
mat_c_t acc = qcom_coop_mat_init(0.0);
#pragma unroll 1
for (int k0 = 0; k0 < K; k0 += 128) {
const int sb = k0 >> 8;
const half dd = src0_d [(ulong)sb * M + lrow];
const half dmm = src0_dm[(ulong)sb * M + lrow];
global const uchar * sp = src0_s + (ulong)(sb * K_SCALE_SIZE) * M + lrow;
const uchar4 c0 = (uchar4)(sp[0], sp[M], sp[2 * M], sp[3 * M]);
const uchar4 c1 = (uchar4)(sp[4 * M], sp[5 * M], sp[6 * M], sp[7 * M]);
const uchar4 c2 = (uchar4)(sp[8 * M], sp[9 * M], sp[10 * M], sp[11 * M]);
const uchar4 sl = c0 & (uchar)0x3F;
const uchar4 ml = c1 & (uchar)0x3F;
const uchar4 sh = (c2 & (uchar)0x0F) | ((c0 >> (uchar)2) & (uchar)0x30);
const uchar4 mh = (c2 >> (uchar)4) | ((c1 >> (uchar)2) & (uchar)0x30);
const int hi = (k0 >> 7) & 1;
const half4 S = dd * convert_half4(hi ? sh : sl);
const half4 Nm = -(dmm * convert_half4(hi ? mh : ml));
STEP64(S.s0, Nm.s0, S.s1, Nm.s1)
STEP64(S.s2, Nm.s2, S.s3, Nm.s3)
}
c_row_t uc;
uc.v = __builtin_astype(acc, half4096);
global float * p = (global float *)(dst + offsetd) + (ulong)tn * 64 * M + row + lane;
const int nv = N - tn * 64;
store_c16(p, uc.s.h0, 0, nv, M);
store_c16(p, uc.s.h1, 16, nv, M);
store_c16(p, uc.s.h2, 32, nv, M);
store_c16(p, uc.s.h3, 48, nv, M);
}
@@ -0,0 +1,145 @@
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
#pragma OPENCL EXTENSION cl_qcom_kernel_cooperative_matrix : enable
// Q6_K x F32 GEMM with cl_qcom_kernel_cooperative_matrix: one 64x64 f16 C tile per wave,
// A dequantized in registers, B packed by kernel_pack_b_f16_coopmat.
typedef half mat_a_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 16, CLK_COOPERATIVE_MATRIX_A_QCOM)));
typedef half mat_b_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 16, 64, CLK_COOPERATIVE_MATRIX_B_QCOM)));
typedef half mat_c_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 64, CLK_COOPERATIVE_MATRIX_ACCUMULATOR_QCOM)));
typedef half half1024 __attribute__((ext_vector_type(1024)));
typedef half half4096 __attribute__((ext_vector_type(4096)));
typedef union {
half1024 v;
half16 h;
} a_row_t;
typedef union {
half4096 v;
struct {
half16 h0;
half16 h1;
half16 h2;
half16 h3;
} s;
} c_row_t;
inline half8 dequant8(uint v, uint hh, half scale) {
uint4 t;
t.s0 = ( v & 0x000F) | ((v << 12) & 0x000F0000);
t.s1 = ((v >> 8) & 0x000F) | ((v << 4) & 0x000F0000);
t.s2 = ((v >> 16) & 0x000F) | ((v >> 4) & 0x000F0000);
t.s3 = ((v >> 24) & 0x000F) | ((v >> 12) & 0x000F0000);
t.s0 |= ((hh & 0x0003u) << 4) | ((hh & 0x000Cu) << 18);
t.s1 |= ( hh & 0x0030u ) | ((hh & 0x00C0u) << 14);
t.s2 |= ((hh & 0x0300u) >> 4) | ((hh & 0x0C00u) << 10);
t.s3 |= ((hh & 0x3000u) >> 8) | ((hh & 0xC000u) << 6);
t |= (uint4)(0x64006400);
return (as_half8(t) - (half8)(1056.0h)) * scale;
}
inline void store_c16(global float * p, half16 h, int j0, int nv, int M) {
float16 f = convert_float16(h);
if (j0 + 0 < nv) p[(ulong)(j0 + 0) * M] = f.s0;
if (j0 + 1 < nv) p[(ulong)(j0 + 1) * M] = f.s1;
if (j0 + 2 < nv) p[(ulong)(j0 + 2) * M] = f.s2;
if (j0 + 3 < nv) p[(ulong)(j0 + 3) * M] = f.s3;
if (j0 + 4 < nv) p[(ulong)(j0 + 4) * M] = f.s4;
if (j0 + 5 < nv) p[(ulong)(j0 + 5) * M] = f.s5;
if (j0 + 6 < nv) p[(ulong)(j0 + 6) * M] = f.s6;
if (j0 + 7 < nv) p[(ulong)(j0 + 7) * M] = f.s7;
if (j0 + 8 < nv) p[(ulong)(j0 + 8) * M] = f.s8;
if (j0 + 9 < nv) p[(ulong)(j0 + 9) * M] = f.s9;
if (j0 + 10 < nv) p[(ulong)(j0 + 10) * M] = f.sa;
if (j0 + 11 < nv) p[(ulong)(j0 + 11) * M] = f.sb;
if (j0 + 12 < nv) p[(ulong)(j0 + 12) * M] = f.sc;
if (j0 + 13 < nv) p[(ulong)(j0 + 13) * M] = f.sd;
if (j0 + 14 < nv) p[(ulong)(j0 + 14) * M] = f.se;
if (j0 + 15 < nv) p[(ulong)(j0 + 15) * M] = f.sf;
}
#define MMA_K32(q0, q1, q2, q3, hA, hB, sc0, sc1, b) { \
a_row_t ua0; \
a_row_t ua1; \
ua0.h = (half16)(dequant8(q0, (hA), sc0), dequant8(q1, (hA) >> 16, sc0)); \
ua1.h = (half16)(dequant8(q2, (hB), sc1), dequant8(q3, (hB) >> 16, sc1)); \
mat_a_t a0 = __builtin_astype(ua0.v, mat_a_t); \
mat_a_t a1 = __builtin_astype(ua1.v, mat_a_t); \
mat_b_t b0 = qcom_coop_mat_load((b), CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
mat_b_t b1 = qcom_coop_mat_load((b) + 1024, CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
acc = qcom_coop_mat_mulAdd(a0, b0, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); \
acc = qcom_coop_mat_mulAdd(a1, b1, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); }
kernel void kernel_gemm_noshuffle_q6_k_f32_coopmat(
global const uint * src0_ql, // A: q6_K low nibbles, [K/8][M] u32
global const uint * src0_qh, // A: q6_K high 2-bit, [K/16][M] u32
global const uint * src0_s, // A: int8 sub-scales, [M][K/16] (4 per u32)
global const half * src0_d, // A: per-superblock scale, [K/256][M] f16
global const half * src1, // B: packed f16 tiles
global char * dst, // C: [N][M] f32
ulong offsetd,
int K,
int M,
int N
) {
const int lane = get_local_id(0);
const int row = get_global_id(1) * 64;
const int tn = get_global_id(2);
const int n_tiles = (N + 63) / 64;
if (tn >= n_tiles) {
return;
}
const int lrow = row + lane;
global const uint * qp = src0_ql + lrow;
global const uint * qhp = src0_qh + lrow;
global const half * bp = src1 + (ulong)tn * (K / 16) * 1024;
global const uint * sp = src0_s + (ulong)lrow * (K / 64);
mat_c_t acc = qcom_coop_mat_init(0.0);
#pragma unroll 1
for (int k0 = 0; k0 < K; k0 += 64) {
const half dd = src0_d[(ulong)(k0 >> 8) * M + lrow];
const half4 g = convert_half4(as_char4(sp[0])) * dd;
sp += 1;
uint8 qv;
qv.s0 = qp[0];
qv.s1 = qp[M];
qv.s2 = qp[2 * M];
qv.s3 = qp[3 * M];
qv.s4 = qp[4 * M];
qv.s5 = qp[5 * M];
qv.s6 = qp[6 * M];
qv.s7 = qp[7 * M];
qp += 8 * M;
uint4 hv;
hv.s0 = qhp[0];
hv.s1 = qhp[M];
hv.s2 = qhp[2 * M];
hv.s3 = qhp[3 * M];
qhp += 4 * M;
MMA_K32(qv.s0, qv.s1, qv.s2, qv.s3, hv.s0, hv.s1, g.s0, g.s1, bp)
MMA_K32(qv.s4, qv.s5, qv.s6, qv.s7, hv.s2, hv.s3, g.s2, g.s3, bp + 2048)
bp += 4096;
}
c_row_t uc;
uc.v = __builtin_astype(acc, half4096);
global float * p = (global float *)(dst + offsetd) + (ulong)tn * 64 * M + row + lane;
const int nv = N - tn * 64;
store_c16(p, uc.s.h0, 0, nv, M);
store_c16(p, uc.s.h1, 16, nv, M);
store_c16(p, uc.s.h2, 32, nv, M);
store_c16(p, uc.s.h3, 48, nv, M);
}
@@ -0,0 +1,157 @@
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
#pragma OPENCL EXTENSION cl_qcom_kernel_cooperative_matrix : enable
// Q8_0 x F32 GEMM with cl_qcom_kernel_cooperative_matrix: one 64x64 f16 C tile per wave,
// A dequantized in registers, B packed by kernel_pack_b_f16_coopmat.
typedef half mat_a_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 16, CLK_COOPERATIVE_MATRIX_A_QCOM)));
typedef half mat_b_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 16, 64, CLK_COOPERATIVE_MATRIX_B_QCOM)));
typedef half mat_c_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 64, CLK_COOPERATIVE_MATRIX_ACCUMULATOR_QCOM)));
typedef half half1024 __attribute__((ext_vector_type(1024)));
typedef half half4096 __attribute__((ext_vector_type(4096)));
typedef union {
half1024 v;
half16 h;
} a_row_t;
typedef union {
half4096 v;
struct {
half16 h0;
half16 h1;
half16 h2;
half16 h3;
} s;
} c_row_t;
inline half4 dequant4(uint v, half d) {
return convert_half4(as_char4(v)) * d;
}
inline void store_c16(global float * p, half16 h, int j0, int nv, int M) {
float16 f = convert_float16(h);
if (j0 + 0 < nv) p[(ulong)(j0 + 0) * M] = f.s0;
if (j0 + 1 < nv) p[(ulong)(j0 + 1) * M] = f.s1;
if (j0 + 2 < nv) p[(ulong)(j0 + 2) * M] = f.s2;
if (j0 + 3 < nv) p[(ulong)(j0 + 3) * M] = f.s3;
if (j0 + 4 < nv) p[(ulong)(j0 + 4) * M] = f.s4;
if (j0 + 5 < nv) p[(ulong)(j0 + 5) * M] = f.s5;
if (j0 + 6 < nv) p[(ulong)(j0 + 6) * M] = f.s6;
if (j0 + 7 < nv) p[(ulong)(j0 + 7) * M] = f.s7;
if (j0 + 8 < nv) p[(ulong)(j0 + 8) * M] = f.s8;
if (j0 + 9 < nv) p[(ulong)(j0 + 9) * M] = f.s9;
if (j0 + 10 < nv) p[(ulong)(j0 + 10) * M] = f.sa;
if (j0 + 11 < nv) p[(ulong)(j0 + 11) * M] = f.sb;
if (j0 + 12 < nv) p[(ulong)(j0 + 12) * M] = f.sc;
if (j0 + 13 < nv) p[(ulong)(j0 + 13) * M] = f.sd;
if (j0 + 14 < nv) p[(ulong)(j0 + 14) * M] = f.se;
if (j0 + 15 < nv) p[(ulong)(j0 + 15) * M] = f.sf;
}
#define LOAD_Q(qv, dd) { \
qv.s0 = qp[0]; \
qv.s1 = qp[M]; \
qv.s2 = qp[2 * M]; \
qv.s3 = qp[3 * M]; \
qv.s4 = qp[4 * M]; \
qv.s5 = qp[5 * M]; \
qv.s6 = qp[6 * M]; \
qv.s7 = qp[7 * M]; \
dd = dp[0]; \
qp += 8 * M; \
dp += M; }
#define MMA_K32(qv, dd, b) { \
a_row_t ua0; \
a_row_t ua1; \
ua0.h = (half16)(dequant4(qv.s0, dd), dequant4(qv.s1, dd), dequant4(qv.s2, dd), dequant4(qv.s3, dd)); \
ua1.h = (half16)(dequant4(qv.s4, dd), dequant4(qv.s5, dd), dequant4(qv.s6, dd), dequant4(qv.s7, dd)); \
mat_a_t a0 = __builtin_astype(ua0.v, mat_a_t); \
mat_a_t a1 = __builtin_astype(ua1.v, mat_a_t); \
mat_b_t b0 = qcom_coop_mat_load((b), CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
mat_b_t b1 = qcom_coop_mat_load((b) + 1024, CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
acc = qcom_coop_mat_mulAdd(a0, b0, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); \
acc = qcom_coop_mat_mulAdd(a1, b1, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); }
kernel void kernel_gemm_noshuffle_q8_0_f32_coopmat(
global const uint * src0_q, // A: q8_0 quants, [K/4][M] u32
global const half * src0_d, // A: q8_0 scales, [K/32][M] f16
global const half * src1, // B: packed f16 tiles
global char * dst, // C: [N][M] f32
ulong offsetd,
int K,
int M,
int N
) {
const int lane = get_local_id(0);
const int row = get_global_id(1) * 64;
const int tn = get_global_id(2);
const int n_tiles = (N + 63) / 64;
if (tn >= n_tiles) {
return;
}
global const uint * qp = src0_q + row + lane;
global const half * dp = src0_d + row + lane;
global const half * bp = src1 + (ulong)tn * (K / 16) * 1024;
mat_c_t acc = qcom_coop_mat_init(0.0);
{
const int odd = (K / 32) & 1;
const int kb = odd ? 0 : 32;
uint8 qa;
uint8 qb;
half da;
half db;
qa.s0 = qp[0];
qa.s1 = qp[M];
qa.s2 = qp[2 * M];
qa.s3 = qp[3 * M];
qa.s4 = qp[4 * M];
qa.s5 = qp[5 * M];
qa.s6 = qp[6 * M];
qa.s7 = qp[7 * M];
da = odd ? 0.0h : dp[0];
qb.s0 = qp[(kb / 4 + 0) * M];
qb.s1 = qp[(kb / 4 + 1) * M];
qb.s2 = qp[(kb / 4 + 2) * M];
qb.s3 = qp[(kb / 4 + 3) * M];
qb.s4 = qp[(kb / 4 + 4) * M];
qb.s5 = qp[(kb / 4 + 5) * M];
qb.s6 = qp[(kb / 4 + 6) * M];
qb.s7 = qp[(kb / 4 + 7) * M];
db = dp[(kb / 32) * M];
MMA_K32(qa, da, bp)
MMA_K32(qb, db, bp + (kb / 16) * 1024)
qp += (kb / 4 + 8) * M;
dp += (kb / 32 + 1) * M;
bp += (kb / 16 + 2) * 1024;
}
#pragma unroll 1
for (int k0 = (K / 32) & 1 ? 32 : 64; k0 < K; k0 += 64) {
uint8 qa;
uint8 qb;
half da;
half db;
LOAD_Q(qa, da)
LOAD_Q(qb, db)
MMA_K32(qa, da, bp)
MMA_K32(qb, db, bp + 2048)
bp += 4096;
}
c_row_t uc;
uc.v = __builtin_astype(acc, half4096);
global float * p = (global float *)(dst + offsetd) + (ulong)tn * 64 * M + row + lane;
const int nv = N - tn * 64;
store_c16(p, uc.s.h0, 0, nv, M);
store_c16(p, uc.s.h1, 16, nv, M);
store_c16(p, uc.s.h2, 32, nv, M);
store_c16(p, uc.s.h3, 48, nv, M);
}
@@ -0,0 +1,48 @@
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
// Pack B f32 [N][K] into f16 64x16 tiles (column-major, ld 16) at (tn * (K/16) + ks) * 1024.
// Columns >= N are zero. A workgroup (64, 4) packs 64 columns x 64 k.
#define PACK_LDS (64 + 8)
kernel void kernel_pack_b_f16_coopmat(
global const char * src1,
ulong offset1,
global half * dst,
int K,
int N
) {
local half tile[64 * PACK_LDS];
const int l = get_local_id(1) * 64 + get_local_id(0);
const int tn = get_group_id(0);
const int kt = get_group_id(1);
const int k0 = kt * 64;
global const float * src = (global const float *)(src1 + offset1);
for (int j = 0; j < 4; j++) {
const int f = j * 256 + l;
const int r = f >> 4;
const int c = (f & 15) * 4;
const int n = tn * 64 + r;
float4 v = (float4)(0.0f);
if (n < N && k0 + c < K) {
v = vload4(0, src + (ulong)n * K + k0 + c);
}
vstore_half4(v, 0, tile + r * PACK_LDS + c);
}
barrier(CLK_LOCAL_MEM_FENCE);
global half * out = dst + ((ulong)tn * (K / 16) + kt * 4) * 1024;
for (int j = 0; j < 2; j++) {
const int q = j * 256 + l;
const int ks = q >> 7;
const int n = (q >> 1) & 63;
const int h = q & 1;
if (kt * 4 + ks < K / 16) {
half8 v = vload8(0, tile + n * PACK_LDS + ks * 16 + h * 8);
vstore8(v, 0, out + q * 8);
}
}
}
+4
View File
@@ -1905,6 +1905,10 @@ static ggml_openvino_op_support is_op_supported_case(const ggml_tensor * op) {
return {false, "FLASH_ATTN_EXT gemma3n pattern on GPU is not supported"};
}
if (ggml_flash_attn_ext_has_indexed_bias(op)) {
return {false, "FLASH_ATTN_EXT with indexed bias is not supported"};
}
// K/V row lists (ggml_flash_attn_ext_rows) are not supported
if (op->src[5] != nullptr) {
return {false, "FLASH_ATTN_EXT with K/V row lists is not supported"};
+1 -2
View File
@@ -3113,9 +3113,8 @@ static ggml_backend_buffer_type_t ggml_backend_rpc_device_get_buffer_type(ggml_b
static bool ggml_backend_rpc_device_supports_op(ggml_backend_dev_t dev, const struct ggml_tensor * op) {
GGML_UNUSED(dev);
GGML_UNUSED(op);
//TODO: call the remote backend and cache the results
return true;
return op->op != GGML_OP_FLASH_ATTN_EXT || !ggml_flash_attn_ext_has_indexed_bias(op);
}
static bool ggml_backend_rpc_device_supports_buft(ggml_backend_dev_t dev, ggml_backend_buffer_type_t buft) {
+2 -2
View File
@@ -7741,8 +7741,8 @@ static bool do_ggml_backend_sycl_device_supports_op(ggml_backend_dev_t dev, cons
case GGML_OP_SOLVE_TRI:
return op->src[0]->ne[0] <= SYCL_SOLVE_TRI_MAX_N && op->src[1]->ne[0] <= SYCL_SOLVE_TRI_MAX_K;
case GGML_OP_FLASH_ATTN_EXT:
// K/V row lists (ggml_flash_attn_ext_rows) are not supported
return op->src[5] == nullptr && ggml_sycl_flash_attn_ext_supported(device, op);
// K/V row lists and indexed bias are not supported
return op->src[5] == nullptr && !ggml_flash_attn_ext_has_indexed_bias(op) && ggml_sycl_flash_attn_ext_supported(device, op);
default:
return false;
}
@@ -30,6 +30,10 @@ static void ggml_backend_remoting_device_get_memory(ggml_backend_dev_t dev, size
}
static bool ggml_backend_remoting_device_supports_op(ggml_backend_dev_t dev, const ggml_tensor * op) {
if (op->op == GGML_OP_FLASH_ATTN_EXT && ggml_flash_attn_ext_has_indexed_bias(op)) {
return false;
}
if ((op->op == GGML_OP_ROPE || op->op == GGML_OP_ROPE_BACK) &&
!ggml_rope_get_truncate(op)) {
return false;
+17 -1
View File
@@ -809,9 +809,25 @@ void ggml_vk_buffer_memset(vk_buffer& dst, size_t offset, uint32_t c, size_t siz
return;
}
// vkCmdFillBuffer requires a 4-byte aligned offset and size, so write the unaligned ends from the matching bytes of c
const size_t end = offset + size;
const size_t begin_aligned = std::min(GGML_PAD(offset, 4), end);
const size_t end_aligned = std::max(end - end % 4, begin_aligned);
const uint8_t * c_bytes = (const uint8_t *) &c;
if (begin_aligned > offset) {
ggml_vk_buffer_write(dst, offset, c_bytes + offset % 4, begin_aligned - offset);
}
if (end > end_aligned) {
ggml_vk_buffer_write(dst, end_aligned, c_bytes, end - end_aligned);
}
if (end_aligned == begin_aligned) {
return;
}
std::lock_guard<std::recursive_mutex> guard(dst->device->mutex);
vk_context subctx = ggml_vk_begin_sync_ctx(dst->device, dst->device->transfer_queue->cmd_pool);
subctx->s->buffer->buf.fillBuffer(dst->buffer, offset, size, c);
subctx->s->buffer->buf.fillBuffer(dst->buffer, begin_aligned, end_aligned - begin_aligned, c);
ggml_vk_submit_sync(dst->device, subctx);
}
+216 -20
View File
@@ -1256,6 +1256,10 @@ static vk_fa_tuning_params get_fa_tuning_params_scalar(const vk_device& device,
result.d_split = std::min(std::min(result.subgroup_size, 8u), D_lsb / 4);
// Staging K/V through shared memory is a loss on AMD GCN: measured -6.7% at depth 16k and
// -7.4% at 32k on Polaris (head size 128). The kvsh stride of D/4+1 dwords is 4 mod 32, which
// costs an 8-way LDS bank conflict on wave64, and the extra shared memory cuts into occupancy
// that this shader is already short of. Keep it to NVIDIA.
result.shmem_staging = (device->vendor_id == VK_VENDOR_ID_NVIDIA && hsk < 256 && hsv < 256) ? 1 : 0;
if (!reduce_block_rows && result.block_rows > 1 && !ggml_vk_flash_attn_scalar_shmem_support(device, result, hsk, hsv, f32acc, k_type, v_type)) {
@@ -2333,7 +2337,12 @@ void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) {
const void * spv_data = nullptr;
size_t spv_size = 0;
const char *name = nullptr;
if (bf16_kv) {
const bool kv_rows = (fa.first.flags & (1u << 31)) != 0;
if (kv_rows) {
spv_data = device->fp16 ? flash_attn_f32_f16_rows_data : flash_attn_f32_f16_rows_fp32_data;
spv_size = device->fp16 ? flash_attn_f32_f16_rows_len : flash_attn_f32_f16_rows_fp32_len;
name = "flash_attn_f32_f16_rows";
} else if (bf16_kv) {
spv_data = flash_attn_f32_f16_fp32_data;
spv_size = flash_attn_f32_f16_fp32_len;
name = aligned ? "flash_attn_f32_bf16_aligned" : "flash_attn_f32_bf16";
@@ -2363,7 +2372,7 @@ void ggml_vk_load_shaders(vk_device& device, vk_pipeline requested) {
}
name = aligned ? "flash_attn_f32_f16_aligned" : "flash_attn_f32_f16";
}
ggml_vk_create_pipeline(device, fa.second, name, spv_size, spv_data, "main", 8,
ggml_vk_create_pipeline(device, fa.second, name, spv_size, spv_data, "main", kv_rows ? 9 : 8,
sizeof(vk_flash_attn_push_constants), {Br, 1, 1},
get_fa_spec_constants(fa.first), aligned ? Bc : 1, true,
!fa_ds, !fa_ds ? fa_sgs : 0);
@@ -9025,10 +9034,12 @@ void ggml_vk_flash_attn(ggml_backend_vk_context * ctx, vk_context& subctx, const
const uint32_t nem2 = mask ? mask->ne[2] : 0;
const uint32_t nem3 = mask ? mask->ne[3] : 0;
const ggml_tensor * bias = dst->src[6];
const uint32_t HSK = nek0;
const uint32_t HSV = nev0;
uint32_t N = neq1;
const uint32_t KV = nek1;
const ggml_tensor * rows = dst->src[5];
const uint32_t KV = rows ? rows->ne[0] : nek1;
GGML_ASSERT(ne0 == HSV);
GGML_ASSERT(ne2 == N);
@@ -9085,7 +9096,7 @@ void ggml_vk_flash_attn(ggml_backend_vk_context * ctx, vk_context& subctx, const
const uint32_t cm2_min_reuse = KV >= 8192 ? 4096 : 8192;
// Query reuse must cover the conversion and synchronization cost.
const bool cm2_dequant_kv = KV >= 4096 && neq1 >= cm2_min_queries && (uint64_t) neq1 * qk_ratio >= cm2_min_reuse;
const bool use_dequant_kv = ((k_quant && v_quant) || kv_raw_strided) && neq1 >= 64 &&
const bool use_dequant_kv = !rows && ((k_quant && v_quant) || kv_raw_strided) && neq1 >= 64 &&
is_dense_kv_cache(k) && is_dense_kv_cache(v) &&
(uint64_t)ggml_nelements(k) * kv_scratch_ts <= ctx->device->properties.limits.maxStorageBufferRange &&
(uint64_t)ggml_nelements(v) * kv_scratch_ts <= ctx->device->properties.limits.maxStorageBufferRange &&
@@ -9105,7 +9116,7 @@ void ggml_vk_flash_attn(ggml_backend_vk_context * ctx, vk_context& subctx, const
vk_fa_tuning_params tuning_params = get_fa_tuning_params(ctx->device, HSK, HSV, 32, KV, k_type_eff, v_type_eff, f32acc, qk_ratio);
const uint32_t max_gqa = std::min(tuning_params.block_rows, 32u);
if (N <= 8 && qk_ratio > 1 && qk_ratio <= max_gqa &&
if (!bias && N <= 8 && qk_ratio > 1 && qk_ratio <= max_gqa &&
qk_ratio * nek2 == neq2 && nek2 == nev2 && nem2 <= 1) {
// grouped query attention - make the N dimension equal to gqa_ratio, reduce
// workgroups proportionally in y dimension. The shader will detect gqa_ratio > 1
@@ -9115,7 +9126,8 @@ void ggml_vk_flash_attn(ggml_backend_vk_context * ctx, vk_context& subctx, const
workgroups_y /= gqa_ratio;
}
tuning_params = get_fa_tuning_params(ctx->device, HSK, HSV, N, KV, k_type_eff, v_type_eff, f32acc, qk_ratio);
tuning_params = bias ? get_fa_tuning_params_scalar(ctx->device, HSK, HSV, N, KV, k_type_eff, v_type_eff, f32acc)
: get_fa_tuning_params(ctx->device, HSK, HSV, N, KV, k_type_eff, v_type_eff, f32acc, qk_ratio);
float scale = 1.0f;
float max_bias = 0.0f;
@@ -9141,7 +9153,7 @@ void ggml_vk_flash_attn(ggml_backend_vk_context * ctx, vk_context& subctx, const
return (t->type != GGML_TYPE_F16 && t->type != GGML_TYPE_BF16) ||
(t->nb[1] | t->nb[2] | t->nb[3]) % (4 * sizeof(ggml_fp16_t)) == 0;
};
const bool use_sparse = !disable_sparse && n_kv_max > 0 && mask &&
const bool use_sparse = !bias && !disable_sparse && n_kv_max > 0 && mask &&
max_bias == 0.0f && logit_softcap == 0.0f &&
(tuning_params.path != FA_COOPMAT2 || (sparse_gather_aligned(k) && sparse_gather_aligned(v))) &&
nem0 == KV &&
@@ -9200,11 +9212,20 @@ void ggml_vk_flash_attn(ggml_backend_vk_context * ctx, vk_context& subctx, const
}
// Only use mask opt when the mask is fairly large. This hasn't been tuned extensively.
bool use_mask_opt = mask && !use_sparse && nem1 >= 32 && nem0 * nem1 > 32768 && nem0 >= tuning_params.block_cols * 16
&& (ctx->device->architecture != vk_device_architecture::AMD_GCN || HSK > 256 || HSV > 256);
// GCN with large head size (>= 256) benefits in high-context prefill (skipping the
// per-block mask add on fully-visible blocks dominates), so enable it there too.
bool use_mask_opt = !bias && mask && !use_sparse && nem1 >= 32 && nem0 * nem1 > 32768 && nem0 >= tuning_params.block_cols * 16
&& (ctx->device->architecture != vk_device_architecture::AMD_GCN || HSK >= 256 || HSV >= 256);
vk_fa_pipeline_state fa_pipeline_state = get_fa_pipeline_state(ctx->device, tuning_params, HSK, HSV, aligned, f32acc,
mask != nullptr, use_mask_opt, logit_softcap != 0, use_sparse, k_type_eff, v_type_eff);
if (bias) {
// Bind the table and indices in the two unused mask optimization slots.
fa_pipeline_state.flags |= (uint32_t) bias->ne[0] << 8;
if (rows) {
fa_pipeline_state.flags |= 1u << 31;
}
}
vk_pipeline pipeline = nullptr;
auto tensor_aligned_with = [&](const ggml_tensor * t, uint32_t in) {
@@ -9219,7 +9240,7 @@ void ggml_vk_flash_attn(ggml_backend_vk_context * ctx, vk_context& subctx, const
tensor_aligned_with(k, 16u) && tensor_aligned_with(v, 16u);
bool xe_fa_supported_dtype = q->type == GGML_TYPE_F32 && k->type == GGML_TYPE_F16 && v->type == GGML_TYPE_F16 &&
mask != nullptr && mask->type == GGML_TYPE_F16;
xe_fa_opt = xe_fa_supported_platform && xe_fa_supported_usage && xe_fa_supported_dtype;
xe_fa_opt = !bias && xe_fa_supported_platform && xe_fa_supported_usage && xe_fa_supported_dtype;
std::pair<vk_pipeline, vk_pipeline> xe_fa_pipeline_dual_phases = { nullptr , nullptr };
vk_pipeline xe_fa_pipeline = nullptr;
size_t size_q_staging = 0;
@@ -9433,6 +9454,10 @@ void ggml_vk_flash_attn(ggml_backend_vk_context * ctx, vk_context& subctx, const
vk_subbuffer sinks_buf = sinks ? ggml_vk_tensor_subbuffer(ctx, sinks) : q_buf;
vk_subbuffer mask_opt_buf = use_mask_opt ? ggml_vk_subbuffer(ctx, ctx->prealloc_y, 0) : q_buf;
vk_subbuffer sparse_buf = use_sparse ? ggml_vk_subbuffer(ctx, ctx->prealloc_y, 0) : q_buf;
if (bias) {
mask_opt_buf = ggml_vk_tensor_subbuffer(ctx, bias);
sparse_buf = ggml_vk_tensor_subbuffer(ctx, dst->src[7]);
}
if (use_mask_opt || use_sparse) {
// the mask opt bits and the sparse index list overwrite a matmul input converted into prealloc_y
ctx->prealloc_y_last_pipeline_used = nullptr;
@@ -9548,6 +9573,16 @@ void ggml_vk_flash_attn(ggml_backend_vk_context * ctx, vk_context& subctx, const
mask_n_head_log2, m0, m1,
gqa_ratio, split_kv, split_k };
auto dispatch_fa = [&](const vk_subbuffer & out, std::array<uint32_t, 3> groups) {
if (rows) {
ggml_vk_dispatch_pipeline(ctx, subctx, pipeline,
{q_buf, k_buf, v_buf, mask_buf, sinks_buf, out, mask_opt_buf, sparse_buf, ggml_vk_tensor_subbuffer(ctx, rows)}, pc, groups);
} else {
ggml_vk_dispatch_pipeline(ctx, subctx, pipeline,
{q_buf, k_buf, v_buf, mask_buf, sinks_buf, out, mask_opt_buf, sparse_buf}, pc, groups);
}
};
if (xe_fa_opt) {
uint32_t head_stride_q = uint32_t(nbq2 / ggml_type_size(q->type));
uint32_t head_stride_k = uint32_t(nbk2 / ggml_type_size(k->type));
@@ -9619,9 +9654,7 @@ void ggml_vk_flash_attn(ggml_backend_vk_context * ctx, vk_context& subctx, const
}
vk_subbuffer split_k_buf = ggml_vk_subbuffer(ctx, ctx->prealloc_split_k, 0);
ggml_vk_dispatch_pipeline(ctx, subctx, pipeline,
{q_buf, k_buf, v_buf, mask_buf, sinks_buf, split_k_buf, mask_opt_buf, sparse_buf},
pc, { dispatch_x, workgroups_y, workgroups_z });
dispatch_fa(split_k_buf, { dispatch_x, workgroups_y, workgroups_z });
ggml_vk_sync_buffers(ctx, subctx);
const vk_op_flash_attn_split_k_reduce_push_constants pc2 = { HSV, (uint32_t)ne1, (uint32_t)ne2, (uint32_t)ne3, split_k, (sinks != nullptr) };
@@ -9634,9 +9667,7 @@ void ggml_vk_flash_attn(ggml_backend_vk_context * ctx, vk_context& subctx, const
// When using gqa, we want one actual workgroup per batch, so cancel out wg_denoms
workgroups_x *= pipeline->wg_denoms[0];
}
ggml_vk_dispatch_pipeline(ctx, subctx, pipeline,
{q_buf, k_buf, v_buf, mask_buf, sinks_buf, dst_buf, mask_opt_buf, sparse_buf},
pc, { workgroups_x, workgroups_y, workgroups_z });
dispatch_fa(dst_buf, { workgroups_x, workgroups_y, workgroups_z });
}
if (use_dequant_kv) {
@@ -9743,7 +9774,21 @@ static vk_pipeline ggml_vk_rms_norm_pipeline(const std::array<vk_pipeline, 3> &
return pipelines[2];
}
// fork: GGML_VK_CONCAT_FASTPATH=0 forces the generic concat kernel (validation / A-B knob, see ggml_vk_concat)
static bool ggml_vk_concat_transpose_enabled() {
static const char * env = getenv("GGML_VK_CONCAT_FASTPATH");
static const bool off = env != nullptr && std::string(env) == "0";
return !off;
}
static bool ggml_vk_concat_is_transposed(const vk_device & device, const ggml_tensor * src0, const ggml_tensor * src1, const ggml_tensor * dst) {
if (!ggml_vk_concat_transpose_enabled()) {
return false;
}
// below one tile (decode, n_tokens < 32) the tiled path is pure overhead, the generic kernel is faster (FORK.md F3)
if (src1->ne[0] < 32) {
return false;
}
if (ggml_get_op_params_i32(dst, 0) != 0) { // dim 0 only
return false;
}
@@ -11595,9 +11640,120 @@ void ggml_vk_opt_step_sgd(ggml_backend_vk_context * ctx, vk_context& subctx, con
ggml_vk_op_f32<vk_op_push_constants>(ctx, subctx, src0, src1, src2, nullptr, dst, GGML_OP_OPT_STEP_SGD, { (uint32_t)n, 0, 0.0f, 0.0f, 0.0f, 0.0f });
}
// Fast path for concat along dim 0 where one source is stored "transposed"
// (nb[1] == type_size, dim1 innermost) and the other source + dst are
// contiguous along dim 0. The generic concat shader reads the transposed
// source with a catastrophic uncoalesced stride; here we instead copy the
// contiguous source with copy.comp and transpose the other source into the
// matching dst sub-region with the tiled copy_transpose shader (shared-memory
// transpose, coalesced read+write). Mirrors the precedent in
// ggml_vk_cpy_to_contiguous: direct dispatch with custom push constants.
static void ggml_vk_concat_transpose_fastpath(ggml_backend_vk_context * ctx, vk_context& subctx,
const ggml_tensor * ctg, const ggml_tensor * trp,
ggml_tensor * dst, uint32_t off_ctg, uint32_t off_trp) {
const uint32_t ts = ggml_type_size(dst->type);
vk_pipeline pipeline_cpy = (ts == 4) ? ctx->device->pipeline_cpy_32
: ctx->device->pipeline_cpy_16;
vk_pipeline pipeline_trp = (ts == 4) ? ctx->device->pipeline_cpy_transpose_32
: ctx->device->pipeline_cpy_transpose_16;
ggml_pipeline_request_descriptor_sets(ctx, pipeline_cpy, 1);
ggml_pipeline_request_descriptor_sets(ctx, pipeline_trp, 1);
vk_subbuffer ctg_buf = ggml_vk_tensor_subbuffer(ctx, ctg, true);
vk_subbuffer trp_buf = ggml_vk_tensor_subbuffer(ctx, trp, true);
vk_subbuffer dst_buf = ggml_vk_tensor_subbuffer(ctx, dst, true);
const uint32_t a_misalign_ctg = get_misalign_bytes(ctx, ctg) / ts;
const uint32_t a_misalign_trp = get_misalign_bytes(ctx, trp) / ts;
const uint32_t d_misalign = get_misalign_bytes(ctx, dst) / ts;
// Dispatch A: contiguous copy of `ctg` into dst[off_ctg : off_ctg + ctg->ne[0], :]
if (ctg->ne[0] > 0) {
const uint32_t ne_ctg = (uint32_t) ggml_nelements(ctg);
vk_op_unary_push_constants pc = vk_op_unary_push_constants_init(ctg, dst, ne_ctg);
pc.ne10 = (uint32_t) ctg->ne[0]; // only ctg's columns in dst
pc.misalign_offsets = (a_misalign_ctg << 16) | (d_misalign + off_ctg);
init_pushconst_fastdiv(pc);
std::array<uint32_t, 3> el = ne_ctg > 262144 ? std::array<uint32_t,3>{512, 512, CEIL_DIV(ne_ctg, 262144)}
: ne_ctg > 512 ? std::array<uint32_t,3>{512, CEIL_DIV(ne_ctg, 512), 1}
: std::array<uint32_t,3>{ne_ctg, 1, 1};
ggml_vk_dispatch_pipeline(ctx, subctx, pipeline_cpy, { ctg_buf, dst_buf }, pc, el);
}
// Dispatch B: tiled transpose of `trp` into dst[off_trp : off_trp + trp->ne[0], :]
if (trp->ne[0] > 0) {
vk_op_unary_push_constants pc = vk_op_unary_push_constants_init(trp, dst, ggml_nelements(trp));
pc.ne10 = (uint32_t) trp->ne[0]; // dst bound = trp columns (NOT dst->ne[0])
pc.misalign_offsets = (a_misalign_trp << 16) | (d_misalign + off_trp);
init_pushconst_fastdiv(pc);
std::array<uint32_t, 3> el = {
(uint32_t) CEIL_DIV(trp->ne[0], 32),
(uint32_t) CEIL_DIV(trp->ne[1], 32),
(uint32_t) (trp->ne[2] * trp->ne[3]),
};
el[0] = std::min(el[0], (uint32_t) ctx->device->properties.limits.maxComputeWorkGroupCount[0]);
el[1] = std::min(el[1], (uint32_t) ctx->device->properties.limits.maxComputeWorkGroupCount[1]);
el[2] = std::min(el[2], (uint32_t) ctx->device->properties.limits.maxComputeWorkGroupCount[2]);
ggml_vk_dispatch_pipeline(ctx, subctx, pipeline_trp, { trp_buf, dst_buf }, pc, el);
}
ggml_vk_sync_buffers(ctx, subctx);
}
void ggml_vk_concat(ggml_backend_vk_context * ctx, vk_context& subctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) {
int * op_params = (int *)dst->op_params;
// Fast path: concat along dim 0 with one source "transposed" (nb[1]==type_size,
// dim1 innermost) and the other source + dst contiguous along dim 0. The generic
// shader reads the transposed source uncoalesced (~5.6ms on RX 580 vs ~130us for
// a coalesced copy); here we instead use a tiled shared-memory transpose.
auto src_transposed_2d = [&](const ggml_tensor * s) {
const uint32_t ts = ggml_type_size(s->type);
return s->nb[1] == ts // dim1 innermost
&& s->nb[0] == (size_t) s->ne[1] * ts // consistent 2D transpose
&& s->ne[0] >= 32 // at least one full transpose tile (TILE_DIM); at 1 token the
// tiled path is pure overhead, the generic copy is faster
&& s->ne[2] == 1 && s->ne[3] == 1;
};
const uint32_t dst_ts = ggml_type_size(dst->type);
const bool dim0_ok = (op_params[0] == 0) && (dst->nb[0] == dst_ts);
// the dispatches below are raw f32/f16 element copies, so quantized and integer
// types stay on the generic path
const bool types_ok = (dst->type == GGML_TYPE_F32 || dst->type == GGML_TYPE_F16)
&& (src0->type == src1->type && src0->type == dst->type);
const bool shapes_ok = (src0->ne[1] == src1->ne[1] && src1->ne[1] == dst->ne[1])
&& (src0->ne[2] == src1->ne[2] && src0->ne[2] == dst->ne[2])
&& (src0->ne[3] == src1->ne[3] && src0->ne[3] == dst->ne[3])
&& (src0->ne[0] + src1->ne[0] == dst->ne[0]);
// doffset is 16-bit for unary shaders; guard against overflow
const bool off_fits = ((get_misalign_bytes(ctx, dst)/dst_ts + dst->ne[0])) < 0xFFFFu;
const bool s1_trans = dim0_ok && types_ok && shapes_ok && off_fits
&& src_transposed_2d(src1) && ggml_is_contiguous(src0);
const bool s0_trans = dim0_ok && types_ok && shapes_ok && off_fits
&& src_transposed_2d(src0) && ggml_is_contiguous(src1);
// validation/A-B knob: GGML_VK_CONCAT_FASTPATH=0 forces the generic path, =log reports every
// fast-path dispatch on stderr
static const char * fastpath_env = getenv("GGML_VK_CONCAT_FASTPATH");
static const bool fastpath_off = fastpath_env != nullptr && std::string(fastpath_env) == "0";
static const bool fastpath_log = fastpath_env != nullptr && std::string(fastpath_env) == "log";
if (!fastpath_off && (s1_trans || s0_trans)) {
const ggml_tensor * ctg = s1_trans ? src0 : src1; // contiguous source
const ggml_tensor * trp = s1_trans ? src1 : src0; // transposed source
if (fastpath_log) {
std::cerr << "ggml_vk_concat: transpose fast path, transposed ne0=" << trp->ne[0] << std::endl;
}
// dst offsets (in elements) where each source region begins
const uint32_t off_ctg = s1_trans ? 0u : (uint32_t) src1->ne[0];
const uint32_t off_trp = s1_trans ? (uint32_t) src0->ne[0] : 0u;
ggml_vk_concat_transpose_fastpath(ctx, subctx, ctg, trp, dst, off_ctg, off_trp);
return;
}
const uint32_t unit_size = ggml_vk_concat_unit_size(dst->type);
const uint32_t units_per_block = ggml_type_size(dst->type) / unit_size;
const uint32_t block_size = ggml_blck_size(dst->type);
@@ -14279,7 +14435,9 @@ bool ggml_backend_vk_buffer_cpy_tensor_n(ggml_backend_buffer_t buffer, const ggm
void ggml_backend_vk_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) {
ggml_backend_vk_buffer_context * ctx = (ggml_backend_vk_buffer_context *)buffer->context;
ggml_vk_buffer_memset(ctx->dev_buffer, 0, value, buffer->size);
// set all bytes of a u32 to the same 1 byte value
uint32_t val32 = (uint32_t)value * 0x01010101;
ggml_vk_buffer_memset(ctx->dev_buffer, 0, val32, buffer->size);
}
const char * ggml_backend_vk_buffer_type_name(ggml_backend_buffer_type_t buft) {
@@ -14402,6 +14560,35 @@ static const char * ggml_backend_vk_name(ggml_backend_t backend) {
return ctx->name.c_str();
}
// Free the compute-scratch device buffers (prealloc_* and the transfer staging buffer) without
// tearing down the backend (pipelines, command pools, fences and device stay alive). These buffers
// are reallocated lazily by ggml_vk_preallocate_buffers() on the next compute, so this just shrinks
// an idle model's device footprint. Mirrors the buffer-freeing subset of ggml_vk_cleanup().
static void ggml_backend_vk_free_scratch(ggml_backend_t backend) {
ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;
VK_LOG_DEBUG("ggml_backend_vk_free_scratch(" << ctx->name << ")");
// discard any unsubmitted command buffer and wait for in-flight work before freeing
ctx->compute_ctx.reset();
ggml_vk_synchronize(ctx);
ggml_vk_destroy_buffer(ctx->prealloc_x);
ggml_vk_destroy_buffer(ctx->prealloc_y);
ggml_vk_destroy_buffer(ctx->prealloc_split_k);
ggml_vk_destroy_buffer(ctx->prealloc_add_rms_partials);
ggml_vk_destroy_buffer(ctx->sync_staging);
ctx->prealloc_y_last_pipeline_used = nullptr;
ctx->prealloc_y_last_tensor_used = nullptr;
ctx->prealloc_y_last_k_padded = false;
ctx->prealloc_size_x = 0;
ctx->prealloc_size_y = 0;
ctx->prealloc_size_split_k = 0;
ctx->prealloc_size_add_rms_partials = 0;
ctx->prealloc_size_add_rms_partials_offset = 0;
}
void ggml_backend_vk_free(ggml_backend_t backend) {
ggml_backend_vk_context * ctx = (ggml_backend_vk_context *)backend->context;
VK_LOG_DEBUG("ggml_backend_vk_free(" << ctx->name << ")");
@@ -16421,6 +16608,7 @@ static ggml_backend_i ggml_backend_vk_interface = {
/* .event_record = */ ggml_backend_vk_event_record,
/* .event_wait = */ ggml_backend_vk_event_wait,
/* .graph_optimize = */ ggml_vk_graph_optimize,
/* .free_scratch = */ ggml_backend_vk_free_scratch,
};
static ggml_guid_t ggml_backend_vk_guid() {
@@ -16772,8 +16960,16 @@ static bool ggml_backend_vk_device_supports_op(ggml_backend_dev_t dev, const ggm
}
case GGML_OP_FLASH_ATTN_EXT:
{
// K/V row lists (ggml_flash_attn_ext_rows) are not supported
if (op->src[5] != nullptr) {
// Only the indexed-bias scalar kernel reads row lists.
const bool indexed_bias = ggml_flash_attn_ext_has_indexed_bias(op);
if (op->src[5] && (!indexed_bias || device->properties.limits.maxPerStageDescriptorStorageBuffers < 9 ||
device->properties.limits.maxDescriptorSetStorageBuffers < 9)) {
return false;
}
if (indexed_bias && (op->src[6]->ne[0] > 0x7fffff || !device->subgroup_shuffle || !device->subgroup_vote ||
ggml_nelements(op->src[6]) > UINT32_MAX || ggml_nelements(op->src[7]) > UINT32_MAX ||
op->src[1]->type != GGML_TYPE_F16 || op->src[2]->type != GGML_TYPE_F16 ||
(op->src[1]->ne[0] != 64 && op->src[1]->ne[0] != 128) || op->src[1]->ne[0] != op->src[2]->ne[0])) {
return false;
}
bool coopmat2 = device->coopmat2;
@@ -17046,7 +17242,7 @@ static bool ggml_backend_vk_device_supports_op(ggml_backend_dev_t dev, const ggm
case GGML_OP_SLEEP:
return device->shader_clock;
case GGML_OP_SCALE:
return ggml_is_contiguous(op->src[0]) && ggml_vk_is_float_elementwise(op);
return ggml_vk_is_float_elementwise(op);
case GGML_OP_PAD:
case GGML_OP_PAD_REFLECT_1D:
case GGML_OP_ROLL:
@@ -212,9 +212,23 @@ void main() {
uint32_t r = (idx + tid) / Bc;
if (idx + tid < Bc * Br) {
uint32_t kcol;
#ifdef KV_ROWS
kcol = j * Bc + c;
bool kv_active = kcol < KV;
#else
bool kv_active = fa_kv_index(j * Bc + c, kcol);
#endif
if (kv_active && (!nem1_bounds_check || i * Br + r < p.nem1)) {
FLOAT_TYPE m = FLOAT_TYPE(data_m[m_offset + (i * Br + r) * m_stride + kcol]);
const uint n_bins = (Flags & 0x7fffffffu) >> 8;
if (n_bins != 0) {
uint query = iq3 * p.N + i * Br + r;
uint bin = uint(data_sparse[query * KV + kcol]);
if (bin < n_bins) {
float bias = uintBitsToFloat(data_mask_opt[(query * p.neq2 + iq2) * n_bins + bin]);
m = FLOAT_TYPE(float16_t(float(m) + bias));
}
}
masksh[c * masksh_stride + r] = m;
max_mask = max(max_mask, float(m));
} else {
@@ -88,6 +88,10 @@ layout (binding = 5) writeonly buffer OV4 {D_TYPEV4 data_ov4[];};
layout (binding = 6) readonly buffer MO {uint32_t data_mask_opt[];};
#ifdef KV_ROWS
layout (binding = 8) readonly buffer KR {int32_t data_kv_rows[];};
#endif
layout (binding = 7) readonly buffer SP {int32_t data_sparse[];};
// Use -FLT_MAX/2 rather than -inf to reduce the possibility of NaNs, e.g. when computing Mold-M.
@@ -237,6 +241,15 @@ void init_indices()
// Resolve a linear KV slot to a real column; false for inactive (sparse padding/-1, or dense OOB).
bool fa_kv_index(uint lin, out uint kv_col) {
#ifdef KV_ROWS
if (lin >= KV) {
kv_col = 0;
return false;
}
int row = data_kv_rows[iq3 * KV + lin];
kv_col = uint(max(row, 0));
return row >= 0;
#endif
if (USE_SPARSE) {
if (lin >= p.split_kv) {
kv_col = 0;
@@ -18,7 +18,7 @@ void main() {
continue;
}
data_d[get_doffset() + idx] = FLOAT_TO_D(A_TO_FLOAT(data_a[get_aoffset() + idx]) * FLOAT_TYPE(p.param1) + FLOAT_TYPE(p.param2));
data_d[get_doffset() + dst_idx(idx)] = FLOAT_TO_D(A_TO_FLOAT(data_a[get_aoffset() + src0_idx(idx)]) * FLOAT_TYPE(p.param1) + FLOAT_TYPE(p.param2));
idx += num_threads;
}
}
@@ -749,6 +749,11 @@ void process_shaders() {
#endif
}
if (!f16acc) {
string_to_spv("flash_attn_f32_f16_rows", "flash_attn.comp",
merge_maps(fa_base_dict, {{"Q_TYPE", "float"}, {"D_TYPE", "float"}, {"D_TYPEV4", "vec4"}, {"KV_ROWS", "1"}}), fp16, false, false, false);
}
string_to_spv("flash_attn_f32_f16", "flash_attn.comp",
merge_maps(fa_base_dict, {{"Q_TYPE", "float"}, {"D_TYPE", "float"}, {"D_TYPEV4", "vec4"}}), fp16, false, false, f16acc);
+10 -5
View File
@@ -4066,7 +4066,7 @@ static void ggml_backend_webgpu_buffer_memset_tensor(ggml_backend_buffer_t buffe
size_t total_offset = ggml_webgpu_tensor_offset(tensor) + offset;
// This is a trick to set all bytes of a u32 to the same 1 byte value.
// set all bytes of a u32 to the same 1 byte value
uint32_t val32 = (uint32_t) value * 0x01010101;
ggml_backend_webgpu_buffer_memset(buf_ctx->global_ctx, buf_ctx->buffer, val32, total_offset, size);
WEBGPU_CPU_PROFILE_TOTAL_END(memset_tensor, buf_ctx->global_ctx);
@@ -4406,7 +4406,10 @@ static void ggml_backend_webgpu_buffer_clear(ggml_backend_buffer_t buffer, uint8
WEBGPU_LOG_DEBUG("ggml_backend_webgpu_buffer_clear(" << buffer << ", " << (uint32_t) value << ")");
WEBGPU_CPU_PROFILE_TOTAL_START(clear);
ggml_backend_webgpu_buffer_context * buf_ctx = (ggml_backend_webgpu_buffer_context *) buffer->context;
ggml_backend_webgpu_buffer_memset(buf_ctx->global_ctx, buf_ctx->buffer, value, 0, buffer->size);
// set all bytes of a u32 to the same 1 byte value
uint32_t val32 = (uint32_t) value * 0x01010101;
ggml_backend_webgpu_buffer_memset(buf_ctx->global_ctx, buf_ctx->buffer, val32, 0, buffer->size);
WEBGPU_CPU_PROFILE_TOTAL_END(clear, buf_ctx->global_ctx);
}
@@ -5080,8 +5083,9 @@ static bool ggml_backend_webgpu_device_supports_op(ggml_backend_dev_t dev, const
{
// conservative support checks for whether the more resource-intensive shader paths
// can be used, to avoid cases where flash_attn is assigned to the CPU later on
// K/V row lists (ggml_flash_attn_ext_rows) are not supported
supports_op = op->src[5] == nullptr && src0->type == GGML_TYPE_F32 &&
// K/V row lists and indexed bias are not supported
supports_op = op->src[5] == nullptr && !ggml_flash_attn_ext_has_indexed_bias(op) &&
src0->type == GGML_TYPE_F32 &&
(src1->type == GGML_TYPE_F32 || src1->type == GGML_TYPE_F16 ||
src1->type == GGML_TYPE_Q4_0 || src1->type == GGML_TYPE_Q8_0) &&
(src2->type == GGML_TYPE_F32 || src2->type == GGML_TYPE_F16 ||
@@ -5262,7 +5266,8 @@ static bool ggml_backend_webgpu_device_supports_op(ggml_backend_dev_t dev, const
supports_op = op->type == GGML_TYPE_F32 && src0->type == GGML_TYPE_F32 && src1->type == GGML_TYPE_F32 &&
src2->type == GGML_TYPE_F32 && op->src[3]->type == GGML_TYPE_F32 &&
op->src[4]->type == GGML_TYPE_F32 && op->src[5]->type == GGML_TYPE_F32 &&
s_v <= ctx->webgpu_global_ctx->capabilities.limits.maxComputeInvocationsPerWorkgroup;
s_v <= ctx->webgpu_global_ctx->capabilities.limits.maxComputeInvocationsPerWorkgroup &&
ggml_are_same_stride(src0, src1);
}
break;
case GGML_OP_CLAMP:
+1 -1
View File
@@ -21,7 +21,7 @@ bool ggml_zdnn_supports_flash_attn_ext(const ggml_backend_zdnn_device_context *
memcpy(&logit_softcap, (const float *) op->op_params + 2, sizeof(float));
// sinks, alibi, softcap, sparse k/v sets and k/v row lists are not implemented
if (op->src[4] != nullptr || op->src[5] != nullptr || max_bias != 0.0f || logit_softcap != 0.0f || ggml_get_op_params_i32(op, 4) != 0) {
if (op->src[4] != nullptr || op->src[5] != nullptr || ggml_flash_attn_ext_has_indexed_bias(op) || max_bias != 0.0f || logit_softcap != 0.0f || ggml_get_op_params_i32(op, 4) != 0) {
return false;
}
+54 -2
View File
@@ -1116,9 +1116,11 @@ static const char * GGML_OP_NAME[GGML_OP_COUNT] = {
"GLU",
"SLEEP",
"SIGMOID_BACK",
};
static_assert(GGML_OP_COUNT == 100, "GGML_OP_COUNT != 100");
static_assert(GGML_OP_COUNT == 101, "GGML_OP_COUNT != 101");
static const char * GGML_OP_SYMBOL[GGML_OP_COUNT] = {
"none",
@@ -1231,9 +1233,11 @@ static const char * GGML_OP_SYMBOL[GGML_OP_COUNT] = {
"glu(x)",
"sleep(x)",
"sigmoid_back(x)",
};
static_assert(GGML_OP_COUNT == 100, "GGML_OP_COUNT != 100");
static_assert(GGML_OP_COUNT == 101, "GGML_OP_COUNT != 101");
static_assert(GGML_OP_POOL_COUNT == 2, "GGML_OP_POOL_COUNT != 2");
@@ -2866,6 +2870,21 @@ struct ggml_tensor * ggml_silu_back(
return result;
}
// ggml_sigmoid_back
struct ggml_tensor * ggml_sigmoid_back(
struct ggml_context * ctx,
struct ggml_tensor * a,
struct ggml_tensor * b) {
struct ggml_tensor * result = ggml_dup_tensor(ctx, a);
result->op = GGML_OP_SIGMOID_BACK;
result->src[0] = a;
result->src[1] = b;
return result;
}
// ggml hardswish
struct ggml_tensor * ggml_hardswish(
@@ -5526,6 +5545,34 @@ void ggml_flash_attn_ext_add_sinks(
a->src[4] = sinks;
}
bool ggml_flash_attn_ext_has_indexed_bias(const struct ggml_tensor * a) {
GGML_ASSERT(a->op == GGML_OP_FLASH_ATTN_EXT);
return ggml_get_op_params_i32(a, 5) != 0;
}
void ggml_flash_attn_ext_add_indexed_bias(
struct ggml_tensor * a,
struct ggml_tensor * table,
struct ggml_tensor * indices) {
GGML_ASSERT(a->op == GGML_OP_FLASH_ATTN_EXT);
GGML_ASSERT((table == NULL) == (indices == NULL));
a->src[6] = table;
a->src[7] = indices;
ggml_set_op_params_i32(a, 5, table != NULL);
if (!table) {
return;
}
const struct ggml_tensor * q = a->src[0];
const struct ggml_tensor * mask = a->src[3];
GGML_ASSERT(mask && mask->ne[0] == (a->src[5] ? a->src[5]->ne[0] : a->src[1]->ne[1]) && mask->ne[1] == q->ne[1]);
GGML_ASSERT(mask->ne[2] == 1 && mask->ne[3] == q->ne[3]);
GGML_ASSERT(table->type == GGML_TYPE_F32 && ggml_is_contiguous(table));
GGML_ASSERT(table->ne[1] == q->ne[2] && table->ne[0] > 0 && table->ne[2] == q->ne[1]*q->ne[3] && table->ne[3] == 1);
GGML_ASSERT(indices->type == GGML_TYPE_I32 && ggml_is_contiguous(indices) && ggml_is_matrix(indices));
GGML_ASSERT(indices->ne[0] == mask->ne[0] && indices->ne[1] == table->ne[2]);
}
// ggml_flash_attn_back
struct ggml_tensor * ggml_flash_attn_back(
@@ -7148,6 +7195,11 @@ static void ggml_compute_backward(
ggml_add_or_set(ctx, cgraph, isrc0, ggml_silu_back(ctx, grad, src0));
}
} break;
case GGML_UNARY_OP_SIGMOID: {
if (src0_needs_grads) {
ggml_add_or_set(ctx, cgraph, isrc0, ggml_sigmoid_back(ctx, grad, src0));
}
} break;
case GGML_UNARY_OP_EXP: {
if (src0_needs_grads) {
ggml_add_or_set(ctx, cgraph, isrc0, ggml_mul(ctx, tensor, grad));
+9 -22
View File
@@ -322,24 +322,6 @@ struct gguf_reader {
return true;
}
bool read(enum ggml_type & dst) const {
int32_t tmp = -1;
if (!read(tmp)) {
return false;
}
dst = ggml_type(tmp);
return true;
}
bool read(enum gguf_type & dst) const {
int32_t tmp = -1;
if (!read(tmp)) {
return false;
}
dst = gguf_type(tmp);
return true;
}
bool read(std::string & dst) const {
uint64_t size = 0;
if (!read(size)) {
@@ -555,7 +537,7 @@ static struct gguf_context * gguf_init_from_reader(const struct gguf_reader & gr
for (int64_t i = 0; ok && i < n_kv; ++i) {
std::string key;
gguf_type type = gguf_type(-1);
int32_t type = -1;
bool is_array = false;
uint64_t n = 1;
@@ -724,15 +706,20 @@ static struct gguf_context * gguf_init_from_reader(const struct gguf_reader & gr
// tensor type
{
ok = ok && gr.read(info.t.type);
int32_t type = -1;
ok = ok && gr.read(type);
if (!ok) {
break;
}
// check that tensor type is within defined range
if (info.t.type < 0 || info.t.type >= GGML_TYPE_COUNT) {
if (type < 0 || type >= GGML_TYPE_COUNT) {
GGML_LOG_ERROR("%s: tensor '%s' has invalid ggml type %d. should be in [0, %d)\n",
__func__, info.t.name, info.t.type, GGML_TYPE_COUNT);
__func__, info.t.name, type, GGML_TYPE_COUNT);
ok = false;
break;
}
info.t.type = ggml_type(type);
const size_t type_size = ggml_type_size(info.t.type);
const int64_t blck_size = ggml_blck_size(info.t.type);
+4 -2
View File
@@ -73,11 +73,13 @@ Tokens allow grammars to match specific tokenizer tokens rather than character s
Tokens can be specified in two ways:
1. **Token ID**: Use angle brackets with the token ID in square brackets: `<[token-id]>`. For example, `<[1000]>` matches the token with ID 1000.
1. **Token ID**: Use angle brackets with the token ID in square brackets: `<[token-id]>`. For example, `<[1000]>` matches the token with ID 1000. Multiple IDs and inclusive ranges can be combined: `<[1000,1005-1010]>` matches token 1000 or any token from 1005 to 1010.
2. **Token string**: Use angle brackets with the token text directly: `<token>`. For example, `<think>` will match the token whose text is exactly `<think>`. This only works if the string tokenizes to exactly one token in the vocabulary, otherwise the grammar will fail to parse.
You can negate token matches using the `!` prefix: `!<[1000]>` or `!<think>` matches any token *except* the specified one.
You can negate token matches using the `!` prefix: `!<[1000]>` or `!<think>` matches any token *except* the specified one, and `!<[1000,1005-1010]>` matches any token outside the set.
Tokens are matched whole, so a token rule can only match at the start of a token, never after a character rule has consumed part of its text. Any token named by a token rule is also excluded from character rules: once `<think>` appears as a token in a grammar, `.*` in that grammar will not match the `<think>` token.
```
# Match a thinking block: <think>...</think>
+17
View File
@@ -582,6 +582,23 @@ extern "C" {
LLAMA_API const struct llama_model * llama_get_model (const struct llama_context * ctx);
LLAMA_API llama_memory_t llama_get_memory (const struct llama_context * ctx);
// On-demand device (VRAM) residency: free the model's GPU weight buffers (keeping a host
// shadow so no reload is needed) to hand VRAM to another model, and rebuild them on demand.
// Any subsequent llama_decode auto-restores. If evict_kv is true, the KV cache device buffers
// are also evicted to a host shadow (for when the KV would not fit alongside the other model),
// at the cost of a D2H/H2D copy of the live KV each cycle; otherwise the KV stays resident.
// Intended for time-sharing a single GPU between several always-loaded models.
LLAMA_API void llama_context_release_device(struct llama_context * ctx, bool evict_kv);
LLAMA_API void llama_context_restore_device(struct llama_context * ctx);
// Free all device buffers (weights, KV, compute) for good, without host copies, e.g. to hand
// the VRAM to another process before exit. The context must not be used for compute afterwards.
LLAMA_API void llama_context_discard_device(struct llama_context * ctx);
// Before exit, after llama_context_release_device(ctx, true): point the evicted KV/recurrent state
// at its host copy (no device use, no extra copy) so llama_state_seq_get_data can read it. The
// context must not be used for compute afterwards.
LLAMA_API bool llama_context_restore_memory_host(struct llama_context * ctx);
LLAMA_API bool llama_context_weights_resident(const struct llama_context * ctx);
LLAMA_API enum llama_pooling_type llama_pooling_type(const struct llama_context * ctx); // TODO: rename to llama_get_pooling_type
LLAMA_API const struct llama_vocab * llama_model_get_vocab(const struct llama_model * model);
File diff suppressed because it is too large. Load diff
+247
View File
@@ -0,0 +1,247 @@
# rx580-bench
Prompt-processing benchmark harness for the RX 580 (Polaris) serving pod.
It exists so that no future session has to rediscover how to measure this
machine. It encodes the production config, fixed corpus slices, the repeat
and median discipline, and the noise floor.
## What it measures
Two things, deliberately:
1. **Corpus prompt processing** (the number we actually care about). Real
Polish prose from `pan-tadeusz.txt` pushed through `llama-server`'s
`/completion` endpoint with `n_predict: 1`, `cache_prompt: false`. This
mimics agentic tool-result parsing: a big blob of real text arriving cold.
Reported as `timings.prompt_per_second` at prompt lengths of about 4k, 16k
and 32k tokens.
2. **`llama-bench` sweep** (the controlled cross-check). Synthetic random
tokens: prefill `pp2048`, `pp8192` at depth 0, and decode `tg256@d4096`
(256 generated tokens on top of a 4096-token context, `-p 0 -n 256
-d 4096`). The prefill cells are the continuity metric with prior work on
this box. Decode used to be `tg32` at the default depth; that is not a valid
metric here (see "Reading the numbers"), and old `tg32` rows in
`results.txt` are history only.
They do not measure the same thing and they do not agree. `llama-bench`
pp2048 is a cold depth-0 batch; the corpus numbers include the cost of a
growing KV cache, so they fall off with prompt length. Both are useful.
## How to run it
One command from the workstation:
```
./scripts/rx580-bench/run.sh --build /root/arms/new-clean/build --label new-clean
```
Interleaved A/B, which is the only valid way to compare two builds:
```
./scripts/rx580-bench/run.sh \
--build /root/arms/new-clean/build --label new-clean \
--build-b /root/arms/new-fork/build --label-b new-fork \
--rounds 3
```
Summarize whatever has accumulated:
```
./scripts/rx580-bench/run.sh --summarize
```
`run.sh` copies the harness onto the pod and execs it there. The wrapper is
thin on purpose: all the methodology lives in the pod-side scripts, so it can
also be driven directly on the pod if kubectl is inconvenient.
The run is detached on the pod and then tailed, so Ctrl-C on the wrapper does
not orphan a half-finished run. `bench.sh` restarts `llama-swap` from an EXIT
trap on success, failure and interrupt alike.
### Pod access
```
export KUBECONFIG=/home/user/Projects/klaster/talos/generated/kubeconfig
kubectl -n llama exec -i deploy/supervisord -- sh -c '<command>'
```
The pod has `curl` and `python3` but **no `jq`**; the harness uses python3 for
all JSON.
## Files
| file | side | what |
|---|---|---|
| `run.sh` | workstation | one-command wrapper; installs and execs the rest |
| `bench.sh` | pod | orchestrator: stops llama-swap, waits for idle GPU, runs both harnesses, restarts llama-swap via trap |
| `ppbench.py` | pod | the corpus prompt-processing harness |
| `lbsweep.sh` | pod | the llama-bench sweep |
| `summarize.py` | pod | turns `results.txt` into a median/min-max table with delta significance |
Results append to `/root/bench/results.txt` as parseable `RESULT` /
`SUMMARY` / `LBRESULT` lines. Server logs land in `/root/bench/server-*.log`.
## The config it assumes
Production serving config, matching `/root/config.yaml`:
```
-t 6 -ngl 99 --n-cpu-moe 40 -b 2048 -ub 2048 -fa 1 -lm none
--ctx-size 40960 --no-warmup
```
Model:
```
/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
```
Qwen3.6-35B-A3B, 20.8 GiB, 40 layers, 256 experts of which 8 are used.
Machine: AMD RX 580 8 GB (Polaris/GCN, RADV, no fp16 accel), PCIe 3.0 x16,
12 CPU cores, 62 GB RAM.
The `llama-bench` equivalent of that config is:
```
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none -p 2048,8192 -n 0 -r 3
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none -p 0 -n 256 -d 4096 -r 3
```
`lbsweep.sh` runs these as two invocations (prefill, then decode). Override
with `LB_P` (prefill sizes), `LB_N` / `LB_D` (decode tokens / depth; `LB_N=0`
skips decode) and `LB_R` (repetitions).
## The corpus slices are constants, not recalibrated
`pan-tadeusz.txt` is 447334 chars / 482907 bytes of Polish text. Against this
model's tokenizer it runs **2.6702 chars/token**, and `/completion` reports
`prompt_n` identical to `/tokenize` (BOS offset 0).
The slices are `corpus[0:NCHARS]` and are baked into `ppbench.py`:
| target tokens | chars | utf-8 bytes |
|---|---|---|
| 4096 | 10776 | 11608 |
| 16384 | 43858 | 47175 |
| 32768 | 87165 | 93843 |
These hit the target `prompt_n` exactly. Every build is therefore measured on
byte-identical input. `ppbench.py` warns loudly if the reported `prompt_n`
ever stops matching the target, which is the signal that the table is stale.
Only recalibrate (`ppbench.py <build> <label> --calibrate`) if the model file
or the corpus changes.
## Reading the numbers
* `prompt_n` is the token count actually prefilled. It must equal the target.
* `prompt_ms` is wall-clock prefill time.
* `tps` / `prompt_per_second` is `prompt_n / prompt_ms * 1000`. Higher better.
* `pp2048` / `pp8192` are llama-bench prefill throughput at depth 0. A depth
suffix (`pp2048@d16384`) means the prompt is processed on top of a context
of that many tokens.
* `tg256@d4096` is the decode metric: 256 generated tokens at depth 4096
(`-p 0 -n 256 -d 4096`). Decode is CPU-bound with `--n-cpu-moe 40` and
swings with node CPU contention (about 60 co-tenant pods), so compare
medians across interleaved rounds and expect occasional catastrophic
samples.
* `tg32` (old rows, no depth suffix) is token generation at the DEFAULT depth
(n_ctx 32). **It is NOT a valid decode metric on this box.** At that size
decode is launch/barrier bound, not compute bound, and measures a regime
serving never reaches: it read -36% between two upstream commits that are at
parity at tg256 (the cause was the view-alias fix in
ggml_vk_graph_optimize). tg is also NOT monotonic in n_ctx here - on one
binary n_ctx 256 measured slower than both 32 and 544 - so only ever compare
same-depth cells. `summarize.py` flags any depth-less tg row.
## Noise floor: deltas under about 5 percent are unproven
This is the single most important thing in this directory.
`llama-bench`'s within-run error bars understate **cross-invocation** variance
by roughly **14x**. The same config measured `276.7 +/- 0.9` in one invocation
and `296.4 +/- 0.6` in another, 7 percent apart, with error bars that claimed
0.3 percent precision.
Therefore:
* Any comparison you want to draw a conclusion from must live **inside a
single `llama-bench` invocation**, using comma-separated sweeps.
* Two different *builds* cannot share an invocation, so run them
**interleaved A/B/A/B/A/B across at least 3 rounds** and compare medians.
That is what `--build-b` does.
* State explicitly that any cross-build delta under about 5 percent is
unproven. `summarize.py` labels them `unproven` for you.
## Traps
* **`-tb` is not a `llama-bench` flag.** It is a `llama-server` flag. Passing
it makes `llama-bench` print usage and exit silently, mid-sweep, which looks
exactly like a run that produced nothing. Always check the row count.
* **A ubatch larger than the prompt never fills.** With `-ub 2048`, prompt
lengths must be multiples of 2048 or the sweep measures nothing meaningful.
* **Nothing else may touch the GPU.** `bench.sh` stops `llama-swap` and
refuses to start if any `llama-*` process is still alive.
* **Never leave the pod not serving.** `bench.sh` restarts `llama-swap` from
an EXIT trap. If you bypass the script, restart it by hand:
`supervisorctl -c /root/supervisord.conf start llama-swap`.
* **Do not run `llama-cli` detached with closed stdin** on this box; it loops
forever. Use the server harness or `llama-bench -d` for depth timing.
* **`--mmap` / `--no-mmap` / `--mlock` and llama-bench's `-mmp` are gone**
upstream (#28334). Use `-lm none` (`--load-mode none`) on both
`llama-server` and `llama-bench`. The old flags now fail: the server refuses
to start, `llama-bench` silently produces no rows. `/root/config.yaml` must
be updated to `-lm none` as well before deploying a build past that commit.
## Optional: the MoE expert cache
Upstream #29887 added `--moe-cache-mib N` (default 0, off): a device cache of
the most recently used experts of the CPU-resident (`--n-cpu-moe`) layers. It
mostly helps decode and small batches; large prefill batches only read from it.
It is a server option only - `llama-bench` has no flag for it - so measure it
with the corpus harness, e.g.
`LLAMA_ARG_MOE_CACHE_MIB=1024 bench.sh --corpus-only --build DIR --label NAME`
(the environment reaches the spawned server). The variable applies to every
arm of the run, so an on/off comparison spans two runs and the noise floor
above applies in full. The cache takes VRAM from the same 8 GB as the KV
cache and compute buffers.
## Known-good reference values
If a fresh run disagrees with these by much more than the noise floor,
something is wrong (something else on the GPU, a thermal problem, a bad
build) before you believe you found a speedup.
Model Qwen3.6-35B-A3B-UD-Q4_K_XL, production config.
llama-bench, current production fork (20261024 rebase, `6c480f7e7`) and the
previous production fork (`5751c977b`), measured 2026-10-24 by
`fork-validate.sh --perf` (2 interleaved arms each, `-r 2` prefill, `-r 3`
decode):
| metric | about |
|---|---|
| pp2048 (depth 0) | 274-286 t/s (274 current, 286 previous; the -4% is inside the noise floor) |
| pp2048@d16384 | 201-206 t/s |
| pp2048@d32768 | 158-160 t/s |
| tg256@d4096 | 17.8 t/s |
Corpus harness, measured 2026-09-08/09 on the then-current upstream
(`f3f1a8f27`) and the fork deployed then (not re-measured since):
| metric | clean upstream | fork |
|---|---|---|
| corpus pp @ 4k | about 240 t/s | about 255 t/s |
| corpus pp @ 16k | about 206 t/s | about 225 t/s |
| corpus pp @ 32k | about 171 t/s | about 195 t/s |
| llama-bench pp8192 | about 222 t/s | about 246 t/s |
Invalid-metric history: the 2026-09 runs recorded `tg32` (7-15 t/s, swinging
by run) and older notes mention "tg32 about 19 t/s". Neither is a decode
reference; see the tg32 warning above.
Model load with `-lm none` takes about 26 s warm, longer cold.
A full 3-round A/B run of both harnesses takes roughly 100 minutes;
add about 16 minutes per extra arm per round. The tg256@d4096 decode cell
costs about 1.2 minutes per arm per round (model load, the 4096-token depth
prefill and 3 x 256 tokens), measured 2026-10-24.
+167
View File
@@ -0,0 +1,167 @@
#!/bin/sh
#
# RX 580 prompt-processing benchmark orchestrator. Runs ON THE POD.
#
# Stops llama-swap, waits for the GPU to go idle, runs the real-corpus
# prompt-processing harness and the llama-bench sweep for every arm, appends
# parseable results, and ALWAYS restarts llama-swap on exit (success, error
# or Ctrl-C).
#
# Usage:
# bench.sh --build DIR --label NAME
# [--build-b DIR --label-b NAME] second arm, interleaved
# [--arm DIR:LABEL ...] extra arms, repeatable (N-way)
# [--rounds N] default 3 multi-arm, 1 otherwise
# [--reps N] corpus repeats per size, default 3
# [--sizes 4096,16384,32768]
# [--corpus-only | --lb-only]
# [--results PATH] default /root/bench/results.txt
# [--keep-swap-down] do not restart llama-swap at exit
#
set -e
BENCH_DIR=/root/bench
SUPCONF=/root/supervisord.conf
RESULTS="$BENCH_DIR/results.txt"
MODEL=/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
ARMS="" # space separated DIR|LABEL entries
BUILD_A=""; LABEL_A=""
BUILD_B=""; LABEL_B=""
ROUNDS=""
REPS=3
SIZES=4096,16384,32768
DO_CORPUS=1
DO_LB=1
KEEP_SWAP_DOWN=0
while [ $# -gt 0 ]; do
case "$1" in
--build) BUILD_A="$2"; shift 2 ;;
--label) LABEL_A="$2"; shift 2 ;;
--build-b) BUILD_B="$2"; shift 2 ;;
--label-b) LABEL_B="$2"; shift 2 ;;
--arm) ARMS="$ARMS ${2%%:*}|${2#*:}"; shift 2 ;;
--rounds) ROUNDS="$2"; shift 2 ;;
--reps) REPS="$2"; shift 2 ;;
--sizes) SIZES="$2"; shift 2 ;;
--results) RESULTS="$2"; shift 2 ;;
--corpus-only) DO_LB=0; shift ;;
--lb-only) DO_CORPUS=0; shift ;;
--keep-swap-down) KEEP_SWAP_DOWN=1; shift ;;
-h|--help) sed -n '2,20p' "$0"; exit 0 ;;
*) echo "unknown arg: $1" >&2; exit 2 ;;
esac
done
if [ -n "$BUILD_A" ] && [ -z "$LABEL_A" ]; then
echo "error: --build needs --label" >&2; exit 2
fi
if [ -n "$BUILD_B" ] && [ -z "$LABEL_B" ]; then
echo "error: --build-b needs --label-b" >&2; exit 2
fi
# --build/--label and --build-b/--label-b are sugar for the first two arms.
if [ -n "$BUILD_A" ]; then ARMS="$BUILD_A|$LABEL_A $ARMS"; fi
if [ -n "$BUILD_B" ]; then ARMS="$ARMS $BUILD_B|$LABEL_B"; fi
ARMS=$(echo $ARMS)
if [ -z "$ARMS" ]; then
echo "error: need at least one arm (--build/--label or --arm DIR:LABEL)" >&2
exit 2
fi
NARMS=$(echo "$ARMS" | wc -w)
if [ -z "$ROUNDS" ]; then
if [ "$NARMS" -gt 1 ]; then ROUNDS=3; else ROUNDS=1; fi
fi
for e in $ARMS; do
d=${e%%|*}
if [ ! -x "$d/bin/llama-server" ] || [ ! -x "$d/bin/llama-bench" ]; then
echo "error: $d lacks bin/llama-server or bin/llama-bench" >&2
exit 2
fi
done
cat <<'WARN'
-------------------------------------------------------------------------
RX 580 benchmark. Read this before trusting any number.
* Cross-invocation variance on this box is about 7 percent. llama-bench
within-run error bars understate it by roughly 14x. Any cross-build
delta under about 5 percent is UNPROVEN noise.
* Comparisons you care about must live inside ONE llama-bench invocation
(comma-separated sweeps), or be interleaved across at least 3 rounds.
That is what multiple arms plus --rounds does.
* Nothing else may touch the GPU while this runs. llama-swap is stopped
for the duration and restarted on exit, including on error or Ctrl-C.
* Traps: -tb is NOT a llama-bench flag. A ubatch larger than -p never
fills, so prompt lengths must be multiples of the 2048 ubatch. A bad
flag makes llama-bench print usage and exit silently mid-sweep.
-------------------------------------------------------------------------
WARN
echo "[bench] $NARMS arm(s), $ROUNDS round(s): $ARMS"
restore_swap() {
rc=$?
if [ "$KEEP_SWAP_DOWN" -eq 0 ]; then
echo ""
echo "[bench] restarting llama-swap"
supervisorctl -c "$SUPCONF" start llama-swap || true
supervisorctl -c "$SUPCONF" status llama-swap || true
else
echo "[bench] --keep-swap-down: llama-swap left STOPPED"
fi
exit $rc
}
trap restore_swap EXIT INT TERM
echo "[bench] stopping llama-swap"
supervisorctl -c "$SUPCONF" stop llama-swap || true
wait_gpu_idle() {
i=0
while [ $i -lt 60 ]; do
if ! pgrep -f 'bin/llama-server|bin/llama-bench|bin/llama-cli' >/dev/null 2>&1; then
return 0
fi
sleep 2
i=$((i+1))
done
echo "error: something is still on the GPU:" >&2
pgrep -af 'bin/llama-server|bin/llama-bench|bin/llama-cli' >&2
exit 3
}
echo "[bench] waiting for the GPU to go idle"
wait_gpu_idle
echo "[bench] GPU idle"
run_corpus() {
[ "$DO_CORPUS" -eq 1 ] || return 0
echo "===== corpus $2 ====="
wait_gpu_idle
python3 "$BENCH_DIR/ppbench.py" "$1" "$2" \
--sizes "$SIZES" --reps "$REPS" --results "$RESULTS"
}
run_lb() {
[ "$DO_LB" -eq 1 ] || return 0
echo "===== llama-bench $2 ====="
wait_gpu_idle
RESULTS="$RESULTS" MODEL="$MODEL" sh "$BENCH_DIR/lbsweep.sh" "$1" "$2"
}
# Interleave: every arm is measured once per round, so slow drift in the
# machine hits all arms roughly equally instead of biasing whichever ran first.
r=1
while [ "$r" -le "$ROUNDS" ]; do
echo ""
echo "########## ROUND $r / $ROUNDS ##########"
for e in $ARMS; do run_corpus "${e%%|*}" "${e#*|}"; done
for e in $ARMS; do run_lb "${e%%|*}" "${e#*|}"; done
r=$((r+1))
done
echo ""
echo "ALL_RUNS_DONE"
echo "[bench] raw results appended to $RESULTS"
python3 "$BENCH_DIR/summarize.py" "$RESULTS" || true
+95
View File
@@ -0,0 +1,95 @@
#!/bin/sh
#
# llama-bench sweep, the controlled cross-check for the corpus harness.
# Runs ON THE POD. Normally invoked by bench.sh.
#
# Usage: lbsweep.sh <build_dir> <label>
# Env overrides: RESULTS, MODEL, LB_P, LB_N, LB_D, LB_R
#
# Two llama-bench invocations per call:
# prefill -p $LB_P -n 0 at depth 0 (default pp2048,pp8192)
# decode -p 0 -n $LB_N -d $LB_D (default tg256 @ d4096)
# LB_N=0 skips the decode invocation.
#
# Everything that must be compared lives inside ONE llama-bench invocation
# (-p takes a comma-separated list), because cross-invocation variance on this
# box is about 7 percent while the within-run error bars are about 0.3 percent.
# Prefill and decode cells are never compared with each other, so they may
# live in separate invocations.
#
# Decode is measured at an explicit depth on purpose: tg32 at the default
# depth (n_ctx 32) is launch/barrier bound on this card and is NOT a valid
# decode metric (it once read -36% between two builds that were at parity at
# tg256). Decode rows are labelled with their depth (tg256@d4096); old tg32
# rows in results.txt are invalid-metric history.
#
# Flag traps, learned the hard way:
# * -tb is NOT a llama-bench flag. It exists on llama-server only.
# * -ncmoe is the llama-bench spelling of --n-cpu-moe.
# * -lm none is the load mode the server runs with. -mmp was removed upstream
# (#28334) and, like any bad flag, now makes the sweep silently empty.
# * A bad flag makes llama-bench print usage and exit silently, which looks
# exactly like a sweep that produced no rows. Always check row count.
# * A ubatch larger than -p never fills, so keep -p a multiple of -ub.
#
set -e
BUILD_DIR="$1"
LABEL="$2"
if [ -z "$BUILD_DIR" ] || [ -z "$LABEL" ]; then
echo "usage: $0 <build_dir> <label>" >&2
exit 2
fi
RESULTS="${RESULTS:-/root/bench/results.txt}"
MODEL="${MODEL:-/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf}"
LB_P="${LB_P:-2048,8192}"
LB_N="${LB_N:-256}"
LB_D="${LB_D:-4096}"
LB_R="${LB_R:-3}"
STAMP=$(date -u +%Y-%m-%dT%H:%M:%S)
export LD_LIBRARY_PATH="$BUILD_DIR/bin"
export LABEL BUILD_DIR STAMP RESULTS
lb() {
"$BUILD_DIR/bin/llama-bench" \
-m "$MODEL" \
-t 6 -ngl 99 -ncmoe 40 -b 2048 -ub 2048 -fa 1 -lm none \
-r "$LB_R" -o json "$@" 2>/dev/null
}
record() { # $1 = what, stdin = llama-bench JSON
WHAT="$1" python3 -c '
import json, sys, os
label = os.environ["LABEL"]
build = os.environ["BUILD_DIR"]
stamp = os.environ["STAMP"]
results = os.environ["RESULTS"]
what = os.environ["WHAT"]
text = sys.stdin.read()
if not text.strip():
sys.stderr.write("ERROR: llama-bench produced no output for %s %s (bad flag?)\n" % (label, what))
sys.exit(3)
rows = json.loads(text)
if not rows:
sys.stderr.write("ERROR: llama-bench returned zero rows for %s %s\n" % (label, what))
sys.exit(3)
out = open(results, "a")
for r in rows:
test = "pp%d" % r["n_prompt"] if r["n_prompt"] else "tg%d" % r["n_gen"]
if r.get("n_depth", 0):
test += "@d%d" % r["n_depth"]
line = ("LBRESULT label=%s build=%s test=%s tps=%.2f stddev=%.2f "
"samples=%s time=%s"
% (label, build, test, r["avg_ts"], r["stddev_ts"],
r["samples_ts"], stamp))
out.write(line + "\n")
print(line)
out.close()
'
}
lb -p "$LB_P" -n 0 | record prefill
if [ "$LB_N" -gt 0 ]; then
lb -p 0 -n "$LB_N" -d "$LB_D" | record decode
fi
+239
View File
@@ -0,0 +1,239 @@
#!/usr/bin/env python3
"""
Real-corpus prompt-processing benchmark for llama-server on the RX 580 pod.
Runs ON THE POD. Normally invoked by bench.sh, not directly.
Starts llama-server from a given build directory with the production config,
waits for /health, POSTs fixed slices of the Pan Tadeusz corpus to /completion,
records timings.prompt_n / prompt_ms / prompt_per_second, then shuts the server
down cleanly.
Prompt slices are FIXED CONSTANTS (see SLICES below), calibrated once against
the Qwen3.6-35B-A3B tokenizer, so that every build is measured on byte-identical
input. Do not re-calibrate for a normal run; use --calibrate only if the model
or the corpus changes.
Usage:
ppbench.py <build_dir> <label> [--sizes 4096,16384,32768] [--reps 3]
[--results PATH] [--calibrate]
"""
import json
import os
import signal
import subprocess
import sys
import time
import urllib.request
from statistics import median
BENCH_DIR = "/root/bench"
CORPUS = os.path.join(BENCH_DIR, "pan-tadeusz.txt")
DEFAULT_RESULTS = os.path.join(BENCH_DIR, "results.txt")
MODEL = ("/root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/"
"snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/"
"Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf")
PORT = 8099
BASE = "http://127.0.0.1:%d" % PORT
# Production serving config. Keep in sync with /root/config.yaml (llama-swap).
# Traps encoded here on purpose:
# -b / -ub 2048 : a ubatch larger than the prompt never fills, so every
# prompt length below must stay a multiple of 2048.
# -lm none : what --no-mmap used to be; --mmap/--no-mmap/--mlock were
# removed upstream (#28334), so the old spelling now fails.
SERVER_ARGS = ["-t", "6", "-ngl", "99", "--n-cpu-moe", "40",
"-b", "2048", "-ub", "2048", "-fa", "1", "-lm", "none",
"--ctx-size", "40960", "--no-warmup",
"--port", str(PORT), "-m", MODEL]
# Calibrated once on 2026-09-08 against Qwen3.6-35B-A3B-UD-Q4_K_XL:
# Polish text of pan-tadeusz.txt runs 2.6702 chars/token, and /completion
# reports prompt_n identical to /tokenize (BOS offset 0).
# Each slice is corpus[0:NCHARS] read as UTF-8 text; NBYTES is the resulting
# UTF-8 byte length, recorded so the slice can be reproduced with byte tools.
# These produce EXACTLY the target prompt_n. Do not edit without recalibrating.
SLICES = {
4096: {"chars": 10776, "bytes": 11608},
16384: {"chars": 43858, "bytes": 47175},
32768: {"chars": 87165, "bytes": 93843},
}
DEFAULT_SIZES = [4096, 16384, 32768]
def post(path, payload, timeout=2400):
req = urllib.request.Request(
BASE + path,
data=json.dumps(payload).encode("utf-8"),
headers={"Content-Type": "application/json"},
method="POST")
with urllib.request.urlopen(req, timeout=timeout) as r:
return json.loads(r.read().decode("utf-8"))
def health_ok():
try:
with urllib.request.urlopen(BASE + "/health", timeout=5) as r:
return json.loads(r.read().decode("utf-8")).get("status") == "ok"
except Exception:
return False
def start_server(build_dir, logpath):
env = dict(os.environ)
env["LD_LIBRARY_PATH"] = os.path.join(build_dir, "bin")
binary = os.path.join(build_dir, "bin", "llama-server")
if not os.path.exists(binary):
raise RuntimeError("no llama-server at %s" % binary)
log = open(logpath, "wb")
p = subprocess.Popen([binary] + SERVER_ARGS, stdout=log,
stderr=subprocess.STDOUT, env=env,
start_new_session=True)
deadline = time.time() + 1200
while time.time() < deadline:
if p.poll() is not None:
raise RuntimeError("llama-server exited early rc=%s, see %s"
% (p.returncode, logpath))
if health_ok():
return p
time.sleep(2)
stop_server(p)
raise RuntimeError("llama-server not healthy within 1200s, see %s" % logpath)
def stop_server(p):
if p is None or p.poll() is not None:
return
try:
os.killpg(os.getpgid(p.pid), signal.SIGTERM)
except Exception:
p.terminate()
for _ in range(120):
if p.poll() is not None:
return
time.sleep(1)
try:
os.killpg(os.getpgid(p.pid), signal.SIGKILL)
except Exception:
p.kill()
p.wait()
def n_tokens(text):
"""Exact token count via /tokenize. Cheap: no prefill."""
return len(post("/tokenize", {"content": text}, timeout=300)["tokens"])
def calibrate(corpus, targets):
"""Recompute the SLICES table. Only needed if model or corpus changes."""
probe = corpus[:2000]
tk = n_tokens(probe)
r = post("/completion", {"prompt": probe, "n_predict": 1,
"cache_prompt": False, "temperature": 0})
offset = r["timings"]["prompt_n"] - tk
cpt = len(probe) / float(tk)
sys.stderr.write("calibrate: %.4f chars/token, prompt_n offset %d\n"
% (cpt, offset))
for target in targets:
want = target - offset
lo, hi = 1, len(corpus)
guess = min(len(corpus), max(1, int(want * cpt)))
best = None
for _ in range(60):
got = n_tokens(corpus[:guess])
if got == want:
best = guess
break
if got < want:
lo = guess + 1
else:
hi = guess - 1
if lo > hi:
break
guess = (lo + hi) // 2
if best is None:
best = guess
nbytes = len(corpus[:best].encode("utf-8"))
sys.stderr.write("calibrate: %d tokens -> chars=%d bytes=%d\n"
% (target, best, nbytes))
def main():
if len(sys.argv) < 3:
sys.stderr.write(__doc__)
return 2
build_dir = os.path.abspath(sys.argv[1])
label = sys.argv[2]
sizes = list(DEFAULT_SIZES)
reps = 3
results_path = DEFAULT_RESULTS
do_calibrate = "--calibrate" in sys.argv
args = sys.argv[3:]
for i, a in enumerate(args):
if a == "--sizes":
sizes = [int(x) for x in args[i + 1].split(",")]
elif a == "--reps":
reps = int(args[i + 1])
elif a == "--results":
results_path = args[i + 1]
corpus = open(CORPUS, encoding="utf-8").read()
logpath = os.path.join(BENCH_DIR, "server-%s-%d.log" % (label, int(time.time())))
p = None
try:
sys.stderr.write("[%s] starting llama-server from %s\n" % (label, build_dir))
t0 = time.time()
p = start_server(build_dir, logpath)
sys.stderr.write("[%s] healthy after %.1fs\n" % (label, time.time() - t0))
if do_calibrate:
calibrate(corpus, sizes)
return 0
stamp = time.strftime("%Y-%m-%dT%H:%M:%S")
out = open(results_path, "a")
for target in sizes:
if target not in SLICES:
sys.stderr.write("no calibrated slice for %d tokens, skipping\n" % target)
continue
nchars = SLICES[target]["chars"]
prompt = corpus[:nchars]
rates, mss, ns = [], [], set()
for rep in range(1, reps + 1):
r = post("/completion", {"prompt": prompt, "n_predict": 1,
"cache_prompt": False, "temperature": 0})
t = r["timings"]
rates.append(t["prompt_per_second"])
mss.append(t["prompt_ms"])
ns.add(t["prompt_n"])
line = ("RESULT corpus label=%s build=%s target=%d chars=%d "
"prompt_n=%d rep=%d prompt_ms=%.2f tps=%.2f time=%s"
% (label, build_dir, target, nchars, t["prompt_n"], rep,
t["prompt_ms"], t["prompt_per_second"], stamp))
out.write(line + "\n")
out.flush()
sys.stderr.write(line + "\n")
if ns != {target}:
sys.stderr.write("WARNING: prompt_n %s != target %d; slice table "
"is stale, rerun with --calibrate\n" % (sorted(ns), target))
s = ("SUMMARY corpus label=%s target=%d prompt_n=%s median_tps=%.2f "
"min_tps=%.2f max_tps=%.2f median_ms=%.1f time=%s"
% (label, target, sorted(ns), median(rates), min(rates),
max(rates), median(mss), stamp))
out.write(s + "\n")
out.flush()
sys.stderr.write(s + "\n")
out.close()
finally:
stop_server(p)
sys.stderr.write("[%s] server stopped\n" % label)
return 0
if __name__ == "__main__":
sys.exit(main())
+91
View File
@@ -0,0 +1,91 @@
#!/usr/bin/env bash
#
# One-command entry point for the RX 580 prompt-processing benchmark.
#
# Runs from your workstation. Copies the pod-side harness into the pod and
# execs it there over kubectl. All the real work happens on the pod; this is
# a thin wrapper so there is exactly one command to remember.
#
# Examples:
# ./scripts/rx580-bench/run.sh --build /root/llama.cpp/build --label fork-before
# ./scripts/rx580-bench/run.sh --build /root/arms/new-clean/build --label new-clean \
# --build-b /root/arms/new-fork/build --label-b new-fork
# ./scripts/rx580-bench/run.sh --summarize
# ./scripts/rx580-bench/run.sh --install-only
#
# Every argument other than the wrapper-only flags below is passed straight
# through to bench.sh on the pod. See bench.sh --help.
#
set -euo pipefail
KUBECONFIG_PATH="${KUBECONFIG:-/home/user/Projects/klaster/talos/generated/kubeconfig}"
NS="${RX580_NS:-llama}"
DEPLOY="${RX580_DEPLOY:-deploy/supervisord}"
POD_BENCH_DIR=/root/bench
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export KUBECONFIG="$KUBECONFIG_PATH"
kx() { kubectl -n "$NS" exec -i "$DEPLOY" -- sh -c "$1"; }
install_harness() {
echo "[run] installing harness into $POD_BENCH_DIR"
kx "mkdir -p $POD_BENCH_DIR"
for f in bench.sh ppbench.py lbsweep.sh summarize.py; do
kx "cat > $POD_BENCH_DIR/$f" < "$HERE/$f"
done
kx "chmod +x $POD_BENCH_DIR/bench.sh $POD_BENCH_DIR/lbsweep.sh $POD_BENCH_DIR/ppbench.py $POD_BENCH_DIR/summarize.py"
# The corpus is large and rarely changes; only push it if missing.
if ! kx "test -s $POD_BENCH_DIR/pan-tadeusz.txt" 2>/dev/null; then
if [ -f "$HERE/pan-tadeusz.txt" ]; then
echo "[run] uploading corpus"
kx "cat > $POD_BENCH_DIR/pan-tadeusz.txt" < "$HERE/pan-tadeusz.txt"
else
echo "[run] ERROR: $POD_BENCH_DIR/pan-tadeusz.txt missing on the pod and" >&2
echo " no local copy at $HERE/pan-tadeusz.txt to upload." >&2
exit 4
fi
fi
}
ARGS=()
INSTALL_ONLY=0
SUMMARIZE=0
for a in "$@"; do
case "$a" in
--install-only) INSTALL_ONLY=1 ;;
--summarize) SUMMARIZE=1 ;;
*) ARGS+=("$a") ;;
esac
done
install_harness
if [ "$INSTALL_ONLY" -eq 1 ]; then
echo "[run] harness installed; not running anything"
exit 0
fi
if [ "$SUMMARIZE" -eq 1 ]; then
kx "python3 $POD_BENCH_DIR/summarize.py ${ARGS[*]:-}"
exit 0
fi
if [ ${#ARGS[@]} -eq 0 ]; then
kx "sh $POD_BENCH_DIR/bench.sh --help"
exit 2
fi
# A full 3-round A/B run takes hours; run detached on the pod and tail it, so a
# dropped kubectl connection cannot orphan a half-finished run with llama-swap
# still down. bench.sh restarts llama-swap from its own EXIT trap either way.
STAMP="$(date -u +%Y%m%d-%H%M%S)"
LOG="$POD_BENCH_DIR/run-$STAMP.log"
echo "[run] starting detached run on the pod, log: $LOG"
kx "cd $POD_BENCH_DIR && nohup sh $POD_BENCH_DIR/bench.sh ${ARGS[*]} > $LOG 2>&1 & echo started"
echo "[run] tailing until ALL_RUNS_DONE (safe to Ctrl-C: the pod keeps running)"
kx "i=0; while [ \$i -lt 100000 ]; do
if grep -qE 'ALL_RUNS_DONE|Traceback|error:' $LOG; then break; fi
sleep 15; i=\$((i+1));
done; cat $LOG"
+101
View File
@@ -0,0 +1,101 @@
#!/usr/bin/env python3
"""
Summarize a results.txt produced by bench.sh into a per-label table.
Usage: summarize.py [results.txt]
Prints median and min-max of every metric for every label, and flags
cross-label deltas against the ~5 percent noise floor of this machine.
"""
import re
import sys
from statistics import median
NOISE_FLOOR_PCT = 5.0
RE_CORPUS = re.compile(
r"^RESULT corpus label=(\S+) build=\S+ target=(\d+) chars=\d+ "
r"prompt_n=(\d+) rep=\d+ prompt_ms=([\d.]+) tps=([\d.]+)")
RE_LB = re.compile(
r"^LBRESULT label=(\S+) build=\S+ test=(\S+) tps=([\d.]+)")
RE_TG_NODEPTH = re.compile(r"^tg\d+-llama-bench$")
def main():
path = sys.argv[1] if len(sys.argv) > 1 else "/root/bench/results.txt"
# data[metric][label] = list of tps
data = {}
order = []
labels = []
for line in open(path):
m = RE_CORPUS.match(line)
if m:
label, target, _pn, _ms, tps = m.groups()
metric = "pp%s-corpus" % target
else:
m = RE_LB.match(line)
if not m:
continue
label, test, tps = m.groups()
metric = "%s-llama-bench" % test
if metric not in data:
data[metric] = {}
order.append(metric)
data[metric].setdefault(label, []).append(float(tps))
if label not in labels:
labels.append(label)
if not data:
print("no parseable results in %s" % path)
return 1
def sort_key(m):
n = re.search(r"(\d+)", m)
return (0 if "corpus" in m else 1, int(n.group(1)) if n else 0)
order.sort(key=sort_key)
w = max(len(l) for l in labels) + 2
head = "metric".ljust(26) + "".join(l.ljust(max(w, 24)) for l in labels)
print("")
print("t/s, median (min-max), n samples")
print(head)
print("-" * len(head))
for metric in order:
row = metric.ljust(26)
for label in labels:
vals = data[metric].get(label)
if not vals:
row += "-".ljust(max(w, 24))
else:
cell = "%.1f (%.1f-%.1f) n=%d" % (
median(vals), min(vals), max(vals), len(vals))
row += cell.ljust(max(w, 24))
print(row)
if any(RE_TG_NODEPTH.match(m) for m in order):
print("note: tgN rows without @d<depth> ran at n_ctx ~N (old default);"
" INVALID decode metric on this box, history only")
if len(labels) >= 2:
base = labels[0]
print("")
print("deltas vs %s (noise floor %.0f%%, anything under it is UNPROVEN)"
% (base, NOISE_FLOOR_PCT))
for other in labels[1:]:
print(" %s vs %s:" % (other, base))
for metric in order:
a = data[metric].get(base)
b = data[metric].get(other)
if not a or not b:
continue
ma, mb = median(a), median(b)
pct = (mb - ma) / ma * 100.0
verdict = "SIGNIFICANT" if abs(pct) >= NOISE_FLOOR_PCT else "unproven"
if RE_TG_NODEPTH.match(metric):
verdict = "INVALID metric (tg at n_ctx ~N), ignore"
print(" %-26s %+6.1f%% %s" % (metric, pct, verdict))
return 0
if __name__ == "__main__":
sys.exit(main())
+8 -6
View File
@@ -69,6 +69,7 @@ These recur often enough in review comments on past add-model PRs that they're w
- Before writing a dedicated tool-call/output parser, check whether the existing autoparser already handles the template (`test-chat-auto-parser <jinja>` shows what it detects).
- Marking a custom EOS/closing-tag token as `eot` at conversion time isn't always sufficient - in long/agentic generations a model can emit the closing sequence as literal text instead of the token, so generation never stops on EOG and raw text leaks past the parser. Verify this case, not just the token path.
- If reusing or aliasing an existing pre-tokenizer for convenience, justify and test that choice explicitly - silent reuse is an easy source of subtle tokenizer bugs.
- The GGUF only stores the pre-tokenizer name (`tokenizer.pre`), the behavior flags are hard-coded per name in `src/llama-vocab.cpp`. When adding a new name, set each flag from the HF `tokenizer.json`, not from the case you copied. Examples: `model.ignore_merges` -> `ignore_merges`, the BOS/EOS of the post-processor -> `special_bos_id` / `add_bos`, prefix space and space cleanup -> `add_space_prefix` / `clean_spaces`. The `chkhsh` check in the converter only identifies the pre-tokenizer on a short text, it may not catch a wrong flag.
- Watch for excessive graph splits caused by building per-layer view/index tensors inside the layer loop - hoist tensors that don't vary per layer out of the loop (relevant if you hit `GGML_SCHED_MAX_SPLIT_INPUTS`).
- A custom KQ mask fed into flash attention must match FA's expected dtype - cast it to F16 before passing it to `build_attn_mha` when FA is enabled.
- When padding a custom KV-cache size to an alignment (e.g. `GGML_PAD(..., 256)`), apply the padding after all other size adjustments, not before - otherwise later logic can un-align it again.
@@ -83,12 +84,13 @@ These recur often enough in review comments on past add-model PRs that they're w
Reference: `examples/model-conversion/README.md`.
1. Convert to GGUF, then inspect/run both the original and converted tensors.
2. Run logits verification (original vs converted). If this model is a new version of an already-supported family, verify the *previous* version still passes logits verification first - numerical differences may be pre-existing, not caused by the new work. The tools to perform full logits validation are available in `examples/model-conversion`.
3. Quantize (including QAT variants if relevant) and re-verify.
4. Run perplexity evaluation (simple and full).
5. Sanity-check across `tools/cli`, `tools/completion`, `tools/imatrix`, `tools/quantize`, and `tools/server`.
6. CPU backend first; other backends (CUDA, Metal, ...) can be separate follow-up PRs per `CONTRIBUTING.md`.
7. Re-review every changed file against the coding/naming guidelines in `AGENTS.md` (and `CONTRIBUTING.md`'s "Coding guidelines"/"Naming guidelines" sections) - this is a separate pass from functional testing and is just as important: no forced line-wrapping, no unicode punctuation, minimal/non-redundant comments, `snake_case` naming (`kebab-case` for file names), matching indentation/brace style, etc.
2. If the model adds a new pre-tokenizer, compare the token ids of `llama-tokenize -f <file> --ids --no-escape` with the HF tokenizer on whole files of prose, code and multilingual text (e.g. wikitext-2 and a few source files).
3. Run logits verification (original vs converted). If this model is a new version of an already-supported family, verify the *previous* version still passes logits verification first - numerical differences may be pre-existing, not caused by the new work. The tools to perform full logits validation are available in `examples/model-conversion`.
4. Quantize (including QAT variants if relevant) and re-verify.
5. Run perplexity evaluation (simple and full).
6. Sanity-check across `tools/cli`, `tools/completion`, `tools/imatrix`, `tools/quantize`, and `tools/server`.
7. CPU backend first; other backends (CUDA, Metal, ...) can be separate follow-up PRs per `CONTRIBUTING.md`.
8. Re-review every changed file against the coding/naming guidelines in `AGENTS.md` (and `CONTRIBUTING.md`'s "Coding guidelines"/"Naming guidelines" sections) - this is a separate pass from functional testing and is just as important: no forced line-wrapping, no unicode punctuation, minimal/non-redundant comments, `snake_case` naming (`kebab-case` for file names), matching indentation/brace style, etc.
## Before opening a PR
+169 -21
View File
@@ -635,7 +635,8 @@ llama_context::~llama_context() {
synchronize();
// when training, ggml_opt allocates extra buffers through the scheduler, so the sizes no longer match the expectation
if (!model.no_alloc() && !opt_ctx) {
// the scheduler is also gone if the context is destroyed while cold (on-demand VRAM eviction, see release_device)
if (sched && !model.no_alloc() && !opt_ctx) {
for (size_t i = 0; i < backend_ptrs.size(); ++i) {
ggml_backend_t backend = backend_ptrs[i];
ggml_backend_buffer_type_t buft = backend_buft[i];
@@ -980,6 +981,86 @@ void llama_context::synchronize() {
t_compute_start_us = 0;
}
bool llama_context::device_resident() const {
return model.weights_resident() && (!moe_cache || moe_cache->resident());
}
void llama_context::release_device(bool evict_kv) {
if (!device_resident() && !(evict_kv && !kv_device_evicted)) {
return;
}
// ensure no compute is in flight before freeing the device buffers
synchronize();
model.release_device_weights();
// the banks are emptied, not shadowed: restored slot maps must not point at zeroed experts
if (moe_cache) {
moe_cache->release_device();
}
if (evict_kv && memory && !kv_device_evicted) {
memory->release_device_buffers();
kv_device_evicted = true;
// also free the scheduler and its worst-case compute buffer (hundreds of MiB) so a cold
// model holds essentially no VRAM. Rebuilt lazily by sched_reserve() on restore.
sched.reset();
sched_need_reserve = true;
// finally, free each backend's own compute-scratch (Vulkan prealloc/staging buffers, which
// are owned by the backend and survive sched.reset()). Reallocated lazily on next compute.
for (auto & backend : backends) {
ggml_backend_free_scratch(backend.get());
}
}
}
void llama_context::discard_device() {
// same as release_device(true), minus the host copies of the weights and the KV
synchronize();
model.release_device_weights(/*keep_shadow =*/ false);
if (moe_cache) {
moe_cache->release_device();
}
if (memory) {
memory->release_device_buffers(/*keep_shadow =*/ false);
}
kv_device_evicted = true;
device_discarded = true;
sched.reset();
sched_need_reserve = true;
for (auto & backend : backends) {
ggml_backend_free_scratch(backend.get());
}
}
bool llama_context::restore_memory_host() {
device_discarded = true;
if (!kv_device_evicted || !memory) {
return true;
}
return memory->restore_device_buffers(/*to_host =*/ true);
}
void llama_context::restore_device() {
GGML_ASSERT(!device_discarded && "device buffers were discarded");
if (device_resident() && !kv_device_evicted && sched) {
return;
}
model.restore_device_weights();
if (moe_cache) {
moe_cache->restore_device();
}
if (kv_device_evicted && memory) {
memory->restore_device_buffers();
kv_device_evicted = false;
}
// rebuild the scheduler + compute buffer if they were freed on release (evict_kv mode)
if (!sched) {
sched_reserve();
}
// make sure all weight/KV uploads have completed before any compute reads them
if (sched) {
ggml_backend_sched_synchronize(sched.get());
}
}
const llama_model & llama_context::get_model() const {
return model;
}
@@ -2164,6 +2245,12 @@ int llama_context::decode(const llama_batch_ext & batch_inp) {
return -1;
}
// on-demand VRAM: if the weights were released while this model was idle, bring them back
// to the device before building the compute graph
if (!device_resident()) {
restore_device();
}
const auto & vocab = model.vocab;
const auto & hparams = model.hparams;
@@ -3143,6 +3230,19 @@ bool llama_context::sched_copy_experts(ggml_backend_t backend, const ggml_tensor
return false;
}
// Later loop passes can select different experts from the same device copy.
// TODO: copy only the experts needed by each pass (see #32807 and #32809).
// [fork F13] with the per-pass host-weight views every pass has its own copy, so the partial
// copy is correct again; LLAMA_LOOP_FULL_COPY=0 skips this guard (A/B knob, only correct with
// the views on, LLAMA_LOOP_HOST_VIEWS unset or 1)
static const bool loop_full_copy = [] {
const char * env = getenv("LLAMA_LOOP_FULL_COPY");
return env == nullptr || strcmp(env, "0") != 0;
}();
if (loop_full_copy && lctx->model.arch == LLM_ARCH_INKLING2_LOOP) {
return false;
}
const ggml_tensor * ids = node->src[2];
if (ggml_nelements(ids) == 0) {
return true;
@@ -3152,16 +3252,40 @@ bool llama_context::sched_copy_experts(ggml_backend_t backend, const ggml_tensor
const size_t expert_size = src->nb[2];
if (ids != st.ids || (int64_t) st.used.size() != n_expert) {
st.ids_data.resize(ggml_nbytes(ids)/sizeof(int32_t));
ggml_backend_tensor_get_async(backend, ids, st.ids_data.data(), 0, ggml_nbytes(ids));
ggml_backend_synchronize(backend);
// Reading the ids back forces a full pipeline flush, since the ids are
// produced on the same device we are about to upload to. Once the batch
// draws enough experts, essentially all of them come back used and the
// readback has bought nothing, so skip it and copy the whole tensor as one
// range. Copying an expert that no id selects cannot change the result:
// mul_mat_id only reads the rows the ids point at, so this stays exact.
// With the MoE cache on, cached experts are still copied device-to-device.
// validation/A-B knob: LLAMA_MOE_IDS_SKIP=0 always reads the ids back (the upstream path),
// =log reports every skip on stderr
static const char * skip_env = getenv("LLAMA_MOE_IDS_SKIP");
static const bool skip_off = skip_env != nullptr && strcmp(skip_env, "0") == 0;
static const bool skip_log = skip_env != nullptr && strcmp(skip_env, "log") == 0;
st.used.assign(n_expert, false);
for (int64_t i1 = 0; i1 < ids->ne[1]; i1++) {
for (int64_t i0 = 0; i0 < ids->ne[0]; i0++) {
const int32_t id = st.ids_data[i1*ids->nb[1]/sizeof(int32_t) + i0*ids->nb[0]/sizeof(int32_t)];
GGML_ASSERT(id >= 0 && id < n_expert);
st.used[id] = true;
const bool all_experts_used = !skip_off && ggml_nelements(ids) >= 4*n_expert;
if (all_experts_used) {
if (skip_log) {
fprintf(stderr, "sched_copy_experts: ids readback skipped, n_ids=%" PRId64 " n_expert=%" PRId64 "\n",
ggml_nelements(ids), n_expert);
}
st.ids_data.clear();
st.used.assign(n_expert, true);
} else {
st.ids_data.resize(ggml_nbytes(ids)/sizeof(int32_t));
ggml_backend_tensor_get_async(backend, ids, st.ids_data.data(), 0, ggml_nbytes(ids));
ggml_backend_synchronize(backend);
st.used.assign(n_expert, false);
for (int64_t i1 = 0; i1 < ids->ne[1]; i1++) {
for (int64_t i0 = 0; i0 < ids->ne[0]; i0++) {
const int32_t id = st.ids_data[i1*ids->nb[1]/sizeof(int32_t) + i0*ids->nb[0]/sizeof(int32_t)];
GGML_ASSERT(id >= 0 && id < n_expert);
st.used[id] = true;
}
}
}
@@ -4124,17 +4248,21 @@ llama_memory_breakdown llama_context::memory_breakdown() const {
ret[buft].context += size;
}
}
if (model.no_alloc()) {
for (size_t i = 0; i < backends.size(); ++i) {
ggml_backend_t backend = backends[i].get();
ggml_backend_buffer_type_t buft = ggml_backend_sched_get_buffer_type(sched.get(), backend);
ret[buft].compute += backend_buf_exp_size[i];
}
} else {
for (const auto & backend_ptr : backends) {
ggml_backend_t backend = backend_ptr.get();
ggml_backend_buffer_type_t buft = ggml_backend_sched_get_buffer_type(sched.get(), backend);
ret[buft].compute += ggml_backend_sched_get_buffer_size(sched.get(), backend);
// the scheduler (and its compute buffers) may have been freed while the model is cold
// (on-demand VRAM eviction, see release_device); it contributes no compute VRAM then.
if (sched) {
if (model.no_alloc()) {
for (size_t i = 0; i < backends.size(); ++i) {
ggml_backend_t backend = backends[i].get();
ggml_backend_buffer_type_t buft = ggml_backend_sched_get_buffer_type(sched.get(), backend);
ret[buft].compute += backend_buf_exp_size[i];
}
} else {
for (const auto & backend_ptr : backends) {
ggml_backend_t backend = backend_ptr.get();
ggml_backend_buffer_type_t buft = ggml_backend_sched_get_buffer_type(sched.get(), backend);
ret[buft].compute += ggml_backend_sched_get_buffer_size(sched.get(), backend);
}
}
}
return ret;
@@ -4652,6 +4780,26 @@ void llama_synchronize(llama_context * ctx) {
ctx->synchronize();
}
void llama_context_release_device(llama_context * ctx, bool evict_kv) {
ctx->release_device(evict_kv);
}
void llama_context_restore_device(llama_context * ctx) {
ctx->restore_device();
}
void llama_context_discard_device(llama_context * ctx) {
ctx->discard_device();
}
bool llama_context_restore_memory_host(llama_context * ctx) {
return ctx->restore_memory_host();
}
bool llama_context_weights_resident(const llama_context * ctx) {
return ctx->device_resident();
}
float * llama_get_logits(llama_context * ctx) {
ctx->synchronize();
+22
View File
@@ -60,6 +60,23 @@ struct llama_context {
void synchronize();
// on-demand device (VRAM) residency: free / rebuild the model's GPU weight buffers so that
// VRAM can be time-shared with another model. release_device() synchronizes first; decode()
// auto-restores when it finds the weights have been released. If evict_kv is set, the KV cache
// device buffers are also evicted to a host shadow (for when the KV would not fit alongside the
// other model) - this adds a D2H/H2D copy of the live KV on each cycle.
void release_device(bool evict_kv = false);
void restore_device();
// free every device buffer for good, without host copies (before exit). The context must not
// compute again: restore_device() aborts.
void discard_device();
// before exit: bring an evicted KV/recurrent state back in host memory so the state API can read
// it without the device. The context must not compute again: restore_device() aborts.
bool restore_memory_host();
// the model weights and the MoE cache are on the device
bool device_resident() const;
const llama_model & get_model() const;
const llama_cparams & get_cparams() const;
@@ -431,6 +448,11 @@ private:
int32_t n_reallocs_same = 0;
int32_t n_reallocs_diff = 0;
// on-demand device residency: true while the KV cache device buffers have been evicted to host
bool kv_device_evicted = false;
// set by discard_device(): the device buffers are gone and cannot be restored
bool device_discarded = false;
ggml_backend_t backend_cpu = nullptr;
std::vector<ggml_backend_ptr> backends;
+3
View File
@@ -112,6 +112,9 @@ using llama_memory_breakdown = std::map<ggml_backend_buffer_type_t, llama_memory
LLAMA_API int32_t llama_model_n_expert (const struct llama_model * model);
LLAMA_API int32_t llama_model_n_devices(const struct llama_model * model);
// number of logical passes that execute each layer's weights (0 for weight-sharing alias slots)
LLAMA_API std::vector<uint32_t> llama_model_layer_visits(const struct llama_model * model);
LLAMA_API ggml_backend_dev_t llama_model_get_device(const struct llama_model * model, int i);
LLAMA_API llama_memory_breakdown llama_get_memory_breakdown(const struct llama_context * ctx);
+165 -51
View File
@@ -7,6 +7,7 @@
#include <cmath>
#include <algorithm>
#include <cstdint>
#include <limits>
#include <set>
#include <numeric>
#include <stdexcept>
@@ -33,9 +34,9 @@ static std::pair<uint32_t, const char *> decode_utf8(const char * src) {
return std::make_pair(value, pos);
}
static std::pair<std::vector<uint32_t>, llama_partial_utf8> decode_utf8(
const std::string & src,
llama_partial_utf8 partial_start) {
std::pair<std::vector<uint32_t>, llama_partial_utf8> llama_grammar_decode_utf8(
const std::string & src,
llama_partial_utf8 partial_start) {
static const int lookup[] = { 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 2, 2, 3, 4 };
const char * pos = src.c_str();
std::vector<uint32_t> code_points;
@@ -185,23 +186,45 @@ static std::pair<uint32_t, const char *> parse_char(const char * src) {
throw std::runtime_error("unexpected end of input");
}
static std::pair<uint32_t, const char *> parse_token(const llama_vocab * vocab, const char * src) {
static std::pair<uint32_t, const char *> parse_token_id(const char * src) {
const char * int_end = parse_int(src);
unsigned long id = std::stoul(std::string(src, int_end - src));
if (id > std::numeric_limits<uint32_t>::max()) {
throw std::runtime_error(std::string("parsed token id is too big at ") + src);
}
return std::make_pair(static_cast<uint32_t>(id), int_end);
}
static const char * parse_token(const llama_vocab * vocab, const char * src, llama_gretype type, llama_grammar_rule & rule) {
const char * pos = src;
if (*pos != '<') {
throw std::runtime_error(std::string("expecting '<' at ") + pos);
}
pos++;
// Parse <[id]>
// Parse <[id,id-id,...]>
if (*pos == '[') {
pos++;
const char * int_end = parse_int(pos);
unsigned long id = std::stoul(std::string(pos, int_end - pos));
if (id > std::numeric_limits<uint32_t>::max()) {
throw std::runtime_error(std::string("parsed token id is too big at ") + pos);
const size_t start = rule.size();
while (true) {
auto id_pair = parse_token_id(pos);
uint32_t token_id = id_pair.first;
pos = id_pair.second;
rule.push_back({rule.size() == start ? type : LLAMA_GRETYPE_TOKEN_ALT, token_id});
if (*pos == '-') {
pos++;
auto end_pair = parse_token_id(pos);
if (end_pair.first < token_id) {
throw std::runtime_error(std::string("invalid token range at ") + pos);
}
pos = end_pair.second;
rule.push_back({LLAMA_GRETYPE_TOKEN_RNG_UPPER, end_pair.first});
}
if (*pos != ',') {
break;
}
pos++;
}
uint32_t token_id = static_cast<uint32_t>(id);
pos = int_end;
if (*pos != ']') {
throw std::runtime_error(std::string("expecting ']' at ") + pos);
}
@@ -210,7 +233,7 @@ static std::pair<uint32_t, const char *> parse_token(const llama_vocab * vocab,
throw std::runtime_error(std::string("expecting '>' at ") + pos);
}
pos++;
return std::make_pair(token_id, pos);
return pos;
}
if (vocab == nullptr) {
@@ -232,7 +255,8 @@ static std::pair<uint32_t, const char *> parse_token(const llama_vocab * vocab,
// must tokenize to exactly 1 token
throw std::runtime_error("invalid token '" + std::string(src, pos - src) + "'");
}
return std::make_pair(tokens[0], pos);
rule.push_back({type, static_cast<uint32_t>(tokens[0])});
return pos;
}
static void print_grammar_char(FILE * file, uint32_t c) {
@@ -244,6 +268,16 @@ static void print_grammar_char(FILE * file, uint32_t c) {
}
}
static bool is_token_element(llama_grammar_element elem) {
switch (elem.type) {
case LLAMA_GRETYPE_TOKEN: return true;
case LLAMA_GRETYPE_TOKEN_NOT: return true;
case LLAMA_GRETYPE_TOKEN_ALT: return true;
case LLAMA_GRETYPE_TOKEN_RNG_UPPER: return true;
default: return false;
}
}
static bool is_char_element(llama_grammar_element elem) {
switch (elem.type) {
case LLAMA_GRETYPE_CHAR: return true;
@@ -258,16 +292,18 @@ static bool is_char_element(llama_grammar_element elem) {
static void print_rule_binary(FILE * file, const llama_grammar_rule & rule) {
for (auto elem : rule) {
switch (elem.type) {
case LLAMA_GRETYPE_END: fprintf(file, "END"); break;
case LLAMA_GRETYPE_ALT: fprintf(file, "ALT"); break;
case LLAMA_GRETYPE_RULE_REF: fprintf(file, "RULE_REF"); break;
case LLAMA_GRETYPE_CHAR: fprintf(file, "CHAR"); break;
case LLAMA_GRETYPE_CHAR_NOT: fprintf(file, "CHAR_NOT"); break;
case LLAMA_GRETYPE_CHAR_RNG_UPPER: fprintf(file, "CHAR_RNG_UPPER"); break;
case LLAMA_GRETYPE_CHAR_ALT: fprintf(file, "CHAR_ALT"); break;
case LLAMA_GRETYPE_CHAR_ANY: fprintf(file, "CHAR_ANY"); break;
case LLAMA_GRETYPE_TOKEN: fprintf(file, "TOKEN"); break;
case LLAMA_GRETYPE_TOKEN_NOT: fprintf(file, "TOKEN_NOT"); break;
case LLAMA_GRETYPE_END: fprintf(file, "END"); break;
case LLAMA_GRETYPE_ALT: fprintf(file, "ALT"); break;
case LLAMA_GRETYPE_RULE_REF: fprintf(file, "RULE_REF"); break;
case LLAMA_GRETYPE_CHAR: fprintf(file, "CHAR"); break;
case LLAMA_GRETYPE_CHAR_NOT: fprintf(file, "CHAR_NOT"); break;
case LLAMA_GRETYPE_CHAR_RNG_UPPER: fprintf(file, "CHAR_RNG_UPPER"); break;
case LLAMA_GRETYPE_CHAR_ALT: fprintf(file, "CHAR_ALT"); break;
case LLAMA_GRETYPE_CHAR_ANY: fprintf(file, "CHAR_ANY"); break;
case LLAMA_GRETYPE_TOKEN: fprintf(file, "TOKEN"); break;
case LLAMA_GRETYPE_TOKEN_NOT: fprintf(file, "TOKEN_NOT"); break;
case LLAMA_GRETYPE_TOKEN_RNG_UPPER: fprintf(file, "TOKEN_RNG_UPPER"); break;
case LLAMA_GRETYPE_TOKEN_ALT: fprintf(file, "TOKEN_ALT"); break;
}
switch (elem.type) {
case LLAMA_GRETYPE_END:
@@ -295,6 +331,10 @@ static void print_rule_binary(FILE * file, const llama_grammar_rule & rule) {
fprintf(file, "%u", elem.value);
fprintf(file, "]> ");
break;
case LLAMA_GRETYPE_TOKEN_RNG_UPPER:
case LLAMA_GRETYPE_TOKEN_ALT:
fprintf(file, "(%u) ", elem.value);
break;
}
}
fprintf(file, "\n");
@@ -354,14 +394,32 @@ static void print_rule(
case LLAMA_GRETYPE_TOKEN:
fprintf(file, "<[");
fprintf(file, "%u", elem.value);
fprintf(file, "]> ");
break;
case LLAMA_GRETYPE_TOKEN_NOT:
fprintf(file, "!");
fprintf(file, "<[");
fprintf(file, "%u", elem.value);
fprintf(file, "]> ");
break;
case LLAMA_GRETYPE_TOKEN_RNG_UPPER:
if (i == 0 || !is_token_element(rule[i - 1])) {
throw std::runtime_error(
"LLAMA_GRETYPE_TOKEN_RNG_UPPER without preceding token: " +
std::to_string(rule_id) + "," + std::to_string(i));
}
fprintf(file, "-%u", elem.value);
break;
case LLAMA_GRETYPE_TOKEN_ALT:
if (i == 0 || !is_token_element(rule[i - 1])) {
throw std::runtime_error(
"LLAMA_GRETYPE_TOKEN_ALT without preceding token: " +
std::to_string(rule_id) + "," + std::to_string(i));
}
fprintf(file, ",%u", elem.value);
break;
}
if (is_token_element(elem) &&
rule[i + 1].type != LLAMA_GRETYPE_TOKEN_ALT && rule[i + 1].type != LLAMA_GRETYPE_TOKEN_RNG_UPPER) {
fprintf(file, "]> ");
}
if (is_char_element(elem)) {
switch (rule[i + 1].type) {
@@ -587,11 +645,9 @@ const char * llama_grammar_parser::parse_sequence(
type = LLAMA_GRETYPE_TOKEN_NOT;
pos++;
}
auto token_pair = parse_token(vocab, pos);
const char * token_end = token_pair.second;
last_sym_start = rule.size();
n_prev_rules = 1;
rule.push_back({type, token_pair.first});
const char * token_end = parse_token(vocab, pos, type, rule);
pos = parse_space(token_end, is_nested);
} else if (is_word_char(*pos)) { // rule reference
const char * name_end = parse_name(pos);
@@ -854,17 +910,58 @@ static bool llama_grammar_match_partial_char(
// returns true iff token matches the rule at pos (regular or inverse)
// asserts that pos is pointing to a token element
static bool llama_grammar_match_token(
static std::pair<bool, const llama_grammar_element *> llama_grammar_match_token(
const llama_grammar_element * pos,
const llama_token token) {
GGML_ASSERT(pos->type == LLAMA_GRETYPE_TOKEN || pos->type == LLAMA_GRETYPE_TOKEN_NOT);
if (pos->type == LLAMA_GRETYPE_TOKEN) {
return pos->value == static_cast<uint32_t>(token);
bool is_positive_token = pos->type == LLAMA_GRETYPE_TOKEN;
GGML_ASSERT(is_positive_token || pos->type == LLAMA_GRETYPE_TOKEN_NOT);
const uint32_t id = static_cast<uint32_t>(token);
bool found = false;
do {
if (pos[1].type == LLAMA_GRETYPE_TOKEN_RNG_UPPER) {
// inclusive range, e.g. <[1-5]>
found = found || (pos->value <= id && id <= pos[1].value);
pos += 2;
} else {
found = found || pos->value == id;
pos += 1;
}
} while (pos->type == LLAMA_GRETYPE_TOKEN_ALT);
return std::make_pair(found == is_positive_token, pos);
}
// every token id a token rule names, merged into sorted inclusive ranges
static std::vector<std::pair<uint32_t, uint32_t>> llama_grammar_opaque_tokens(const llama_grammar_rules & rules) {
std::vector<std::pair<uint32_t, uint32_t>> ranges;
for (const auto & rule : rules) {
for (size_t i = 0; i < rule.size(); i++) {
const auto type = rule[i].type;
if (type == LLAMA_GRETYPE_TOKEN || type == LLAMA_GRETYPE_TOKEN_NOT || type == LLAMA_GRETYPE_TOKEN_ALT) {
const bool is_range = i + 1 < rule.size() && rule[i + 1].type == LLAMA_GRETYPE_TOKEN_RNG_UPPER;
ranges.emplace_back(rule[i].value, is_range ? rule[i + 1].value : rule[i].value);
}
}
}
if (pos->type == LLAMA_GRETYPE_TOKEN_NOT) {
return pos->value != static_cast<uint32_t>(token);
std::sort(ranges.begin(), ranges.end());
std::vector<std::pair<uint32_t, uint32_t>> merged;
for (const auto & r : ranges) {
if (!merged.empty() && r.first <= (uint64_t) merged.back().second + 1) {
merged.back().second = std::max(merged.back().second, r.second);
} else {
merged.push_back(r);
}
}
return false;
return merged;
}
bool llama_grammar_is_opaque(const llama_grammar & grammar, llama_token token) {
const auto id = static_cast<uint32_t>(token);
auto it = std::upper_bound(grammar.opaque_tokens.begin(), grammar.opaque_tokens.end(), std::make_pair(id, UINT32_MAX));
return it != grammar.opaque_tokens.begin() && std::prev(it)->second >= id;
}
// transforms a grammar pushdown stack into N possible stacks, all ending
@@ -946,8 +1043,7 @@ static void llama_grammar_advance_stack(
break;
default:
// end of alternate (LLAMA_GRETYPE_END, LLAMA_GRETYPE_ALT) or middle of char range
// (LLAMA_GRETYPE_CHAR_ALT, LLAMA_GRETYPE_CHAR_RNG_UPPER); stack should never be left on
// those
// (LLAMA_GRETYPE_CHAR_ALT, LLAMA_GRETYPE_CHAR_RNG_UPPER) or token set (LLAMA_GRETYPE_TOKEN_ALT, LLAMA_GRETYPE_TOKEN_RNG_UPPER); stack should never be left on those
GGML_ABORT("fatal error");
}
}
@@ -1238,13 +1334,13 @@ static bool llama_grammar_reject_step(
if (stack_pos->type == LLAMA_GRETYPE_TOKEN || stack_pos->type == LLAMA_GRETYPE_TOKEN_NOT) {
for (const uint32_t i : cur) {
const auto & tok = candidates[i];
if (tok.code_points[depth] == 0) {
// reached the end of a token consumed by char rules, reject iff it ended
// in a partial response
if (tok.partial_utf8.n_remain != 0) {
if (depth > 0) {
// char rules already consumed part of this token, so it cannot match by id.
// accept it only if they consumed all of it and did not end in a partial sequence.
if (tok.code_points[depth] != 0 || tok.partial_utf8.n_remain != 0) {
rejects.push_back(i);
}
} else if (!llama_grammar_match_token(stack_pos, tok.id)) {
} else if (tok.partial_utf8.n_remain < 0 || !llama_grammar_match_token(stack_pos, tok.id).first) {
rejects.push_back(i);
}
}
@@ -1255,7 +1351,10 @@ static bool llama_grammar_reject_step(
for (const uint32_t i : cur) {
const auto & tok = candidates[i];
if (tok.code_points[depth] == 0) {
if (tok.opaque) {
// a token named by a token rule only matches token rules
rejects.push_back(i);
} else if (tok.code_points[depth] == 0) {
// reached end of full codepoints in token, reject iff it ended in a partial sequence
// that cannot satisfy this position in grammar
if (tok.partial_utf8.n_remain != 0 &&
@@ -1425,6 +1524,8 @@ struct llama_grammar * llama_grammar_init_impl(
}
} while (true);
auto opaque_tokens = llama_grammar_opaque_tokens(vec_rules);
// Important: vec_rules has to be moved here, not copied, because stacks contains
// pointers to elements of vec_rules. If vec_rules were copied into llama_grammar
// then the pointers would be invalidated when the local vec_rules goes out of scope.
@@ -1439,6 +1540,7 @@ struct llama_grammar * llama_grammar_init_impl(
/* .trigger_buffer_positions = */ {},
/* .trigger_tokens = */ {},
/* .trigger_patterns = */ {},
/* .opaque_tokens = */ std::move(opaque_tokens),
};
}
@@ -1540,6 +1642,8 @@ struct llama_grammar * llama_grammar_init_impl(
trigger.regex = std::regex(trigger.pattern);
}
auto opaque_tokens = llama_grammar_opaque_tokens(vec_rules);
// Important: vec_rules has to be moved here, not copied, because stacks contains
// pointers to elements of vec_rules. If vec_rules were copied into llama_grammar
// then the pointers would be invalidated when the local vec_rules goes out of scope.
@@ -1554,6 +1658,7 @@ struct llama_grammar * llama_grammar_init_impl(
/* .trigger_buffer_positions = */ {},
std::move(vec_trigger_tokens),
std::move(vec_trigger_patterns),
std::move(opaque_tokens),
};
}
@@ -1601,8 +1706,9 @@ void llama_grammar_apply_impl(const struct llama_grammar & grammar, llama_token_
} else if (piece.empty() || piece[0] == 0) {
cur_p->data[i].logit = -INFINITY;
} else {
candidates_decoded.push_back(decode_utf8(piece, grammar.partial_utf8));
candidates_grammar.push_back({ i, candidates_decoded.back().first.data(), candidates_decoded.back().second, id });
candidates_decoded.push_back(llama_grammar_decode_utf8(piece, grammar.partial_utf8));
candidates_grammar.push_back({ i, candidates_decoded.back().first.data(), candidates_decoded.back().second, id,
llama_grammar_is_opaque(grammar, id) });
}
}
@@ -1673,7 +1779,7 @@ void llama_grammar_accept_impl(struct llama_grammar & grammar, llama_token token
void llama_grammar_accept_str(struct llama_grammar & grammar, const std::string & piece) {
// Note terminating 0 in decoded string
const auto decoded = decode_utf8(piece, grammar.partial_utf8);
const auto decoded = llama_grammar_decode_utf8(piece, grammar.partial_utf8);
const auto & code_points = decoded.first;
for (auto it = code_points.begin(), end = code_points.end() - 1; it != end; ++it) {
@@ -1688,9 +1794,16 @@ void llama_grammar_accept_str(struct llama_grammar & grammar, const std::string
void llama_grammar_accept_token(struct llama_grammar & grammar, llama_token token, const std::string & piece) {
// Note terminating 0 in decoded string
const auto decoded = decode_utf8(piece, grammar.partial_utf8);
const auto decoded = llama_grammar_decode_utf8(piece, grammar.partial_utf8);
const auto & code_points = decoded.first;
// the mask rejects a piece that does not continue a partial character, so accepting one has to fail too
if (decoded.second.n_remain < 0) {
throw std::runtime_error("Invalid UTF-8 continuation in piece: " + piece + " (" + std::to_string(token) + ")");
}
const bool opaque = llama_grammar_is_opaque(grammar, token);
llama_grammar_stacks stacks_new;
stacks_new.reserve(grammar.stacks.size());
@@ -1702,14 +1815,15 @@ void llama_grammar_accept_token(struct llama_grammar & grammar, llama_token toke
const llama_grammar_element * pos = stack.back();
if (pos->type == LLAMA_GRETYPE_TOKEN || pos->type == LLAMA_GRETYPE_TOKEN_NOT) {
if (llama_grammar_match_token(pos, token)) {
auto match = llama_grammar_match_token(pos, token);
if (match.first) {
llama_grammar_stack new_stack(stack.begin(), stack.end() - 1);
if (!llama_grammar_is_end_of_sequence(pos + 1)) {
new_stack.push_back(pos + 1);
if (!llama_grammar_is_end_of_sequence(match.second)) {
new_stack.push_back(match.second);
}
llama_grammar_advance_stack(*grammar.rules, new_stack, stacks_new);
}
} else {
} else if (!opaque) {
llama_grammar_stacks current_stacks = {stack};
for (auto it = code_points.begin(), end = code_points.end() - 1; it != end; ++it) {
+18 -2
View File
@@ -38,11 +38,17 @@ enum llama_gretype {
// any character (.)
LLAMA_GRETYPE_CHAR_ANY = 7,
// terminal element: token (<[token-id]>)
// terminal element: token (<[token-id]>, <[token-id,token-id-token-id]>)
LLAMA_GRETYPE_TOKEN = 8,
// inverse token (!<[token-id]>)
// inverse token (!<[token-id]>, !<[token-id,token-id-token-id]>)
LLAMA_GRETYPE_TOKEN_NOT = 9,
// modifies a preceding LLAMA_GRETYPE_TOKEN, LLAMA_GRETYPE_TOKEN_NOT, or LLAMA_GRETYPE_TOKEN_ALT to be an inclusive range (<[1-5]>)
LLAMA_GRETYPE_TOKEN_RNG_UPPER = 10,
// modifies a preceding LLAMA_GRETYPE_TOKEN, LLAMA_GRETYPE_TOKEN_NOT, or LLAMA_GRETYPE_TOKEN_RNG_UPPER to add an alternate token to match (<[1,5]>, <[1-5,9]>)
LLAMA_GRETYPE_TOKEN_ALT = 11,
};
typedef struct llama_grammar_element {
@@ -60,6 +66,7 @@ struct llama_grammar_candidate {
const uint32_t * code_points;
llama_partial_utf8 partial_utf8;
llama_token id;
bool opaque;
};
using llama_grammar_rule = std::vector< llama_grammar_element>;
@@ -84,6 +91,10 @@ std::vector<llama_grammar_candidate> llama_grammar_reject_candidates_for_stack(
const llama_grammar_stack & stack,
const llama_grammar_candidates & candidates);
std::pair<std::vector<uint32_t>, llama_partial_utf8> llama_grammar_decode_utf8(
const std::string & src,
llama_partial_utf8 partial_start);
struct llama_grammar_parser {
const llama_vocab * vocab;
std::map<std::string, uint32_t> symbol_ids;
@@ -151,6 +162,8 @@ struct llama_grammar {
trigger_patterns; // Regular expressions that trigger a lazy grammar. Must be a full match of the entire generated
// string, and the grammar will be given the string from the first match group onwards.
// token ids named by token rules, as sorted inclusive ranges. char rules never match these tokens.
std::vector<std::pair<uint32_t, uint32_t>> opaque_tokens;
};
//
@@ -195,3 +208,6 @@ void llama_grammar_accept_token(
struct llama_grammar & grammar,
llama_token token,
const std::string & piece);
// true if a token rule names the token, char rules do not match it
bool llama_grammar_is_opaque(const llama_grammar & grammar, llama_token token);
+54 -32
View File
@@ -1415,7 +1415,7 @@ void llm_graph_input_mem_hybrid_iswa_rel::set_input(const llama_ubatch * ubatch)
}
if (self_rel_pos->buffer) {
attn_ctx->get_base()->set_input_relative_pos(self_rel_pos, ubatch, extent);
attn_ctx->get_base()->set_input_relative_pos(self_rel_pos, inp_attn->self_attn_mask.rows, ubatch, extent);
}
// swa tensors may not be allocated if there are no SWA attention layers
@@ -1429,7 +1429,7 @@ void llm_graph_input_mem_hybrid_iswa_rel::set_input(const llama_ubatch * ubatch)
}
if (self_rel_pos_swa->buffer) {
attn_ctx->get_swa()->set_input_relative_pos(self_rel_pos_swa, ubatch, extent_swa);
attn_ctx->get_swa()->set_input_relative_pos(self_rel_pos_swa, inp_attn->self_attn_mask_swa.rows, ubatch, extent_swa);
}
const int64_t n_rs = mctx->get_recr()->get_n_rs();
@@ -1454,16 +1454,13 @@ bool llm_graph_input_mem_hybrid_iswa_rel::can_reuse(const llm_graph_params & par
const auto * attn_ctx = mctx->get_attn();
// the masks are always dense, see build_inp_mem_hybrid_iswa_rel()
// base tensors may not be allocated if there are no non-SWA attention layers
if (inp_attn->self_k_idxs && inp_attn->self_k_idxs->buffer) {
res &= inp_attn->self_k_idxs->ne[0] == params.ubatch.n_tokens;
}
res &= can_reuse_kq_mask(inp_attn->self_attn_mask.raw, attn_ctx->get_base(), params.ubatch, params.cparams);
res &= can_reuse_attn_mask(inp_attn->self_attn_mask, attn_ctx->get_base(), params.ubatch, params.cparams);
res &= self_rel_pos->ne[0] == attn_ctx->get_base()->get_n_kv();
res &= self_rel_pos->ne[1] == params.ubatch.n_tokens;
// swa tensors may not be allocated if there are no SWA attention layers
@@ -1471,9 +1468,8 @@ bool llm_graph_input_mem_hybrid_iswa_rel::can_reuse(const llm_graph_params & par
res &= inp_attn->self_k_idxs_swa->ne[0] == params.ubatch.n_tokens;
}
res &= can_reuse_kq_mask(inp_attn->self_attn_mask_swa.raw, attn_ctx->get_swa(), params.ubatch, params.cparams);
res &= can_reuse_attn_mask(inp_attn->self_attn_mask_swa, attn_ctx->get_swa(), params.ubatch, params.cparams);
res &= self_rel_pos_swa->ne[0] == attn_ctx->get_swa()->get_n_kv();
res &= self_rel_pos_swa->ne[1] == params.ubatch.n_tokens;
res &= inp_rs->s_copy->ne[0] == mctx->get_recr()->get_n_rs();
@@ -3114,7 +3110,9 @@ ggml_tensor * llm_graph_context::build_attn_mha(
ggml_tensor * v_mla,
int64_t n_kv_max,
float kq_scale,
int il) const {
int il,
ggml_tensor * bias_table,
ggml_tensor * bias_indices) const {
const bool v_trans = v->nb[1] > v->nb[2];
// split the batch into streams if needed
@@ -3129,6 +3127,21 @@ ggml_tensor * llm_graph_context::build_attn_mha(
k = ggml_permute(ctx0, k, 0, 2, 1, 3);
v = ggml_permute(ctx0, v, 0, 2, 1, 3);
auto dense_bias_mask = [&]() {
ggml_tensor * table = ggml_cont(ctx0, ggml_permute(ctx0, bias_table, 1, 0, 2, 3));
ggml_tensor * bias = ggml_get_rows(ctx0, table, bias_indices);
bias = ggml_reshape_4d(ctx0, bias, q->ne[2], bias_indices->ne[0], q->ne[1], n_stream);
bias = ggml_cont(ctx0, ggml_permute(ctx0, bias, 2, 0, 1, 3));
cb(bias, "kq_b", il);
ggml_tensor * mask = kq_mask->type == GGML_TYPE_F32 ? kq_mask : ggml_cast(ctx0, kq_mask, GGML_TYPE_F32);
mask = ggml_add(ctx0, bias, mask);
if (cparams.flash_attn) {
mask = ggml_cast(ctx0, mask, GGML_TYPE_F16);
}
cb(mask, "kq_mask_rel_pos", il);
return mask;
};
ggml_tensor * cur;
const bool use_flash_attn = cparams.flash_attn && kq_b == nullptr;
@@ -3162,6 +3175,23 @@ ggml_tensor * llm_graph_context::build_attn_mha(
GGML_ASSERT(n_kv_max >= 0 && n_kv_max <= INT32_MAX);
ggml_flash_attn_ext_set_n_kv_max(cur, static_cast<int32_t>(n_kv_max));
ggml_prec_set_acc(cur, GGML_PREC_F32);
if (bias_table) {
ggml_flash_attn_ext_add_indexed_bias(cur, bias_table, bias_indices);
const auto dev = model.dev_layer(il);
if (!cparams.offload_kqv || !dev || !ggml_backend_dev_supports_op(dev, cur)) {
ggml_flash_attn_ext_add_indexed_bias(cur, nullptr, nullptr);
cur->src[3] = dense_bias_mask();
} else if (kv_rows) {
// Looped layers can project Q and the bias on a different device from their KV cache.
for (int i = 0; i < ggml_backend_sched_get_n_backends(sched); ++i) {
auto backend = ggml_backend_sched_get_backend(sched, i);
if (ggml_backend_get_device(backend) == dev) {
ggml_backend_sched_set_tensor_backend(sched, cur, backend);
break;
}
}
}
}
if (v_mla) {
#if 0
@@ -3182,6 +3212,9 @@ ggml_tensor * llm_graph_context::build_attn_mha(
cur = ggml_reshape_2d(ctx0, cur, cur->ne[0]*cur->ne[1], cur->ne[2]*cur->ne[3]);
} else {
if (bias_table) {
kq_mask = dense_bias_mask();
}
ggml_tensor * kq = ggml_mul_mat(ctx0, k, q);
cb(kq, "kq", il);
@@ -3691,7 +3724,9 @@ static ggml_tensor * build_attn_iswa_impl(
ggml_tensor * sinks,
ggml_tensor * v_mla,
float kq_scale,
int il) {
int il,
ggml_tensor * bias_table = nullptr,
ggml_tensor * bias_indices = nullptr) {
ggml_context * ctx0 = gctx.ctx0;
ggml_cgraph * gf = gctx.gf;
@@ -3757,7 +3792,7 @@ static ggml_tensor * build_attn_iswa_impl(
ggml_tensor * k = use_kv_cur ? k_cur : mctx_cur->get_k(ctx0, il);
ggml_tensor * v = use_kv_cur ? v_cur : mctx_cur->get_v(ctx0, il);
ggml_tensor * cur = gctx.build_attn_mha(q, k, v, kq_b, kq_mask, kv_rows, sinks, v_mla, 0, kq_scale, il);
ggml_tensor * cur = gctx.build_attn_mha(q, k, v, kq_b, kq_mask, kv_rows, sinks, v_mla, 0, kq_scale, il, bias_table, bias_indices);
cur = build_attn_scale_kv_output(ctx0, mctx_cur, cur, il);
gctx.cb(cur, "kqv_out", il);
@@ -4351,19 +4386,18 @@ llm_graph_input_mem_hybrid_iswa_rel * llm_graph_context::build_inp_mem_hybrid_is
auto inp_attn = std::make_unique<llm_graph_input_attn_kv_iswa>(hparams, cparams, attn_ctx);
// dense masks: the bias is gathered by the distance to each cell of the K/V view
{
inp_attn->self_k_idxs = attn_ctx->get_base()->build_input_k_idxs(ctx0, ubatch);
inp_attn->self_v_idxs = attn_ctx->get_base()->build_input_v_idxs(ctx0, ubatch);
inp_attn->self_attn_mask.raw = build_attn_inp_kq_mask(ctx0, attn_ctx->get_base(), ubatch, cparams);
inp_attn->self_attn_mask = build_attn_inp_attn_mask(ctx0, attn_ctx->get_base(), ubatch, cparams);
}
{
inp_attn->self_k_idxs_swa = attn_ctx->get_swa()->build_input_k_idxs(ctx0, ubatch);
inp_attn->self_v_idxs_swa = attn_ctx->get_swa()->build_input_v_idxs(ctx0, ubatch);
inp_attn->self_attn_mask_swa.raw = build_attn_inp_kq_mask(ctx0, attn_ctx->get_swa(), ubatch, cparams);
inp_attn->self_attn_mask_swa = build_attn_inp_attn_mask(ctx0, attn_ctx->get_swa(), ubatch, cparams);
}
inp_attn->self_k_rot = attn_ctx->get_base()->get_k_rot();
@@ -4374,11 +4408,11 @@ llm_graph_input_mem_hybrid_iswa_rel * llm_graph_context::build_inp_mem_hybrid_is
auto inp = std::make_unique<llm_graph_input_mem_hybrid_iswa_rel>(cparams, std::move(inp_attn), std::move(inp_rs), mctx_cur, extent, extent_swa);
inp->self_rel_pos = ggml_new_tensor_2d(ctx0, GGML_TYPE_I32, attn_ctx->get_base()->get_n_kv(), ubatch.n_tokens);
inp->self_rel_pos = ggml_new_tensor_2d(ctx0, GGML_TYPE_I32, inp->get_attn()->self_attn_mask.raw->ne[0], ubatch.n_tokens);
ggml_set_input(inp->self_rel_pos);
ggml_set_name(inp->self_rel_pos, "attn_inp_rel_pos");
inp->self_rel_pos_swa = ggml_new_tensor_2d(ctx0, GGML_TYPE_I32, attn_ctx->get_swa()->get_n_kv(), ubatch.n_tokens);
inp->self_rel_pos_swa = ggml_new_tensor_2d(ctx0, GGML_TYPE_I32, inp->get_attn()->self_attn_mask_swa.raw->ne[0], ubatch.n_tokens);
ggml_set_input(inp->self_rel_pos_swa);
ggml_set_name(inp->self_rel_pos_swa, "attn_inp_rel_pos_swa");
@@ -4393,7 +4427,7 @@ ggml_tensor * llm_graph_context::build_attn(
ggml_tensor * q_cur,
ggml_tensor * k_cur,
ggml_tensor * v_cur,
ggml_tensor * kq_b,
ggml_tensor * bias_table,
ggml_tensor * sinks,
float kq_scale,
int il) const {
@@ -4403,22 +4437,10 @@ ggml_tensor * llm_graph_context::build_attn(
const auto & attn_mask = is_swa ? inp_attn->self_attn_mask_swa : inp_attn->self_attn_mask;
// TODO: support kv_rows - fill the distances over the kv_rows cell lists and pass the rows to build_attn_mha,
// needs a kv_rows flash attention with a per-head mask (CUDA has none)
GGML_ASSERT(attn_mask.rows == nullptr);
ggml_tensor * indices = is_swa ? inp->self_rel_pos_swa : inp->self_rel_pos;
// a mask per head, in the dense layout of the input mask, which flash attention takes in F16
ggml_tensor * kq_mask = attn_mask.raw;
if (kq_mask->type != GGML_TYPE_F32) {
kq_mask = ggml_cast(ctx0, kq_mask, GGML_TYPE_F32);
}
kq_mask = ggml_add(ctx0, kq_b, kq_mask);
if (cparams.flash_attn) {
kq_mask = ggml_cast(ctx0, kq_mask, GGML_TYPE_F16);
}
cb(kq_mask, "kq_mask_rel_pos", il);
return build_attn_iswa_impl(*this, inp_attn, wo, wo_b, wo_s, q_cur, k_cur, v_cur, nullptr, kq_mask, nullptr, sinks, nullptr, kq_scale, il);
return build_attn_iswa_impl(*this, inp_attn, wo, wo_b, wo_s, q_cur, k_cur, v_cur, nullptr, attn_mask.raw, attn_mask.rows,
sinks, nullptr, kq_scale, il, bias_table, indices);
}
void llm_graph_context::build_dense_out(
+6 -5
View File
@@ -774,8 +774,7 @@ public:
};
// hybrid iswa memory with a per-head attention bias that depends on the distance from the query to the cell (Inkling)
// the KQ masks are always dense (rows == nullptr) and are filled from the same cells as the distances, so that the bias
// gathered by distance lines up with the mask
// The visibility mask and distances use the same cache cell order.
class llm_graph_input_mem_hybrid_iswa_rel : public llm_graph_input_i {
public:
llm_graph_input_mem_hybrid_iswa_rel(
@@ -1336,7 +1335,9 @@ struct llm_graph_context {
ggml_tensor * v_mla, // [n_embd_head_v_mla, n_embd_head_v, n_head_v]
int64_t n_kv_max,
float kq_scale,
int il) const;
int il,
ggml_tensor * bias_table = nullptr,
ggml_tensor * bias_indices = nullptr) const;
llm_graph_input_attn_no_cache * build_attn_inp_no_cache() const;
@@ -1534,7 +1535,7 @@ struct llm_graph_context {
uint32_t extent,
uint32_t extent_swa) const;
// kq_b is added to the KQ mask, after the KQ scale
// Gather the per-head bias by distance and add it to the visibility mask.
ggml_tensor * build_attn(
llm_graph_input_mem_hybrid_iswa_rel * inp,
ggml_tensor * wo,
@@ -1543,7 +1544,7 @@ struct llm_graph_context {
ggml_tensor * q_cur, // [n_embd_head_q, n_head_q, n_tokens]
ggml_tensor * k_cur, // [n_embd_head_k, n_head_k, n_tokens]
ggml_tensor * v_cur, // [n_embd_head_v, n_head_v, n_tokens]
ggml_tensor * kq_b, // [n_kv, n_tokens/n_stream, n_head_q, n_stream]
ggml_tensor * bias_table, // [n_bins, n_head_q, n_tokens]
ggml_tensor * sinks, // [n_head_q]
float kq_scale,
int il) const;
+11
View File
@@ -264,6 +264,17 @@ void llama_kv_cache_iswa::state_read(llama_io_read_i & io, llama_seq_id seq_id,
kv_swa ->state_read(io, seq_id, flags);
}
void llama_kv_cache_iswa::release_device_buffers(bool keep_shadow) {
kv_base->release_device_buffers(keep_shadow);
kv_swa->release_device_buffers(keep_shadow);
}
bool llama_kv_cache_iswa::restore_device_buffers(bool to_host) {
bool ok = kv_base->restore_device_buffers(to_host);
ok = kv_swa->restore_device_buffers(to_host) && ok;
return ok;
}
llama_kv_cache * llama_kv_cache_iswa::get_base() const {
return kv_base.get();
}
+3
View File
@@ -83,6 +83,9 @@ public:
void state_write(llama_io_write_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) const override;
void state_read (llama_io_read_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) override;
void release_device_buffers(bool keep_shadow = true) override;
bool restore_device_buffers(bool to_host = false) override;
//
// llama_kv_cache_iswa specific API
//
+96 -6
View File
@@ -429,7 +429,9 @@ void llama_kv_cache::clear(bool data) {
if (data) {
for (auto & [_, buf] : ctxs_bufs) {
ggml_backend_buffer_clear(buf.get(), 0);
if (buf) { // may be null if evicted for on-demand VRAM sharing
ggml_backend_buffer_clear(buf.get(), 0);
}
}
}
}
@@ -2007,14 +2009,18 @@ bool llama_kv_cache::has_cell_ext() const {
return hparams.n_pos_per_embd() > 1 || hparams.ple_n_heads > 0;
}
void llama_kv_cache::set_input_relative_pos(ggml_tensor * dst, const llama_ubatch * ubatch, uint32_t extent) const {
void llama_kv_cache::set_input_relative_pos(ggml_tensor * dst, const ggml_tensor * rows, const llama_ubatch * ubatch, uint32_t extent) const {
GGML_ASSERT(dst->type == GGML_TYPE_I32);
const int64_t n_kv = dst->ne[0];
GGML_ASSERT(!rows || (rows->ne[0] == n_kv && ggml_backend_buffer_is_host(rows->buffer)));
const int64_t n_tps = ubatch->n_tokens / (rows ? rows->ne[1] : 1);
std::vector<int32_t> data(n_kv * ubatch->n_tokens);
for (uint32_t i = 0; i < ubatch->n_tokens; ++i) {
const auto & cells = v_cells.at(seq_to_stream.at(ubatch->seq_id[i][0]));
const int32_t * list = rows ? (const int32_t *) rows->data + (i/n_tps)*n_kv : nullptr;
for (int64_t j = 0; j < n_kv; ++j) {
const int64_t distance = cells.is_empty(j) ? -1 : int64_t(ubatch->pos[i]) - cells.pos_get(j);
const int64_t cell = list ? list[j] : j;
const int64_t distance = cell < 0 || cells.is_empty(cell) ? -1 : int64_t(ubatch->pos[i]) - cells.pos_get(cell);
data[i*n_kv + j] = distance >= 0 && distance < extent ? distance : extent;
}
}
@@ -2084,12 +2090,96 @@ size_t llama_kv_cache::total_size() const {
size_t size = 0;
for (const auto & [_, buf] : ctxs_bufs) {
size += ggml_backend_buffer_get_size(buf.get());
if (buf) { // may be null if evicted for on-demand VRAM sharing
size += ggml_backend_buffer_get_size(buf.get());
}
}
return size;
}
void llama_kv_cache::release_device_buffers(bool keep_shadow) {
// NOTE: the caller must have synchronized the backend so nothing references these buffers.
// The KV cache is read-write, so its host shadow is (re)captured fresh on every release.
if (dev_released) {
return;
}
dev_shadows.assign(ctxs_bufs.bufs.size(), device_buffer_shadow{});
size_t freed = 0;
for (size_t i = 0; i < ctxs_bufs.bufs.size(); ++i) {
ggml_context * ctx = ctxs_bufs.bufs[i].first.get();
ggml_backend_buffer_t buf = ctxs_bufs.bufs[i].second.get();
if (buf == nullptr || ggml_backend_buffer_is_host(buf) || ggml_backend_buffer_get_size(buf) == 0) {
continue; // only real device (VRAM) buffers are evictable
}
auto & sh = dev_shadows[i];
sh.releasable = true;
sh.buft = ggml_backend_buffer_get_type(buf);
// capture live contents compactly in stable iteration order (skip views, which alias a base)
if (keep_shadow) {
size_t total = 0;
for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
if (t->view_src == nullptr) { total += ggml_nbytes(t); }
}
sh.data.resize(total);
size_t off = 0;
for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
if (t->view_src != nullptr) { continue; }
const size_t n = ggml_nbytes(t);
ggml_backend_tensor_get(t, sh.data.data() + off, 0, n);
off += n;
}
}
freed += ggml_backend_buffer_get_size(buf);
ctxs_bufs.bufs[i].second.reset(); // free the device buffer
for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
t->buffer = nullptr;
t->data = nullptr;
}
}
dev_released = true;
if (freed > 0) {
LLAMA_LOG_INFO("%s: released %.2f MiB of KV cache from device\n", __func__, freed / 1024.0 / 1024.0);
}
}
bool llama_kv_cache::restore_device_buffers(bool to_host) {
if (!dev_released) {
return true;
}
for (size_t i = 0; i < ctxs_bufs.bufs.size(); ++i) {
auto & sh = dev_shadows[i];
if (!sh.releasable) {
continue;
}
ggml_context * ctx = ctxs_bufs.bufs[i].first.get();
if (to_host) {
ctxs_bufs.bufs[i].second.reset(llama_memory_wrap_host(ctx, sh.data));
continue;
}
ggml_backend_buffer_t buf = ggml_backend_alloc_ctx_tensors_from_buft(ctx, sh.buft);
if (buf == nullptr) {
LLAMA_LOG_ERROR("%s: failed to reallocate KV cache device buffer (out of VRAM?)\n", __func__);
return false;
}
size_t off = 0;
for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != nullptr; t = ggml_get_next_tensor(ctx, t)) {
if (t->view_src != nullptr) { continue; }
const size_t n = ggml_nbytes(t);
ggml_backend_tensor_set(t, sh.data.data() + off, 0, n);
off += n;
}
ctxs_bufs.bufs[i].second.reset(buf);
}
dev_released = false;
if (!to_host) {
dev_shadows.clear(); // recaptured on next release; to_host buffers point into the shadows
}
return true;
}
size_t llama_kv_cache::size_k_bytes() const {
size_t size_k_bytes = 0;
@@ -3124,8 +3214,8 @@ void llama_kv_cache_context::set_input_pos_bucket(ggml_tensor * dst, const llama
kv->set_input_pos_bucket(dst, ubatch);
}
void llama_kv_cache_context::set_input_relative_pos(ggml_tensor * dst, const llama_ubatch * ubatch, uint32_t extent) const {
kv->set_input_relative_pos(dst, ubatch, extent);
void llama_kv_cache_context::set_input_relative_pos(ggml_tensor * dst, const ggml_tensor * rows, const llama_ubatch * ubatch, uint32_t extent) const {
kv->set_input_relative_pos(dst, rows, ubatch, extent);
}
void llama_kv_cache_context::get_prev_tokens(const llama_ubatch & ubatch, uint32_t n, std::vector<llama_token> & res) const {
+16 -2
View File
@@ -153,6 +153,10 @@ public:
void state_write(llama_io_write_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) const override;
void state_read (llama_io_read_i & io, llama_seq_id seq_id = -1, llama_state_seq_flags flags = 0) override;
// on-demand device (VRAM) residency (see llama_memory_i)
void release_device_buffers(bool keep_shadow = true) override;
bool restore_device_buffers(bool to_host = false) override;
//
// llama_kv_cache specific API
//
@@ -249,7 +253,7 @@ public:
void set_input_attn_mask (const llm_attn_mask & attn_mask, const llama_ubatch * ubatch, bool causal_attn) const;
void set_input_pos_bucket(ggml_tensor * dst, const llama_ubatch * ubatch) const;
void set_input_relative_pos(ggml_tensor * dst, const llama_ubatch * ubatch, uint32_t extent) const;
void set_input_relative_pos(ggml_tensor * dst, const ggml_tensor * rows, const llama_ubatch * ubatch, uint32_t extent) const;
// true if llama_kv_cell_ext holds information that has to survive a state save/restore
bool has_cell_ext() const;
@@ -320,6 +324,16 @@ private:
// ggml contexts for the KV cache along with the allocated backend buffers:
llama_memory_bufs ctxs_bufs;
// on-demand device eviction: host shadow of each device buffer's live contents (re-captured on
// every release since the KV is read-write), plus the buffer type needed to reallocate it.
struct device_buffer_shadow {
ggml_backend_buffer_type_t buft = nullptr;
bool releasable = false; // device (VRAM) buffer that was freed
std::vector<uint8_t> data; // captured tensor bytes (compact, iteration order)
};
std::vector<device_buffer_shadow> dev_shadows; // parallel to ctxs_bufs
bool dev_released = false;
// the current index from where we start searching for a free slot in the ring buffer of KV cells (see find_slot())
// note: this is not part of the KV state and it's only used to speed-up the find_slot() method
std::vector<uint32_t> v_heads;
@@ -454,7 +468,7 @@ public:
void set_input_k_shift (ggml_tensor * dst) const;
void set_input_attn_mask (const llm_attn_mask & attn_mask, const llama_ubatch * ubatch, bool causal_attn) const;
void set_input_pos_bucket(ggml_tensor * dst, const llama_ubatch * ubatch) const;
void set_input_relative_pos(ggml_tensor * dst, const llama_ubatch * ubatch, uint32_t extent) const;
void set_input_relative_pos(ggml_tensor * dst, const ggml_tensor * rows, const llama_ubatch * ubatch, uint32_t extent) const;
// see llama_kv_cache::get_prev_tokens()
void get_prev_tokens(const llama_ubatch & ubatch, uint32_t n, std::vector<llama_token> & res) const;
+12
View File
@@ -206,6 +206,18 @@ void llama_memory_hybrid::state_read(llama_io_read_i & io, llama_seq_id seq_id,
}
}
void llama_memory_hybrid::release_device_buffers(bool keep_shadow) {
// evict both the attention KV (grows with context) and the recurrent/SSM state
mem_attn->release_device_buffers(keep_shadow);
mem_recr->release_device_buffers(keep_shadow);
}
bool llama_memory_hybrid::restore_device_buffers(bool to_host) {
bool ok = mem_attn->restore_device_buffers(to_host);
ok = mem_recr->restore_device_buffers(to_host) && ok;
return ok;
}
llama_kv_cache * llama_memory_hybrid::get_mem_attn() const {
return mem_attn.get();
}
Loaded 100 of 148 files, more files were not shown because too many files have changed in this diff. Show more