Compare commits

..
Author SHA1 Message Date
LumpiastyandClaude Opus 5.5 2caae7e113 docs: FORK.md/runbook after the f3397cded deploy - F14 measured (35B TTFT A/B, batch layout, edit cost, disk cache), pointer 16
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 10:48:22 +01:00
LumpiastyandClaude Opus 5.5 f3397cded0 docs: FORK.md F14, user-message batch-break merge (not deployed)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 10:15:30 +01:00
LumpiastyandClaude Opus 5.5 510aa6e090 server: merge user-message batch breaks when the rest of the prompt fits in one ubatch
Upstream (#22929, #24176, #25420) ends a prompt batch at the start of the last
user message (and of earlier ones on a fresh prompt) so that a context
checkpoint lands there. On a hybrid/recurrent model with MoE experts on the
host, that turns a typical chat turn (previous answer + new user message, under
one ubatch) into two partial ubatches, and each one of >= 32 tokens uploads the
host experts again.

When the rest of the prompt from the current batch start fits in one ubatch,
skip the user-start break. The near-end rule still checkpoints the batch start
and the end-4 rule the prompt end, so a later prompt that diverges inside the
last user message re-processes less than one ubatch more, in the same ubatch.
Prompts longer than one ubatch, and batches after a media chunk, keep the
upstream behaviour. LLAMA_USERSTART_MERGE=0 restores it everywhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 10:15:30 +01:00
LumpiastyandClaude Opus 5.5 166ec22e2b docs: FORK.md section 5, Inkling vs Qwen3.6-35B quality comparison
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 10:13:29 +01:00
LumpiastyandClaude Opus 5.5 e8386267ec docs: FORK.md/runbook after the b13c41355 deploy - F13 measured (KLD bug-window test, fit and placement guide, quants, profile), Inkling -ot k=4 without the offload knob, nothink fix, pointer 15
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 09:14:29 +01:00
LumpiastyandClaude Opus 5.5 b13c413555 docs: FORK.md F13, per-pass views of host weights in inkling2_loop
What and why (#32755 and the 6.5 GiB of shared expert copies), the knob,
F1/F4/MoE-cache interactions, the extra upload cost, how to drop it, and the
pod validation plan (reference with the loop experts on the GPU, views on,
views off).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 05:40:39 +01:00
LumpiastyandClaude Opus 5.5 312c29c2b1 llama: inkling2_loop - per-pass views of host weights for the later loop passes
inkling2_loop runs the core layers (6..17 on Inkling-2-Mini) three times, and
post_arch_tensors makes every cache slot share the tensors of its physical
layer. With the experts in host memory (--n-cpu-moe, -ot exps=CPU) the
scheduler makes one device copy per (host tensor, backend) per graph, at the
first split that reads it, and keeps it until the last reader. So:

- the expert copies of all 12 loop layers are alive at once (about 6.5 GiB on
  UD-Q4_K_XL), which is most of the ~7 GiB Vulkan0 compute buffer;
- sched_copy_experts runs only for the first reader. When it reads the ids back
  (ubatches of 32-63 tokens: MUL_MAT_ID is offloaded from 32 tokens, the F1
  skip starts at 4*n_expert ids), it uploads only the experts that pass 0
  selected, and passes 1 and 2 read stale VRAM for the others (upstream issue
  #32755, "GGGG" output).

Add a post_load_tensors hook to llama_model (called after the weights are
loaded, and on the no_alloc path after the dummy buffers are set). In
inkling2_loop it replaces, for each slot of a later pass, every tensor in a
host buffer with a view of it (name "<name>.visit<pass>", buffer and data of
the original). The scheduler hashes tensors by pointer, so each pass gets its
own copy, freed after that pass, and the copy callback sees the ids of that
pass. Weights in device buffers and CPU extra buffers (CPU_REPACK, not host)
stay shared. The pointer-keyed weight maps (w_scales, prec_policy, hdmd.rot)
get entries for the views. The views live in a model-owned context outside
ctxs_bufs, so the F4 release/restore, tensor counts and the memory breakdown
do not see them; they point only at host buffers, which F4 never releases.

LLAMA_LOOP_HOST_VIEWS=0 keeps the shared tensors (A/B knob).

Not built or run yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 05:40:39 +01:00
LumpiastyandClaude Opus 5.5 24b605331c docs: FORK.md/runbook after the 3621025e8 deploy - F12 measurements, Inkling ub1024 + ctx 65536 and the offload knob, nothink finding, pointer 14
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 05:38:11 +01:00
LumpiastyandClaude Opus 5.5 3621025e8f docs: FORK.md F12 temporary upstream carries (#32735, #32740), Inkling config; runbook carry procedure
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 02:33:14 +01:00
Piotr WilkinandClaude Opus 5.5 72da62cea0 upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan
Temporary carry of the open upstream PR
https://github.com/ggml-org/llama.cpp/pull/32740 at head 808fd1c4e,
its four commits (1fa93f902, d29f9d489, ff91e4f84, 808fd1c4e) squashed.
Inkling's per-query bias table is applied inside flash attention
instead of a dense [n_kv, n_tokens, n_head] bias tensor, which cuts
Inkling's compute buffers and long-context prefill time on the RX 580.
Drop this commit when the PR is merged upstream (FORK.md, "Temporary
upstream carries").

One conflict, ggml-vulkan.cpp use_mask_opt: kept the fork's GCN
`HSK >= 256 || HSV >= 256` clause and its comment (F2) and upstream's
`!use_sparse`, added the PR's `!bias &&`.

Original commit trailer: Assisted-by: Codex

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 02:30:27 +01:00
RevertronandClaude Opus 5.5 d2396e4e36 upstream-carry #32735: vulkan : support non-contiguous src in SCALE
Temporary carry of the open upstream PR
https://github.com/ggml-org/llama.cpp/pull/32735 (head 0ea4ffd00),
cherry-picked unchanged. The Vulkan backend runs SCALE on a
non-contiguous source instead of rejecting it (CPU fallback). Carried
together with #32740 and evaluated as a pair.
Drop this commit when the PR is merged upstream (FORK.md, "Temporary
upstream carries").

Original commit trailer: Assisted-By: Opus 5.5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-22 02:30:27 +01:00
LumpiastyandClaude Sonnet a27217e278 docs: runbook/FORK.md fixes from the 2026-11-21 rebase, validate and deploy cycle
HF_UI_VERSION must use the tag after the '|' in .ui-stamp; bundle path
for agents told not to push; build backup and copy-based rollback;
status, pointers and last-run record for 3bf6b55b3; q35.py build path
note; disk-cache restore test wording; open questions.

Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-11-21 21:02:14 +01:00
LumpiastyandClaude Sonnet 5.5 3bf6b55b39 docs: runbook and FORK.md status for the 2026-11-21 rebase onto e6d624b9a
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-21 19:40:51 +01:00
LumpiastyandClaude Sonnet 5.5 3ab97b345f ggml-vulkan: gate upstream's tiled concat below one tile, honour the fast-path knob
Upstream #30591 added its own tiled-transpose concat pipelines, selected
for any dim-0 concat whose src1 is transposed. Behind the fork's fast path
(which already takes ne0 >= 32 and honours GGML_VK_CONCAT_FASTPATH) that
selector still caught the decode case (ne0 < 32), bringing back the
decode regression the TILE_DIM gate fixed (FORK.md F3), and
GGML_VK_CONCAT_FASTPATH=0 no longer gave the generic kernel.

Apply the same gate and the knob to upstream's selector.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-21 19:39:57 +01:00
LumpiastyandClaude Sonnet 1df5ced65e docs: runbook/FORK.md after the adc2aa3ec deploy
Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
2026-11-21 19:39:57 +01:00
LumpiastyandClaude Opus 5.5 cf217b0318 docs: F11 mmproj incident and fix; runbook - production command line, build targets, detached scripts
FORK.md F11 records the rolled-back deploy (the GGML_ASSERT(!has_mtmd) abort
with the production mmproj), the fix, the new mmproj cases (15 PASS), and
that the earlier 35B numbers were measured with --no-mmproj. The 35B run with
production's exact command line (diskcache-window3.sh) is ready, not run, and
a precondition for deploying --cache-disk-path.

Runbook:
- tests must use production's exact model command line, mmproj included;
- the validated build has all three targets;
- the deploy mtime list includes libllama-server-impl.so and
  libllama-common.so (llama-server is a stub);
- deploy and rollback steps run as one setsid-wrapped script on the pod,
  never as a backgrounded cat && sh -n && setsid chain from kubectl exec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:57 +01:00
LumpiastyandClaude Opus 5.5 383dbf4e6c scripts: fork-validate.sh --diskcache - the mmproj cases
Production runs the 35B and gemma with an mmproj, and the first deploy of the
disk prompt cache aborted on that path. Every case had used --no-mmproj. Now
--no-mmproj is in DC_MM, separate from DC_ARGS, and two cases run with the
0.8B's mmproj (--no-mmproj-offload, arbiter env, LLAMA_MMPROJ_ONDEMAND=1),
against their own reference:
- restore-mmproj: turn 1 is saved on SIGTERM and restored after the restart,
  prompt_n equal to the reference, output identical;
- mmproj-image: an image request (no restore, answer "red"), then the text
  chat (restored, identical), then an image again, then SIGTERM with media in
  the slot (no state copied, exit 0, the file count unchanged).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:57 +01:00
LumpiastyandClaude Opus 5.5 060e46516e server: disk prompt cache - text-only prompts with an mmproj loaded
The deploy of cb48bf224 with --cache-disk-path was rolled back: the first
request to the production 35B aborted in disk_cache_load().
server_tokens::get_tokens() has GGML_ASSERT(!has_mtmd), and has_mtmd is true
whenever an mmproj is loaded, even for a text-only prompt. Production's 35B
and gemma-26B always load one (-hf pulls it). disk_cache_collect() and
server_prompt_disk_cache::save() made the same call, so the shutdown save
would have aborted too. Every test had used --no-mmproj, which hid it.

disk_cache_text() takes the list from get_text_tokens(). A prompt is
text-only when that list is as long as size(), because media positions hold
LLAMA_TOKEN_NULL. So a text-only prompt is saved and restored with an mmproj
loaded, which is the main case (OpenWebUI chats on the 35B and gemma), and a
prompt with media is never saved or restored. When the slot itself holds
media, its reuse estimate is server_tokens::get_common_prefix(). These four
calls were the only get_tokens()/mtmd-unsafe accesses in the feature.

fork-validate --diskcache with the new mmproj cases, beside production:
15 PASS, 0 FAIL, GPU legs included. With the 0.8B's mmproj loaded, a text
chat is restored: prompt_n 58 of 596, output identical to the reference.
An image request is answered "red" with no restore, the text chat after it
is restored, and a SIGTERM with media in the slot copies no state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:57 +01:00
LumpiastyandClaude Opus 5.5 35d58d62cd docs: FORK.md F11 - 35B results, the two leg-c anomalies explained
The swapdown window: --diskcache --arbiter 29 PASS. The 35B, 18760 tokens:
83.4 s prefill, turn 2 after a restart 3.0 s with identical output. Warm
save: token free in 328 ms, exit 1952 ms, peak RSS +430 MiB. Cold save:
peak +432 MiB.

Leg c, output != reference: c sent turn 2 again, so the server rolled back
to the checkpoint at the prompt end and processed 4 tokens, while the
reference processed 27 from turn 1. These are different reuse points. A
controlled same-reuse-point check on the 0.8B (X never restarts vs Z after a
restore, a cold save and a restore) is bit-identical, top-3 logprobs
included, on CPU and GPU. The 35B version is ready as diskcache-window2.sh
and has not run.

Leg c, +3055 MiB and token free in 1881 ms: the driver's fake waiter (flock
-s file sleep) leaked the lock into sleep, so the SIGTERM landed in an
ordinary evict_kv release (the full KV shadow, F4). Not the cache; the
driver is fixed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:57 +01:00
LumpiastyandClaude Opus 5.5 da244dc458 docs: FORK.md F11 disk prompt cache, the CPU_REPACK release fix; runbook notes
F11 in the per-feature format: design, invariants, the measured costs (CPU
only) and llama-swap's stop behaviour (it waits for the exit, SIGKILL at
30 s). Also: the recommended config (not applied), the commit map entries, the
F4 incident note for the CPU_REPACK crash, F7's shutdown time with the cache,
the F9 log lines, and the section 7 gaps (GPU leg, 35B and --arbiter not run
yet). Runbook: --diskcache in phase 1, the pod window script, the deploy
notes (every deploy empties the cache, /root free space, a slower stop), a
post-deploy check, and the cache dir.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:56 +01:00
LumpiastyandClaude Opus 5.5 062a025714 scripts: fork-validate.sh --diskcache, the disk prompt cache
Runs beside production. The model is the small hybrid (Qwen3.5-0.8B), CPU only
(-dev none -ngl 0, no VRAM), with chat requests and greedy decoding. A
reference server that never restarts runs turn 1 and turn 2. The candidate
saves turn 1 on SIGTERM and restarts. Turn 2 must then be restored from disk
with the same prompt_n as the reference (only the suffix processed) and the
same answer. Cases:
- restore plain, with the arbiter env warm, and forced cold by a fake waiter
  on want.lock (the save from a released state). Each case records the exit
  time and SIGTERM-to-token-free, also once without the cache;
- no restore for another conversation, or with another n_ctx, KV type or
  model path (other key);
- a truncated file is removed at startup and a byte-flipped one at load,
  never used;
- the size limit evicts the oldest file;
- a write error (RLIMIT_FSIZE 1 MiB, SIGXFSZ ignored: EFBIG) is logged, the
  exit stays 0 with no file left, and the next start serves;
- the same restore on the GPU (-ngl 99, arbiter env, warm and cold) when
  >= 1.5 GiB of VRAM is free, else SKIP.
start_server gains SRV_WRAP, a command prefix that execs the server.

First run (2026-10-25, beside production): 11 PASS, 0 FAIL; the GPU leg
SKIPped (1219 MiB free). The first run found the CPU_REPACK release crash
(exit 139 in the cold case).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:56 +01:00
LumpiastyandClaude Opus 5.5 7a233df171 server: disk prompt cache, save slots on shutdown and restore after a restart
llama-swap SIGTERMs a model when the owner switches models (only one large
model fits in RAM), and switching back re-prefills the whole conversation:
about 150 s for a 30k-token chat on the 35B. With --cache-disk-path DIR,
each slot with a prompt is saved to DIR on shutdown. After the restart, a
request whose prompt starts with a saved prompt is restored from the file,
so only the suffix is processed. Off by default.

Save, in server_context::start_loop() after the loop returns:
- disk_cache_collect() copies the state of each slot with >= 256 text tokens
  to host memory (llama_state_seq_get_data_ext, flags NONE) while the device
  buffers still exist. A cold server reads its released state through
  llama_context_restore_memory_host(), with no device use and no extra copy.
- vram_share_exit() then frees the VRAM and releases the GPU token as before,
  so only the D2H copy delays the token release, never the disk IO.
- disk_cache_flush() writes the files. A slot is skipped if its KV does not
  cover its token list (seq_pos_max + 1 != n_tokens: the last batch was not
  decoded) or if the same token list is already on disk.

File (server_prompt_disk_cache, server-task.cpp):
- A 48-byte header (magic, version, key hash, sizes, payload hash), then a
  payload of 8-byte padded sections: tokens, checkpoint table, state, the
  checkpoint states. The hash is FNV-1a over the 64-bit words of the payload.
- Plain fwrite to <name>.tmp, fdatasync, POSIX_FADV_DONTNEED, rename. The read
  is a plain fread plus DONTNEED. There is no mmap: upstream #28092 livelocked
  the kernel on AMD + Vulkan by DMA-ing into a MAP_SHARED file.
- The index is built at startup from the headers. A bad magic, a size that
  does not match the header, or an inconsistent table means the file is
  removed. A hash mismatch at load also removes it, and nothing is restored.
  Other keys and other versions are kept. A .tmp older than 1 h is removed.
- --cache-disk-size N MiB (default 4096, 0 = no limit): before a write, the
  least recently used *.lpcd files of every model in DIR are removed until the
  new file fits; a restore updates the file mtime. Entries the new prompt
  extends are dropped after the write and do not count against the limit.
  A write also needs its size + 1 GiB free on the file system. A write error
  or a full disk is logged; the file is not saved and the exit stays clean.

Key: the build commit, the model path + size + mtime, LoRA and control
vectors, n_ctx, n_ctx_seq, n_seq_max, n_ubatch, kv_unified, swa_full, the
K/V types, the flash-attn type, and the RoPE/YaRN parameters.

Restore, in get_available_slot() after the RAM cache:
- Only while warm (holding the GPU token), with cache_prompt on and no media
  tokens. The file must reuse at least 256 more tokens than the slot holds.
- Reusable means one of two things. Either the saved token list is an exact
  prefix of the new prompt, or (for a recurrent/SWA model) a saved checkpoint
  at or below the common prefix can be used. A plain-KV model can be cut back
  to any prefix. Exact prefix alone is not enough for OpenWebUI on Qwen: the
  template rewrites the previous assistant turn (the reasoning is dropped), so
  the saved list is almost never a prefix of the next turn. The checkpoints
  are exact snapshots, and update_slots() rolls back to one exactly as it does
  for the RAM cache. They are restored with id_task -1, so one that fails to
  load means a re-prefill, not an abort.
- The slot's current prompt goes to the RAM cache first. A set_data failure
  clears the slot and removes the file.

Not supported with speculative decoding (warning, disabled): the draft state
is not saved. Only the slots are saved, not the RAM cache entries.

Measured on the pod, CPU only, Qwen3.5-0.8B (hybrid, 6 of 24 layers attention):
- 573-token chat: 64.5 MiB file with 2 checkpoints, copy 18 ms, write 171 ms,
  exit 314 ms (125 ms without the cache). After the restart: read 166 ms,
  restore 3 ms, turn 2 prompt_n 58 of 596 in 462 ms against a 2913 ms full
  prefill. The answer is identical to a server that never restarted.
- 12k tokens: 200.7 MiB, copy 110 ms, write 350 ms, exit 641 ms, peak RSS
  +162 MiB during the exit save. After the restart: read 307 ms, restore 18 ms,
  turn 2 prompt_n 35 of 12188 in 1.2 s against a 72 s prefill.
- The /root PVC writes at about 0.66 GB/s with fdatasync. /tmp (the node
  overlay) stalled once: 5.8 s for 64.5 MiB.
llama-swap v239 waits for the old process to exit before it starts the next
model, and SIGKILLs a process that is still there after 30 s. So the exit
save adds to the model switch time: about 1 s per GB on the PVC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:56 +01:00
LumpiastyandClaude Opus 5.5 7dbe7fdafa llama: read a released KV/recurrent state from its host shadow
After llama_context_release_device(ctx, true) the KV cache and the recurrent
state live only in host shadows, and their tensors have no buffer, so
llama_state_seq_get_data cannot read them. The disk prompt cache must save
a cold server's slots at exit, and a shutting-down server must not restore
to the device (it may not hold the GPU token, F7).

llama_context_restore_memory_host() makes the state readable without the
device. restore_device_buffers(to_host = true) wraps each shadow as a
CPU_Mapped buffer (ggml_backend_cpu_buffer_from_ptr) and points the tensors
into it: there is no copy and no allocation. The first version copied into a
CPU buffer instead, which needs the full KV allocation a second time, about
2.7 GB for the 35B at ctx 131072; the node has had an OOM-controller kill
before. The buffer base is rounded down to the 32-byte alignment that
cpu_buffer_from_ptr asserts (a large std::vector is only 16-byte aligned);
those leading bytes are never accessed. With to_host the shadows are kept,
because the buffers point into them.

It sets device_discarded, so restore_device() aborts afterwards. Exit only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:56 +01:00
LumpiastyandClaude Opus 5.5 2766d7977b llama: do not release CPU extra buffers (CPU_REPACK) on device release
A model with weights on the CPU device (-dev none / -ngl 0) segfaulted in
llama_model::release_device_weights() the first time the VRAM arbiter put
it to sleep: CPU_REPACK buffers are not host buffers, so load_tensors marked
them releasable, and capturing the shadow called their NULL get_tensor.
Reproduced with the production build 2fc9f9c35 (Qwen3.5-0.8B, -dev none
-ngl 0, LLAMA_SLEEP_VRAM_ONLY=1 LLAMA_SLEEP_EVICT_KV=1, force a yield with a
waiter on want.lock): exit 139 at the release, before any SIGTERM. The
faulting call (rip 0) returned into release_device_weights.

Only buffers on a non-CPU device are releasable now. These buffers are in RAM
already, so there is nothing to free. Production is not affected: every
arbiter model there has its weights on the GPU, and a releasable CPU_REPACK
buffer would have crashed it. Found by the --diskcache check, which runs
its arbiter cases CPU only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 4b29f38b92 docs: FORK.md and runbook after the 2fc9f9c35 deploy
Feedback from the 2026-10-25 deploy, which used only these docs:
- status: production is 2fc9f9c35; F10 deployed and verified on a 35B
  GPU load; --moe-exact and median --perf have run on the GPU; last
  full fork-validate run (37 PASS) and its timings; the first-candidate
  -arm VRAM spike seen twice is an open question.
- runbook: a no-rebase redeploy path; --arbiter beside production only
  if 1536 MiB VRAM is free, else in the swapdown window; full --moecache
  model path; page-cache eviction before and after phase 2; deploy
  steps reordered so the memory check follows stopping llama-swap;
  in-place rebuild and system_fingerprint check; mtime check through the
  versioned symlinks; rollback pointer numbering and one format; disk
  space; nothink alias and a PPM image helper for post-deploy requests;
  power1_cap via sysfs; restart and OOM checks; harmless UI warning.
- fork-validate.sh header: note the --arbiter VRAM requirement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 dbae58b893 docs: FORK.md F10 page-cache drop, -lm dio dead end; runbook note
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 f58520aa4f llama: drop the model file's page cache after a copying load
With -lm none the GGUF is read() into our own buffers, but the kernel keeps
the file in page cache afterwards, charged to the pod's cgroup: about 20 GB
after a 35B load. On 2026-10-24 the Talos OOM controller killed the pod right
after a load under exactly that pressure. Those pages are pure waste once the
data is copied.

llama_file::drop_cache() wraps posix_fadvise(POSIX_FADV_DONTNEED) (Linux only,
errors logged at debug). load_all_data() drops each copied tensor's range right
after reading it, so the cache never grows past about one tensor, and drops the
whole file at the final cleanup to catch the metadata and partial pages at
tensor edges. Both only when the loader holds no mapping of the files
(mappings.empty(): not mmap, not mmap+mlock, no lazy tensors), so nothing that
is mapped is ever advised away. Split GGUFs are covered (every file in files).
clip.cpp always copies the mmproj via ifstream, so it drops the whole file
after loading it.

Measured on the pod, gemma-4-E2B UD-Q4_K_XL (2.5 GiB) + mmproj BF16 (0.9 GiB),
CPU only, -lm none, cold cache, same binary with the drops disabled as the
control (4 rounds, rotated order):
- file pages resident after load (mincore): 2431-2444 / 940 MiB -> 0 / 0
- peak model-file residency during load: 2.5 GiB -> 1.2-1.5 GiB (largest tensor)
- pod cgroup file after load: +3.37 GiB -> +0; host MemFree after load
  0.6 GB -> 2.5-2.8 GB (MemAvailable counts clean cache, so it cannot show this)
- load time: 5.87 s -> 6.82 s median (+0.95 s); a post-load-only drop is
  6.56 s (+0.68 s). The cost is synchronous page freeing that reclaim would
  otherwise do later; -lm dio (direct IO) costs the same (+0.9 s) and does not
  cover metadata or the mmproj.
- greedy output byte-identical; -lm mmap keeps the model file fully resident
  and mapped (2498 MiB, no refaults); -lzm on skips the drop (file mapped).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 263d75cba3 docs: runbook - new fork-validate checks, gdb gone, scratch lost on restart
Phase 1 now covers the VRAM release/restore check and the handoff latency
A/B against --ref (which must be the production build); phase 2 adds
--moe-exact on the 35B and keeps its scratch dir. Explains median gating and
--perf-report, that a pod restart empties /tmp, that the manual checks are
now automated, and that gdb is not in the pod image (lost in the 2026-10-24
restart). States the commit trailer convention and what llama-swap does with
a child that exits vs one that wedges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 5f753f7a0c docs(readme): short fork summary, FORK.md is the source of truth
The README fork section still said "the two code changes below are auto-on"
(the fork has three prefill changes plus the VRAM sharing and server fixes)
and recommended --n-cpu-moe 28 on the 35B, which FORK.md section 4 records as
worse at depth and unable to hold a long context. Replace it with a short
accurate summary that points to FORK.md, the runbook and the scripts. The
Laguna time profile, the iSWA KV note and Laguna's --parallel 1 saving were
only in the README and move to FORK.md sections 4 and 5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 4e455b79df docs: FORK.md - review fixes, new checks, llama-swap restart behaviour
- status: no hardcoded fork-commit count (the list includes docs and
  scripts; count git log origin/master..HEAD); what is after the production
  commit; the commit-trailer convention of this private fork vs upstream's
  AGENTS.md rules
- commit map: the commits after the production commit
- F1: validated by the new --moe-exact check and the LLAMA_MOE_IDS_SKIP knob
- F3: every graph that builds concat(state, transpose(x), 0) (bailingmoe3,
  clef, kimi-linear, kimi-k3, glm5-next, mamba-base, plamo2, lfm-base,
  qwen4exp besides delta-net); only the delta-net path is measured
- F4/F6: the new v-vram-* and a-handoff-latency checks with first results
- F5: encode_mtmd_chunk was split out of process_mtmd_chunk, not renamed
- F6: the vram_cold ordering rule restated from the code (the measured
  failure was under the doorbell design; with the inline probe should_sleep
  cannot run inside the restore; keep it, and A/B handoff latency before any
  change) and why vram_acquire_for_load sets the flag early
- F7: what llama-swap does when a child dies, verified with a scratch
  llama-swap: "upstream exited unexpectedly", restart on the next request
- F9: new flags, phases, median gating, 2026-10-24 reference values, the
  /tmp and gdb loss on the pod restart, new grep targets and gaps
- section 7: stale comments fixed, what is not yet run on the GPU

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 9537f885e0 scripts: fork-validate.sh - --moe-exact, greedy identity for the readback skip
F1 (the routing-ids readback skip in sched_copy_experts) had no automated
check. --moe-exact MODEL reads expert_count / expert_used_count from the GGUF
header, computes the skip threshold ceil(4 * n_expert / n_expert_used) tokens
per ubatch (128 on the 35B), and runs one greedy request with a prompt of
about 2x that in a single ubatch, every expert on the host, so the prefill
takes the skip and each decode step takes the readback. The candidate's 64
tokens must equal the reference's byte for byte; with the LLAMA_MOE_IDS_SKIP
knob in the candidate, also the candidate with the skip forced off (=0), which
proves the skip exact even when the reference carries it, and =log must show
skips, all at >= 4 * n_expert ids. Without the knob skip-engaged SKIPs with
the batch-size argument. It needs the GPU to itself (same guard as --perf,
now shared as gpu_exclusive); FV_TEST_SHARED_GPU=1 bypasses the guard for
script tests against stub binaries only.

Also unset inherited LLAMA_MOE* variables.

Not run on the GPU yet (needs a swapdown window). Dry-run on the pod
against stub servers and the real 35B GGUF header: threshold 128, all-PASS
when outputs match, FAIL on a differing reference, SKIP without the knob,
FAIL for a dense model and while llama-swap runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 12a2add2c5 scripts: fork-validate.sh - check that a cold server gives its VRAM back
Nothing checked that a release actually frees VRAM. --arbiter now takes a
baseline of mem_info_vram_used before any scratch server starts, then with A
warm and alone on the scratch arena makes it release through a fake waiter
on want.lock. v-vram-release requires the card-wide delta to drop by at
least 70% of what A added warm; v-vram-restore requires it back (and the
output equal to the reference) after the next request. Production's own
share is re-read at every point from the DRM fdinfo of every process the
script did not start (deduplicated by drm-client-id); if it moved by more
than 64 MiB the check SKIPs instead of blaming the candidate.

Run beside production on the pod (reb1024 vs itself): warm +546 MiB, cold
+4 MiB, restored +546 MiB, production steady at 5053 MiB; 16 PASS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 deb09fb305 scripts: fork-validate.sh - A/B ordinary handoff latency against --ref
--arbiter only reported the candidate's handoff latency, while the fork's
process rule (two production deploys lost) is to A/B the ordinary handoff,
not only the pathological cases. After the arbiter cases, a-handoff-latency
starts fresh A/B server pairs of each build on the scratch arena and runs
blocks ref, cand, cand, ref of 16 alternating handoffs each. It FAILs if the
candidate median exceeds the reference median by more than max(25%, 200 ms)
and reports both medians, maxima and per-block medians.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 3b2c7188fb scripts: fork-validate.sh - gate --perf on medians, not means
--perf averaged the two per-arm llama-bench means of each build, so one
catastrophic sample (CPU contention from the ~60 other pods on the node) moved
the result, against the fork's own rule of medians, never means. llama-bench
now writes -o jsonl (every repetition sample) next to the markdown table, and
the gates use the median of all samples of a build pooled over its two arms
(4 per prefill cell, 6 for decode); the table shows min-max and n.

--perf-report TSV re-aggregates a saved <scratch>/perf.tsv without a GPU.

Checked offline on the pod: the 2026-09 A/B samples in results.txt give
pp2048 reference 250.24 (old mean) vs 247.76 (new median), candidate 280.15
vs 276.93; a synthetic 2026-10-24-like set with one 180 t/s candidate sample
read -9.3% (FAIL) the old way and -0.2% (PASS) with medians.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 3141ad2a9e scripts: rx580-bench - decode at tg256 depth 4096, not tg32
The llama-bench sweep measured decode as tg32 at the default depth and the
README called it the continuity metric, while tg32 at n_ctx 32 is launch /
barrier bound on this card and is not a valid decode metric (it once read
-36% between builds at parity at tg256). lbsweep.sh now runs prefill
(-p $LB_P -n 0, depth 0) and decode (-p 0 -n 256 -d 4096, LB_N / LB_D) as
two invocations and labels depth cells (tg256@d4096). summarize.py marks
old depth-less tg rows as invalid-metric history. The README examples and
the reference values are updated to the 2026-10-24 production numbers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 0ecd1a43cc llama: env knob for the routing-ids readback skip
LLAMA_MOE_IDS_SKIP=0 forces the ids readback in sched_copy_experts (the
upstream path) and =log reports each skip on stderr, so validation can show
that the skip engages at prefill and that output is byte-identical with and
without it in one build. Read once (function-local statics); unset changes
nothing. Not set in production.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 bf3bda9f65 server: fix stale VRAM-arbiter comments and the sleep log line
Comments only, plus one log string:
- the arbiter block comment said the KV cache is never evicted; with
  LLAMA_SLEEP_EVICT_KV it is (and the scheduler and scratch with it)
- vram_go_cold said vram_ensure_warm() rebuilds the encoder; only
  encode_mtmd_chunk() restores it, and only for an encode
- vram_ensure_warm and the mtmd init block named process_mtmd_chunk; the
  restore lives in encode_mtmd_chunk
- handle_sleeping_state said waking happens lazily before decode; it
  restores eagerly on the wake transition, before update_slots()
- the vram_cold ordering note now says why it still holds with the inline
  want-lock probe
- the sleep log line said "keeping KV cache" even when LLAMA_SLEEP_EVICT_KV
  evicts it; it now names what actually happens (the prefix
  "server entering sleeping state" that fork-validate.sh counts is kept)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 bdf2b81b81 docs: FORK.md and rebase runbook for the RX 580 fork
FORK.md documents every fork feature (what and why with measured numbers,
where, how it works, invariants, past failures, rebase hazards, validation),
the dead ends measured on this hardware, the production configuration and
the measurement traps. docs/fork-rebase-runbook.md is the rebase, validation,
deploy, rollback and GPU-recovery procedure plus an operations reference.
The README fork section links both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 a213514855 scripts: add fork-validate.sh, post-rebase validation of the fork
Checks the op kernels, the concat fast path, the VRAM arbiter incl. every
shutdown path that has caused a production incident, and prefill/decode
against a reference build. Prints PASS/FAIL per check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 8742446dba ggml-vulkan: env knob for the concat transpose fast path
GGML_VK_CONCAT_FASTPATH=0 forces the generic path and =log reports each
fast-path dispatch, so validation can prove the path fires at prefill and
not at decode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 79e6ade8b3 server: release the GPU token promptly on shutdown
Two ways a shutting-down server kept the VRAM arbiter's token, stalling every
other model:

- SIGTERM with a request in flight: terminate() stopped only the task queue.
  The HTTP handler kept waiting for results the loop would never send, the main
  thread blocked joining it, and the token was released only in the destructor
  after that join - i.e. never, until the client went away. Terminate the
  result queue too (readers now get "server is shutting down": a 503, or an SSE
  error event), and let wait_until_no_sleep give up once stopping.

- SIGTERM while waiting for the token: the loop thread sat in a blocking flock
  inside the wake callback, holding mutex_tasks, so terminate() waited for the
  current holder to finish and then for a pointless restore of our weights.
  Poll the lock non-blocking and abort the wait on shutdown; the aborted wake
  stays cold, skips the restore, and the loop exits before update_slots.

Also release the token right after start_loop() returns rather than in the
destructor, and only after the device buffers are actually freed: the old
order unlocked before destroy(), so the next model could start uploading into
VRAM that was still occupied. Freeing at exit uses a new
llama_context_discard_device(), which drops weights, KV, MoE cache and scratch
without capturing host shadows first.

Off production: SIGTERM mid-generation or mid-wait now exits in ~0.2 s with
exit 0 and frees the token within 5-70 ms (previously never / after the
holder's whole generation). Handoff latency and greedy output are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 3b068bdb3a server: run the shutdown handler off the signal-interrupted thread
A SIGTERM that lands while the start_loop() thread is inside a sleeping
callback deadlocks the server. The callbacks run under
server_queue::mutex_tasks, and the kernel delivers the signal to that same
thread; the handler calls terminate(), which locks mutex_tasks again, and the
thread waits on itself forever.

With the VRAM arbiter that window is wide - a whole device release, or a
blocking wait for the GPU token - and llama-swap sends SIGTERM to evict
models, so it hit production: an evicted model wedged mid-release, never
released token.lock, and every other model queued behind it. SIGTERM could
not stop it either; only SIGKILL did.

Make the handler only write the signal number to a self-pipe, which is
async-signal-safe, and run shutdown_handler on a dedicated thread. terminate()
there simply waits until the loop thread drops mutex_tasks, after which the
loop exits normally. The thread is joined before the objects the handler uses
are destroyed. If pipe() fails the old direct call is kept.

Reproduced 6/6 off production (SIGTERM during release and during the token
wait); with the fix 8/8 clean exits, and handoffs, image requests and greedy
output are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 3d4e7bc065 scripts: rx580-bench - use -lm none, document the MoE cache option
Upstream #28334 removed --mmap/--no-mmap/--mlock and llama-bench's -mmp, so the
harness flags no longer parse: the server would refuse to start and llama-bench
would silently produce no rows. Switch both harnesses to -lm none, the same
load mode, and note --moe-cache-mib as a server-only option worth measuring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 a620add3b7 llama: release/restore the MoE cache with the device weights
The MoE cache (--moe-cache-mib, #29887) keeps the most recently used host
experts in device banks. The on-demand VRAM path (release_device /
restore_device, used by the server VRAM arbiter and the on-demand mmproj path)
did not know about it, so its banks were never freed on handoff. Restoring them
by reallocation alone would also be wrong: the banks come back zeroed while the
slot maps still mark experts as cached, so the model would silently read zero
weights.

Free the banks on release and empty the cache instead of shadowing it: re-init
every LRU, re-point its slot-map pointers at the host slot maps and clear those
to -1. The banks are reallocated (which also re-inits their views) and cleared
on restore, before the scheduler is rebuilt, and the cache refills from the
host experts as before. Release runs in both modes, weights-only and evict_kv.

device_resident() now covers the cache too, for the early returns in
release/restore, the auto-restore in decode() and
llama_context_weights_resident().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5 9c042626b3 llama: skip the compute-buffer size check when the context is cold
release_device(evict_kv) drops the scheduler, so a context evicted by the VRAM
arbiter and then destroyed reaches the destructor with a null sched. The upstream
"compute buffer size matches expectation" loop calls
ggml_backend_sched_get_buffer_size() unconditionally, whose GGML_ASSERT(sched)
then aborts: the server exits 134 instead of 0 on every shutdown taken while
cold, which under llama-swap makes an ordinary stop look like a crashed child and
leaves a ggml backtrace in the log each time. It also skipped the rest of the
destructor, so the cold teardown path had never actually run to completion.

Guard at the call site rather than relaxing the assert - the assert is right, and
every other caller reserves the scheduler first. Same shape as the null guards in
synchronize(), memory_breakdown() and the released-buffer iteration.

The loop is a diagnostic size comparison, already reported by sched_reserve() at
load, so skipping it when cold loses nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5 2c6aa3b6e7 server: replace the VRAM doorbell with a want-lock
The doorbell was edge-triggered: a waiter wrote <arena>/doorbell/<pid> and the
holder's warden turned the inotify event into a queue flag. A ring carried no
notion of "still wanted", so it was lost whenever the holder was not already warm
and listening - during its model load (the watch does not exist yet, and the
kernel does not queue events for a watch that is not there) or inside
restore_device - and any ring that survived past the point its sender had been
satisfied caused a spurious release. Lost rings wedged the waiter permanently,
because flock(LOCK_EX) never times out and, with idle-sleep disabled, the holder
had no other reason to release. The wait blocks under mutex_tasks, so the whole
server stopped answering while /health still returned 200.

Express the request as kernel state instead. A waiter holds <arena>/want.lock
shared while it waits and drops it once it owns the token; the sleep decision
probes that lock non-blocking and releases the GPU while anyone is waiting. The
probe needs its own fd - flock treats two open file descriptions of one file
independently, so probing on the waiter's fd would convert our own lock rather
than conflict with it. Nothing can be missed, nothing goes stale, and a waiter
that dies is cleaned up by the kernel.

Probe from should_sleep() on the loop thread rather than from a warden thread.
Routing it through a flag is what made the first attempts fail: start_loop()
holds mutex_tasks from should_sleep() through the callbacks to the wait, so a
warden's request_yield() blocks on that mutex and is admitted only after the flag
has been consumed, latching a release for the next wake. Reading live state where
the decision is made has no edge to latch, and drops the warden's poll latency.

Two sleep-path bugs this exposed: wait_until_no_sleep() waited on !sleeping but
the loop clears req_stop_sleeping on the way in, so a loop that slept again
before the waiter ran stranded it forever - re-ask on every wake. And a task
queued after the waiter saw us awake could not wake us by itself, so sleep now
also breaks on a non-empty queue. Hold the GPU for 100 ms after a wake: the
request that woke us is not queued yet, and yielding at once only sends it round
again.

Measured on the RX 580 pod, two servers contending, 60 alternating handoffs:
0 stranded, median 0.309 s against the doorbell's 0.314 s.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5 c61db474d8 scripts: add the RX 580 benchmark harness
Encodes the production config, fixed corpus slices, repeat/median discipline
and the noise floor, so the measurement method does not have to be
rediscovered each time. Runs the corpus prefill test through llama-server and
the llama-bench sweep as a controlled cross-check, with interleaved A/B.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5 7194d47311 ggml-vulkan: skip the concat transpose fast path below one tile
The tiled-transpose fast path in ggml_vk_concat is a large prefill win but a
loss at decode. On a hybrid model like Qwen3.6-35B (gated delta-net on 30 of
40 layers) build_conv_state emits a transposed concat per recurrent layer, so
the fast path runs 30 times per decoded token.

At one token the transposed source is a single column, so there is no
uncoalesced stride left to fix, but the path still costs two dispatches and an
unconditional ggml_vk_sync_buffers - a full pipeline barrier that serializes
the command stream. Measured -17% tg256 on RX 580; the barrier, not the
half-empty transpose dispatch, is nearly all of it.

Require the transposed source to be at least one TILE_DIM wide. That source is
qkv_mixed transposed, so ne[0] is the ubatch token count: prefill keeps the
fast path, decode and small speculative drafts take the generic one.

pp8192 +11% retained, tg256 back to parity, generation byte-identical.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-21 19:39:51 +01:00
Lumpiasty 04552507fe docs(readme): Polaris MoE profile - GCN mask_opt, the ids-readback win, tuning and dead ends 2026-11-21 19:39:51 +01:00
Lumpiasty d773ab0ffc server: on-demand mmproj - free encoder VRAM on the text path
The vision encoder sits idle in VRAM on every text request. With
LLAMA_MMPROJ_ONDEMAND=1 the server keeps the encoder in RAM (released
after load) and brings it into VRAM just-in-time before mtmd_batch_encode,
releasing it again afterwards - so the encoder's VRAM is free for KV /
expert cache on the common text-only path.

When the encoder does not fit (that VRAM has been claimed), evict the LLM
backbone weights first: they are not needed while the encoder runs (the
decode that uses them happens afterwards), their host shadow is read-only
(cheap free, no D2H), and the KV / prompt cache is left untouched. Order
on the way back matters: free the encoder before re-uploading the weights
so the peak stays within VRAM.

Also wire the encoder release/restore into the VRAM arbiter: vram_go_cold
releases it; the just-in-time encode-path restore replaces the previous
restore in vram_ensure_warm.

Assisted-by: Claude
2026-11-21 19:39:51 +01:00
Lumpiasty 58e693b794 mtmd: release/restore the encoder weights from VRAM on demand
Add clip_release_device/clip_restore_device (and mtmd_release_device/
mtmd_restore_device wrappers over the vision + audio contexts) that free
the multimodal encoder's device weight buffer to a read-only host shadow
and rebuild it on demand, using the same shadow/free/reallocate pattern
as llama_model weights. No-op for a CPU-backed encoder. This lets the
server drop the ~hundreds-of-MiB vision encoder from VRAM when it is not
encoding an image.

Assisted-by: Claude
2026-11-21 19:39:51 +01:00
Lumpiasty b7cb284e54 llama: free backend compute scratch on cold; guard memory_breakdown
On device release with KV eviction, also call ggml_backend_free_scratch()
on each backend so a cold model drops its Vulkan compute preallocations
(reallocated lazily on the next compute via restore).

Also guard llama_context::memory_breakdown() against a freed scheduler:
evict_kv release resets sched to null, so a cold model that is then torn
down (e.g. terminated by the process manager) hit GGML_ASSERT(sched) in
ggml_backend_sched_get_buffer_type. Skip the compute-buffer accounting
when sched is null.

Assisted-by: Claude
2026-11-21 19:39:51 +01:00
Lumpiasty fa24747925 ggml: add ggml_backend_free_scratch to drop Vulkan compute prealloc
Add an optional backend interface method free_scratch (with a public
ggml_backend_free_scratch wrapper) that frees transient/scratch device
memory a backend holds outside of any allocated buffer, keeping the
backend usable - the scratch is reallocated lazily on the next compute.

Implement it for the Vulkan backend (ggml_backend_vk_free_scratch): free
the prealloc_x/y/split_k/add_rms_partials and sync_staging device buffers
and reset their sizes, so an idle/cold model does not hold the vision or
matmul compute preallocations in VRAM. All other backends leave the hook
null (no-op).

Assisted-by: Claude
2026-11-21 19:39:51 +01:00
Lumpiasty 98c7e109b7 llama: evict recurrent/SSM state on device release
The recurrent (SSM/conv) state of hybrid models (e.g. Qwen3.5) was left
resident when a model's device buffers were released for on-demand VRAM
sharing - llama_memory_recurrent::release_device_buffers() was a no-op
default. Implement it (and restore_device_buffers) with the same
capture-host-shadow / free / reallocate pattern as llama_kv_cache, so
llama_memory_hybrid now evicts both its attention KV and its recurrent
state. The state is read-write, so its shadow is recaptured on every
release.

Assisted-by: Claude
2026-11-21 19:39:51 +01:00
Lumpiasty bc55fb9990 server: also free the compute-graph scheduler on KV eviction
Under LLAMA_SLEEP_EVICT_KV a cold model still held the scheduler's worst-case
compute buffer (hundreds of MiB, e.g. ~700 MiB for gemma-26B) plus the resident
experts. release_device(evict_kv) now also frees the sched (sched.reset() +
sched_need_reserve), and restore_device() rebuilds it via sched_reserve(), so a
fully-evicted model holds essentially no VRAM (gemma cold: 7147 -> 51 MiB).
Trades a heavier re-warm (weights+KV H2D + sched reserve) for the freed VRAM;
weights-only mode is unchanged (fast switch, keeps KV+compute).

Assisted-by: Claude
2026-11-21 19:39:46 +01:00
Lumpiasty 25c12e56ca server: restore weights/KV on wake, not at decode, so KV eviction is safe
With LLAMA_SLEEP_EVICT_KV the KV cache device buffers are freed when a model
goes cold. update_slots() touches the KV before the decode-time vram_ensure_warm
(e.g. SWA models create a checkpoint that reads the KV via ggml_backend_tensor_get),
so the KV must already be resident by then. Move the restore to the sleep-wake
handler (handle_sleeping_state(false)), which runs before update_slots, fixing a
GGML_ASSERT(buffer) crash on iSWA models (gemma) when KV eviction is enabled.

Assisted-by: Claude
2026-11-21 19:39:46 +01:00
Lumpiasty 8e575c9e54 llama: guard buffer iteration against released device buffers
release_device_weights()/release_device_buffers() leave a null buffer in
ctxs_bufs while the device memory is released for on-demand VRAM sharing.
The memory-breakdown / total_size / clear paths iterated these and called
ggml_backend_buffer_get_size()/get_type()/clear() on the null buffer, tripping
GGML_ASSERT(buffer) and aborting on shutdown of a cold model. Skip null buffers
in llama_model::memory_breakdown and llama_kv_cache::{memory_breakdown,
total_size,clear}.

Assisted-by: Claude
2026-11-21 19:39:46 +01:00
Lumpiasty 1ee9806b3d server: coordinate model load with the VRAM arbiter to avoid load-time OOM
Before uploading a model's weights, acquire the shared VRAM token (ring the
doorbell so any resident model releases first). Previously load uploaded weights
to VRAM before the arbiter was active, so loading a large model while another
(e.g. the warm 4B task model) held VRAM could exceed the budget and OOM.

- add vram_arena_open() (idempotent flock/doorbell setup) and
  vram_acquire_for_load(), called from load_model() before
  common_init_from_params().
- a coordinated load now stays warm holding the token and serves its first
  request without a re-warm (drop the init-time go_cold cold-start).

Assisted-by: Claude
2026-11-21 19:39:46 +01:00
Lumpiasty 558f7f65a2 server: keep the sleep-wake path active when the VRAM arbiter is enabled
The request handler only calls wait_until_no_sleep() (which wakes a server out
of its sleeping state) when sleep_idle_seconds >= 0. But the VRAM arbiter's
cross-process doorbell can put a server to sleep even when idle-sleep is
disabled, so without this a doorbell-slept server would never wake and requests
to it would hang until timeout.

Do not bypass the wake path when LLAMA_SLEEP_VRAM_ONLY is set, so the arbiter no
longer depends on --sleep-idle-seconds being configured.

Assisted-by: Claude
2026-11-21 19:39:46 +01:00
Lumpiasty cb471f9b2f server: optionally evict the KV cache too (Phase 2 of VRAM sharing)
Extend on-demand device residency to the KV cache so that when a model's KV
plus another model would not fit in VRAM, the KV can also be evicted to a host
shadow (D2H on release, H2D on restore) instead of only the weights.

- llama_memory_i: add release_device_buffers()/restore_device_buffers()
  (default no-op). Implemented in llama_kv_cache (D2H shadow of the live
  ctxs_bufs, freed and reallocated like the weights); llama_memory_hybrid and
  llama_kv_cache_iswa delegate to their child caches.
- llama_context::release_device(evict_kv): also evict the memory's device
  buffers when requested; restore_device() rebuilds them. Public API
  llama_context_release_device gains an evict_kv flag.
- server: LLAMA_SLEEP_EVICT_KV=1 enables it. Off by default (weights-only),
  since the KV shadow adds a D2H/H2D copy of the live cache each cycle.

Validated on RX 580 (Vulkan), 4B @ 32k ctx: weights-only cold VRAM 1750 MB
(KV stays); weights+KV cold VRAM 726 MB (KV freed, ~1 GB reclaimed). KV
survives the round-trip: prompt cache reused after the cycle (prompt_n 4 vs
42), correct output.

Assisted-by: Claude
2026-11-21 19:39:46 +01:00
Lumpiasty 56a1010bbc server: on-demand VRAM sharing to time-share one GPU between models
Add release/restore of a model's GPU weight buffers (keeping a host shadow
and the KV cache) so several always-loaded llama-server processes can
time-share a single GPU without reloading or losing the prompt cache.

- llama-model: release_device_weights()/restore_device_weights() capture a
  compact host shadow (stable iteration order, view-skipping) and free then
  realloc the device weight buffers; weights_resident() query.
- llama-context: release_device()/restore_device() wrappers; decode() auto-
  restores; public C API llama_context_release_device/restore_device.
- server: LLAMA_SLEEP_VRAM_ONLY makes idle-sleep release only the VRAM weights
  (not a full unload/reload). A cross-process flock token in LLAMA_VRAM_ARENA
  enforces "resident iff holds token"; an inotify doorbell forces the holder
  to release on contention. The warden thread only touches the task queue, so
  releases run on the loop thread and never race a decode.

Validated on RX 580 (Vulkan): two models share 8GB, never both resident,
correct output under contention, KV cache preserved (no re-prefill).

Assisted-by: Claude
2026-11-21 19:32:30 +01:00
Lumpiasty cfda297cbb ggml-vulkan: tiled transpose fast-path for concat with transposed source
The generic concat shader reads a transposed source (nb[1]==type_size) with an
uncoalesced stride, which is catastrophically slow on discrete GPUs (~5.6ms for
a 16MB concat on the RX 580 vs ~130us of memory bandwidth), a ~9% prefill
hotspot on Qwen3.5-4B (delta-net state concat).

Add a fast path for concat along dim 0 where one source is stored transposed and
the other source + dst are contiguous along dim 0: copy the contiguous source
with copy.comp and transpose the other source into the matching dst sub-region
with the existing tiled copy_transpose shader (shared-memory 32x32 transpose,
coalesced read+write). Reuses pipeline_cpy_* / pipeline_cpy_transpose_* with
custom push constants + doffset, no new shader. Falls back to the generic path
otherwise (gated on type/shape/contiguity + 16-bit doffset bound).

Assisted-by: opencode
2026-11-21 19:32:13 +01:00
LumpiastyandClaude Opus 5 e1bc1f7398 vulkan: record why flash-attn shmem staging stays off for GCN
Staging K/V through shared memory looks like an obvious win on GCN: without it
each rowgroup re-reads the whole K/V block from global memory, and with row_split
4 that pulls one 16KB block through a 16KB vector L1 four times.

Measured, it loses: -6.7% pp2048 at depth 16k and -7.4% at 32k on Polaris at head
size 128. The kvsh stride of D/4+1 dwords is 4 mod 32, which costs an 8-way LDS
bank conflict on wave64 - the +1 padding is tuned for warp32 - and the extra
shared memory eats occupancy this shader is already short of.

Comment only, no behaviour change. Leaving a note so the next person does not
spend a GPU on it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-11-21 19:32:13 +01:00
Lumpiasty b5085151f2 ggml-vulkan: enable flash-attn mask_opt for GCN large head sizes
mask_opt was disabled on AMD GCN for head sizes <= 256, but it is
beneficial there in high-context prefill: on fully-visible causal
blocks it skips the per-block mask load+add, and it skips fully-masked
blocks entirely. The attention op on GCN is compute-bound on the
softmax path (no matrix cores), so this cuts real work.

Verified lossless (perplexity bit-identical with it on vs off) and a
measured prefill win on Qwen3.5-35B-A3B (head_dim 256) on an RX 580:
pp2048 unchanged at short context, +8% @ 16k, +12% @ 32k, growing with
depth. Enable for GCN when HSK/HSV >= 256; the existing large-mask
conditions keep it off for decode.

Assisted-by: Claude
2026-11-21 19:32:13 +01:00
LumpiastyandClaude Opus 5.5 ab7e03ec6c llama: skip the routing-ids readback in sched_copy_experts when the batch uses every expert
Offloaded expert weights are uploaded per split, and to decide which experts to
upload the copy callback reads the routing ids back to the host. The ids are
produced on the same device we are about to upload to, so the readback forces a
full pipeline flush - 47 of them per eval on a 48-layer MoE.

Once the batch draws enough experts the readback stops telling us anything. At
2048 tokens x 10 experts over 256 experts every expert comes back used, so the
used set is all true and the copy is a single whole-tensor range anyway.

Skip the readback when ggml_nelements(ids) >= 4 * n_expert and mark every expert
used. Uploading an expert that no id selects cannot change the result, since
mul_mat_id only reads the rows the ids point at, so this stays exact. With the
MoE cache on, the all-used set still goes through moe_cache->copy_experts, which
copies the cached experts device-to-device from the host slot maps and uploads
the rest - still exact, and no readback is needed for it. Decode is unaffected:
n_ids there is the number of experts per token, far below the threshold. The
small-batch MoE cache slot path has its own readback and is not touched.

The st.ids/st.used caching is unchanged: a repeat call with the same ids tensor
reuses the all-true set, which is what the threshold would decide again.

Port of dbc449afd ("sched: skip the MoE routing-ids readback when the batch
uses every expert"), whose target block in ggml_backend_sched_compute_splits was
removed upstream (#29943) in favour of this copy callback.

Measured (original, before the port) on an RX 580 (Vulkan, Polaris) running
Laguna-S-2.1 118B IQ2_M with all experts host-resident: pp2048 +4.7% at depth
32k, +0.9% at 16k, tg32 +3%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-21 19:32:13 +01:00
85 changed files with 730 additions and 4441 deletions

No files matched your search

+1 -1
View File
@@ -72,7 +72,7 @@ jobs:
- name: Fetch emdawnwebgpu
run: |
DAWN_TAG="v20261120.121236"
DAWN_TAG="v20260908.214631"
EMDAWN_PKG="emdawnwebgpu_pkg-${DAWN_TAG}.zip"
echo "Downloading ${EMDAWN_PKG}"
curl -L -o emdawn.zip \
-2
View File
@@ -1,7 +1,5 @@
# Instructions for llama.cpp
> **Private fork: read [FORK.md](FORK.md) and [docs/fork-rebase-runbook.md](docs/fork-rebase-runbook.md) before any work.**
> [!IMPORTANT]
>
> AI-generated code is allowed. What is **not** allowed is submitting code you do not understand. You are 100% responsible for every line, however it was produced.
+1 -157
View File
@@ -6,12 +6,6 @@ which also holds the operations reference (pod access, files, recovery).
Status at the time of writing (2026-11-22):
- Candidate, not deployed: branch `fable5/vram-ondemand-rebase-20261122` on upstream `5bacb7c86`
(19 upstream commits past `e6d624b9a`, including upstream's #32755 fix #32807 and the merge of
#32735), with the F14 first-user-message break, the `LLAMA_LOOP_FULL_COPY` knob (F13) and the
carry of #32777 (F12). Pod build `/root/arms/reb20261122/build`. Its validation and the Inkling
A/Bs are in runbook section 3 ("2026-11-22") and in F12, F13 and F14 below; production stays on
`f3397cded` until they pass.
- Production runs `f3397cded` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12, F13, the
per-pass loop views for Inkling, and F14, the user-message batch-break merge), deployed 2026-11-22
@@ -147,13 +141,6 @@ Oldest first. "Feature" points at the section below.
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
| 312c29c2b | llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as `b13c41355`) | F13 |
| 510aa6e09 | server: merge user-message batch breaks when the rest of the prompt fits in one ubatch (deployed 2026-11-22 as `f3397cded`) | F14 |
| e7e907b7c | server: F14 - keep the batch break at the first user message (system-prompt checkpoint) (branch `-20261122` only) | F14 |
| 5f88124b3 | llama: env knob LLAMA_LOOP_FULL_COPY=0 to skip upstream's inkling2_loop full-copy guard (#32807) (branch `-20261122` only) | F13 |
| 6c895a491 | upstream-carry #32777: fit : prioritize looped MoE layers in device memory (branch `-20261122` only) | F12 |
| 70d39bdbc | scripts: fork-validate.sh - scratch arenas on /dev/shm, lock_id failed on overlayfs (branch `-20261122` only) | F9 |
On `fable5/vram-ondemand-rebase-20261122` the #32735 carry (d2396e4e3) is gone (merged upstream) and
the other hashes in this table differ; the last four rows are hashes of that branch.
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
@@ -1133,69 +1120,6 @@ Carried since 2026-11-22 (both on upstream `e6d624b9a`, for Inkling-2-Mini, sect
`!bias` (F2). Vulkan supports it only for F16 K/V, head size 64 or 128, with subgroup shuffle and
vote (RX 580 RADV has both; Inkling is f16 KV, head size 128).
State after the 2026-11-22 rebase onto `5bacb7c86` (branch `fable5/vram-ondemand-rebase-20261122`):
- #32735: merged upstream as `8a97fa98a`, patch identical to the carried head. Carry dropped.
- #32740: still open, but the PR was split on maintainer request (one backend per PR). Its head
`8c6f0c0ca` (8 commits) is now only the GGML contract: `ggml_flash_attn_ext_add_indexed_bias`,
a CPU reference (the CPU backend now accepts indexed bias) and test-backend-ops cases (the
standalone `test-flash-attn-indexed-bias` is gone). `be0bf1fbd` removed the Vulkan support
(supports_op returns false for `src[5]` or indexed bias), `9cd195304` moved the CUDA/HIP kernels
and the Inkling graph changes (`llama-graph.cpp`, `llama-kv-cache.cpp`, `inkling2.cpp`) out of
it. No open PR carries the Vulkan part (Metal is #32808). The contract also changed in wording:
the bias is added "in F32" (before: "before its F16 rounding"); the API and the `src[6]`/`src[7]`
layout are the same. **Decision: the carry of head `808fd1c4e` is kept unchanged.** Carrying the
new head would remove the Vulkan kernel and the graph use, so Inkling would fall back to the
dense mask: compute buffer about 15.8 GiB at ctx 65536 instead of 7 GiB, and the k=4 placement of
F13 would no longer fit. Owner: decide whether to keep carrying `808fd1c4e` until a Vulkan
follow-up PR appears, or to port the Vulkan part onto the new contract. Once the contract merges
upstream, the carry will conflict with it in `ggml.h`, `ggml.c`, `ggml-cpu/ops.cpp` and every
backend's supports_op, and has to be reduced to the Vulkan and graph parts by hand.
- #32777 (new carry, owner's wish): `fit : prioritize looped MoE layers in device memory`, head
`01945dcb0`, one commit, applied cleanly. See "#32777 and `--fit` for Inkling" below.
**#32777 and `--fit` for Inkling.** `llama_model_layer_visits()` counts how many slots share each
layer's tensors (by pointer: `ffn_up_exps`, else the next present weight), so inkling2_loop gets 3
for the loop layers 6..17, 1 for the others, 0 for the 24 repeat slots. `--fit` then (a) probes
"only the non-loop experts on the CPU" against "all experts on the CPU" and keeps the loop experts
on the GPU if that leaves more free memory, and (b) when it moves experts back to the GPU layer by
layer, ranks the layers by visit count, so loop layers go first (the order of the F13 placement
guide).
- F13 interaction, checked by reading: fit's first probe, the only one that computes the visits,
runs with fit's own defaults (fit refuses to run with a user `-ngl`, `-ot`, `-ncmoe` or `-ts`), so
every weight is in a device (dummy) buffer, no F13 view exists, and the pointer sharing is
intact. Views appear only in the later probes with CPU experts, which do not recompute the
visits. If the visits were ever computed with the loop experts on the host, the views would make
every slot count 1 and the loop ranking would silently turn off (fix then: key on
`t->view_src ? t->view_src : t`); not reachable through `--fit` today, so not changed.
- Memory estimate: fit does not model the per-pass copies; it measures every candidate placement
with a no_alloc load and a graph reserve (`common_get_device_memory_data_impl`). The reserve sees
the F13 views, so its compute buffer is the F13 one (about 1.4 GiB at ctx 16384, not 7 GiB). The
estimate therefore already accounts for F13, and the PR's comment ("CPU-offloading repeated core
layers can inflate the compute buffer") describes the build without views; with views step (a)
will usually pick "all experts on the CPU" as the base and let step (b) add loop layers back.
Not changed.
- The mmproj: llama-server adds the mmproj estimate (about 0.8 GiB) to the fit margin of its
device before fitting, on top of `--fit-target` (default 1024 MiB). F5 releases the backbone
while the encoder runs, so that headroom is conservative.
- Fit plans against the VRAM free when it runs. Under llama-swap it runs inside
`common_init_from_params`, after `vram_acquire_for_load()` took the GPU token, so the previous
holder has released its VRAM; a process outside the arbiter still counts against it.
How to use it for Inkling (production command line `inkling_args` in `/root/config.yaml`):
- Remove `-ngl 99` and the `-ot blk[.](...)[.]ffn_.*_exps=CPU` pattern (either makes fit abort with
"already set by user"), keep `--ctx-size 65536` (a user context is not changed by fit), and add
`--fit on` (the default, but explicit) and `-lv 4` (fit's log and the memory breakdown are info
level). Optionally `--fit-target 512` to give back part of the 1024 MiB margin (the mmproj
estimate is still added).
- Dry run without serving: `build/bin/llama-fit-params` with the same arguments prints the chosen
`-c ... -ngl ... -ot "..."` line on stdout. Compare its `-ot` list with production's k=4 (CPU
experts for layers 0-5 and 10-23, GPU for 6-9).
- A/B in the window (GPU to itself): the k=4 `-ot` line against the `--fit` line, with F13's
drivers (`/root/arms/overnight/sweep.sh` for llama-bench pp2048 @ d8192/d16384 and tg256 @
d4096; give llama-bench the explicit `-ot` that fit-params printed rather than its own `-fitt`,
which is off by default and fits for the bench context, not 65536), plus the process VRAM/GTT after load and the `-lv 4` breakdown (the free column must keep
about 0.8 GiB for the mmproj).
Measured on the RX 580 (2026-11-21, the previous PR head: `1fa93f902`, `d29f9d489`, `ff91e4f84`,
without `808fd1c4e`; base `3bf6b55b3` vs base + both PRs; logs `/root/arms/pr-out/` on the pod):
- Inkling pp2048 at ub1024: 91.6 -> 110.5 t/s. pp2048 at depth 16384 (ub512): 48.4 -> 73.0 t/s.
@@ -1437,35 +1361,6 @@ only for the first split; not investigated.
- Model saving is not supported for `inkling2_loop` (`llama_arch_supports_save`), so duplicate data
in a saved file cannot happen.
**Upstream #32807 (in `5bacb7c86`, branch `-20261122`).** Upstream's fix for #32755: in
`sched_copy_experts`, after the first-node check and before F1's logic, `if (arch ==
LLM_ARCH_INKLING2_LOOP) return false;`, so the scheduler copies every expert of a shared host
weight (a TODO points at per-pass copies; the scheduler-side alternative #32809, "copy host experts
again for each split that reads them", was closed unmerged). That fixes correctness but keeps one
device copy per (host tensor, backend) per graph, alive until the last pass: the 6.5 GiB of held
copies and the ~7 GiB compute buffer stay. So F13 is still needed for memory; for correctness
it is now redundant with upstream's guard.
- With both, each pass uploads its own copy (F13) and every copy is a whole tensor (the guard):
in the 32-63-token window the readback path, which would upload only the selected experts, no
longer runs. Ubatches of 64+ tokens are unchanged (F1 copies every expert anyway), below 32 the
experts run on the CPU.
- Knob (fork, commit "env knob LLAMA_LOOP_FULL_COPY=0"): `LLAMA_LOOP_FULL_COPY=0` skips upstream's
guard. Only correct with the views on: with `LLAMA_LOOP_HOST_VIEWS=0` it brings #32755 back.
- On this branch, "upstream's fix alone" is `LLAMA_LOOP_HOST_VIEWS=0` (shared tensors plus the
guard), "F13 plus upstream" is the default, "F13 alone" is `LLAMA_LOOP_FULL_COPY=0`.
Window A/B to run (GPU to itself, candidate build, Inkling production command line):
1. Memory: process VRAM/GTT after load and the `-lv 4` breakdown at ctx 65536 with production's
k=4 `-ot`, default vs `LLAMA_LOOP_HOST_VIEWS=0`. Expected: upstream-only spills (compute about
7 GiB, as before F13); F13 fits with about 0.8 GiB free.
2. Correctness: `/root/arms/overnight/kld48.sh` (48-token ubatches, KLD against the knob-64 base)
for default, `LLAMA_LOOP_HOST_VIEWS=0` and `LLAMA_LOOP_FULL_COPY=0`. Expected: all three at the
GPU-vs-CPU noise level (mean KLD about 0.003); before #32807 views off gave 0.032.
3. Speed: llama-bench pp2048 @ d8192 and tg256 @ d4096 (`sweep.sh` settings) and a 48-token-tail
server prompt (the window where the guard costs uploads), default vs `LLAMA_LOOP_FULL_COPY=0`,
arms interleaved in one invocation (section 6). Keep the guard unless the knob wins clearly.
Then decide whether F13 stays (expected: yes, for memory).
**How to drop it.** If upstream fixes #32755 (for example per-reader copies or a copy-callback call for
every reader in the scheduler, or real per-pass tensors in the loader), revert 312c29c2b; the hook has
no other user. Check that upstream's fix also bounds the copy lifetime, otherwise the 6.5 GiB comes back.
@@ -1527,8 +1422,7 @@ which splits off the system prompt. Tool results count as user messages (Qwen op
**Where.** `tools/server/server-context.cpp`, `update_slots`, the prompt-batch loop:
`merge_user_starts` before the loop and the `!merge_user_starts` term of the user-start break.
Commit 510aa6e09. On branch `-20261122` the refinement adds `first_user_pos` before the loop, and the
user-start break checks `merge_user_starts` first (merged: break only at `first_user_pos`).
Commit 510aa6e09.
**How it works.** When the rest of the prompt from the current batch start fits in one ubatch
(`task.n_tokens() - prompt.n_tokens() <= n_ubatch`) and no media chunk was processed in this
@@ -1543,38 +1437,6 @@ whose rest is longer than one ubatch keep the upstream breaks (there the break d
whole of a ubatch, and the fallback would not be bounded). `LLAMA_USERSTART_MERGE=0` restores the
upstream behaviour everywhere.
**Refinement: the system-prompt checkpoint (owner's wish, commit "F14 - keep the batch break at the
first user message", branch `-20261122`, not deployed).** The merged path keeps one user-start
break: the one at the first user message, which is the end of the system prompt. Upstream breaks
there on a fresh prompt (no checkpoints yet), and the plain F14 had dropped it, so a new chat with
the same system prompt found no checkpoint at the system-prompt end and re-processed the system
prompt (a hybrid/recurrent state cannot be cut back without one). Now, merged or not, a prompt
processed from before its first user message ends a batch there; the next batch starts at the
user message and gets a checkpoint (user-start batch, near-end rule). A later new chat with the same
system prompt diverges inside the first user message, restores that checkpoint and processes only
its own user message and the generation prompt.
- Only fresh prompts are affected: checkpoints past the restore point are erased on reuse, so the
batch reaches the first user message only when the prompt is processed from before it (a new
chat, or a changed system prompt). Follow-up turns start past it and keep the full merge,
including no break at the last user message (the edit cost below is unchanged).
- First user messages below 8 tokens are skipped (no system prompt, only a BOS before it).
- Cost: turn 1 of every new chat runs the system prompt as its own batch again. That was part of
the U arm's turn-1 layout below (TTFT 6.08 vs 3.99 s for M, which also merged the last-user
break; the system-prompt split alone was not measured). A system prompt of 32+ tokens runs on the
GPU with one extra expert upload, below 32 tokens on the CPU.
- Lifetime of that checkpoint: it is the oldest one of the slot. When the list is full
(`--ctx-checkpoints`, default 32), the min-step thinning keeps the oldest checkpoint and erases
later ones within `--checkpoint-min-step` (8192) of it, and only then the FIFO drops the oldest. A
very long chat can still lose it; the RAM prompt cache (and F11 on disk) keeps the earlier
conversation's copy.
- Disk cache (F11), read: `disk_cache_load` replaces the slot prompt only with an entry that
reuses at least `DISK_CACHE_MIN_TOKENS` (256) more of the new prompt, and `n_reuse` counts a
saved prompt that diverges inside it as its newest checkpoint at or below the divergence. A saved
chat therefore offers the system-prompt checkpoint to a new chat after a restart, but only for a
system prompt of 256+ tokens; it never replaces a better slot prompt. The checkpoints are saved in
the file appendix and trimmed to `--ctx-checkpoints` from the front on load (only if the file has
more). Nothing in it removes the break; check it in the window anyway (validation plan item 5).
**Invariants that must hold.**
- The merge only applies when the rest fits in one ubatch: that bounds the extra re-processing on a
divergence inside the last user message to less than `n_ubatch` tokens.
@@ -1671,24 +1533,6 @@ measured above.
restore after a restart, `prompt_n` and output identical to the reference built from the same
commit.
Additions for the refinement (branch `-20261122`, to run in the window with the 35B think entry's
exact command line plus `-lv 4`, scratch port, arena and cache dir):
5. New chat #2 with the same system prompt. Chat 1: a system prompt of about 300 tokens (above the
256-token disk-cache minimum), then 3 turns as in item 2. Expected at trace level on chat 1 turn
1: a batch `[0, first user start)`, then a `created context checkpoint ... n_tokens = <first
user start>`, then the merged batch to end minus 4. Then chat 2: the same system prompt, a new
first user message. Expected: `restored context checkpoint ... n_tokens = <first user start>`
and `timings.prompt_n` = chat 2's user message plus the generation prompt only (not the system
prompt); no `forcing full prompt re-processing`. Follow-up turns of chat 1 must still run as one
batch (no break at the last user message): the M layout of the table above.
6. The same after a restart (disk cache): run chat 1, stop the server with SIGTERM (`disk cache:
saved the prompt`), start it again, send chat 2. Expected: `disk cache: restored`, then the
restore of the system-prompt checkpoint and the same `prompt_n` as in item 5. Also once with a
system prompt below 256 tokens: no disk restore is expected there (the minimum), only a fresh
prompt; that is the known limit, not a failure.
7. The 6-turn TTFT table of item 2 with the candidate, M only: turn 1 may be slower than 3.99 s
(the system-prompt split is back), turns 2-6 must match the M column within noise.
---
## 4. Dead ends
+5 -5
View File
@@ -326,7 +326,7 @@ static bool parse_bool_value(const std::string & value) {
// common_models_handler
//
std::string common_get_default_local_path(const std::string & url) {
static std::string get_default_local_path(const std::string & url) {
auto f = string_split<std::string>(url, '#').front();
f = string_split<std::string>(f, '?').front();
return fs_get_cache_file(string_split<std::string>(f, '/').back()).string();
@@ -436,7 +436,7 @@ static std::vector<common_download_task> build_url_tasks(const common_params_mod
if (parts.size() == 1) {
common_download_task task;
task.url = parts[0];
task.local_path = model.path.empty() ? common_get_default_local_path(parts[0]) : model.path;
task.local_path = model.path.empty() ? get_default_local_path(parts[0]) : model.path;
task.opts = opts;
tasks.push_back(std::move(task));
return tasks;
@@ -454,7 +454,7 @@ static std::vector<common_download_task> build_url_tasks(const common_params_mod
task.url = part;
task.opts = opts;
std::string local = common_get_default_local_path(part);
std::string local = get_default_local_path(part);
if (!base_dir.empty()) {
auto pos = local.rfind('/');
std::string name = pos == std::string::npos ? local : local.substr(pos + 1);
@@ -487,7 +487,7 @@ void common_models_handler_apply(common_models_handler & handler, common_params
auto handle_url = [&](common_params_model & model) {
if (!model.url.empty()) {
if (model.path.empty()) {
model.path = common_get_default_local_path(model.url);
model.path = get_default_local_path(model.url);
}
}
};
@@ -498,7 +498,7 @@ void common_models_handler_apply(common_models_handler & handler, common_params
// optionally, if docker repo is set, resolve it
if (!params.model.docker_repo.empty()) {
params.model.url = common_docker_resolve_model(params.model.docker_repo);
params.model.path = common_get_default_local_path(params.model.url);
params.model.path = get_default_local_path(params.model.url);
}
// handle plain "url" tasks (non-hf)
-3
View File
@@ -149,9 +149,6 @@ common_models_handler common_models_handler_init(const common_params & params, l
// check if the model is a preset repo (i.e. has a preset file)
bool common_models_handler_is_preset_repo(const common_models_handler & handler);
// the cache path a model downloaded from a plain url is stored at
std::string common_get_default_local_path(const std::string & url);
// download and update params with the downloaded model path
void common_models_handler_apply(common_models_handler & handler, common_params & params, common_download_callback * callback = nullptr);
+8 -24
View File
@@ -69,21 +69,9 @@ common_chat_params peg_generator::generate_parser(const common_chat_template &
std::string trigger_marker = !autoparser.tools.format.section_start.empty() ? autoparser.tools.format.section_start :
autoparser.tools.format.per_call_start;
// bare JSON calls without a start marker (e.g. Llama 3.x): trigger the lazy grammar on the call's opening instead
std::string trigger_pattern;
{
const auto & fmt = autoparser.tools.format;
bool top_level_name = (fmt.function_field.empty() || fmt.function_field == "function") &&
fmt.name_field.find('.') == std::string::npos;
if (trigger_marker.empty() && fmt.mode == tool_format::JSON_NATIVE && !fmt.fun_name_is_key &&
!fmt.tools_array_wrapped && top_level_name) {
trigger_pattern = "\\{\\s*\"" + regex_escape(fmt.name_field) + "\"\\s*:";
}
}
bool has_response_format = !inputs.json_schema.empty() && inputs.json_schema.is_object();
bool include_grammar = has_response_format || (has_tools &&
((inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO && (!trigger_marker.empty() || !trigger_pattern.empty())) ||
((inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO && !trigger_marker.empty()) ||
inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
if (include_grammar) {
@@ -94,11 +82,9 @@ common_chat_params peg_generator::generate_parser(const common_chat_template &
// Set grammar triggers based on tool section markers (fall back to per-call markers)
if (data.grammar_lazy) {
if (!trigger_marker.empty()) {
data.grammar_triggers = { { COMMON_GRAMMAR_TRIGGER_TYPE_WORD, trigger_marker } };
} else {
data.grammar_triggers = { { COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN, trigger_pattern } };
}
data.grammar_triggers = {
{ COMMON_GRAMMAR_TRIGGER_TYPE_WORD, trigger_marker }
};
if (autoparser.tools.format.openai_wrapper_trigger) {
// model emits the OpenAI function wrapper, trigger on it
data.grammar_triggers.push_back({ COMMON_GRAMMAR_TRIGGER_TYPE_WORD, "{\"type\": \"function\"," });
@@ -373,9 +359,7 @@ common_peg_parser analyze_tools::build_tool_parser_tag_tagged(parser_build_conte
auto & p = ctx.p;
const auto & inputs = ctx.inputs;
// A raw string value needs a suffix to end on, without one it is parsed as JSON
bool raw_strings = !arguments.value_suffix.empty();
auto until_suffix = raw_strings ? p.rule("until-suffix", p.until(arguments.value_suffix)) : p.eps();
auto until_suffix = p.rule("until-suffix", p.until(arguments.value_suffix));
common_peg_parser tool_choice = p.choice();
@@ -391,9 +375,9 @@ common_peg_parser analyze_tools::build_tool_parser_tag_tagged(parser_build_conte
p.tool_arg(p.tool_arg_open(arguments.name_prefix + p.tool_arg_name(p.literal(param.name)) +
arguments.name_suffix) +
arguments.value_prefix +
(raw_strings && param.schema->may_be_string() ?
p.through(p.tool_arg_string_value(until_suffix),
p.tool_arg_close(p.literal(arguments.value_suffix))) :
(param.schema->may_be_string() ?
p.ac(p.tool_arg_string_value(until_suffix) +
p.tool_arg_close(p.literal(arguments.value_suffix)), arguments.value_suffix) :
(p.tool_arg_json_value(p.schema(
p.json(), "tool-" + std::to_string(tool_index) + "-arg-" + std::to_string(param_index) + "-schema", doc, *param.schema)) +
p.tool_arg_close(p.literal(arguments.value_suffix)))));
-1
View File
@@ -71,7 +71,6 @@ struct generation_params {
bool is_inference = true;
bool add_inference = false;
bool mark_input = true; // whether to mark input strings in the jinja context
common_peg_tokens specials;
bool has_continuation() const {
return continue_final_message != COMMON_CHAT_CONTINUATION_NONE && !continue_msg.empty();
-11
View File
@@ -71,9 +71,6 @@ constexpr size_t COMMON_CHAT_MAX_PERMUTE = 6;
class common_chat_peg_builder : public common_peg_parser_builder {
public:
common_chat_peg_builder() = default;
explicit common_chat_peg_builder(common_peg_tokens tokens) : common_peg_parser_builder(std::move(tokens)) {}
// Tag constants (from former common_chat_peg_base_builder)
static constexpr const char * REASONING_BLOCK = "reasoning-block";
static constexpr const char * REASONING = "reasoning";
@@ -203,14 +200,6 @@ inline common_peg_arena build_chat_peg_parser(
return builder.build();
}
inline common_peg_arena build_chat_peg_parser(
const common_peg_tokens & specials,
const std::function<common_peg_parser(common_chat_peg_builder & builder)> & fn) {
common_chat_peg_builder builder(specials);
builder.set_root(fn(builder));
return builder.build();
}
class tag_based_peg_mapper {
public:
std::map<std::string, std::string> tags;
+2 -18
View File
@@ -315,7 +315,6 @@ struct common_chat_templates {
bool add_bos;
bool add_eos;
bool has_explicit_template; // Model had builtin template or template overridden was specified.
common_peg_tokens specials;
std::unique_ptr<common_chat_template> template_default; // always set (defaults to chatml)
std::unique_ptr<common_chat_template> template_tool_use;
std::unique_ptr<common_chat_template> template_tts; // the prompt of a text to speak
@@ -803,10 +802,6 @@ void common_chat_templates_free(struct common_chat_templates * tmpls) {
delete tmpls;
}
void common_chat_templates_set_specials(struct common_chat_templates * tmpls, common_peg_tokens specials) {
tmpls->specials = std::move(specials);
}
bool common_chat_templates_was_explicit(const struct common_chat_templates * tmpls) {
return tmpls->has_explicit_template;
}
@@ -983,7 +978,6 @@ common_chat_templates_ptr common_chat_templates_init(const struct llama_model *
std::string token_eos = eos_token_override;
bool add_bos = false;
bool add_eos = false;
common_peg_tokens specials;
if (model) {
const auto * vocab = llama_model_get_vocab(model);
const auto get_token = [&](llama_token token, const char * name, const char * jinja_variable_name) {
@@ -1003,13 +997,11 @@ common_chat_templates_ptr common_chat_templates_init(const struct llama_model *
token_eos = get_token(llama_vocab_eos(vocab), "EOS", "eos_token");
add_bos = llama_vocab_get_add_bos(vocab);
add_eos = llama_vocab_get_add_eos(vocab);
specials = common_peg_tokens(vocab);
}
common_chat_templates_ptr tmpls(new common_chat_templates());
tmpls->has_explicit_template = has_explicit_template;
tmpls->add_bos = add_bos;
tmpls->add_eos = add_eos;
tmpls->specials = std::move(specials);
try {
tmpls->template_default = std::make_unique<common_chat_template>(default_template_src, token_bos, token_eos);
} catch (const std::exception & e) {
@@ -1551,7 +1543,6 @@ static common_chat_params common_chat_templates_apply_jinja(const struct common_
params.add_generation_prompt = inputs.add_generation_prompt;
params.add_bos = tmpls->add_bos;
params.add_eos = tmpls->add_eos;
params.specials = tmpls->specials;
params.continue_final_message = inputs.continue_final_message;
if (params.continue_final_message != COMMON_CHAT_CONTINUATION_NONE) {
@@ -1834,7 +1825,6 @@ common_chat_input common_chat_input_tokenize(const llama_vocab * vocab, const st
}
if (input.text != text) {
// the pieces do not give back the same text, keep the text without tokens
LOG_WRN("%s: token pieces do not match the text, parsing without token ids\n", __func__);
return common_chat_input(text);
}
return input;
@@ -1945,17 +1935,11 @@ common_chat_session::common_chat_session(const common_chat_templates * tm
thinking_ends = std::move(applied.thinking_end_tags);
parser_params.format = applied.format;
parser_params.generation_prompt = vocab ? common_chat_input_tokenize(vocab, applied.generation_prompt)
: common_chat_input(applied.generation_prompt);
parser_params.debug = params.debug;
parser_params.parser = std::move(applied.parser);
if (vocab) {
parser_params.generation_prompt = common_chat_input_tokenize(vocab, applied.generation_prompt);
} else if (params.tokenize) {
parser_params.generation_prompt = params.tokenize(applied.generation_prompt);
} else {
parser_params.generation_prompt = common_chat_input(applied.generation_prompt);
}
delimiters = std::move(applied.message_delimiters);
if (vocab) {
-6
View File
@@ -372,9 +372,6 @@ common_chat_templates_ptr common_chat_templates_init(const struct llama_model *
const std::string & bos_token_override = "",
const std::string & eos_token_override = "");
// override special tokens for testing
void common_chat_templates_set_specials(struct common_chat_templates * tmpls, common_peg_tokens specials);
bool common_chat_templates_was_explicit(const struct common_chat_templates * tmpls);
std::string common_chat_templates_source(const struct common_chat_templates * tmpls, const std::string & variant = "");
@@ -400,9 +397,6 @@ common_chat_msg common_chat_peg_parse(const common_peg_arena & src_parser, const
struct common_chat_session_params {
bool echo = false; // include the assistant prefill in the output when continuing a message
bool debug = false; // enable debug output for the PEG parser
// tokenize the generation prompt when there is no vocab for testing
std::function<common_chat_input(const std::string &)> tokenize;
};
class common_chat_session {
+6 -20
View File
@@ -98,9 +98,6 @@ class ProgressBar : public common_download_callback {
std::string filename;
size_t len = 0;
// the caller's callback, if any, still receives every event
common_download_callback * next;
static void cleanup(const ProgressBar * line) {
lines.erase(line);
if (lines.empty()) {
@@ -113,12 +110,9 @@ class ProgressBar : public common_download_callback {
}
public:
ProgressBar(common_download_callback * next) : next(next) {}
ProgressBar() = default;
void on_start(const common_download_progress & p) override {
if (next) {
next->on_start(p);
}
filename = url_get_filename(p.url);
for (size_t i = 0; i < filename.size(); ++i) {
@@ -132,22 +126,12 @@ public:
}
}
void on_done(const common_download_progress & p, bool ok) override {
if (next) {
next->on_done(p, ok);
}
void on_done(const common_download_progress &, bool) override {
std::lock_guard<std::mutex> lock(mutex);
cleanup(this);
}
bool is_cancelled() const override {
return next && next->is_cancelled();
}
void on_update(const common_download_progress & p) override {
if (next) {
next->on_update(p);
}
if (!p.total || !is_output_a_tty()) {
return;
}
@@ -535,9 +519,11 @@ int common_download_file_single(const std::string & url,
const std::string & lock_path,
const std::string & alt_path) {
if (!opts.offline) {
ProgressBar tty_cb(opts.callback);
ProgressBar tty_cb;
common_download_opts online_opts = opts;
online_opts.callback = &tty_cb;
if (!online_opts.callback) {
online_opts.callback = &tty_cb;
}
return common_download_file_single_online(url, path, online_opts, skip_etag, lock_path, alt_path);
}
+16 -144
View File
@@ -7,7 +7,6 @@
#include "../src/llama-ext.h"
#include <algorithm>
#include <array>
#include <cassert>
#include <stdexcept>
@@ -65,8 +64,7 @@ static std::vector<llama_device_memory_data> common_get_device_memory_data_impl(
uint32_t & hp_ngl_skip,
uint32_t & hp_n_ctx_train,
uint32_t & hp_n_expert,
ggml_log_level log_level,
std::vector<uint32_t> * hp_layer_visits = nullptr) {
ggml_log_level log_level) {
common_fit_log_scope log_scope(log_level);
llama_model_params mparams_copy = *mparams;
@@ -159,9 +157,6 @@ static std::vector<llama_device_memory_data> common_get_device_memory_data_impl(
hp_ngl_skip = mparams->load_mtp ? 0 : llama_model_n_layer_nextn(model);
hp_n_ctx_train = llama_model_n_ctx_train(model);
hp_n_expert = llama_model_n_expert(model);
if (hp_layer_visits != nullptr) {
*hp_layer_visits = llama_model_layer_visits(model);
}
common_memory_breakdown_print(ctx);
@@ -282,7 +277,6 @@ static void common_params_fit_impl(
uint32_t hp_nsk = 0; // number of skipped NextN layers, counted by the model loader but not by hp_ngl
uint32_t hp_nct = 0; // hparams.n_ctx_train
uint32_t hp_nex = 0; // hparams.n_expert
std::vector<uint32_t> hp_layer_visits;
// with non-unified kv, we need to take into account n_streams
// for example, if memory can hold more than model's trained context size, we must extend the n_ctx to hold enough n_streams
@@ -354,7 +348,7 @@ static void common_params_fit_impl(
// step 1: get data for default parameters and check whether any changes are necessary in the first place
LOG_TRC("%s: getting device memory data for initial parameters:\n", __func__);
dmds_t dmds_full = common_get_device_memory_data_impl(paths_model, mparams, cparams, devs, hp_ngl, hp_nsk, hp_nct, hp_nex, log_level, &hp_layer_visits);
dmds_t dmds_full = common_get_device_memory_data_impl(paths_model, mparams, cparams, devs, hp_ngl, hp_nsk, hp_nct, hp_nex, log_level);
// saturate instead of overflowing, this also preserves the UINT32_MAX sentinel of n_ctx_min:
const uint32_t n_ctx_max = (uint32_t) std::min<uint64_t>(uint64_t(hp_nct) * n_streams, UINT32_MAX);
@@ -637,42 +631,6 @@ static void common_params_fit_impl(
const size_t ntbo = llama_max_tensor_buft_overrides();
const bool has_looped_layers = std::any_of(
hp_layer_visits.begin(), hp_layer_visits.end(),
[](uint32_t v) { return v != 1; });
// For looped architectures (e.g. inkling2_loop), logical layers share physical weights across passes
// and extra KV slots have visits == 0. Rank physical layers by descending visit count so hot looped
// MoE layers stay in device memory before single-pass prelude/coda layers.
auto get_device_phys_layers = [&](const uint32_t dev_il0, const uint32_t dev_il1) -> std::vector<uint32_t> {
std::vector<uint32_t> phys;
const uint32_t limit = std::min(dev_il1, (uint32_t) hp_layer_visits.size());
for (uint32_t il = dev_il0; il < limit; ++il) {
if (hp_layer_visits[il] > 0) {
phys.push_back(il);
}
}
std::stable_sort(phys.begin(), phys.end(), [&](uint32_t a, uint32_t b) {
return hp_layer_visits[a] > hp_layer_visits[b];
});
return phys;
};
auto device_layer_slice = [&](const std::vector<ngl_t> & cfg, const size_t target_id) -> std::pair<uint32_t, uint32_t> {
uint32_t total_ngl = 0;
for (size_t id = 0; id < nd; ++id) {
total_ngl += cfg[id].n_layer;
}
uint32_t cur_il0 = hp_ngl + 1 - total_ngl;
for (size_t id = 0; id < target_id; ++id) {
cur_il0 += cfg[id].n_layer;
}
const uint32_t dev_n_tf = (target_id == nd - 1 && cfg[target_id].n_layer > 0)
? cfg[target_id].n_layer - 1
: cfg[target_id].n_layer;
return { cur_il0, cur_il0 + dev_n_tf };
};
// utility function to set n_gpu_layers and tensor_split
auto set_ngl_tensor_split_tbo = [&](
const std::vector<ngl_t> & ngl_per_device,
@@ -705,45 +663,22 @@ static void common_params_fit_impl(
mparams.tensor_split = tensor_split;
size_t itbo = 0;
auto push_tbo = [&](const char * pattern, ggml_backend_buffer_type_t buft) {
if (itbo + 1 >= ntbo) {
tensor_buft_overrides[itbo].pattern = nullptr;
tensor_buft_overrides[itbo].buft = nullptr;
itbo++;
mparams.tensor_buft_overrides = tensor_buft_overrides;
throw common_params_fit_exception("llama_max_tensor_buft_overrides() == "
+ std::to_string(ntbo) + " is insufficient for model");
}
tensor_buft_overrides[itbo].pattern = pattern;
tensor_buft_overrides[itbo].buft = buft;
itbo++;
};
for (size_t id = 0; id < nd; id++) {
if (!has_looped_layers) {
il0 += ngl_per_device[id].n_full();
for (uint32_t il = il0; il < il0 + ngl_per_device[id].n_part; il++) {
push_tbo(
get_overflow_pattern(il, il == il0 ? ngl_per_device[id].overflow_type : LAYER_FRACTION_MOE),
il == il0 ? overflow_bufts[id] : ggml_backend_cpu_buffer_type());
il0 += ngl_per_device[id].n_full();
for (uint32_t il = il0; il < il0 + ngl_per_device[id].n_part; il++) {
if (itbo + 1 >= ntbo) {
tensor_buft_overrides[itbo].pattern = nullptr;
tensor_buft_overrides[itbo].buft = nullptr;
itbo++;
mparams.tensor_buft_overrides = tensor_buft_overrides;
throw common_params_fit_exception("llama_max_tensor_buft_overrides() == "
+ std::to_string(ntbo) + " is insufficient for model");
}
il0 += ngl_per_device[id].n_part;
continue;
tensor_buft_overrides[itbo].pattern = get_overflow_pattern(il, il == il0 ? ngl_per_device[id].overflow_type : LAYER_FRACTION_MOE);
tensor_buft_overrides[itbo].buft = il == il0 ? overflow_bufts[id] : ggml_backend_cpu_buffer_type();
itbo++;
}
const uint32_t dev_n_tf = (id == nd - 1 && ngl_per_device[id].n_layer > 0)
? ngl_per_device[id].n_layer - 1
: ngl_per_device[id].n_layer;
const std::vector<uint32_t> phys = get_device_phys_layers(il0, il0 + dev_n_tf);
const uint32_t n_part_tf = std::min(dev_n_tf, ngl_per_device[id].n_part);
const uint32_t n_full_tf = dev_n_tf - n_part_tf;
for (size_t i = n_full_tf; i < phys.size(); ++i) {
const bool is_first = i == n_full_tf;
push_tbo(
get_overflow_pattern(phys[i], is_first ? ngl_per_device[id].overflow_type : LAYER_FRACTION_MOE),
is_first ? overflow_bufts[id] : ggml_backend_cpu_buffer_type());
}
il0 += ngl_per_device[id].n_layer;
il0 += ngl_per_device[id].n_part;
}
tensor_buft_overrides[itbo].pattern = nullptr;
tensor_buft_overrides[itbo].buft = nullptr;
@@ -780,7 +715,6 @@ static void common_params_fit_impl(
};
int64_t global_surplus_cpu_moe = 0;
bool keep_looped_moe_in_step3 = false;
if (hp_nex > 0) {
const static std::string pattern_moe_all = "blk\\.\\d+\\.ffn_(up|down|gate_up|gate)_(ch|)exps"; // matches all MoE tensors
ggml_backend_buffer_type_t cpu_buft = ggml_backend_cpu_buffer_type();
@@ -806,43 +740,6 @@ static void common_params_fit_impl(
__func__, -global_surplus_cpu_moe/MiB);
}
// For looped MoE models (e.g. inkling2_loop), CPU-offloading repeated core layers can inflate the
// compute buffer enough that keeping looped experts on GPU uses less total VRAM than offloading all MoEs.
if (has_looped_layers) {
size_t itbo = 0;
bool overrides_fit = true;
for (uint32_t il = 0; il < hp_layer_visits.size(); ++il) {
if (hp_layer_visits[il] == 1) {
if (itbo + 1 >= ntbo) {
overrides_fit = false;
break;
}
tensor_buft_overrides[itbo++] = {get_overflow_pattern(il, LAYER_FRACTION_MOE), cpu_buft};
}
}
if (overrides_fit && itbo > 0) {
tensor_buft_overrides[itbo] = {nullptr, nullptr};
mparams->tensor_buft_overrides = tensor_buft_overrides;
LOG_TRC("%s: getting device memory data with only non-looped MoE tensors moved to system memory:\n", __func__);
dmds_t dmds_looped_gpu = common_get_device_memory_data_impl(
paths_model, mparams, cparams, devs, hp_ngl, hp_nsk, hp_nct, hp_nex, log_level);
add_extra_memory(dmds_looped_gpu);
int64_t surplus_looped_gpu = 0;
for (size_t id = 0; id < nd; id++) {
surplus_looped_gpu += dmds_looped_gpu[id].free;
surplus_looped_gpu -= int64_t(dmds_looped_gpu[id].mb.total()) + margins[id];
}
if (surplus_looped_gpu > 0 && surplus_looped_gpu >= global_surplus_cpu_moe) {
LOG_TRC("%s: keeping looped MoE weights in device memory yields a larger surplus of %" PRId64 " MiB\n",
__func__, surplus_looped_gpu/MiB);
global_surplus_cpu_moe = surplus_looped_gpu;
keep_looped_moe_in_step3 = true;
}
}
}
// reset
tensor_buft_overrides[0] = {nullptr, nullptr};
mparams->tensor_buft_overrides = tensor_buft_overrides;
@@ -887,17 +784,6 @@ static void common_params_fit_impl(
ngl_per_device_high[id].n_layer = n_unassigned;
if (hp_nex > 0) {
ngl_per_device_high[id].n_part = size_t(id) < nd - 1 ? ngl_per_device_high[id].n_layer : ngl_per_device_high[id].n_layer - 1;
if (keep_looped_moe_in_step3) {
const auto [dev_il0, dev_il1] = device_layer_slice(ngl_per_device_high, id);
uint32_t n_hot = 0;
const uint32_t limit = std::min(dev_il1, (uint32_t) hp_layer_visits.size());
for (uint32_t il = dev_il0; il < limit; ++il) {
if (hp_layer_visits[il] > 1) {
n_hot++;
}
}
ngl_per_device_high[id].n_part -= std::min(ngl_per_device_high[id].n_part, n_hot);
}
}
if (ngl_per_device_high[id].n_layer > 0) {
std::vector<int64_t> mem_high = get_memory_for_layers(__func__, ngl_per_device_high, overflow_bufts);
@@ -972,12 +858,6 @@ static void common_params_fit_impl(
ngl_per_device_high[jd].n_part = 0;
}
size_t id_dense_start_high = nd - 1;
if (has_looped_layers && id == id_dense_start && ngl_per_device_high[id].n_layer == ngl_per_device[id].n_layer) {
const auto [dev_il0, dev_il1] = device_layer_slice(ngl_per_device_high, id);
const uint32_t n_phys_dev = (uint32_t) get_device_phys_layers(dev_il0, dev_il1).size();
const uint32_t dev_n_tf = dev_il1 - dev_il0;
ngl_per_device_high[id].n_part = dev_n_tf - std::min(dev_n_tf, n_phys_dev);
}
std::vector<int64_t> mem_high = get_memory_for_layers(__func__, ngl_per_device_high, overflow_bufts);
if (mem_high[id] > targets[id]) {
@@ -1029,15 +909,7 @@ static void common_params_fit_impl(
}
// try to fit at least part of one more layer
bool can_fit_partial = ngl_per_device[id_dense_start].n_layer > (id < nd - 1 ? 0 : 1);
if (can_fit_partial && has_looped_layers && id == id_dense_start) {
const auto [dev_il0, dev_il1] = device_layer_slice(ngl_per_device, id);
const uint32_t n_phys_dev = (uint32_t) get_device_phys_layers(dev_il0, dev_il1).size();
const uint32_t dev_n_tf = dev_il1 - dev_il0;
const uint32_t n_full_tf = dev_n_tf - std::min(dev_n_tf, ngl_per_device[id].n_part);
can_fit_partial = n_full_tf < n_phys_dev;
}
if (can_fit_partial) {
if (ngl_per_device[id_dense_start].n_layer > (id < nd - 1 ? 0 : 1)) {
std::vector<ngl_t> ngl_per_device_test = ngl_per_device;
size_t id_dense_start_test = id_dense_start;
ngl_per_device_test[id_dense_start_test].n_layer--;
+2 -5
View File
@@ -593,7 +593,7 @@ caps caps_get(jinja::program & prog) {
if (result.supports_reasoning_effort) {
JJ_DEBUG("%s\n", ">>> Running capability check: reasoning effort levels");
static const std::vector<std::string> candidates = { "none", "minimal", "low", "medium", "high", "xhigh", "max" };
static const std::vector<std::string> candidates = { "minimal", "low", "medium", "high", "xhigh", "max" };
// renders a single user turn with the given effort, nothing when the template raises
const auto render = [&](const std::string & effort) {
@@ -631,10 +631,7 @@ caps caps_get(jinja::program & prog) {
for (const auto & level : candidates) {
const auto rendered = render(level);
if (!rendered) {
// "none" also turns thinking off, so the server does not answer it with an error
if (level != "none") {
rejected.push_back(level);
}
rejected.push_back(level);
continue;
}
if (rendered == unknown) {
+2 -2
View File
@@ -676,8 +676,8 @@ private:
const auto & node = trie.nodes[idx];
std::string rejects;
auto first = true;
for (const auto & [sym, child] : node.children) {
std::string c = common_unicode_cpt_to_utf8(sym.value);
for (const auto & [cpt, child] : node.children) {
std::string c = common_unicode_cpt_to_utf8(cpt);
rejects += c;
if (first) {
first = false;
+26 -29
View File
@@ -24,9 +24,7 @@ struct depth_guard {
// Gemma 4 value parsers
void gemma4_value_rules(common_chat_peg_builder & p) {
// The tags are what gemma4_to_json() in the mapper reads, the rules may wrap them freely.
auto string_delim = p.token("<|\"|>");
p.rule("gemma4-string", p.tag(gemma4::STRING, string_delim + p.tag(gemma4::STRING_CONTENT, p.until(string_delim)) + string_delim));
p.rule("gemma4-string", p.tag(gemma4::STRING, p.literal("<|\"|>") + p.tag(gemma4::STRING_CONTENT, p.until("<|\"|>")) + p.literal("<|\"|>")));
p.rule("gemma4-bool", p.tag(gemma4::BOOL, p.json_bool()));
@@ -36,7 +34,7 @@ void gemma4_value_rules(common_chat_peg_builder & p) {
// Keys are bare in the template, but Gemma 4 occasionally wraps them in quotes
p.rule("gemma4-dict-key", p.tag(gemma4::DICT_KEY, p.choice({
string_delim + p.tag(gemma4::DICT_KEY_NAME, p.until(string_delim)) + string_delim,
p.literal("<|\"|>") + p.tag(gemma4::DICT_KEY_NAME, p.until("<|\"|>")) + p.literal("<|\"|>"),
p.literal("\"") + p.tag(gemma4::DICT_KEY_NAME, p.chars("[^\"]", 1, -1)) + p.literal("\""),
p.literal("'") + p.tag(gemma4::DICT_KEY_NAME, p.chars("[^']", 1, -1)) + p.literal("'"),
p.tag(gemma4::DICT_KEY_NAME, p.chars("[^:}]", 1, -1)),
@@ -146,7 +144,7 @@ class gemma4_schema_parser {
alts |= name();
}
// Gemma 4 may emit keys in its string delimiters or quotes
alts |= p_.token("<|\"|>") + name() + p_.token("<|\"|>");
alts |= p_.literal("<|\"|>") + name() + p_.literal("<|\"|>");
if (key.find('\'') == std::string::npos) {
alts |= p_.literal("'") + name() + p_.literal("'");
}
@@ -333,23 +331,18 @@ common_chat_params common_chat_params_init_gemma4(const common_chat_template &
auto include_grammar = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
auto extract_reasoning = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
data.parser = build_chat_peg_parser(inputs.specials, [&](common_chat_peg_builder & p) {
auto start = p.rule("start", p.optional(p.token("<|turn>") + p.literal("model\n")));
auto channel_start = p.token("<|channel>");
auto channel_end = p.token("<channel|>");
auto think_start = channel_start + p.literal("thought");
auto tool_call_start = p.token("<|tool_call>");
// the model may start a tool call before closing the thought channel
auto think_end = channel_end | p.peek(tool_call_start);
data.parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
auto start = p.rule("start", p.optional(p.literal("<|turn>model\n")));
if (extract_reasoning) {
p.rule("thought", think_start + p.space() + p.reasoning(p.until(channel_end | tool_call_start)) + think_end);
// the model may start a tool call before closing the thought channel
p.rule("thought", p.literal("<|channel>thought") + p.space() + p.reasoning(p.until_one_of({"<channel|>", "<|tool_call>"})) + (p.literal("<channel|>") | p.peek(p.literal("<|tool_call>"))));
} else {
p.rule("thought", p.content(think_start + p.space() + p.until(channel_end | tool_call_start) + think_end));
p.rule("thought", p.content(p.literal("<|channel>thought") + p.space() + p.until_one_of({"<channel|>", "<|tool_call>"}) + (p.literal("<channel|>") | p.peek(p.literal("<|tool_call>")))));
}
auto consume_empty_channels = p.gbnf(p.zero_or_more(channel_start + p.negate(p.literal("thought"))), "");
auto thought = (p.peek(channel_start) + consume_empty_channels + p.ref("thought")) | p.negate(channel_start);
auto consume_empty_channels = p.gbnf(p.zero_or_more(p.literal("<|channel>") + p.negate(p.literal("thought"))), "");
auto thought = (p.peek(p.literal("<|channel>")) + consume_empty_channels + p.ref("thought")) | p.negate(p.literal("<|channel>"));
if (has_response_format) {
auto response_format = p.literal("```json") <<
@@ -385,35 +378,39 @@ common_chat_params common_chat_params_init_gemma4(const common_chat_template &
})));
});
auto tool_call_rest = p.literal("call:") + tool_choice + p.token("<tool_call|>");
auto tool_call = p.rule("tool-call", tool_call_start + tool_call_rest);
auto more = inputs.parallel_tool_calls ? p.zero_or_more(tool_call) : p.eps();
auto tool_calls = p.trigger_rule("tool-calls", tool_call_start, tool_call_rest + more);
auto tool_call = p.trigger_rule("tool-call", p.repeat(
"<|tool_call>call:" + tool_choice + "<tool_call|>",
/* min = */ inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED ? 1 : 0,
/* max = */ inputs.parallel_tool_calls ? -1 : 1
));
if (inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED) {
return start + thought + tool_calls;
return start + thought + tool_call;
}
auto scan_to_toolcall = p.rule("scan-to-toolcall", p.until(tool_call_start));
auto content = p.rule("content", p.content(p.until(channel_start | channel_end | tool_call_start)));
auto scan_to_toolcall = p.rule("scan-to-toolcall", p.until("<|tool_call>"));
auto content = p.rule("content", p.content(p.until_one_of({"<|channel>", "<channel|>", "<|tool_call>"})));
auto message = p.rule("message", thought + content);
return start + p.zero_or_more(message) + scan_to_toolcall + p.optional(tool_calls);
return start + p.zero_or_more(message) + scan_to_toolcall + tool_call;
}
// Gemma 4 may emit an extra <|channel>thought\n<channel|> at the end of the content. It may
// also emit a single trailing <channel|> token. Consume all complete reasoning blocks and
// then stop at the first unmatched <channel|> token.
auto content = p.rule("content", p.content(p.until(channel_start | channel_end)));
auto content = p.rule("content", p.content(p.until_one_of({"<|channel>", "<channel|>"})));
auto message = p.rule("message", thought + content);
return start + p.one_or_more(message);
});
if (include_grammar) {
data.grammar_lazy = false;
data.grammar_lazy = !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
data.grammar = build_grammar([&](const common_grammar_builder & builder) {
data.parser.build_grammar(builder, !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED)));
data.parser.build_grammar(builder, data.grammar_lazy);
});
data.grammar_triggers = {
{ COMMON_GRAMMAR_TRIGGER_TYPE_WORD, "<|tool_call>" },
};
}
return data;
+30 -31
View File
@@ -65,18 +65,17 @@ common_chat_params common_chat_params_init_gpt_oss(const common_chat_template &
auto include_grammar = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
auto extract_reasoning = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
data.parser = build_chat_peg_parser(inputs.specials, [&](common_chat_peg_builder & p) {
auto start_assistant = p.token("<|start|>") + p.literal("assistant");
auto start = p.rule("start", start_assistant);
auto end = p.rule("end", p.token("<|end|>"));
auto content = p.rule("message-content", p.until(p.token("<|end|>")));
auto channel = p.token("<|channel|>") + (p.literal("commentary") | p.literal("analysis"));
data.parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
auto start = p.rule("start", p.literal("<|start|>assistant"));
auto end = p.rule("end", p.literal("<|end|>"));
auto content = p.rule("message-content", p.until("<|end|>"));
auto channel = p.literal("<|channel|>") + (p.literal("commentary") | p.literal("analysis"));
auto constrain_type = p.chars("[A-Za-z0-9_-]", 1, -1);
auto constraint = p.optional(p.chars("[ ]", 1, -1) + p.optional(p.token("<|constrain|>")) + constrain_type);
auto constraint = p.optional(p.space() + p.optional(p.literal("<|constrain|>")) + constrain_type);
// Occasionally, gpt-oss-20b will prefix channels with this commentary
auto stray_commentary = p.optional(p.token("<|channel|>") + p.literal("commentary") + p.optional(p.literal(" to=assistant")));
auto start_analysis = stray_commentary + p.token("<|channel|>") + p.literal("analysis") + p.token("<|message|>");
auto stray_commentary = p.optional(p.literal("<|channel|>commentary") + p.optional(p.literal(" to=assistant")));
auto start_analysis = stray_commentary + p.literal("<|channel|>analysis<|message|>");
if (extract_reasoning) {
p.rule("analysis", start_analysis + p.reasoning(content) + end);
@@ -85,8 +84,8 @@ common_chat_params common_chat_params_init_gpt_oss(const common_chat_template &
}
auto analysis = p.ref("analysis");
auto preamble = p.rule("preamble", p.token("<|channel|>") + p.literal("commentary") + p.token("<|message|>") + p.content(content) + end);
auto final_msg = p.rule("final", stray_commentary + p.token("<|channel|>") + p.literal("final") + constraint + p.token("<|message|>") + p.content(content));
auto preamble = p.rule("preamble", p.literal("<|channel|>commentary<|message|>") + p.content(content) + end);
auto final_msg = p.rule("final", stray_commentary + p.literal("<|channel|>final") + constraint + p.literal("<|message|>") + p.content(content));
// Consume any unsolicited tool calls, e.g. builtin functions
auto unsolicited = p.rule("unsolicited", p.atomic(p.optional(channel) + p.literal(" to=") + content + end));
@@ -95,58 +94,58 @@ common_chat_params common_chat_params_init_gpt_oss(const common_chat_template &
if (has_response_format) {
auto response_format = p.rule("response-format",
p.token("<|channel|>") + p.literal("final") + constraint + p.token("<|message|>") +
p.literal("<|channel|>final") + constraint + p.literal("<|message|>") +
p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema)));
return p.zero_or_more(start + analysis) + start + response_format;
}
if (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE) {
auto in_role = p.choice();
auto in_channel = p.choice();
auto tool_choice = p.choice();
foreach_function(inputs.tools, [&](size_t tool_index, const json & tool) {
const auto & function = tool.at("function");
std::string name = function.at("name");
const auto params = common_chat_tool_parameters(function);
auto func_name = p.tool_name(p.literal(name));
auto func_name = p.literal(" to=functions.") + p.tool_name(p.literal(name));
auto args = p.tool_args(p.schema(p.json(), "tool-" + std::to_string(tool_index) + "-schema", params));
// recipient in role header
// <|start|>assistant to=functions.NAME<|channel|>(commentary|analysis)[constraint]<|message|>ARGS
in_role |= p.rule("tool-" + std::to_string(tool_index) + "-in-role",
p.atomic(func_name + channel + constraint + p.token("<|message|>")) + args);
auto tool_in_role = p.tool(p.tool_open(func_name + channel + constraint + p.literal("<|message|>")) + args);
// recipient in channel header
// <|start|>assistant<|channel|>(commentary|analysis) to=functions.NAME[constraint]<|message|>ARGS
in_channel |= p.rule("tool-" + std::to_string(tool_index) + "-in-channel",
p.atomic(func_name + constraint + p.token("<|message|>")) + args);
// <|channel|>(commentary|analysis) to=functions.NAME[constraint]<|message|>ARGS
auto tool_in_channel = p.tool(p.tool_open(channel + func_name + constraint + p.literal("<|message|>")) + args);
tool_choice |= p.rule("tool-" + std::to_string(tool_index), tool_in_role | tool_in_channel);
});
// The recipient prefix is the trigger, so the grammar constrains the name that follows it
auto to_functions = p.tool_open(p.literal(" to=functions."));
auto tool_call = p.choice({
p.trigger_rule("tool-call-in-role", start_assistant + to_functions, in_role),
p.trigger_rule("tool-call-in-channel", start_assistant + channel + to_functions, in_channel)
});
auto tool_call = p.trigger_rule("tool-call", tool_choice);
if (inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED) {
return p.zero_or_more(start + any) + tool_call;
return p.zero_or_more(start + any) + start + tool_call;
}
return p.zero_or_more(start + any) + (tool_call | (start + final_msg));
return p.zero_or_more(start + any) + start + (tool_call | final_msg);
}
return p.zero_or_more(start + any) + start + (final_msg | unsolicited);
});
if (include_grammar) {
// The grammar scans for the triggers itself, so the sampler runs it from the first token
data.grammar_lazy = false;
data.grammar_lazy = !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
data.grammar = build_grammar([&](const common_grammar_builder & builder) {
data.parser.build_grammar(builder, !has_response_format && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO);
data.parser.build_grammar(builder, data.grammar_lazy);
});
data.grammar_triggers = {
{ COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN, "^\\s+to$" },
{ COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN, "^<\\|channel\\|>(?:commentary|analysis)\\s+to=functions$" },
{ COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN, "<\\|start\\|>assistant(\\s+to)" },
{ COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN, "<\\|start\\|>assistant(<\\|channel\\|>(?:commentary|analysis)\\s+to)" }
};
}
return data;
+2 -5
View File
@@ -99,11 +99,8 @@ common_chat_params common_chat_params_init_harmony_xml(const common_chat_templat
auto arg_close = p.tool_arg_close(p.gbnf(
p.literal(PARAM_CLOSE) | (p.literal("\n") + p.peek(p.optional(p.literal("\n")) + next_open)),
gbnf_format_literal("\n") + " " + gbnf_format_literal(PARAM_CLOSE.substr(1)) + "?"));
auto string_value = p.tool_arg_string_value(p.until_one_of(string_ends));
auto arg_string = p.rule("rfl-arg-string", p.through(string_value, p.tool_arg_close(p.literal(PARAM_CLOSE))));
// the parser also accepts a value with no closing tag, the grammar still requires the closing tag
auto missing_close = string_value + p.tool_arg_close(p.literal("\n") + p.peek(p.optional(p.literal("\n")) + next_open));
arg_string = p.gbnf(arg_string | missing_close, "rfl-arg-string");
auto arg_string = p.rule("rfl-arg-string",
p.ac(p.tool_arg_string_value(p.until_one_of(string_ends)) + arg_close, PARAM_CLOSE));
auto arg_sep = p.optional(p.literal("\n"));
std::string call_close_gbnf = gbnf_format_literal("<|end|>");
+2 -2
View File
@@ -91,7 +91,7 @@ common_chat_params common_chat_params_init_k2_horizon(const common_chat_template
}
auto think_body = p.until_one_of(data.thinking_end_tags);
auto think_block = [&](const common_peg_parser & body) {
return p.optional(THINK_START + p.space() + p.through(body, think_end));
return p.optional(THINK_START + p.space() + p.ac(body + think_end, data.thinking_end_tags));
};
auto reasoning = extract_reasoning ? think_block(p.reasoning(think_body)) : p.eps();
@@ -110,7 +110,7 @@ common_chat_params common_chat_params_init_k2_horizon(const common_chat_template
tool_choice = p.standard_json_tools(CALL_START, CALL_END, inputs.tools, false, true);
} else {
auto arg_close = p.tool_arg_close(p.literal(ARG_VAL_END));
auto arg_string = p.rule("xml-arg-string", p.through(p.tool_arg_string_value(p.until(ARG_VAL_END)), arg_close));
auto arg_string = p.rule("xml-arg-string", p.ac(p.tool_arg_string_value(p.until(ARG_VAL_END)) + arg_close, ARG_VAL_END));
// The models leave out <ifm|arg_type> even when asked for xml_typed
auto arg_type = call_format == "xml_typed" ? p.optional(ARG_TYPE + p.until(ARG_TYPE_END) + ARG_TYPE_END + p.space()) : p.eps();
+2 -2
View File
@@ -63,8 +63,8 @@ common_chat_params common_chat_params_init_minicpm5(const common_chat_template &
// CDATA lets a value carry characters that would otherwise close the tag (e.g.
// </param>); capture the inner text only, excluding the CDATA markers.
auto string_value = p.choice({
p.literal("<![CDATA[") + p.through(p.tool_arg_string_value(p.until("]]>")), "]]>") + p.tool_arg_close(p.literal("</param>")),
p.negate(p.literal("<![CDATA[")) + p.through(p.tool_arg_string_value(p.until("</param>")), p.tool_arg_close(p.literal("</param>")))
p.literal("<![CDATA[") + p.ac(p.tool_arg_string_value(p.until("]]>")) + p.literal("]]>"), "]]>") + p.tool_arg_close(p.literal("</param>")),
p.negate(p.literal("<![CDATA[")) + p.ac(p.tool_arg_string_value(p.until("</param>")) + p.tool_arg_close(p.literal("</param>")), "</param>")
});
auto tool_choice = p.choice();
+3 -3
View File
@@ -64,8 +64,8 @@ common_chat_params common_chat_params_init_minimax_m3(const common_chat_template
if (extract_reasoning) {
auto block = inputs.enable_thinking
? p.literal(THINK_START) + p.space() +
p.through(p.reasoning(p.until(THINK_END)), THINK_END)
: p.literal(THINK_START) + p.through(p.until(THINK_END), THINK_END);
p.ac(p.reasoning(p.until(THINK_END)) + p.literal(THINK_END), THINK_END)
: p.literal(THINK_START) + p.ac(p.until(THINK_END) + p.literal(THINK_END), THINK_END);
// A turn without reasoning is prefixed with a bare </mm:think>, written either by the
// generation prompt (thinking_mode = "disabled") or by the model itself.
@@ -113,7 +113,7 @@ common_chat_params common_chat_params_init_minimax_m3(const common_chat_template
// A string accepts anything, so a union with a string alternative is a string
if (schema.may_be_string()) {
return p.through(p.tool_arg_string_value(p.until(close)), close_tag);
return p.ac(p.tool_arg_string_value(p.until(close)) + close_tag, close);
}
if (schema.kind() == common_chat_schema::KIND_ANY_OF) {
+29 -25
View File
@@ -17,7 +17,7 @@ common_chat_params common_chat_params_init_muse_glimmer(const common_chat_templa
data.supports_thinking = true;
data.preserved_tokens = {
"<|start|>", "<|message|>", "<|eom|>",
"<|start|>", "<|message|>", "<|eom|>", "<|eot|>",
// ATEM tool-call markup emitted on " to=<tool>" turns.
"<atem:function_calls>", "<atem:invoke", "<atem:parameter", "</atem:parameter>",
"</atem:invoke>", "</atem:function_calls>",
@@ -48,38 +48,37 @@ common_chat_params common_chat_params_init_muse_glimmer(const common_chat_templa
// Constrained grammar whenever tools are offered or a response format is requested.
auto include_grammar = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
data.parser = build_chat_peg_parser(inputs.specials, [&](common_chat_peg_builder & p) {
auto start_token = p.token("<|start|>");
auto message = p.token("<|message|>");
auto eom = p.token("<|eom|>");
auto start = p.rule("start", start_token + p.literal("assistant"));
data.parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
auto start = p.rule("start", p.literal("<|start|>assistant"));
if (!extract_reasoning && !include_grammar) {
return start + p.content(p.rest());
}
if (extract_reasoning) {
p.rule("analysis", p.literal(" to=self") + message + p.reasoning(p.until(eom)) + eom);
p.rule("analysis", p.literal(" to=self<|message|>") + p.reasoning(p.until("<|eom|>")) + p.literal("<|eom|>"));
} else {
p.rule("analysis", p.literal(" to=self") + message + p.content(p.until(eom)) + eom);
p.rule("analysis", p.literal(" to=self<|message|>") + p.content(p.until("<|eom|>")) + p.literal("<|eom|>"));
}
auto analysis = p.ref("analysis");
auto recipient = p.optional(p.literal(" to=user"));
auto final_msg = p.rule("final", recipient + message + p.content(p.until(eom)));
auto final_msg = p.rule("final", recipient + p.literal("<|message|>") +
p.content(p.until_one_of({ "<|eot|>", "<|eom|>" })));
if (has_response_format) {
auto response_json = p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema));
auto response_format = p.rule("response-format",
recipient + message +
recipient + p.literal("<|message|>") +
((p.literal("```json") + p.space() + response_json + p.space() + p.literal("```")) | response_json));
return p.zero_or_more(start + analysis) + start + response_format;
}
if (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE) {
auto string_value = p.through(p.tool_arg_string_value(p.until("</atem:parameter>")),
p.tool_arg_close(p.literal("</atem:parameter>")));
auto string_value = p.ac(
p.tool_arg_string_value(p.until("</atem:parameter>")) + p.tool_arg_close(p.literal("</atem:parameter>")),
"</atem:parameter>");
auto tool_choice = p.choice();
foreach_function(inputs.tools, [&](size_t tool_index, const json & tool) {
@@ -107,35 +106,40 @@ common_chat_params common_chat_params_init_muse_glimmer(const common_chat_templa
args = p.zero_or_more(p.choice(arg_rules) + p.space());
}
auto open = p.tool_open(start_token + p.literal("assistant to=") + p.tool_name(p.literal(name)) + message);
auto body = p.rule("tool-" + std::to_string(tool_index),
p.literal("<atem:function_calls>") + p.space() + p.literal("<atem:invoke name=\"" + name + "\">") + p.space()
auto tool_parser = p.tool(
p.tool_open(p.literal(" to=") + p.until("<|message|>") +
p.literal("<|message|><atem:function_calls>") + p.space() +
p.literal("<atem:invoke name=\"") + p.tool_name(p.literal(name)) + p.literal("\">") + p.space())
<< p.tool_args(args)
<< p.tool_close(p.literal("</atem:invoke>") + p.space() + p.literal("</atem:function_calls>")));
tool_choice |= open + body;
tool_choice |= p.rule("tool-" + std::to_string(tool_index), tool_parser);
});
auto more = inputs.parallel_tool_calls ? p.zero_or_more(eom + tool_choice) : p.eps();
auto tool_calls = p.rule("tool-calls", tool_choice + more);
auto tool_calls = inputs.parallel_tool_calls
? p.trigger_rule("tool-call", tool_choice + p.zero_or_more(p.literal("<|eom|>") + start + tool_choice))
: p.trigger_rule("tool-call", tool_choice);
if (inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED) {
return p.zero_or_more(start + analysis) + tool_calls;
return p.zero_or_more(start + analysis) + start + tool_calls;
}
auto trailing_calls = p.optional(eom + tool_calls);
return p.zero_or_more(start + analysis) + (tool_calls | (start + final_msg + trailing_calls));
auto trailing_calls = p.optional(p.literal("<|eom|>") + start + tool_calls);
return p.zero_or_more(start + analysis) + start + (tool_calls | (final_msg + trailing_calls));
}
return p.zero_or_more(start + analysis) + start + final_msg;
});
if (include_grammar) {
// Every message starts with a recipient header, so the grammar is not lazy.
// The model cannot generate a recipient that is not self, user, or a tool.
data.grammar_lazy = false;
data.grammar_lazy = !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
data.grammar = build_grammar([&](const common_grammar_builder & builder) {
data.parser.build_grammar(builder);
data.parser.build_grammar(builder, data.grammar_lazy);
});
data.grammar_triggers = {
{ COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN,
"(?:^|<\\|start\\|>assistant)( to=(?!self<\\|message\\|>)(?!user<\\|message\\|>)[^<]*?<\\|message\\|>)" },
};
}
return data;
+3 -2
View File
@@ -86,8 +86,9 @@ common_chat_params common_chat_params_init_muse_spark(const common_chat_template
}
if (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE) {
auto string_value = p.through(p.tool_arg_string_value(p.until("</atem:parameter>")),
p.tool_arg_close(p.literal("</atem:parameter>")));
auto string_value = p.ac(
p.tool_arg_string_value(p.until("</atem:parameter>")) + p.tool_arg_close(p.literal("</atem:parameter>")),
"</atem:parameter>");
auto tool_choice = p.choice();
foreach_function(inputs.tools, [&](size_t tool_index, const json & tool) {
+48 -31
View File
@@ -59,16 +59,27 @@ common_chat_params common_chat_params_init_qwen3_coder(const common_chat_templat
data.prompt.append(data.generation_prompt);
}
data.parser = build_chat_peg_parser(inputs.specials, [&](common_chat_peg_builder & p) {
std::vector<std::string> tool_call_starts = { "<tool_call>" };
if (is_qwen3_coder) {
// Match complete <function=name> opener for Qwen3-Coder models that occasionally omit the
// starting <tool_call>. The model may hallucinate a tool name, but it is preferable over
// constraining on <function which may occur in valid content generation, e.g. #include <functional>
foreach_function(inputs.tools, [&](size_t, const json & tool) {
const std::string name = tool.at("function").at("name");
tool_call_starts.push_back("<function=" + name + ">");
});
}
data.parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
auto generation_prompt = p.literal(GEN_PREFIX);
// The template strips the newline before </think>, so leave it out of the reasoning
auto think_end = (p.literal("\n") + p.token("</think>")) | p.token("</think>");
auto think_body = p.until(think_end | p.token("<tool_call>"));
auto think_close = think_end | p.peek(p.token("<tool_call>"));
auto think_body = p.until_one_of({ "\n</think>", "</think>", "<tool_call>" });
auto think_close = p.literal("\n</think>") | p.literal("</think>") | p.peek(p.literal("<tool_call>"));
auto reasoning_extracted = p.optional(p.token("<think>") + p.space() + p.reasoning(think_body) + think_close);
auto reasoning_skipped = p.optional(p.token("<think>") + p.space() + think_body + think_close);
auto reasoning_extracted = p.optional("<think>" + p.space() + p.reasoning(think_body) + think_close);
auto reasoning_skipped = p.optional("<think>" + p.space() + think_body + think_close);
auto reasoning = p.eps();
if (supports_reasoning && extract_reasoning) {
@@ -88,13 +99,7 @@ common_chat_params common_chat_params_init_qwen3_coder(const common_chat_templat
if (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE) {
auto arg_close = p.tool_arg_close(p.literal("\n</parameter>\n"));
auto arg_string = p.rule("xml-arg-string",
p.through(p.tool_arg_string_value(p.until("\n</parameter>\n")), arg_close));
auto min_calls = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED ? 1 : 0;
auto more = inputs.parallel_tool_calls ? p.zero_or_more(p.ref("tool-call")) : p.eps();
auto starts = p.token("<tool_call>");
auto tool_calls = p.choice();
p.ac(p.tool_arg_string_value(p.until("\n</parameter>\n")) + arg_close, "\n</parameter>\n"));
auto tool_choice = p.choice();
foreach_function(inputs.tools, [&](size_t tool_index, const json & tool) {
@@ -150,29 +155,35 @@ common_chat_params common_chat_params_init_qwen3_coder(const common_chat_templat
args = args + p.zero_or_more(p.choice(optional_args));
}
auto opener = p.tool_open("<function=" + p.tool_name(p.literal(name)) + ">");
auto body = p.literal("\n") + p.tool_args(args) + p.tool_close(p.literal("</function>\n"));
auto func = p.tool(p.tool_open("<function=" + p.tool_name(p.literal(name)) + ">\n") +
p.tool_args(args) +
p.tool_close(p.literal("</function>\n")));
tool_choice |= p.rule("tool-" + std::to_string(tool_index), p.tool(opener + body));
if (is_qwen3_coder) {
// Qwen3-Coder models sometimes omit <tool_call>, so the full <function=name> opener is also a trigger.
// Do not trigger on <function alone, it occurs in normal content, e.g. #include <functional>.
tool_calls |= p.trigger_rule("tool-calls-" + std::to_string(tool_index), opener, body + p.token("</tool_call>") + p.space() + more);
starts |= opener;
}
tool_choice |= p.rule("tool-" + std::to_string(tool_index), func);
});
auto tool_call_body = tool_choice + p.token("</tool_call>") + p.space();
p.rule("tool-call", p.token("<tool_call>") + p.literal("\n") + tool_call_body);
auto min_calls = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED ? 1 : 0;
tool_calls |= p.trigger_rule("tool-calls", p.token("<tool_call>"), p.literal("\n") + tool_call_body + more);
auto tool_call_body = tool_choice + "</tool_call>" + p.space();
auto tool_call = p.rule("tool-call", "<tool_call>\n" + tool_call_body);
// Qwen3-Coder models may occasionally omit the <tool_call> token.
auto tool_call_first = is_qwen3_coder ?
p.rule("tool-call-first", p.optional(p.literal("<tool_call>\n")) + tool_call_body) :
tool_call;
auto calls = inputs.parallel_tool_calls ? tool_call_first + p.zero_or_more(tool_call) : tool_call_first;
auto tool_calls = p.trigger_rule("tool-call-root", p.repeat(calls, min_calls, 1));
// The templates separate content from the tool calls with one or two newlines,
// stop the content before them
auto content_end = (p.literal("\n\n") + starts) | (p.literal("\n") + starts) | starts;
std::vector<std::string> content_ends;
for (const auto & start : tool_call_starts) {
content_ends.insert(content_ends.end(), { "\n\n" + start, "\n" + start, start });
}
return generation_prompt + (reasoning << p.content(p.until(content_end)) << p.repeat(tool_calls, min_calls, 1));
return generation_prompt +
(reasoning << p.content(p.until_one_of(content_ends)) << tool_calls);
}
// Content only parser
@@ -180,11 +191,17 @@ common_chat_params common_chat_params_init_qwen3_coder(const common_chat_templat
});
if (include_grammar) {
// The grammar scans for the triggers itself, so the sampler runs it from the first token
data.grammar_lazy = false;
data.grammar_lazy = !has_response_format && has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO;
data.grammar = build_grammar([&](const common_grammar_builder & builder) {
data.parser.build_grammar(builder, !has_response_format && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO);
data.parser.build_grammar(builder, data.grammar_lazy);
});
if (data.grammar_lazy) {
for (const auto & start : tool_call_starts) {
data.grammar_triggers.push_back({ COMMON_GRAMMAR_TRIGGER_TYPE_WORD, start });
}
}
}
return data;
+115 -543
View File
@@ -208,89 +208,6 @@ void common_peg_ast_arena::visit(const common_peg_parse_result & result, const c
}
}
common_peg_tokens::common_peg_tokens(const llama_vocab * vocab) {
const llama_token n_tokens = llama_vocab_n_tokens(vocab);
for (llama_token id = 0; id < n_tokens; ++id) {
const auto attr = llama_vocab_get_attr(vocab, id);
if (attr & (LLAMA_TOKEN_ATTR_CONTROL | LLAMA_TOKEN_ATTR_USER_DEFINED)) {
std::string text = llama_vocab_get_text(vocab, id);
ids.emplace(text, id);
tokens.emplace(id, common_peg_token{ std::move(text), attr });
}
}
}
llama_token common_peg_tokens::token_id(const std::string & text) const {
auto it = ids.find(text);
return it != ids.end() ? it->second : LLAMA_TOKEN_NULL;
}
// Insert every delimiter the parser matches into the trie, starting from each node in from. Returns the end nodes.
// A choice adds the delimiters of each branch. A sequence joins every combination of its children.
static std::set<size_t> insert_delimiter(common_trie & trie, const common_peg_arena & arena, common_peg_parser_id id,
std::set<size_t> from = { 0 }) {
return std::visit([&](const auto & p) -> std::set<size_t> {
using T = std::decay_t<decltype(p)>;
if constexpr (std::is_same_v<T, common_peg_tag_parser> || std::is_same_v<T, common_peg_atomic_parser>) {
return insert_delimiter(trie, arena, p.child, std::move(from));
} else if constexpr (std::is_same_v<T, common_peg_literal_parser>) {
std::set<size_t> ends;
for (size_t node : from) {
size_t pos = 0;
while (pos < p.literal.size()) {
auto result = common_parse_utf8_codepoint(p.literal, pos);
if (result.status != utf8_parse_result::SUCCESS) {
break;
}
node = trie.extend(node, common_trie::symbol::codepoint(result.codepoint));
pos += result.bytes_consumed;
}
ends.insert(node);
}
return ends;
} else if constexpr (std::is_same_v<T, common_peg_token_parser>) {
std::set<size_t> ends;
for (size_t node : from) {
ends.insert(trie.extend(node, common_trie::symbol::token(p.token)));
}
return ends;
} else if constexpr (std::is_same_v<T, common_peg_choice_parser>) {
std::set<size_t> ends;
for (auto child : p.children) {
ends.merge(insert_delimiter(trie, arena, child, from));
}
return ends;
} else if constexpr (std::is_same_v<T, common_peg_sequence_parser>) {
for (auto child : p.children) {
from = insert_delimiter(trie, arena, child, std::move(from));
}
return from;
} else {
throw std::invalid_argument("a delimiter may only be a sequence of literals, tokens, and choices of them, possibly wrapped in tags and atomics");
}
}, arena.get(id));
}
// Build the matcher of an until parser, each pattern valued by the branch it belongs to
static common_trie build_until_matcher(const common_peg_arena & arena,
const std::vector<common_peg_until_branch> & branches,
bool include) {
common_trie matcher;
for (size_t b = 0; b < branches.size(); b++) {
for (size_t end : insert_delimiter(matcher, arena, branches[b].delimiter)) {
if (end == 0) {
throw std::invalid_argument("delimiter must not match the empty string");
}
// nothing follows a delimiter without include, so the first branch to claim it wins
auto owner = matcher.mark(end, (int32_t) b);
if (owner != (int32_t) b && include) {
throw std::invalid_argument("until branches " + std::to_string(owner) + " and " + std::to_string(b) + " must not share a delimiter");
}
}
}
return matcher;
}
struct parser_executor;
common_peg_parser_id common_peg_arena::add_parser(common_peg_parser_variant parser) {
@@ -381,49 +298,6 @@ struct parser_executor {
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, pos);
}
common_peg_parse_result operator()(const common_peg_token_parser & p) {
if (start_pos >= ctx.input.size()) {
if (!ctx.is_lenient()) {
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_FAIL, start_pos);
}
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_NEED_MORE_INPUT, start_pos);
}
if (start_pos >= ctx.tokens.size() || ctx.tokens[start_pos] != p.token) {
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_FAIL, start_pos);
}
// Skip the rest of the piece, up to the next token or the end of the input
auto pos = start_pos + 1;
while (pos < ctx.input.size() && ctx.tokens[pos] == LLAMA_TOKEN_NULL) {
++pos;
}
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, pos);
}
common_peg_parse_result operator()(const common_peg_token_not_parser & p) {
if (start_pos >= ctx.input.size()) {
if (!ctx.is_lenient()) {
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_FAIL, start_pos);
}
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_NEED_MORE_INPUT, start_pos);
}
if (start_pos >= ctx.tokens.size() || ctx.tokens[start_pos] == LLAMA_TOKEN_NULL ||
std::find(p.tokens.begin(), p.tokens.end(), ctx.tokens[start_pos]) != p.tokens.end()) {
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_FAIL, start_pos);
}
// Skip the rest of the piece, up to the next token or the end of the input
auto pos = start_pos + 1;
while (pos < ctx.input.size() && ctx.tokens[pos] == LLAMA_TOKEN_NULL) {
++pos;
}
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, pos);
}
common_peg_parse_result operator()(const common_peg_sequence_parser & p) {
if (ctx.is_debug()) {
LOG_DBG("%sSEQ start at %zu '%s' (%zu children)\n", debug_indent().c_str(), start_pos,
@@ -815,11 +689,7 @@ struct parser_executor {
}
common_peg_parse_result operator()(const common_peg_until_parser & p) const {
if (p.include) {
return parse_through(p);
}
const auto & matcher = p.matcher;
common_trie matcher(p.delimiters);
// Scan input and check for delimiters
size_t pos = start_pos;
@@ -827,24 +697,24 @@ struct parser_executor {
std::vector<common_peg_invalid_utf8> invalid_utf8;
while (pos < ctx.input.size()) {
auto step = matcher.next(ctx.input, ctx.tokens, pos);
auto utf8_result = common_parse_utf8_codepoint(ctx.input, pos);
if (step.status == utf8_parse_result::INCOMPLETE && ctx.is_lenient()) {
if (utf8_result.status == utf8_parse_result::INCOMPLETE && ctx.is_lenient()) {
// The rest of the sequence may still arrive, return what we have so far
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_NEED_MORE_INPUT, start_pos, last_valid_pos, {}, std::move(invalid_utf8));
}
if (step.status != utf8_parse_result::SUCCESS) {
if (utf8_result.status != utf8_parse_result::SUCCESS) {
// Malformed UTF-8, or a sequence truncated by the end of a complete input.
// A delimiter cannot start inside bytes that fail to decode, so consume them and move on
invalid_utf8.push_back({pos, step.bytes_consumed});
pos += step.bytes_consumed;
invalid_utf8.push_back({pos, utf8_result.bytes_consumed});
pos += utf8_result.bytes_consumed;
last_valid_pos = pos;
continue;
}
// Check if a delimiter starts at this position
auto match = matcher.check_at(ctx.input, ctx.tokens, pos);
auto match = matcher.check_at(ctx.input, pos);
if (match == common_trie::COMPLETE_MATCH) {
// Found a complete delimiter, return everything before it
@@ -857,7 +727,7 @@ struct parser_executor {
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, pos, {}, std::move(invalid_utf8));
}
pos += step.bytes_consumed;
pos += utf8_result.bytes_consumed;
last_valid_pos = pos;
}
@@ -868,48 +738,6 @@ struct parser_executor {
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, last_valid_pos, {}, std::move(invalid_utf8));
}
// Parse content, then the first branch whose delimiter and rest follow it, like C (D1 R1 | D2 R2 | ...)
common_peg_parse_result parse_through(const common_peg_until_parser & p) const {
auto content = arena.parse(p.content, ctx, start_pos);
if (content.fail() || content.need_more_input()) {
return content;
}
auto pos = content.end;
for (const auto & branch : p.branches) {
auto nodes = content.nodes;
auto invalid_utf8 = content.invalid_utf8;
auto end = pos;
bool matched = true;
for (auto id : { branch.delimiter, branch.rest }) {
if (id == COMMON_PEG_INVALID_PARSER_ID) {
continue;
}
auto result = arena.parse(id, ctx, end);
if (result.fail()) {
matched = false;
break;
}
nodes.insert(nodes.end(), result.nodes.begin(), result.nodes.end());
invalid_utf8.insert(invalid_utf8.end(), result.invalid_utf8.begin(), result.invalid_utf8.end());
end = result.end;
if (result.need_more_input()) {
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_NEED_MORE_INPUT, start_pos, end, std::move(nodes), std::move(invalid_utf8));
}
}
if (matched) {
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, end, std::move(nodes), std::move(invalid_utf8));
}
}
if (p.optional && pos == ctx.input.size()) {
auto type = ctx.is_lenient() ? COMMON_PEG_PARSE_RESULT_NEED_MORE_INPUT : COMMON_PEG_PARSE_RESULT_SUCCESS;
return common_peg_parse_result(type, start_pos, pos, std::move(content.nodes), std::move(content.invalid_utf8));
}
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_FAIL, start_pos, pos);
}
common_peg_parse_result operator()(const common_peg_schema_parser & p) {
return arena.parse(p.child, ctx, start_pos);
}
@@ -995,6 +823,10 @@ struct parser_executor {
common_peg_parse_result operator()(const common_peg_gbnf_parser & p) {
return arena.parse(p.child, ctx, start_pos);
}
common_peg_parse_result operator()(const common_peg_ac_parser & p) {
return arena.parse(p.child, ctx, start_pos);
}
};
common_peg_parse_result common_peg_arena::parse(common_peg_parse_context & ctx, size_t start) const {
@@ -1067,33 +899,19 @@ void common_peg_arena::resolve_refs() {
std::is_same_v<T, common_peg_not_parser> ||
std::is_same_v<T, common_peg_tag_parser> ||
std::is_same_v<T, common_peg_atomic_parser> ||
std::is_same_v<T, common_peg_gbnf_parser>) {
std::is_same_v<T, common_peg_gbnf_parser> ||
std::is_same_v<T, common_peg_ac_parser>) {
p.child = resolve_ref(p.child);
} else if constexpr (std::is_same_v<T, common_peg_until_parser>) {
if (p.content != COMMON_PEG_INVALID_PARSER_ID) {
p.content = resolve_ref(p.content);
}
for (auto & branch : p.branches) {
branch.delimiter = resolve_ref(branch.delimiter);
if (branch.rest != COMMON_PEG_INVALID_PARSER_ID) {
branch.rest = resolve_ref(branch.rest);
}
}
} else if constexpr (std::is_same_v<T, common_peg_rule_parser>) {
p.child = resolve_ref(p.child);
if (p.start != COMMON_PEG_INVALID_PARSER_ID) {
p.start = resolve_ref(p.start);
p.rest = resolve_ref(p.rest);
}
} else if constexpr (std::is_same_v<T, common_peg_schema_parser>) {
p.child = resolve_ref(p.child);
} else if constexpr (std::is_same_v<T, common_peg_epsilon_parser> ||
std::is_same_v<T, common_peg_start_parser> ||
std::is_same_v<T, common_peg_end_parser> ||
std::is_same_v<T, common_peg_ref_parser> ||
std::is_same_v<T, common_peg_until_parser> ||
std::is_same_v<T, common_peg_literal_parser> ||
std::is_same_v<T, common_peg_token_parser> ||
std::is_same_v<T, common_peg_token_not_parser> ||
std::is_same_v<T, common_peg_string_parser> ||
std::is_same_v<T, common_peg_chars_parser> ||
std::is_same_v<T, common_peg_any_parser> ||
@@ -1137,14 +955,6 @@ std::string common_peg_arena::dump_impl(common_peg_parser_id
return "End";
} else if constexpr (std::is_same_v<T, common_peg_literal_parser>) {
return "Literal(" + p.literal + ")";
} else if constexpr (std::is_same_v<T, common_peg_token_parser>) {
return "Token(" + p.piece + ", " + std::to_string(p.token) + ")";
} else if constexpr (std::is_same_v<T, common_peg_token_not_parser>) {
std::vector<std::string> parts;
for (auto token : p.tokens) {
parts.push_back(std::to_string(token));
}
return "TokenNot(" + string_join(parts, ", ") + ")";
} else if constexpr (std::is_same_v<T, common_peg_sequence_parser>) {
std::vector<std::string> parts;
for (const auto & child : p.children) {
@@ -1171,6 +981,8 @@ std::string common_peg_arena::dump_impl(common_peg_parser_id
return "Atomic(" + dump_impl(p.child, visited) + ")";
} else if constexpr (std::is_same_v<T, common_peg_gbnf_parser>) {
return "Gbnf(" + p.grammar + ", " + dump_impl(p.child, visited) + ")";
} else if constexpr (std::is_same_v<T, common_peg_ac_parser>) {
return "Ac(" + string_join(p.delimiters, " | ") + ", " + dump_impl(p.child, visited) + ")";
} else if constexpr (std::is_same_v<T, common_peg_any_parser>) {
return "Any";
} else if constexpr (std::is_same_v<T, common_peg_space_parser>) {
@@ -1183,18 +995,7 @@ std::string common_peg_arena::dump_impl(common_peg_parser_id
} else if constexpr (std::is_same_v<T, common_peg_string_parser>) {
return "String(" + std::string(1, p.delimiter) + ")";
} else if constexpr (std::is_same_v<T, common_peg_until_parser>) {
std::vector<std::string> branches;
for (const auto & branch : p.branches) {
auto text = dump_impl(branch.delimiter, visited);
if (branch.rest != COMMON_PEG_INVALID_PARSER_ID) {
text += " " + dump_impl(branch.rest, visited);
}
branches.push_back(std::move(text));
}
if (p.include) {
return "Until(" + dump_impl(p.content, visited) + ", " + string_join(branches, " | ") + (p.optional ? ", optional" : "") + ")";
}
return "Until(" + string_join(branches, " | ") + ")";
return "Until(" + string_join(p.delimiters, " | ") + ")";
} else if constexpr (std::is_same_v<T, common_peg_schema_parser>) {
return "Schema(" + dump_impl(p.child, visited) + ", " + (p.node ? common_chat_schema::kind_name(p.node->kind()) : "null") + ")";
} else if constexpr (std::is_same_v<T, common_peg_rule_parser>) {
@@ -1288,67 +1089,6 @@ common_peg_parser operator|(const std::string & str, const common_peg_parser & p
common_peg_parser_builder::common_peg_parser_builder() {}
common_peg_parser_builder::common_peg_parser_builder(common_peg_tokens tokens) {
arena_.tokens_ = std::move(tokens);
}
common_peg_parser common_peg_parser_builder::until_one_of(const std::vector<std::string> & delimiters) {
common_peg_until_parser p;
for (const auto & d : delimiters) {
p.branches.push_back({ literal(d).id() });
}
p.matcher = build_until_matcher(arena_, p.branches, p.include);
return add(std::move(p));
}
common_peg_parser common_peg_parser_builder::until(const common_peg_parser & delimiter) {
common_peg_until_parser p;
p.branches = { { delimiter.id() } };
p.matcher = build_until_matcher(arena_, p.branches, p.include);
return add(std::move(p));
}
common_peg_parser common_peg_parser_builder::through(const common_peg_parser & content, const std::vector<common_peg_until_branch> & branches, bool optional) {
if (branches.empty()) {
throw std::runtime_error("through requires at least one branch");
}
common_peg_until_parser p;
p.branches = branches;
p.include = true;
p.optional = optional;
p.content = content.id();
p.matcher = build_until_matcher(arena_, p.branches, p.include);
return add(std::move(p));
}
common_peg_parser common_peg_parser_builder::token(const std::string & piece) {
auto token = arena_.tokens_.token_id(piece);
if (token == LLAMA_TOKEN_NULL) {
return literal(piece);
}
return add(common_peg_token_parser{token, piece});
}
common_peg_parser common_peg_parser_builder::token(const std::vector<std::string> & pieces, bool negate) {
if (!negate) {
std::vector<common_peg_parser> alts;
for (const auto & piece : pieces) {
alts.push_back(token(piece));
}
return choice(alts);
}
common_peg_token_not_parser p;
for (const auto & piece : pieces) {
auto token = arena_.tokens_.token_id(piece);
if (token == LLAMA_TOKEN_NULL) {
throw std::invalid_argument("token is not registered with the builder: " + piece);
}
p.tokens.push_back(token);
}
return add(p);
}
common_peg_parser common_peg_parser_builder::sequence(const std::vector<common_peg_parser_id> & parsers) {
// Flatten nested sequences
std::vector<common_peg_parser_id> flattened;
@@ -1455,17 +1195,6 @@ common_peg_parser common_peg_parser_builder::rule(const std::string & name, cons
return ref(clean_name);
}
common_peg_parser common_peg_parser_builder::trigger_rule(const std::string & name, const common_peg_parser & start, const common_peg_parser & rest) {
common_trie scratch;
if (insert_delimiter(scratch, arena_, start.id()).count(0)) {
throw std::invalid_argument("trigger start must not match the empty string");
}
auto clean_name = gbnf_format_rule_name(name);
auto rule_id = arena_.add_parser(common_peg_rule_parser{clean_name, (start + rest).id(), true, start.id(), rest.id()});
arena_.add_rule(clean_name, rule_id);
return ref(clean_name);
}
void common_peg_parser_builder::set_root(const common_peg_parser & p) {
arena_.set_root(p.id());
}
@@ -1663,6 +1392,13 @@ common_peg_parser common_peg_parser_builder::json_member(const std::string & key
});
}
common_peg_parser common_peg_parser_builder::ac(const common_peg_parser & p, const std::vector<std::string> & delimiters) {
if (delimiters.empty()) {
throw std::runtime_error("ac parser requires at least one delimiter");
}
return add(common_peg_ac_parser{p, delimiters});
}
static std::string gbnf_escape_char_class(uint32_t c) {
if (c == '-' || c == ']' || c == '[' || c == '\\') {
return "\\" + std::string(1, (char) c);
@@ -1721,63 +1457,15 @@ static std::string gbnf_char_class(const std::vector<uint32_t> & chars, bool neg
return s + "]";
}
// Token set such as <[1,5-9]>, with consecutive ids collapsed into ranges
static std::string gbnf_token_set(const std::vector<uint32_t> & tokens, bool negate) {
std::string s = negate ? "!<[" : "<[";
for (size_t i = 0; i < tokens.size();) {
size_t j = i;
while (j + 1 < tokens.size() && tokens[j + 1] == tokens[j] + 1) {
j++;
}
s += (i == 0 ? "" : ",") + std::to_string(tokens[i]);
if (j > i) {
s += "-" + std::to_string(tokens[j]);
}
i = j + 1;
}
return s + "]>";
}
// Symbols on the transitions out of an automaton state, codepoints and tokens kept apart
struct gbnf_ac_symbols {
std::vector<uint32_t> chars;
std::vector<uint32_t> tokens;
void add(const common_trie::symbol & sym) {
(sym.is_token() ? tokens : chars).push_back(sym.value);
}
// One alternative for the codepoints and one for the tokens, each followed by next if given
std::vector<std::string> alternatives(const std::string & next = "") const {
std::vector<std::string> alts;
const std::string tail = next.empty() ? "" : " " + next;
if (!chars.empty()) {
alts.push_back(gbnf_char_class(chars, false) + tail);
}
if (!tokens.empty()) {
alts.push_back(gbnf_token_set(tokens, false) + tail);
}
return alts;
}
};
static std::string gbnf_ac_grammar(
const common_grammar_builder & builder,
const std::string & prefix,
const common_trie & trie,
const std::set<uint32_t> & opaque,
const std::function<std::string(const gbnf_ac_symbols &,
const std::map<size_t, gbnf_ac_symbols> &,
const std::map<size_t, gbnf_ac_symbols> &,
const std::string &,
const common_grammar_builder & builder,
const std::string & prefix,
const std::vector<std::string> & strings,
const std::function<std::string(const std::vector<uint32_t> &,
const std::map<size_t, std::vector<uint32_t>> &,
const std::vector<uint32_t> &,
const std::function<std::string(size_t)> &)> & build_rule) {
common_aho_corasick ac(trie);
// With only tokens in the alphabet, the grammar can step a whole token at a time
bool token_only = true;
for (const auto & sym : ac.alphabet) {
token_only = token_only && sym.is_token();
}
common_aho_corasick ac(strings);
auto state_name = [&](size_t s) -> std::string {
if (s == 0) {
@@ -1793,43 +1481,27 @@ static std::string gbnf_ac_grammar(
continue; // match states
}
std::map<size_t, gbnf_ac_symbols> buckets;
std::map<size_t, gbnf_ac_symbols> completing_to; // completing symbols by the match state they reach
gbnf_ac_symbols completing; // symbols that complete a delimiter
gbnf_ac_symbols specific; // symbols with an explicit transition
for (const auto & sym : ac.alphabet) {
size_t d = ac.next(q, sym);
std::map<size_t, std::vector<uint32_t>> buckets;
std::vector<uint32_t> completing; // chars that complete a delimiter
std::vector<uint32_t> specific; // chars with an explicit transition
for (uint32_t c : ac.alphabet) {
size_t d = ac.next(q, c);
if (ac.is_terminal(d)) {
completing.add(sym);
completing_to[d].add(sym);
specific.add(sym);
completing.push_back(c);
specific.push_back(c);
} else if (d != 0) {
buckets[d].add(sym); // specific non-root destination
specific.add(sym);
buckets[d].push_back(c); // specific non-root destination
specific.push_back(c);
}
}
// every other symbol goes back to the start
std::string fallback;
if (token_only) {
fallback = gbnf_token_set(specific.tokens, true);
} else {
fallback = specific.chars.empty() ? "." : gbnf_char_class(specific.chars, true);
builder.add_rule(state_name(q), build_rule(completing, buckets, specific, state_name));
}
// Char rules never match a token named by a token rule, so allow the other named tokens by name
std::vector<uint32_t> others;
for (auto token : opaque) {
if (std::find(specific.tokens.begin(), specific.tokens.end(), token) == specific.tokens.end()) {
others.push_back(token);
}
}
if (!others.empty()) {
fallback = "(" + fallback + " | " + gbnf_token_set(others, false) + ")";
}
}
fallback += " " + state_name(0);
builder.add_rule(state_name(q), build_rule(completing, completing_to, buckets, fallback, state_name));
// An empty delimiter makes the start state terminal. Emit an entry rule
// that matches the empty string so the returned reference stays valid.
if (ac.is_terminal(0)) {
builder.add_rule(prefix, "|");
}
return state_name(0);
@@ -1842,104 +1514,52 @@ static std::string gbnf_ac_grammar(
// ref: https://github.com/ggml-org/llama.cpp/pull/24839
static std::string gbnf_excluding_grammar(const common_grammar_builder & builder,
const std::string & prefix,
const common_trie & trie,
const std::set<uint32_t> & opaque) {
return gbnf_ac_grammar(builder, prefix, trie, opaque,
[](const gbnf_ac_symbols & /*completing*/,
const std::map<size_t, gbnf_ac_symbols> & /*completing_to*/,
const std::map<size_t, gbnf_ac_symbols> & buckets,
const std::string & fallback,
const std::vector<std::string> & strings) {
return gbnf_ac_grammar(builder, prefix, strings,
[](const std::vector<uint32_t> & /*completing*/,
const std::map<size_t, std::vector<uint32_t>> & buckets,
const std::vector<uint32_t> & specific,
const std::function<std::string(size_t)> & state_name) {
// every state accepts and completing symbols get no alternative, so a forbidden string never matches
// every state is accepting and completing chars get no
// alternative, so a forbidden string can never be matched
std::string rhs = "|";
for (const auto & [d, symbols] : buckets) {
for (const auto & alt : symbols.alternatives(state_name(d))) {
rhs += " " + alt + " |";
}
for (const auto & [d, chars] : buckets) {
rhs += " " + gbnf_char_class(chars, false) + " " + state_name(d) + " |";
}
rhs += " " + fallback;
rhs += " " + gbnf_char_class(specific, true) + " " + state_name(0);
return rhs;
});
}
// GBNF grammar matching up to the first pattern in the trie, then the rest rule of that pattern's value.
// For the pattern "ab" with rest "x":
//
// prefix ::= [a] prefix-01 | [^a] prefix
// prefix-01 ::= [b] rest | [a] prefix-01 | [^ab] prefix
// rest ::= "x"
static std::string gbnf_including_grammar(const common_grammar_builder & builder,
const std::string & prefix,
const common_trie & trie,
const std::set<uint32_t> & opaque,
const std::vector<std::string> & rests,
bool optional) {
common_aho_corasick ac(trie);
// every pattern ending at a match state, the longest and the ones reached by suffix links
auto rests_at = [&](size_t s, bool & terminal) {
std::vector<std::string> alts;
terminal = false;
for (; s != 0; s = ac.fail[s]) {
auto pattern = ac.t.nodes[s].pattern;
if (pattern < 0) {
continue;
}
if (rests[pattern].empty()) {
terminal = true;
} else if (std::find(alts.begin(), alts.end(), rests[pattern]) == alts.end()) {
alts.push_back(rests[pattern]);
}
}
return alts;
};
return gbnf_ac_grammar(builder, prefix, trie, opaque,
[&](const gbnf_ac_symbols & /*completing*/,
const std::map<size_t, gbnf_ac_symbols> & completing_to,
const std::map<size_t, gbnf_ac_symbols> & buckets,
const std::string & fallback,
const std::function<std::string(size_t)> & state_name) {
// GBNF grammar matching everything up to and including the first occurrence of
// any string in `strings`. Emits the Aho-Corasick automaton DFA and returns
// the start state rule name.
static std::string gbnf_including_grammar(const common_grammar_builder & builder,
const std::string & prefix,
const std::vector<std::string> & strings) {
return gbnf_ac_grammar(builder, prefix, strings,
[](const std::vector<uint32_t> & completing,
const std::map<size_t, std::vector<uint32_t>> & buckets,
const std::vector<uint32_t> & specific,
const std::function<std::string(size_t)> & state_name) {
std::vector<std::string> alts;
gbnf_ac_symbols terminating; // symbols that complete a pattern with no rest
for (const auto & [d, symbols] : completing_to) {
bool terminal = false;
auto rest = rests_at(d, terminal);
if (!rest.empty()) {
for (auto & alt : symbols.alternatives(rest.size() == 1 ? rest[0] : "(" + string_join(rest, " | ") + ")")) {
alts.push_back(std::move(alt));
}
}
if (terminal) {
terminating.chars.insert(terminating.chars.end(), symbols.chars.begin(), symbols.chars.end());
terminating.tokens.insert(terminating.tokens.end(), symbols.tokens.begin(), symbols.tokens.end());
}
if (!completing.empty()) {
alts.push_back(gbnf_char_class(completing, false)); // terminate on match
}
for (auto & alt : terminating.alternatives()) {
alts.push_back(std::move(alt));
for (const auto & [d, chars] : buckets) {
alts.push_back(gbnf_char_class(chars, false) + " " + state_name(d));
}
for (const auto & [d, symbols] : buckets) {
for (auto & alt : symbols.alternatives(state_name(d))) {
alts.push_back(std::move(alt));
}
}
// every other symbol keeps scanning from the start state
alts.push_back(fallback);
return (optional ? "| " : "") + string_join(alts, " | ");
// every other character keeps scanning from the start state
alts.push_back(gbnf_char_class(specific, true) + " " + state_name(0));
return string_join(alts, " | ");
});
}
// The rules and the tokens named by token rules that the grammar of a parser reaches
struct common_peg_reachable {
std::set<std::string> rules;
std::set<uint32_t> tokens;
};
static void collect_reachable(
static std::set<std::string> collect_reachable_rules(
const common_peg_arena & arena,
const common_peg_parser_id & root,
common_peg_reachable & reachable
const common_peg_parser_id & rule
) {
std::set<std::string> reachable;
std::set<std::string> visited;
std::function<void(common_peg_parser_id)> visit = [&](common_peg_parser_id id) {
@@ -1951,27 +1571,13 @@ static void collect_reachable(
if constexpr (std::is_same_v<T, common_peg_epsilon_parser> ||
std::is_same_v<T, common_peg_start_parser> ||
std::is_same_v<T, common_peg_end_parser> ||
std::is_same_v<T, common_peg_until_parser> ||
std::is_same_v<T, common_peg_literal_parser> ||
std::is_same_v<T, common_peg_chars_parser> ||
std::is_same_v<T, common_peg_space_parser> ||
std::is_same_v<T, common_peg_any_parser> ||
std::is_same_v<T, common_peg_string_parser>) {
// These parsers do not have any children
} else if constexpr (std::is_same_v<T, common_peg_token_parser>) {
reachable.tokens.insert(p.token);
} else if constexpr (std::is_same_v<T, common_peg_token_not_parser>) {
reachable.tokens.insert(p.tokens.begin(), p.tokens.end());
} else if constexpr (std::is_same_v<T, common_peg_until_parser>) {
// The delimiters name their tokens in the grammar
if (p.content != COMMON_PEG_INVALID_PARSER_ID) {
visit(p.content);
}
for (const auto & branch : p.branches) {
visit(branch.delimiter);
if (branch.rest != COMMON_PEG_INVALID_PARSER_ID) {
visit(branch.rest);
}
}
} else if constexpr (std::is_same_v<T, common_peg_sequence_parser>) {
for (auto child : p.children) {
visit(child);
@@ -1986,12 +1592,13 @@ static void collect_reachable(
std::is_same_v<T, common_peg_tag_parser> ||
std::is_same_v<T, common_peg_atomic_parser> ||
std::is_same_v<T, common_peg_gbnf_parser> ||
std::is_same_v<T, common_peg_ac_parser> ||
std::is_same_v<T, common_peg_schema_parser>) {
visit(p.child);
} else if constexpr (std::is_same_v<T, common_peg_rule_parser>) {
if (visited.find(p.name) == visited.end()) {
visited.insert(p.name);
reachable.rules.insert(p.name);
reachable.insert(p.name);
visit(p.child);
}
} else if constexpr (std::is_same_v<T, common_peg_ref_parser>) {
@@ -2004,13 +1611,12 @@ static void collect_reachable(
}, parser);
};
visit(root);
visit(rule);
return reachable;
}
// GBNF generation implementation
void common_peg_arena::build_grammar(const common_grammar_builder & builder, bool lazy) const {
common_peg_reachable reachable;
// A raw string value is parsed by the child rather than constrained by the schema
auto schema_delegates = [](const common_peg_schema_parser & s) -> bool {
return !s.node || (s.raw && s.node->may_be_string());
@@ -2049,10 +1655,6 @@ void common_peg_arena::build_grammar(const common_grammar_builder & builder, boo
return "";
} else if constexpr (std::is_same_v<T, common_peg_literal_parser>) {
return gbnf_format_literal(p.literal);
} else if constexpr (std::is_same_v<T, common_peg_token_parser>) {
return "<[" + std::to_string(p.token) + "]>";
} else if constexpr (std::is_same_v<T, common_peg_token_not_parser>) {
return gbnf_token_set(std::vector<uint32_t>(p.tokens.begin(), p.tokens.end()), true);
} else if constexpr (std::is_same_v<T, common_peg_sequence_parser>) {
std::string s;
for (const auto & child : p.children) {
@@ -2144,19 +1746,10 @@ void common_peg_arena::build_grammar(const common_grammar_builder & builder, boo
const std::string delim(1, p.delimiter);
return R"(( [^)" + delim + R"(\\] | "\\" ( [)" + delim + R"(\\/ bfnrt] | "u" [0-9a-fA-F]{4} ) )*)";
} else if constexpr (std::is_same_v<T, common_peg_until_parser>) {
auto prefix = "until-" + std::to_string(id);
if (!p.include) {
if (p.branches.empty()) {
return ".*";
}
return gbnf_excluding_grammar(builder, prefix, p.matcher, reachable.tokens);
if (p.delimiters.empty()) {
return ".*";
}
std::vector<std::string> branch_rests;
for (size_t b = 0; b < p.branches.size(); b++) {
auto rest = p.branches[b].rest;
branch_rests.push_back(rest == COMMON_PEG_INVALID_PARSER_ID ? "" : builder.add_rule(prefix + "-rest-" + std::to_string(b), to_gbnf(rest)));
}
return gbnf_including_grammar(builder, prefix, p.matcher, reachable.tokens, branch_rests, p.optional);
return gbnf_excluding_grammar(builder, "until-" + std::to_string(id), p.delimiters);
} else if constexpr (std::is_same_v<T, common_peg_schema_parser>) {
if (schema_delegates(p)) {
return to_gbnf(p.child);
@@ -2173,52 +1766,38 @@ void common_peg_arena::build_grammar(const common_grammar_builder & builder, boo
return to_gbnf(p.child);
} else if constexpr (std::is_same_v<T, common_peg_gbnf_parser>) {
return p.grammar;
} else if constexpr (std::is_same_v<T, common_peg_ac_parser>) {
return gbnf_including_grammar(builder, "ac-" + std::to_string(id), p.delimiters);
} else {
static_assert(is_always_false_v<T>);
}
}, parser);
};
// Trigger rules, sorted for a predictable order
std::vector<const common_peg_rule_parser *> trigger_rules;
for (const auto & [name, id] : rules_) {
if (auto rule = std::get_if<common_peg_rule_parser>(&parsers_.at(id))) {
if (rule->trigger) {
trigger_rules.push_back(rule);
}
}
}
std::sort(trigger_rules.begin(), trigger_rules.end(), [](const auto * a, const auto * b) { return a->name < b->name; });
// Collect reachable rules
std::set<std::string> reachable_rules;
// A lazy grammar scans for the starts when the trigger rules were built from a start and rest
bool scans = false;
for (const auto * rule : trigger_rules) {
scans = scans || rule->start != COMMON_PEG_INVALID_PARSER_ID;
}
if (scans) {
for (const auto * rule : trigger_rules) {
if (rule->start == COMMON_PEG_INVALID_PARSER_ID) {
throw std::runtime_error("trigger rule without a start in a scanning grammar: " + rule->name);
if (lazy) {
// Collect rules reachable from trigger rules
for (const auto & [name, id] : rules_) {
const auto & parser = parsers_.at(id);
if (auto rule = std::get_if<common_peg_rule_parser>(&parser)) {
if (rule->trigger) {
// Mark trigger as reachable and visit it
reachable_rules.insert(name);
auto add_rules = collect_reachable_rules(*this, id);
reachable_rules.insert(add_rules.begin(), add_rules.end());
}
}
}
}
if (lazy && scans) {
for (const auto * rule : trigger_rules) {
collect_reachable(*this, rule->start, reachable);
collect_reachable(*this, rule->rest, reachable);
}
} else if (lazy) {
for (const auto * rule : trigger_rules) {
collect_reachable(*this, rules_.at(rule->name), reachable);
}
} else {
collect_reachable(*this, root_, reachable);
// Collect rules reachable from root
reachable_rules = collect_reachable_rules(*this, root_);
}
// Create GBNF rules for all reachable rules
for (const auto & [name, rule_id] : rules_) {
if (reachable.rules.find(name) == reachable.rules.end()) {
if (reachable_rules.find(name) == reachable_rules.end()) {
continue;
}
@@ -2228,27 +1807,20 @@ void common_peg_arena::build_grammar(const common_grammar_builder & builder, boo
}
}
if (lazy && scans) {
// The root scans for a trigger start and then constrains its rest
common_trie starts;
std::vector<std::string> rests;
for (const auto * rule : trigger_rules) {
auto value = (int32_t) rests.size();
rests.push_back(builder.add_rule(rule->name + "-rest", to_gbnf(rule->rest)));
for (size_t end : insert_delimiter(starts, *this, rule->start)) {
auto owner = starts.mark(end, value);
if (owner != value) {
throw std::runtime_error("trigger rules " + trigger_rules[owner]->name + " and " + rule->name + " must not share a start");
if (lazy) {
// Generate root rule from trigger rules only
std::vector<std::string> trigger_names;
for (const auto & [name, rule_id] : rules_) {
const auto & parser = parsers_.at(rule_id);
if (auto rule = std::get_if<common_peg_rule_parser>(&parser)) {
if (rule->trigger) {
trigger_names.push_back(rule->name);
}
}
}
builder.add_rule("root", gbnf_including_grammar(builder, "trigger", starts, reachable.tokens, rests, /* optional = */ true));
} else if (lazy) {
// Generate root rule from trigger rules only
std::vector<std::string> trigger_names;
for (const auto * rule : trigger_rules) {
trigger_names.push_back(rule->name);
}
// Sort for predictable order
std::sort(trigger_names.begin(), trigger_names.end());
builder.add_rule("root", string_join(trigger_names, " | "));
} else if (root_ != COMMON_PEG_INVALID_PARSER_ID) {
builder.add_rule("root", to_gbnf(root_));
+17 -77
View File
@@ -3,7 +3,6 @@
#include "json-schema.h"
#include "json.h"
#include "llama.h"
#include "trie.h"
#include <memory>
#include <set>
@@ -189,22 +188,6 @@ struct common_peg_recursion_error : std::invalid_argument {
using std::invalid_argument::invalid_argument;
};
struct common_peg_token {
std::string text;
llama_token_attr attr;
};
struct common_peg_tokens {
std::unordered_map<llama_token, common_peg_token> tokens;
std::unordered_map<std::string, llama_token> ids;
common_peg_tokens() = default;
explicit common_peg_tokens(const llama_vocab * vocab);
// LLAMA_TOKEN_NULL if text is not a special token
llama_token token_id(const std::string & text) const;
};
struct common_peg_parse_context {
std::string input; // [h, e, l, l, o, _, w, o, r, l, d]
std::vector<llama_token> tokens; // [id, -1, -1, -1, -1, id, -1, -1, -1, -1, -1]
@@ -243,16 +226,6 @@ struct common_peg_literal_parser {
std::string literal;
};
struct common_peg_token_parser {
llama_token token;
std::string piece;
};
// Matches one token that is not one of the tokens
struct common_peg_token_not_parser {
std::vector<llama_token> tokens;
};
struct common_peg_sequence_parser {
std::vector<common_peg_parser_id> children;
};
@@ -297,18 +270,8 @@ struct common_peg_string_parser {
char delimiter;
};
// A delimiter an until parser stops at, and the parser that follows it when the delimiter is consumed
struct common_peg_until_branch {
common_peg_parser_id delimiter;
common_peg_parser_id rest = COMMON_PEG_INVALID_PARSER_ID;
};
struct common_peg_until_parser {
std::vector<common_peg_until_branch> branches;
bool include = false; // consume the first delimiter and parse the rest of its branch
bool optional = false; // with include, the input may end before any delimiter
common_peg_parser_id content = COMMON_PEG_INVALID_PARSER_ID; // with include, parsed up to the delimiter
common_trie matcher; // each pattern valued by its branch
std::vector<std::string> delimiters;
};
struct common_peg_schema_parser {
@@ -325,9 +288,6 @@ struct common_peg_rule_parser {
std::string name;
common_peg_parser_id child;
bool trigger;
// set when the rule has a start and rest, the lazy grammar scans for the start
common_peg_parser_id start = COMMON_PEG_INVALID_PARSER_ID;
common_peg_parser_id rest = COMMON_PEG_INVALID_PARSER_ID;
};
struct common_peg_ref_parser {
@@ -348,14 +308,17 @@ struct common_peg_gbnf_parser {
std::string grammar;
};
struct common_peg_ac_parser {
common_peg_parser_id child;
std::vector<std::string> delimiters;
};
// Variant holding all parser types
using common_peg_parser_variant = std::variant<
common_peg_epsilon_parser,
common_peg_start_parser,
common_peg_end_parser,
common_peg_literal_parser,
common_peg_token_parser,
common_peg_token_not_parser,
common_peg_sequence_parser,
common_peg_choice_parser,
common_peg_repetition_parser,
@@ -371,14 +334,14 @@ using common_peg_parser_variant = std::variant<
common_peg_ref_parser,
common_peg_atomic_parser,
common_peg_tag_parser,
common_peg_gbnf_parser
common_peg_gbnf_parser,
common_peg_ac_parser
>;
class common_peg_arena {
std::vector<common_peg_parser_variant> parsers_;
std::unordered_map<std::string, common_peg_parser_id> rules_;
common_peg_parser_id root_ = COMMON_PEG_INVALID_PARSER_ID;
common_peg_tokens tokens_;
public:
const common_peg_parser_variant & get(common_peg_parser_id id) const { return parsers_.at(id); }
@@ -393,8 +356,6 @@ class common_peg_arena {
common_peg_parser_id root() const { return root_; }
void set_root(common_peg_parser_id id) { root_ = id; }
const common_peg_tokens & tokens() const { return tokens_; }
common_peg_parse_result parse(common_peg_parse_context & ctx, size_t start = 0) const;
common_peg_parse_result parse(common_peg_parser_id id, common_peg_parse_context & ctx, size_t start) const;
@@ -423,7 +384,6 @@ class common_peg_parser_builder {
public:
common_peg_parser_builder();
explicit common_peg_parser_builder(common_peg_tokens tokens);
// Match nothing, always succeed.
// S -> ε
@@ -441,16 +401,6 @@ class common_peg_parser_builder {
// S -> "hello"
common_peg_parser literal(const std::string & literal) { return add(common_peg_literal_parser{literal}); }
// Matches a token. Falls back to a literal if the token is not registered with the builder.
// S -> <[token-id]>
common_peg_parser token(const std::string & piece);
// Matches one of the tokens. With negate, matches one token that is not in the set.
// A negated set may only name tokens registered with the builder.
// S -> <[id1]> | <[id2]> | ... or !<[id1,id2,...]>
common_peg_parser token(const std::vector<std::string> & pieces, bool negate = false);
common_peg_parser token(std::initializer_list<std::string> pieces, bool negate = false) { return token(std::vector<std::string>(pieces), negate); }
// Matches a sequence of parsers in order, all must succeed.
// S -> A B C
common_peg_parser sequence() { return add(common_peg_sequence_parser{}); }
@@ -507,29 +457,16 @@ class common_peg_parser_builder {
// Matches all characters until a delimiter is found (delimiter not consumed).
// Invalid UTF-8 is consumed and recorded on the AST nodes.
// S -> (!delim .)*
common_peg_parser until(const std::string & delimiter) { return until_one_of({delimiter}); }
common_peg_parser until(const std::string & delimiter) { return add(common_peg_until_parser{{delimiter}}); }
// Matches all characters until one of the delimiters in the list is found (delimiter not consumed).
// S -> (!delim .)*
common_peg_parser until_one_of(const std::vector<std::string> & delimiters);
// Matches all characters until the delimiter is found (delimiter not consumed).
// The delimiter may only be a sequence of literals, tokens, and choices of them. Tags and atomics may wrap them.
// S -> (!delim .)*
common_peg_parser until(const common_peg_parser & delimiter);
common_peg_parser until_one_of(const std::vector<std::string> & delimiters) { return add(common_peg_until_parser{delimiters}); }
// Matches everything
// S -> .*
common_peg_parser rest() { return until_one_of({}); }
// Matches content, then the first delimiter, then the rest of the branch of that delimiter.
// Branches must not share a delimiter. With optional, the input may end before any delimiter.
// S -> C (D1 R1 | D2 R2 | ...)
common_peg_parser through(const common_peg_parser & content, const std::vector<common_peg_until_branch> & branches, bool optional = false);
common_peg_parser through(const common_peg_parser & content, std::initializer_list<common_peg_until_branch> branches, bool optional = false) { return through(content, std::vector<common_peg_until_branch>(branches), optional); }
common_peg_parser through(const common_peg_parser & content, const common_peg_parser & delimiter) { return through(content, { { delimiter } }); }
common_peg_parser through(const common_peg_parser & content, const std::string & delimiter) { return through(content, literal(delimiter)); }
// Matches between min and max repetitions of a parser (inclusive).
// S -> A{m,n}
// Use -1 for max to represent unbounded repetition (equivalent to {m,})
@@ -600,10 +537,6 @@ class common_peg_parser_builder {
common_peg_parser trigger_rule(const std::string & name, const common_peg_parser & p) { return rule(name, p, true); }
common_peg_parser trigger_rule(const std::string & name, const std::function<common_peg_parser()> & builder) { return rule(name, builder, true); }
// Creates a trigger rule that matches start followed by rest. A lazy grammar scans for the start of any such rule, then only allows its rest.
// The start has the same limits as an until() delimiter.
common_peg_parser trigger_rule(const std::string & name, const common_peg_parser & start, const common_peg_parser & rest);
// Creates an atomic parser. Atomic parsers do not create an AST node if
// the child results in a partial parse, i.e. NEEDS_MORE_INPUT. This is
// intended for situations where partial output is undesirable.
@@ -617,6 +550,13 @@ class common_peg_parser_builder {
// the child's grammar. Parsing delegates entirely to the child.
common_peg_parser gbnf(const common_peg_parser & p, const std::string & grammar) { return add(common_peg_gbnf_parser{p, grammar}); }
// Wraps a child parser but emits a GBNF grammar built from the Aho-Corasick
// automaton of `delimiters`, matching everything up to and including the
// first delimiter. Parsing delegates entirely to the child, which is
// responsible for consuming the delimiter (e.g. until(D) + literal(D)).
common_peg_parser ac(const common_peg_parser & p, const std::vector<std::string> & delimiters);
common_peg_parser ac(const common_peg_parser & p, const std::string & delimiter) { return ac(p, std::vector<std::string>{delimiter}); }
void set_root(const common_peg_parser & p);
common_peg_arena build();
+2 -6
View File
@@ -31,18 +31,14 @@ struct token_matcher {
static common_trie build_trie(const std::vector<llama_tokens> & seqs) {
common_trie t;
for (const auto & seq : seqs) {
std::vector<common_trie::symbol> symbols;
for (llama_token token : seq) {
symbols.push_back(common_trie::symbol::token(token));
}
t.insert(symbols);
t.insert(std::vector<uint32_t>(seq.begin(), seq.end()));
}
return t;
}
// returns the index into seqs of the longest sequence ending at this token, or -1
int32_t advance(llama_token token) {
state = ac.next(state, common_trie::symbol::token(token));
state = ac.next(state, (uint32_t) token);
const int32_t p = ac.match_pattern(state);
if (p >= 0) {
state = 0;
+20 -54
View File
@@ -4,46 +4,26 @@
#include <deque>
common_trie::step common_trie::next(std::string_view sv, const std::vector<int32_t> & token_map, size_t pos) const {
if (!tokens.empty() && pos < token_map.size()) {
const int32_t id = token_map[pos];
if (id >= 0 && tokens.find(id) != tokens.end()) {
size_t end = pos + 1;
while (end < token_map.size() && token_map[end] < 0) {
++end;
}
return { utf8_parse_result::SUCCESS, symbol::token(id), end - pos };
}
}
auto result = common_parse_utf8_codepoint(sv, pos);
return { result.status, symbol::codepoint(result.codepoint), result.bytes_consumed };
}
common_trie::match_result common_trie::check_at(std::string_view sv, size_t start_pos) const {
return check_at(sv, {}, start_pos);
}
common_trie::match_result common_trie::check_at(std::string_view sv, const std::vector<int32_t> & token_map, size_t start_pos) const {
size_t current = 0; // Start at root
size_t pos = start_pos;
// LOG_DBG("%s: checking at pos %zu, sv='%s'\n", __func__, start_pos, std::string(sv).c_str());
while (pos < sv.size()) {
auto step = next(sv, token_map, pos);
if (step.status != utf8_parse_result::SUCCESS) {
auto result = common_parse_utf8_codepoint(sv, pos);
if (result.status != utf8_parse_result::SUCCESS) {
break;
}
auto it = nodes[current].children.find(step.symbol);
auto it = nodes[current].children.find(result.codepoint);
if (it == nodes[current].children.end()) {
// Can't continue matching
return match_result{match_result::NO_MATCH};
}
current = it->second;
pos += step.bytes_consumed;
pos += result.bytes_consumed;
// Check if we've matched a complete word
if (nodes[current].pattern >= 0) {
@@ -62,7 +42,7 @@ common_trie::match_result common_trie::check_at(std::string_view sv, const std::
}
int32_t common_trie::insert(const std::string & word) {
std::vector<symbol> symbols;
std::vector<uint32_t> symbols;
size_t pos = 0;
while (pos < word.length()) {
auto result = common_parse_utf8_codepoint(word, pos);
@@ -70,42 +50,28 @@ int32_t common_trie::insert(const std::string & word) {
break;
}
symbols.push_back(symbol::codepoint(result.codepoint));
symbols.push_back(result.codepoint);
pos += result.bytes_consumed;
}
return insert(symbols);
}
int32_t common_trie::insert(const std::vector<symbol> & symbols) {
int32_t common_trie::insert(const std::vector<uint32_t> & symbols) {
size_t current = 0;
for (symbol ch : symbols) {
current = extend(current, ch);
for (uint32_t ch : symbols) {
auto it = nodes[current].children.find(ch);
if (it == nodes[current].children.end()) {
size_t child = create_node();
nodes[current].children[ch] = child;
current = child;
} else {
current = it->second;
}
}
int32_t pattern = mark(current, n_patterns);
if (pattern == n_patterns) {
n_patterns++;
if (nodes[current].pattern < 0) {
nodes[current].pattern = n_patterns++;
}
return pattern;
}
size_t common_trie::extend(size_t node, symbol ch) {
if (ch.is_token()) {
tokens.insert((int32_t) ch.value);
}
auto it = nodes[node].children.find(ch);
if (it != nodes[node].children.end()) {
return it->second;
}
size_t child = create_node();
nodes[node].children[ch] = child;
return child;
}
int32_t common_trie::mark(size_t node, int32_t value) {
if (nodes[node].pattern < 0) {
nodes[node].pattern = value;
}
return nodes[node].pattern;
return nodes[current].pattern;
}
common_aho_corasick::common_aho_corasick(common_trie trie) : t(std::move(trie)) {
@@ -147,7 +113,7 @@ common_aho_corasick::common_aho_corasick(common_trie trie) : t(std::move(trie))
}
}
size_t common_aho_corasick::next(size_t state, common_trie::symbol ch) const {
size_t common_aho_corasick::next(size_t state, uint32_t ch) const {
const auto & nodes = t.nodes;
while (state && nodes[state].children.find(ch) == nodes[state].children.end()) {
state = fail[state];
+10 -51
View File
@@ -1,7 +1,5 @@
#pragma once
#include "unicode.h"
#include <cstdint>
#include <map>
#include <set>
@@ -12,42 +10,13 @@
// Trie for matching multiple literals.
// This is used in common_peg_until_parser and to build a GBNF exclusion grammar
struct common_trie {
// A symbol is either a UTF-8 codepoint or a token
struct symbol {
enum kind_type : uint8_t { CODEPOINT, TOKEN };
kind_type kind = CODEPOINT;
uint32_t value = 0;
static symbol codepoint(uint32_t cpt) { return { CODEPOINT, cpt }; }
static symbol token(int32_t id) { return { TOKEN, (uint32_t) id }; }
bool is_token() const { return kind == TOKEN; }
bool operator<(const symbol & other) const {
return kind != other.kind ? kind < other.kind : value < other.value;
}
bool operator==(const symbol & other) const {
return kind == other.kind && value == other.value;
}
};
struct step {
enum utf8_parse_result::status status;
common_trie::symbol symbol;
size_t bytes_consumed;
};
struct node {
std::map<symbol, size_t> children;
std::map<uint32_t, size_t> children; // Use uint32_t to store Unicode codepoints
int32_t pattern = -1; // index of the pattern ending at this node, -1 if none
};
std::vector<node> nodes;
// every token id used by a pattern, the input is split so these match as single symbols
std::set<int32_t> tokens;
common_trie() {
create_node(); // root node
}
@@ -60,25 +29,15 @@ struct common_trie {
enum match_result { NO_MATCH, PARTIAL_MATCH, COMPLETE_MATCH };
// Read the symbol at pos. It is a token if a pattern uses the token at pos, else a UTF-8 codepoint.
step next(std::string_view sv, const std::vector<int32_t> & token_map, size_t pos) const;
// Check if a delimiter starts at the given position
match_result check_at(std::string_view sv, size_t start_pos) const;
match_result check_at(std::string_view sv, const std::vector<int32_t> & token_map, size_t start_pos) const;
// Insert a word as a sequence of Unicode codepoints, returns its pattern index
int32_t insert(const std::string & word);
// Insert a symbol sequence, returns its pattern index (insertion order, duplicates keep the first index)
int32_t insert(const std::vector<symbol> & symbols);
// Child of node on a symbol, created if missing
size_t extend(size_t node, symbol ch);
// Mark node as the end of a pattern with a value from the caller. Returns the value already there, or value if the node had none.
// A trie must take its values from mark or from insert, not both.
int32_t mark(size_t node, int32_t value);
// Insert a raw symbol sequence, returns its pattern index (insertion order,
// duplicates keep the first index)
int32_t insert(const std::vector<uint32_t> & symbols);
private:
int32_t n_patterns = 0;
@@ -92,11 +51,11 @@ struct common_trie {
// Aho-Corasick automaton
struct common_aho_corasick {
common_trie t;
std::vector<size_t> fail; // failure links
std::vector<size_t> order; // states in BFS order
std::vector<int32_t> match; // longest pattern ending at each state (directly or via a suffix link), -1 if none
std::set<common_trie::symbol> alphabet; // every symbol with a transition
common_trie t;
std::vector<size_t> fail; // failure links
std::vector<size_t> order; // states in BFS order
std::vector<int32_t> match; // longest pattern ending at each state (directly or via a suffix link), -1 if none
std::set<uint32_t> alphabet; // every character with a transition
common_aho_corasick(common_trie trie);
@@ -110,5 +69,5 @@ struct common_aho_corasick {
int32_t match_pattern(size_t s) const { return match[s]; }
// follow failure links until a transition on `ch` exists.
size_t next(size_t state, common_trie::symbol ch) const;
size_t next(size_t state, uint32_t ch) const;
};
-131
View File
@@ -67,8 +67,6 @@ For a more complete example, see `test_example_native()` in
- **`end()`** - Matches the end of input (anchor `$`)
- **`literal(string)`** - Matches an exact literal string
- **`any()`** - Matches any single character (`.`)
- **`token(piece)`** - Matches a special token, or the literal `piece` if it is not registered with the builder (see [Special Tokens](#special-tokens))
- **`token(pieces, negate)`** - Matches one of the tokens, or with `negate` one token that is not one of them
### Combinators
@@ -91,8 +89,6 @@ For a more complete example, see `test_example_native()` in
- **`space()`** - Matches zero or more whitespace characters (space, tab, newline)
- **`until(delimiter)`** - Matches characters until delimiter is found (delimiter not consumed)
- **`until_one_of(delimiters)`** - Matches characters until any delimiter in the list is found
- **`until(parser)`** - Matches characters until a delimiter built from literals and tokens is found (delimiter not consumed, see [until(parser)](#untilparser))
- **`through(content, branches, optional)`** - Matches content, then the first delimiter found and the rest of its branch (delimiter consumed). The grammar scans for the delimiters instead of following content
- **`rest()`** - Matches everything remaining (`.*`)
### JSON Parsers
@@ -112,7 +108,6 @@ For a more complete example, see `test_example_native()` in
- **`ref(name)`** - Creates a lightweight reference to a named rule (for recursive grammars)
- **`rule(name, p, trigger)`** - Creates a named rule and returns a reference
- **`trigger_rule(name, p)`** - Creates a trigger rule (entry point for lazy grammar generation)
- **`trigger_rule(name, start, rest)`** - Creates a trigger rule matching `start` then `rest`, gating the lazy grammar on the start
- **`schema(p, name, schema, raw)`** - Wraps parser with JSON schema metadata for grammar generation
- **`schema(p, name, doc, node, raw)`** - Same, for a node of a `common_chat_schema_document` built earlier, e.g. one tool parameter
@@ -121,110 +116,6 @@ For a more complete example, see `test_example_native()` in
- **`atomic(p)`** - Prevents AST node creation for partial parses
- **`tag(tag, p)`** - Creates AST nodes with semantic tags (multiple nodes can share tags)
## Special Tokens
Matching a special token by its text alone is ambiguous: a model may emit
`<|end|>` as a single control token, or spell it out as plain text, such as
when quoting a chat template. The parser can tell them apart when the builder
knows the special tokens of the vocabulary, and the input carries the ids of
the tokens that produced it.
A `common_peg_tokens` table constructed from a `llama_vocab` registers every
control and user-defined token. Pass it to the builder:
```cpp
common_peg_tokens specials(vocab);
auto parser = build_chat_peg_parser(specials, [&](common_chat_peg_builder & p) {
return p.content(p.until(p.token("<|end|>"))) + p.token("<|end|>");
});
```
Chat templates initialized with a model build this table from its vocabulary,
and format handlers receive it as `inputs.specials`.
### token()
`token(piece)` matches only where the input holds the id of that token, and
consumes the whole piece. The same characters without the id do not match.
If `piece` is not registered with the builder, it falls back to
`literal(piece)`.
`token(pieces)` matches any of the tokens, and `token(pieces, true)` matches one
token that is not in the set. Every piece of a negated set must be registered,
otherwise it throws `std::invalid_argument`.
In the grammar, a token emits `<[id]>` and a negated set emits `!<[...]>`:
```cpp
p.token({ "<a>", "<b>" }) + p.token({ "<a>", "<b>" }, true)
```
```
root ::= (<[1]> | <[2]>) !<[1-2]>
```
### until(parser)
`until(delimiter)` with a parser instead of a string stops at a delimiter that
may contain tokens, or several literals joined in a sequence:
```cpp
auto end = p.token("<|end|>");
auto final = p.token("<|channel|>") + p.literal("final");
p.reasoning(p.until(end | final))
```
The delimiter may only be built from:
- `literal()` and `token()`
- `sequence()` and `choice()` of the above
- `tag()` and `atomic()` wrapping any of the above
Any other parser throws `std::invalid_argument`. A choice inside a sequence
expands to every combination, so `"a" + ("b" | "d")` stops at both `ab` and
`ad`.
The grammar scans for the delimiters the same way as `until_one_of()`:
```cpp
p.until((p.literal("a") + p.literal("b")) | p.literal("cd"))
```
```
root ::= until-5
until-5 ::= | [a] until-5-01 | [c] until-5-03 | [^ac] until-5
until-5-01 ::= | [a] until-5-01 | [c] until-5-03 | [^abc] until-5
until-5-03 ::= | [a] until-5-01 | [c] until-5-03 | [^acd] until-5
```
A token never matches a character rule such as `.` or `[^a]`, so the scan lets
the other tokens named elsewhere in the grammar through by name. Here `<b>` is
token 2:
```cpp
p.until(p.token("<a>") + p.literal("x")) + p.token("<a>") + p.literal("x") + p.token("<b>")
```
```
root ::= until-3 <[1]> "x" <[2]>
until-3 ::= | <[1]> until-3-01 | (. | <[2]>) until-3
until-3-01 ::= | <[1]> until-3-01 | ([^x] | <[2]>) until-3
```
When the delimiters are made of tokens alone, the scan consumes whole tokens
with a negated token set instead:
```cpp
p.until(p.token("<a>")) + p.token("<a>")
```
```
root ::= until-1 <[1]>
until-1 ::= | !<[1]> until-1
```
## GBNF Grammar Generation
The PEG parser also acts as a convenient DSL for generating GBNF grammars, with
@@ -252,28 +143,6 @@ are emitted in the grammar. All trigger rules are added as alternations in the
root rule. It is still necessary to define trigger patterns, as the parser has
no interaction with the grammar sampling.
With `trigger_rule(name, start, rest)`, the grammar instead scans for a start and
then constrains the rest. It runs from the first token, so `grammar_lazy` is false
and no trigger patterns are needed. For example:
```cpp
auto ab = p.literal("ab");
auto cd = p.literal("cd");
return p.until(ab | cd) + (p.trigger_rule("one", ab, p.literal("x")) | p.trigger_rule("two", cd, p.literal("y")));
```
```
root ::= trigger
trigger ::= | [a] trigger-01 | [c] trigger-03 | [^ac] trigger
trigger-01 ::= | [b] one-rest | [a] trigger-01 | [c] trigger-03 | [^abc] trigger
trigger-03 ::= | [d] two-rest | [a] trigger-01 | [c] trigger-03 | [^acd] trigger
one-rest ::= "x"
two-rest ::= "y"
```
A start may only be a sequence of literals, tokens, and choices of them, possibly
wrapped in tags and atomics.
### JSON Schema
The `schema(p, name, schema, raw)` parser will use the `json-schema-to-grammar`
+6 -70
View File
@@ -61,9 +61,7 @@ git log --oneline "$OLD_BASE"..origin/master -- \
**Temporary upstream carries (FORK.md F12)** come first: `git log --oneline --grep '^upstream-carry'
"$OLD_BASE"..HEAD`. For each: merged into the new upstream -> do not cherry-pick it (check that the
merged version is the carried head or newer); still open -> re-fetch the PR head and re-carry it if
it moved; closed -> ask the owner. A PR head can also shrink: on 2026-11-22 #32740 dropped the Vulkan part the fork needs, so the old
head stayed carried (FORK.md F12). Compare `git diff --stat` of the old and new head before re-carrying.
To carry a PR (2026-11-21: #32735, #32740; 2026-11-22: #32777):
it moved; closed -> ask the owner. To carry a PR (2026-11-22: #32735, #32740):
```sh
git fetch https://github.com/ggml-org/llama.cpp pull/<N>/head:pr-<N>
git cherry-pick $(git merge-base pr-<N> HEAD)..pr-<N> # the PR's commits, oldest first; resolve by hand
@@ -108,10 +106,6 @@ git switch -c "$NEW" origin/master
git cherry-pick <first-fork-commit> # one at a time, in the original order
```
**A rebase is not done until `fork-validate.sh` phases 1 and 2 (section 6) pass on a pod build of
the new branch.** A rebased branch that is only picked, checked by reading, or built is unfinished
work, whatever the cherry-picks looked like. Report it as unfinished until both phases pass.
Rules:
- Cherry-pick in the original order, keep authors and messages. New commits you author get the
fork's trailer (`Co-Authored-By: Claude ...` for agent-written commits, FORK.md status block),
@@ -181,28 +175,7 @@ built or run locally; every item below is by reading only and needs the pod buil
`llama_kv_cache_dsa`, `_dsa_iswa`, `_dsv4`, `_msa`, `llama_memory_hybrid_iswa`. The KV rotation
tensors (`attn_rot_bufs`, #31073) are small and are not released either.
**What the 2026-11-22 rebase (upstream `5bacb7c86`, 19 commits past `e6d624b9a`) changed.** Branch
`fable5/vram-ondemand-rebase-20261122`. The 64 picks (65 fork commits minus the #32735 carry) all
applied cleanly; the range-diff against the old branch is identical except for that dropped carry and
the new commits at the end. Reading checks:
- #32735 merged as `8a97fa98a`, patch identical to the carried head `0ea4ffd00`: carry dropped.
- #32740 is still open but was split: its head `8c6f0c0ca` is now only the GGML contract (op API,
CPU reference, test-backend-ops cases); the Vulkan kernel, the `kv_rows` support and the Inkling
graph use were removed from it and are in no open PR (Metal is #32808). Carrying that head would
drop Inkling's in-kernel bias (back to the 15.8 GiB compute buffer at ctx 65536, FORK.md F12). The
validated carry of head `808fd1c4e` is kept as is (FORK.md F12 has the owner decision).
- #32807 (`5bacb7c86`) adds an `LLM_ARCH_INKLING2_LOOP` guard to `sched_copy_experts` (whole-tensor
copies), upstream's fix for #32755. It sits beside F1 in the same function; no conflict. It does
not remove the held per-pass copies, so F13 stays (FORK.md F13, "Upstream #32807").
- #32713 (non-causal prompts fit in one ubatch) touches `update_slots` near the F14 loop but not
the user-start break; F14 unchanged by it.
- #32799 (`ignore_merges` for inkling2) changes Inkling's tokenization: Inkling prompts cached by
the previous build no longer match. The disk cache key holds the build commit, so nothing stale is
restored.
- New commits: the F14 first-user-message break (FORK.md F14), the `LLAMA_LOOP_FULL_COPY` knob
(F13) and the carry of #32777 (F12).
Open questions from the 2026-11-21 rebase:
Open questions from that rebase:
- Is the fork's own concat fast path (F3) now redundant with upstream's tiled concat (#30591)?
Benchmark `GGML_VK_CONCAT_FASTPATH=0` against the default on pp and decode before deciding.
- Does `n_inflight` blocking the want-lock yield delay handoffs? Validation says no: handoff median
@@ -375,35 +348,6 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f
`<scratch>/perf.tsv`, and `fork-validate.sh --perf-report <file>` re-aggregates them offline. A pod
restart empties `/tmp`, so copy a scratch dir you want to keep to `/root/arms/` (the 2026-10-24
scratch dirs were lost that way).
- Overlayfs (since the persistent rootfs, 2026-11-22): `/tmp` is on overlayfs, where `stat`
reports another device than `/proc/locks` prints, so a scratch arena under `/tmp` made every
token check fail (`arbiter/start: A serves but does not hold token.lock` while A held the lock).
`fork-validate.sh` from `70d39bdbc` on keeps its arenas on `/dev/shm`. Use the script of that
commit or later, also for a `--ref` run of an older build.
- Phase 1 of the 2026-11-22 rebase (candidate build `/root/arms/reb20261122/build`, code
`6c895a491`, scripts `70d39bdbc`, vs production `f3397cded`, beside production with 3579 MiB
free; logs `/root/arms/reb20261122-p1.log`, `-p1b.log`): `--ops --concat --diskcache` 23 PASS
(ops 5/5 with FLASH_ATTN_EXT 5693/5693, concat off/cand 5.3x, diskcache 15/15 including the GPU
legs), `--arbiter` 16 PASS on the rerun with the fixed script (handoff median cand 789.5 / ref
768 ms; the first run's single FAIL was the overlayfs mismatch above). F12: test-backend-ops
SCALE and FLASH_ATTN_EXT OK, `test-flash-attn-indexed-bias` 16 Vulkan0 cases, max error 1.3e-4.
Still to run in the window: phase 2 (`--arbiter --perf --moe-exact --moecache`, the runbook
command with this build path) and the Inkling/F14 items below.
- What the 2026-11-22 window must run, in this order (GPU to itself under swapdown; about 75-90
minutes in all):
1. Phase 2 as written above with `W=/root/arms/reb20261122` (about 32 minutes).
2. F13 vs upstream's #32807 (FORK.md F13, "Window A/B to run": memory, `kld48.sh`, speed with
`LLAMA_LOOP_HOST_VIEWS=0` and `LLAMA_LOOP_FULL_COPY=0`).
3. `--fit` vs the k=4 `-ot` for Inkling (FORK.md F12, "How to use it for Inkling"). Smoke run
beside production (3.5 GiB free, so not a placement result; `/root/arms/reb20261122-fit-smoke.log`):
fit's loop path runs on Inkling; the probing took 27.8 s, which every Inkling cold start would
pay with `--fit` in the config (freeze the printed `-ot` instead); the projected memory is not
monotonic in `n_layer` (5173 MiB at 44, 6736 at 43) and fit still reported success with -3116
MiB free. Check both with the full card.
4. F14 refinement (FORK.md F14, validation plan items 5-7): the new-chat checkpoint, its disk-cache
variant and the M-only TTFT table, with the 35B think entry's exact command line.
5. The Inkling post-deploy items of section 8 step 8 against a scratch server of the candidate:
#32799 changed Inkling's tokenization, so judge the answer text.
- F14 run (2026-11-22, candidate `f3397cded` = `166ec22e2` (docs on `b13c41355`) + F14, server-only change; one
swapdown window, `/root/arms/usw-window.sh`, log `/root/arms/usw-window.log`, about 15 minutes):
the 35B TTFT A/B with and without `LLAMA_USERSTART_MERGE=0` (FORK.md F14, M U U M, production 35B
@@ -634,8 +578,8 @@ A wedged arbiter looks like this: requests hang while `/health` still returns 20
shows a process holding WRITE on `token.lock` that is not generating. llama-swap restarts a child
that exits (on the next request for its model, FORK.md F7) but never one that wedges. Inspect it with
`gdb -p <pid> -batch -ex 'thread apply all bt'` before killing it. gdb is not in
the pod image; it was lost in the 2026-10-24 pod restart, but the rootfs persists since 2026-11-22 (section 11), so
an apt install now stays. Check `command -v gdb` and install it first if needed.
the pod image: it was installed by hand and was gone after the 2026-10-24 pod restart, so check
`command -v gdb` and reinstall it first if needed.
## 9. Rollback
@@ -726,14 +670,6 @@ kubectl -n llama exec -i deploy/supervisord -- sh -c '<command>'
kubectl get pods -n llama -l app=llama-swap -o name # the pod name changes across restarts
```
Alternative (since 2026-11-22): the pod runs sshd, so `ssh root@2001:470:61a3:400::7` works instead
of `kubectl exec` (`scp 'root@[2001:470:61a3:400::7]:/path' .` for files). The same rules apply:
detach long runs with `setsid nohup ... </dev/null &` and never `pkill -f`.
The pod's root filesystem now persists across pod restarts: packages installed with apt (gdb,
for example) stay installed. `/tmp` and `/run` are still wiped on every boot, so copy a scratch dir
you want to keep (fork-validate's `/tmp/fork-validate.*`) to `/root/arms/`.
Wiring: k8s service `llama:11434` -> pod port 8080 (selector `app=llama-swap`) -> llama-swap `:8080`;
a Caddy proxy fronts `llama.lumpiasty.xyz`. llama-swap must listen on `:8080`.
@@ -768,8 +704,8 @@ Useful commands:
- Token holder and waiters: `/proc/locks` filtered by the inode of `token.lock` / `want.lock`
(WRITE = holder on token.lock, READ = registered waiters on want.lock).
- 120 W cap active: `cat /sys/class/drm/card*/device/hwmon/hwmon*/power1_cap` = `120000000`.
- The pod has `curl`, `python3`, `flock`; no `jq`, and `gdb` only if installed with apt (it persists
since 2026-11-22; without it `fork-validate.sh` runs its SIGTERM-in-release case by polling the log).
- The pod has `curl`, `python3`, `flock`; no `jq`, and no `gdb` unless reinstalled by hand since the
last pod restart (`fork-validate.sh` then runs its SIGTERM-in-release case by polling the log).
Second test box (optional): `ssh user@gaming-pc`, the owner's NixOS desktop with an RX 6600 (RDNA2).
Pod-built binaries can run there through the host glibc loader without compiling. Numbers from it do
+2 -2
View File
@@ -8,10 +8,10 @@ extern "C" {
#define RPC_PROTO_MAJOR_VERSION 10
#define RPC_PROTO_MINOR_VERSION 0
#define RPC_PROTO_PATCH_VERSION 2
#define RPC_PROTO_PATCH_VERSION 1
#ifdef __cplusplus
static_assert(GGML_OP_COUNT == 101, "GGML_OP_COUNT has changed - update RPC_PROTO_PATCH_VERSION");
static_assert(GGML_OP_COUNT == 100, "GGML_OP_COUNT has changed - update RPC_PROTO_PATCH_VERSION");
#endif
#define GGML_RPC_MAX_SERVERS 16
-9
View File
@@ -433,8 +433,6 @@ extern "C" {
GGML_OP_SLEEP,
GGML_OP_SIGMOID_BACK,
GGML_OP_COUNT,
};
@@ -1040,13 +1038,6 @@ extern "C" {
struct ggml_tensor * a,
struct ggml_tensor * b);
// a - dy
// b - x
GGML_API struct ggml_tensor * ggml_sigmoid_back(
struct ggml_context * ctx,
struct ggml_tensor * a,
struct ggml_tensor * b);
// hardswish(x) = x * relu6(x + 3) / 6
GGML_API struct ggml_tensor * ggml_hardswish(
struct ggml_context * ctx,
-1
View File
@@ -35,7 +35,6 @@ bool ggml_op_can_inplace(enum ggml_op op) {
case GGML_OP_ROPE:
case GGML_OP_ROPE_BACK:
case GGML_OP_SILU_BACK:
case GGML_OP_SIGMOID_BACK:
case GGML_OP_RMS_NORM:
case GGML_OP_RMS_NORM_BACK:
case GGML_OP_CLAMP:
+1 -2
View File
@@ -975,8 +975,7 @@ static struct ggml_backend_meta_split_state ggml_backend_meta_get_split_state(
case GGML_OP_CONCAT: {
split_state = handle_concat(src_ss);
} break;
case GGML_OP_SILU_BACK:
case GGML_OP_SIGMOID_BACK: {
case GGML_OP_SILU_BACK: {
split_state = handle_generic(src_ss, /*scalar_only =*/ false);
} break;
case GGML_OP_NORM:
-5
View File
@@ -1884,10 +1884,6 @@ static void ggml_compute_forward(struct ggml_compute_params * params, struct ggm
{
ggml_compute_forward_silu_back(params, tensor);
} break;
case GGML_OP_SIGMOID_BACK:
{
ggml_compute_forward_sigmoid_back(params, tensor);
} break;
case GGML_OP_NORM:
{
ggml_compute_forward_norm(params, tensor);
@@ -2396,7 +2392,6 @@ static int ggml_get_n_tasks(struct ggml_tensor * node, int n_threads) {
}
break;
case GGML_OP_SILU_BACK:
case GGML_OP_SIGMOID_BACK:
case GGML_OP_MUL:
case GGML_OP_DIV:
case GGML_OP_NORM:
-112
View File
@@ -2898,118 +2898,6 @@ void ggml_compute_forward_silu_back(
}
}
// ggml_compute_forward_sigmoid_back
static void ggml_compute_forward_sigmoid_back_f32(
const ggml_compute_params * params,
ggml_tensor * dst) {
const ggml_tensor * grad = dst->src[0];
const ggml_tensor * src1 = dst->src[1];
GGML_ASSERT(ggml_is_contiguous_1(grad));
GGML_ASSERT(ggml_is_contiguous_1(src1));
GGML_ASSERT(ggml_is_contiguous_1(dst));
GGML_ASSERT(ggml_are_same_shape(src1, dst));
GGML_ASSERT(ggml_are_same_shape(src1, grad));
const int ith = params->ith;
const int nth = params->nth;
const int nc = src1->ne[0];
const int nr = ggml_nrows(src1);
// rows per thread
const int dr = (nr + nth - 1)/nth;
// row range for this thread
const int ir0 = dr*ith;
const int ir1 = MIN(ir0 + dr, nr);
for (int i1 = ir0; i1 < ir1; i1++) {
ggml_vec_sigmoid_backward_f32(nc,
(float *) ((char *) dst->data + i1*( dst->nb[1])),
(float *) ((char *) src1->data + i1*(src1->nb[1])),
(float *) ((char *) grad->data + i1*(grad->nb[1])));
#ifndef NDEBUG
for (int k = 0; k < nc; k++) {
const float x = ((float *) ((char *) dst->data + i1*( dst->nb[1])))[k];
GGML_UNUSED(x);
assert(!isnan(x));
assert(!isinf(x));
}
#endif // NDEBUG
}
}
static void ggml_compute_forward_sigmoid_back_f16(
const ggml_compute_params * params,
ggml_tensor * dst) {
const ggml_tensor * grad = dst->src[0];
const ggml_tensor * src1 = dst->src[1];
GGML_ASSERT(ggml_is_contiguous_1(grad));
GGML_ASSERT(ggml_is_contiguous_1(src1));
GGML_ASSERT(ggml_is_contiguous_1(dst));
GGML_ASSERT(ggml_are_same_shape(src1, dst));
GGML_ASSERT(ggml_are_same_shape(src1, grad));
const int ith = params->ith;
const int nth = params->nth;
const int nc = src1->ne[0];
const int nr = ggml_nrows(src1);
// rows per thread
const int dr = (nr + nth - 1)/nth;
// row range for this thread
const int ir0 = dr*ith;
const int ir1 = MIN(ir0 + dr, nr);
for (int i1 = ir0; i1 < ir1; i1++) {
ggml_vec_sigmoid_backward_f16(nc,
(ggml_fp16_t *) ((char *) dst->data + i1*( dst->nb[1])),
(ggml_fp16_t *) ((char *) src1->data + i1*(src1->nb[1])),
(ggml_fp16_t *) ((char *) grad->data + i1*(grad->nb[1])));
#ifndef NDEBUG
for (int k = 0; k < nc; k++) {
const float x = ((ggml_fp16_t *) ((char *) dst->data + i1*( dst->nb[1])))[k];
const float v = GGML_CPU_FP16_TO_FP32(x);
GGML_UNUSED(v);
assert(!isnan(v));
assert(!isinf(v));
}
#endif // NDEBUG
}
}
void ggml_compute_forward_sigmoid_back(
const ggml_compute_params * params,
ggml_tensor * dst) {
const ggml_tensor * src0 = dst->src[0];
switch (src0->type) {
case GGML_TYPE_F32:
{
ggml_compute_forward_sigmoid_back_f32(params, dst);
} break;
case GGML_TYPE_F16:
{
ggml_compute_forward_sigmoid_back_f16(params, dst);
} break;
default:
{
GGML_ABORT("fatal error");
}
}
}
// ggml_compute_forward_reglu
// row loop shared by the BF16 GLU ops
-1
View File
@@ -41,7 +41,6 @@ void ggml_compute_forward_repeat(const struct ggml_compute_params * params, stru
void ggml_compute_forward_repeat_back(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_concat(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_silu_back(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_sigmoid_back(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_norm(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_rms_norm(const struct ggml_compute_params * params, struct ggml_tensor * dst);
void ggml_compute_forward_rms_norm_mul_fused(const struct ggml_compute_params * params, struct ggml_tensor * dst_rms_norm, struct ggml_tensor * dst_mul);
-23
View File
@@ -1536,29 +1536,6 @@ inline static void ggml_vec_silu_backward_f16(const int n, ggml_fp16_t * dx, con
}
}
inline static float ggml_sigmoid_backward_f32(float x, float dy) {
const float s = 1.0f/(1.0f + expf(-x));
return dy*s*(1.0f - s);
}
inline static ggml_fp16_t ggml_sigmoid_backward_f16(ggml_fp16_t x, ggml_fp16_t dy) {
const float v = GGML_CPU_FP16_TO_FP32(x);
const float s = 1.0f/(1.0f + expf(-v));
return GGML_CPU_FP32_TO_FP16(GGML_CPU_FP16_TO_FP32(dy)*s*(1.0f - s));
}
inline static void ggml_vec_sigmoid_backward_f32(const int n, float * dx, const float * x, const float * dy) {
for (int i = 0; i < n; ++i) {
dx[i] = ggml_sigmoid_backward_f32(x[i], dy[i]);
}
}
inline static void ggml_vec_sigmoid_backward_f16(const int n, ggml_fp16_t * dx, const ggml_fp16_t * x, const ggml_fp16_t * dy) {
for (int i = 0; i < n; ++i) {
dx[i] = ggml_sigmoid_backward_f16(x[i], dy[i]);
}
}
inline static void ggml_vec_reglu_f32 (const int n, float * y, const float * x, const float * g) {
for (int i = 0; i < n; ++i) {
y[i] = (x[i] > 0.f) ? x[i] * g[i] : 0.f;
+1 -1
View File
@@ -911,7 +911,7 @@ static __device__ __forceinline__ float ggml_cuda_ue4m3_to_fp32(uint8_t x) {
// but CDNA3 supports only e4m3_fnuz (no inf).
const uint32_t bits = x * (x != 0x7F && x != 0xFF); // Convert NaN to 0.0f to match CPU implementation.
const __hip_fp8_e4m3_fnuz xf = *reinterpret_cast<const __hip_fp8_e4m3_fnuz *>(&bits);
return static_cast<float>(xf);
return static_cast<float>(xf) / 2;
#else
#if defined(FP8_AVAILABLE) && !defined(GGML_USE_HIP)
const uint32_t bits = x * (x != 0x7F && x != 0xFF); // Convert NaN to 0.0f to match CPU implementation.
-7
View File
@@ -227,13 +227,6 @@ set(GGML_OPENCL_KERNELS
gemm_coopk_q4_0_q8_1_dp4a
gemm_coopk_q4_k_q8_1_dp4a
gemm_coopk_q6_k_q8_1_dp4a
pack_b_f16_coopmat
gemm_noshuffle_q4_0_f32_coopmat
gemm_noshuffle_q5_0_f32_coopmat
gemm_noshuffle_q8_0_f32_coopmat
gemm_noshuffle_q4_k_f32_coopmat
gemm_noshuffle_q5_k_f32_coopmat
gemm_noshuffle_q6_k_f32_coopmat
quant_a_q8_1
gemv_moe_q4_k_f32_ns
gemm_moe_q5_k_f32_ns
+4 -219
View File
@@ -1495,13 +1495,6 @@ struct ggml_backend_opencl_context {
cl_kernel kernel_gemm_noshuffle_iq4_nl_q8_1_dp4a = nullptr; // dp4a (int8) dense IQ4_NL prefill GEMM
cl_kernel kernel_gemm_noshuffle_q4_0_q8_1_dp4a = nullptr; // dp4a (int8) dense q4_0 prefill GEMM
cl_kernel kernel_gemm_noshuffle_q4_0_q8_1_dp4a_act_vload4 = nullptr;
cl_kernel kernel_pack_b_f16_coopmat = nullptr;
cl_kernel kernel_gemm_noshuffle_q4_0_f32_coopmat = nullptr;
cl_kernel kernel_gemm_noshuffle_q5_0_f32_coopmat = nullptr;
cl_kernel kernel_gemm_noshuffle_q8_0_f32_coopmat = nullptr;
cl_kernel kernel_gemm_noshuffle_q4_k_f32_coopmat = nullptr;
cl_kernel kernel_gemm_noshuffle_q5_k_f32_coopmat = nullptr;
cl_kernel kernel_gemm_noshuffle_q6_k_f32_coopmat = nullptr;
cl_kernel kernel_gemm_coopk_q4_0_q8_1_dp4a_c4 = nullptr;
cl_kernel kernel_gemm_coopk_q4_0_q8_1_dp4a_c2 = nullptr;
int nsg_coopk_dp4a_q4_0 = 8;
@@ -1986,137 +1979,6 @@ static void load_cl_kernels_gemm_coopk(ggml_backend_opencl_context * backend_ctx
}
}
static void load_cl_kernels_coopmat(ggml_backend_opencl_context * backend_ctx) {
if (backend_ctx->gpu_family != GPU_FAMILY::ADRENO || !strstr(backend_ctx->device_name.c_str(), "850")) {
return;
}
cl_int err;
const std::string opts = "-cl-std=CL" + std::to_string(backend_ctx->opencl_c_version.major) + "." +
std::to_string(backend_ctx->opencl_c_version.minor) +
(backend_ctx->adreno_use_large_buffer ? " -qcom-enable-large-buffer" : "");
// pack_b_f16_coopmat
{
#ifdef GGML_OPENCL_EMBED_KERNELS
const std::string kernel_src {
#include "pack_b_f16_coopmat.cl.h"
};
#else
const std::string kernel_src = read_file("pack_b_f16_coopmat.cl");
#endif
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
if (!prog) {
return;
}
CL_CHECK((backend_ctx->kernel_pack_b_f16_coopmat = clCreateKernel(prog, "kernel_pack_b_f16_coopmat", &err), err));
CL_CHECK(clReleaseProgram(prog));
GGML_LOG_CONT(".");
}
// gemm_noshuffle_q4_0_f32_coopmat
{
#ifdef GGML_OPENCL_EMBED_KERNELS
const std::string kernel_src {
#include "gemm_noshuffle_q4_0_f32_coopmat.cl.h"
};
#else
const std::string kernel_src = read_file("gemm_noshuffle_q4_0_f32_coopmat.cl");
#endif
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
if (prog) {
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q4_0_f32_coopmat = clCreateKernel(prog, "kernel_gemm_noshuffle_q4_0_f32_coopmat", &err), err));
CL_CHECK(clReleaseProgram(prog));
GGML_LOG_CONT(".");
}
}
// gemm_noshuffle_q5_0_f32_coopmat
{
#ifdef GGML_OPENCL_EMBED_KERNELS
const std::string kernel_src {
#include "gemm_noshuffle_q5_0_f32_coopmat.cl.h"
};
#else
const std::string kernel_src = read_file("gemm_noshuffle_q5_0_f32_coopmat.cl");
#endif
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
if (prog) {
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q5_0_f32_coopmat = clCreateKernel(prog, "kernel_gemm_noshuffle_q5_0_f32_coopmat", &err), err));
CL_CHECK(clReleaseProgram(prog));
GGML_LOG_CONT(".");
}
}
// gemm_noshuffle_q8_0_f32_coopmat
{
#ifdef GGML_OPENCL_EMBED_KERNELS
const std::string kernel_src {
#include "gemm_noshuffle_q8_0_f32_coopmat.cl.h"
};
#else
const std::string kernel_src = read_file("gemm_noshuffle_q8_0_f32_coopmat.cl");
#endif
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
if (prog) {
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q8_0_f32_coopmat = clCreateKernel(prog, "kernel_gemm_noshuffle_q8_0_f32_coopmat", &err), err));
CL_CHECK(clReleaseProgram(prog));
GGML_LOG_CONT(".");
}
}
// gemm_noshuffle_q4_k_f32_coopmat
{
#ifdef GGML_OPENCL_EMBED_KERNELS
const std::string kernel_src {
#include "gemm_noshuffle_q4_k_f32_coopmat.cl.h"
};
#else
const std::string kernel_src = read_file("gemm_noshuffle_q4_k_f32_coopmat.cl");
#endif
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
if (prog) {
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q4_k_f32_coopmat = clCreateKernel(prog, "kernel_gemm_noshuffle_q4_k_f32_coopmat", &err), err));
CL_CHECK(clReleaseProgram(prog));
GGML_LOG_CONT(".");
}
}
// gemm_noshuffle_q5_k_f32_coopmat
{
#ifdef GGML_OPENCL_EMBED_KERNELS
const std::string kernel_src {
#include "gemm_noshuffle_q5_k_f32_coopmat.cl.h"
};
#else
const std::string kernel_src = read_file("gemm_noshuffle_q5_k_f32_coopmat.cl");
#endif
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
if (prog) {
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q5_k_f32_coopmat = clCreateKernel(prog, "kernel_gemm_noshuffle_q5_k_f32_coopmat", &err), err));
CL_CHECK(clReleaseProgram(prog));
GGML_LOG_CONT(".");
}
}
// gemm_noshuffle_q6_k_f32_coopmat
{
#ifdef GGML_OPENCL_EMBED_KERNELS
const std::string kernel_src {
#include "gemm_noshuffle_q6_k_f32_coopmat.cl.h"
};
#else
const std::string kernel_src = read_file("gemm_noshuffle_q6_k_f32_coopmat.cl");
#endif
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
if (prog) {
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q6_k_f32_coopmat = clCreateKernel(prog, "kernel_gemm_noshuffle_q6_k_f32_coopmat", &err), err));
CL_CHECK(clReleaseProgram(prog));
GGML_LOG_CONT(".");
}
}
}
#endif
static std::string ggml_opencl_make_compile_opts(const ggml_backend_opencl_context * backend_ctx) {
@@ -7951,7 +7813,6 @@ static void load_cl_kernels(ggml_backend_opencl_context *backend_ctx) {
}
}
load_cl_kernels_gemm_coopk(backend_ctx);
load_cl_kernels_coopmat(backend_ctx);
#endif // GGML_OPENCL_USE_ADRENO_KERNELS
GGML_LOG_CONT("\n");
backend_ctx->kernels_loaded = true;
@@ -11878,8 +11739,7 @@ inline bool use_q4_0_bin_kernels(const ggml_backend_opencl_context *backend_ctx,
#ifdef GGML_OPENCL_USE_ADRENO_KERNELS
if (!backend_ctx->kernel_gemv_noshuffle_q4_0_f32_32b_trans ||
!(backend_ctx->kernel_gemm_noshuffle_q4_0_f32_32b_trans_ila_a8_bin ||
backend_ctx->kernel_gemm_noshuffle_q4_0_f32_32b_trans_ila_a7_bin ||
backend_ctx->kernel_gemm_noshuffle_q4_0_f32_coopmat)) {
backend_ctx->kernel_gemm_noshuffle_q4_0_f32_32b_trans_ila_a7_bin)) {
return false;
}
@@ -12033,8 +11893,7 @@ inline bool use_q6_k_bin_kernels(const ggml_backend_opencl_context *backend_ctx,
#ifdef GGML_OPENCL_USE_ADRENO_KERNELS
if (!backend_ctx->kernel_gemv_noshuffle_q6_k_f32_32b_trans ||
!(backend_ctx->kernel_gemm_noshuffle_q6_k_f32_32b_trans_ila_a8_bin ||
backend_ctx->kernel_gemm_noshuffle_q6_k_f32_32b_trans_ila_a7_bin ||
backend_ctx->kernel_gemm_noshuffle_q6_k_f32_coopmat)) {
backend_ctx->kernel_gemm_noshuffle_q6_k_f32_32b_trans_ila_a7_bin)) {
return false;
}
// bin kernels require ne2 == 1 and ne3 == 1 for weights
@@ -12052,8 +11911,7 @@ inline bool use_q4_k_bin_kernels(const ggml_backend_opencl_context *backend_ctx,
#ifdef GGML_OPENCL_USE_ADRENO_KERNELS
if (!backend_ctx->kernel_gemv_noshuffle_q4_k_f32_32b_trans ||
!(backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a8_bin ||
backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a7_bin ||
backend_ctx->kernel_gemm_noshuffle_q4_k_f32_coopmat)) {
backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a7_bin)) {
return false;
}
// bin kernels require ne2 == 1 and ne3 == 1 for weights
@@ -12071,8 +11929,7 @@ inline bool use_q5_k_bin_kernels(const ggml_backend_opencl_context *backend_ctx,
#ifdef GGML_OPENCL_USE_ADRENO_KERNELS
if (!backend_ctx->kernel_gemv_noshuffle_q5_k_f32_32b_trans ||
!(backend_ctx->kernel_gemm_noshuffle_q5_k_f32_32b_trans_ila_a8_bin ||
backend_ctx->kernel_gemm_noshuffle_q5_k_f32_32b_trans_ila_a7_bin ||
backend_ctx->kernel_gemm_noshuffle_q5_k_f32_coopmat)) {
backend_ctx->kernel_gemm_noshuffle_q5_k_f32_32b_trans_ila_a7_bin)) {
return false;
}
return (tensor->ne[0] % 256 == 0) && (tensor->ne[1] % 64 == 0);
@@ -25762,48 +25619,6 @@ static void ggml_cl_mul_mat_q2_0_f32_adreno(ggml_backend_t backend, const ggml_t
CL_CHECK(clReleaseMemObject(b_sub_buf));
}
static bool ggml_cl_mul_mat_coopmat(ggml_backend_opencl_context * backend_ctx, cl_kernel kernel,
std::initializer_list<cl_mem> weights,
const ggml_tensor * src1, ggml_tensor * dst, int K, int M, int N) {
if (kernel == nullptr || M % 64 != 0 || N < 2) {
return false;
}
ggml_tensor_extra_cl * extra1 = (ggml_tensor_extra_cl *)src1->extra;
ggml_tensor_extra_cl * extrad = (ggml_tensor_extra_cl *)dst->extra;
cl_ulong offset1 = extra1->offset + src1->view_offs;
cl_ulong offsetd = extrad->offset + dst->view_offs;
const int n_tiles = CEIL_DIV(N, 64);
backend_ctx->prealloc_act_trans.allocate(backend_ctx->context, (size_t)n_tiles * 64 * K * sizeof(cl_half));
cl_kernel pk = backend_ctx->kernel_pack_b_f16_coopmat;
CL_CHECK(clSetKernelArg(pk, 0, sizeof(cl_mem), &extra1->data_device));
CL_CHECK(clSetKernelArg(pk, 1, sizeof(cl_ulong), &offset1));
CL_CHECK(clSetKernelArg(pk, 2, sizeof(cl_mem), &backend_ctx->prealloc_act_trans.buffer));
CL_CHECK(clSetKernelArg(pk, 3, sizeof(int), &K));
CL_CHECK(clSetKernelArg(pk, 4, sizeof(int), &N));
size_t p_local[2] = { 64, 4 };
size_t p_global[2] = { (size_t)n_tiles * 64, (size_t)CEIL_DIV(K, 64) * 4 };
backend_ctx->enqueue_ndrange_kernel(pk, 2, p_global, p_local, dst);
cl_uint ka = 0;
for (cl_mem w : weights) {
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(cl_mem), &w));
}
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(cl_mem), &backend_ctx->prealloc_act_trans.buffer));
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(cl_mem), &extrad->data_device));
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(cl_ulong), &offsetd));
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(int), &K));
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(int), &M));
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(int), &N));
size_t c_local[3] = { 64, 1, 4 };
size_t c_global[3] = { 64, (size_t)(M / 64), (size_t)CEIL_DIV(n_tiles, 4) * 4 };
backend_ctx->enqueue_ndrange_kernel(kernel, 3, c_global, c_local, dst);
return true;
}
static void ggml_cl_mul_mat_q4_0_f32_adreno_ila(ggml_backend_t backend, const ggml_tensor * src0,
const ggml_tensor * src1, ggml_tensor * dst) {
GGML_ASSERT(src0);
@@ -26119,11 +25934,6 @@ static void ggml_cl_mul_mat_q4_0_f32_adreno(ggml_backend_t backend, const ggml_t
const bool use_bin = use_q4_0_bin_kernels(backend_ctx, src0);
if (use_bin && ggml_cl_mul_mat_coopmat(backend_ctx, backend_ctx->kernel_gemm_noshuffle_q4_0_f32_coopmat,
{ extra0_q4_0->q, extra0_q4_0->d }, src1, dst, K, M, N)) {
return;
}
if (use_bin) {
if (use_q40_mc3) {
static bool warned = false;
@@ -26688,11 +26498,6 @@ static void ggml_cl_mul_mat_q5_0_f32_adreno(ggml_backend_t backend, const ggml_t
int N = ne1;
int K = ne00;
if (ggml_cl_mul_mat_coopmat(backend_ctx, backend_ctx->kernel_gemm_noshuffle_q5_0_f32_coopmat,
{ extra0_q5_0->qs, extra0_q5_0->qh, extra0_q5_0->d }, src1, dst, K, M, N)) {
return;
}
if (ne1 == 1) {
cl_mem qs_img = nullptr;
cl_mem b_sub_buf = nullptr;
@@ -27535,11 +27340,6 @@ static void ggml_cl_mul_mat_q8_0_f32_adreno(ggml_backend_t backend, const ggml_t
int N = ne1;
int K = ne00;
if (ggml_cl_mul_mat_coopmat(backend_ctx, backend_ctx->kernel_gemm_noshuffle_q8_0_f32_coopmat,
{ extra0_q8_0->q, extra0_q8_0->d }, src1, dst, K, M, N)) {
return;
}
// Multi-column verify GEMV for ne1 2..4, reuses the ne1 == 1 image setup.
// Default on for X2E, GGML_OPENCL_Q8_0_MC=0/1 forces it.
static const char * q8_mc_env = getenv("GGML_OPENCL_Q8_0_MC");
@@ -28521,11 +28321,6 @@ static void ggml_cl_mul_mat_q4_k_f32_adreno(ggml_backend_t backend, const ggml_t
const bool use_bin = use_q4_k_bin_kernels(backend_ctx, src0);
if (use_bin && ggml_cl_mul_mat_coopmat(backend_ctx, backend_ctx->kernel_gemm_noshuffle_q4_k_f32_coopmat,
{ extra0_q4_k->q, extra0_q4_k->s, extra0_q4_k->d, extra0_q4_k->dm }, src1, dst, K, M, N)) {
return;
}
if (use_bin) {
if (use_mc3) {
static bool warned = false;
@@ -29436,11 +29231,6 @@ static void ggml_cl_mul_mat_q6_K_f32_adreno(ggml_backend_t backend, const ggml_t
const bool use_bin = use_q6_k_bin_kernels(backend_ctx, src0);
if (use_bin && ggml_cl_mul_mat_coopmat(backend_ctx, backend_ctx->kernel_gemm_noshuffle_q6_k_f32_coopmat,
{ extra0_q6_K->ql, extra0_q6_K->qh, extra0_q6_K->s, extra0_q6_K->d }, src1, dst, ne00, ne01, ne1)) {
return;
}
if (use_bin) {
if (use_q6k_mc3 || use_q6k_tiled_mc) {
static bool warned = false;
@@ -30181,11 +29971,6 @@ static void ggml_cl_mul_mat_q5_K_f32_adreno(ggml_backend_t backend, const ggml_t
const bool use_bin = use_q5_k_bin_kernels(backend_ctx, src0);
if (use_bin && ggml_cl_mul_mat_coopmat(backend_ctx, backend_ctx->kernel_gemm_noshuffle_q5_k_f32_coopmat,
{ extra0_q5_k->q, extra0_q5_k->qh, extra0_q5_k->s, extra0_q5_k->d, extra0_q5_k->dm }, src1, dst, K, M, N)) {
return;
}
if (use_bin) {
if (use_q5k_mc3) {
static bool warned = false;
@@ -1,145 +0,0 @@
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
#pragma OPENCL EXTENSION cl_qcom_kernel_cooperative_matrix : enable
// Q4_0 x F32 GEMM with cl_qcom_kernel_cooperative_matrix: one 64x64 f16 C tile per wave,
// A dequantized in registers, B packed by kernel_pack_b_f16_coopmat.
typedef half mat_a_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 16, CLK_COOPERATIVE_MATRIX_A_QCOM)));
typedef half mat_b_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 16, 64, CLK_COOPERATIVE_MATRIX_B_QCOM)));
typedef half mat_c_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 64, CLK_COOPERATIVE_MATRIX_ACCUMULATOR_QCOM)));
typedef half half1024 __attribute__((ext_vector_type(1024)));
typedef half half4096 __attribute__((ext_vector_type(4096)));
typedef union {
half1024 v;
half16 h;
} a_row_t;
typedef union {
half4096 v;
struct {
half16 h0;
half16 h1;
half16 h2;
half16 h3;
} s;
} c_row_t;
inline half8 dequant8(uint v, half d) {
uint4 t;
t.s0 = ( v & 0x000F) | ((v << 12) & 0x000F0000);
t.s1 = ((v >> 8) & 0x000F) | ((v << 4) & 0x000F0000);
t.s2 = ((v >> 16) & 0x000F) | ((v >> 4) & 0x000F0000);
t.s3 = ((v >> 24) & 0x000F) | ((v >> 12) & 0x000F0000);
t |= (uint4)(0x64006400);
return (as_half8(t) - (half8)(1032.0h)) * d;
}
inline void store_c16(global float * p, half16 h, int j0, int nv, int M) {
float16 f = convert_float16(h);
if (j0 + 0 < nv) p[(ulong)(j0 + 0) * M] = f.s0;
if (j0 + 1 < nv) p[(ulong)(j0 + 1) * M] = f.s1;
if (j0 + 2 < nv) p[(ulong)(j0 + 2) * M] = f.s2;
if (j0 + 3 < nv) p[(ulong)(j0 + 3) * M] = f.s3;
if (j0 + 4 < nv) p[(ulong)(j0 + 4) * M] = f.s4;
if (j0 + 5 < nv) p[(ulong)(j0 + 5) * M] = f.s5;
if (j0 + 6 < nv) p[(ulong)(j0 + 6) * M] = f.s6;
if (j0 + 7 < nv) p[(ulong)(j0 + 7) * M] = f.s7;
if (j0 + 8 < nv) p[(ulong)(j0 + 8) * M] = f.s8;
if (j0 + 9 < nv) p[(ulong)(j0 + 9) * M] = f.s9;
if (j0 + 10 < nv) p[(ulong)(j0 + 10) * M] = f.sa;
if (j0 + 11 < nv) p[(ulong)(j0 + 11) * M] = f.sb;
if (j0 + 12 < nv) p[(ulong)(j0 + 12) * M] = f.sc;
if (j0 + 13 < nv) p[(ulong)(j0 + 13) * M] = f.sd;
if (j0 + 14 < nv) p[(ulong)(j0 + 14) * M] = f.se;
if (j0 + 15 < nv) p[(ulong)(j0 + 15) * M] = f.sf;
}
#define LOAD_Q(qv, dd, o) { \
qv.s0 = qp[(4 * (o) + 0) * M]; \
qv.s1 = qp[(4 * (o) + 1) * M]; \
qv.s2 = qp[(4 * (o) + 2) * M]; \
qv.s3 = qp[(4 * (o) + 3) * M]; \
dd = dp[(o) * M]; }
#define MMA_K32(qv, dd, b) { \
a_row_t ua0; \
a_row_t ua1; \
ua0.h = (half16)(dequant8(qv.s0, dd), dequant8(qv.s1, dd)); \
ua1.h = (half16)(dequant8(qv.s2, dd), dequant8(qv.s3, dd)); \
mat_a_t a0 = __builtin_astype(ua0.v, mat_a_t); \
mat_a_t a1 = __builtin_astype(ua1.v, mat_a_t); \
mat_b_t b0 = qcom_coop_mat_load((b), CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
mat_b_t b1 = qcom_coop_mat_load((b) + 1024, CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
acc = qcom_coop_mat_mulAdd(a0, b0, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); \
acc = qcom_coop_mat_mulAdd(a1, b1, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); }
kernel void kernel_gemm_noshuffle_q4_0_f32_coopmat(
global const uint * src0_q, // A: q4_0 quants, [K/8][M] u32
global const half * src0_d, // A: q4_0 scales, [K/32][M] f16
global const half * src1, // B: packed f16 tiles
global char * dst, // C: [N][M] f32
ulong offsetd,
int K,
int M,
int N
) {
const int lane = get_local_id(0);
const int row = get_global_id(1) * 64;
const int tn = get_global_id(2);
const int n_tiles = (N + 63) / 64;
if (tn >= n_tiles) {
return;
}
global const uint * qp = src0_q + row + lane;
global const half * dp = src0_d + row + lane;
global const half * bp = src1 + (ulong)tn * (K / 16) * 1024;
mat_c_t acc = qcom_coop_mat_init(0.0);
{
const int odd = (K / 32) & 1;
const int ob = odd ? 0 : 1;
uint4 qa;
uint4 qb;
half da;
half db;
LOAD_Q(qa, da, 0)
LOAD_Q(qb, db, ob)
da = odd ? 0.0h : da;
MMA_K32(qa, da, bp)
MMA_K32(qb, db, bp + ob * 2048)
qp += (ob + 1) * 4 * M;
dp += (ob + 1) * M;
bp += (ob + 1) * 2048;
}
#pragma unroll 1
for (int k0 = (K / 32) & 1 ? 32 : 64; k0 < K; k0 += 64) {
uint4 qa;
uint4 qb;
half da;
half db;
LOAD_Q(qa, da, 0)
LOAD_Q(qb, db, 1)
MMA_K32(qa, da, bp)
MMA_K32(qb, db, bp + 2048)
qp += 8 * M;
dp += 2 * M;
bp += 4096;
}
c_row_t uc;
uc.v = __builtin_astype(acc, half4096);
global float * p = (global float *)(dst + offsetd) + (ulong)tn * 64 * M + row + lane;
const int nv = N - tn * 64;
store_c16(p, uc.s.h0, 0, nv, M);
store_c16(p, uc.s.h1, 16, nv, M);
store_c16(p, uc.s.h2, 32, nv, M);
store_c16(p, uc.s.h3, 48, nv, M);
}
@@ -1,148 +0,0 @@
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
#pragma OPENCL EXTENSION cl_qcom_kernel_cooperative_matrix : enable
// Q4_K x F32 GEMM with cl_qcom_kernel_cooperative_matrix: one 64x64 f16 C tile per wave,
// A dequantized in registers, B packed by kernel_pack_b_f16_coopmat.
#define K_SCALE_SIZE 12
typedef half mat_a_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 16, CLK_COOPERATIVE_MATRIX_A_QCOM)));
typedef half mat_b_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 16, 64, CLK_COOPERATIVE_MATRIX_B_QCOM)));
typedef half mat_c_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 64, CLK_COOPERATIVE_MATRIX_ACCUMULATOR_QCOM)));
typedef half half1024 __attribute__((ext_vector_type(1024)));
typedef half half4096 __attribute__((ext_vector_type(4096)));
typedef union {
half1024 v;
half16 h;
} a_row_t;
typedef union {
half4096 v;
struct {
half16 h0;
half16 h1;
half16 h2;
half16 h3;
} s;
} c_row_t;
inline half8 dequant8(uint v, half scale, half nminv) {
uint4 t;
t.s0 = ( v & 0x000F) | ((v << 12) & 0x000F0000);
t.s1 = ((v >> 8) & 0x000F) | ((v << 4) & 0x000F0000);
t.s2 = ((v >> 16) & 0x000F) | ((v >> 4) & 0x000F0000);
t.s3 = ((v >> 24) & 0x000F) | ((v >> 12) & 0x000F0000);
t |= (uint4)(0x64006400);
return (as_half8(t) - (half8)(1024.0h)) * scale + nminv;
}
inline void store_c16(global float * p, half16 h, int j0, int nv, int M) {
float16 f = convert_float16(h);
if (j0 + 0 < nv) p[(ulong)(j0 + 0) * M] = f.s0;
if (j0 + 1 < nv) p[(ulong)(j0 + 1) * M] = f.s1;
if (j0 + 2 < nv) p[(ulong)(j0 + 2) * M] = f.s2;
if (j0 + 3 < nv) p[(ulong)(j0 + 3) * M] = f.s3;
if (j0 + 4 < nv) p[(ulong)(j0 + 4) * M] = f.s4;
if (j0 + 5 < nv) p[(ulong)(j0 + 5) * M] = f.s5;
if (j0 + 6 < nv) p[(ulong)(j0 + 6) * M] = f.s6;
if (j0 + 7 < nv) p[(ulong)(j0 + 7) * M] = f.s7;
if (j0 + 8 < nv) p[(ulong)(j0 + 8) * M] = f.s8;
if (j0 + 9 < nv) p[(ulong)(j0 + 9) * M] = f.s9;
if (j0 + 10 < nv) p[(ulong)(j0 + 10) * M] = f.sa;
if (j0 + 11 < nv) p[(ulong)(j0 + 11) * M] = f.sb;
if (j0 + 12 < nv) p[(ulong)(j0 + 12) * M] = f.sc;
if (j0 + 13 < nv) p[(ulong)(j0 + 13) * M] = f.sd;
if (j0 + 14 < nv) p[(ulong)(j0 + 14) * M] = f.se;
if (j0 + 15 < nv) p[(ulong)(j0 + 15) * M] = f.sf;
}
#define MMA_K32(q0, q1, q2, q3, sc, nm, b) { \
a_row_t ua0; \
a_row_t ua1; \
ua0.h = (half16)(dequant8(q0, sc, nm), dequant8(q1, sc, nm)); \
ua1.h = (half16)(dequant8(q2, sc, nm), dequant8(q3, sc, nm)); \
mat_a_t a0 = __builtin_astype(ua0.v, mat_a_t); \
mat_a_t a1 = __builtin_astype(ua1.v, mat_a_t); \
mat_b_t b0 = qcom_coop_mat_load((b), CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
mat_b_t b1 = qcom_coop_mat_load((b) + 1024, CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
acc = qcom_coop_mat_mulAdd(a0, b0, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); \
acc = qcom_coop_mat_mulAdd(a1, b1, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); }
#define STEP64(sa, na, sb_, nb_) { \
uint8 qv; \
qv.s0 = qp[0]; \
qv.s1 = qp[M]; \
qv.s2 = qp[2 * M]; \
qv.s3 = qp[3 * M]; \
qv.s4 = qp[4 * M]; \
qv.s5 = qp[5 * M]; \
qv.s6 = qp[6 * M]; \
qv.s7 = qp[7 * M]; \
qp += 8 * M; \
MMA_K32(qv.s0, qv.s1, qv.s2, qv.s3, sa, na, bp) \
MMA_K32(qv.s4, qv.s5, qv.s6, qv.s7, sb_, nb_, bp + 2048) \
bp += 4096; }
kernel void kernel_gemm_noshuffle_q4_k_f32_coopmat(
global const uint * src0_q, // A: q4_K quants, [K/8][M] u32
global const uchar * src0_s, // A: 6-bit scale/min codes, [K/256*12][M] u8
global const half * src0_d, // A: per-superblock scale, [K/256][M] f16
global const half * src0_dm, // A: per-superblock min, [K/256][M] f16
global const half * src1, // B: packed f16 tiles
global char * dst, // C: [N][M] f32
ulong offsetd,
int K,
int M,
int N
) {
const int lane = get_local_id(0);
const int row = get_global_id(1) * 64;
const int tn = get_global_id(2);
const int n_tiles = (N + 63) / 64;
if (tn >= n_tiles) {
return;
}
const int lrow = row + lane;
global const uint * qp = src0_q + lrow;
global const half * bp = src1 + (ulong)tn * (K / 16) * 1024;
mat_c_t acc = qcom_coop_mat_init(0.0);
#pragma unroll 1
for (int k0 = 0; k0 < K; k0 += 128) {
const int sb = k0 >> 8;
const half dd = src0_d [(ulong)sb * M + lrow];
const half dmm = src0_dm[(ulong)sb * M + lrow];
global const uchar * sp = src0_s + (ulong)(sb * K_SCALE_SIZE) * M + lrow;
const uchar4 c0 = (uchar4)(sp[0], sp[M], sp[2 * M], sp[3 * M]);
const uchar4 c1 = (uchar4)(sp[4 * M], sp[5 * M], sp[6 * M], sp[7 * M]);
const uchar4 c2 = (uchar4)(sp[8 * M], sp[9 * M], sp[10 * M], sp[11 * M]);
const uchar4 sl = c0 & (uchar)0x3F;
const uchar4 ml = c1 & (uchar)0x3F;
const uchar4 sh = (c2 & (uchar)0x0F) | ((c0 >> (uchar)2) & (uchar)0x30);
const uchar4 mh = (c2 >> (uchar)4) | ((c1 >> (uchar)2) & (uchar)0x30);
const int hi = (k0 >> 7) & 1;
const half4 S = dd * convert_half4(hi ? sh : sl);
const half4 Nm = -(dmm * convert_half4(hi ? mh : ml));
STEP64(S.s0, Nm.s0, S.s1, Nm.s1)
STEP64(S.s2, Nm.s2, S.s3, Nm.s3)
}
c_row_t uc;
uc.v = __builtin_astype(acc, half4096);
global float * p = (global float *)(dst + offsetd) + (ulong)tn * 64 * M + row + lane;
const int nv = N - tn * 64;
store_c16(p, uc.s.h0, 0, nv, M);
store_c16(p, uc.s.h1, 16, nv, M);
store_c16(p, uc.s.h2, 32, nv, M);
store_c16(p, uc.s.h3, 48, nv, M);
}
@@ -1,157 +0,0 @@
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
#pragma OPENCL EXTENSION cl_qcom_kernel_cooperative_matrix : enable
// Q5_0 x F32 GEMM with cl_qcom_kernel_cooperative_matrix: one 64x64 f16 C tile per wave,
// A dequantized in registers, B packed by kernel_pack_b_f16_coopmat.
typedef half mat_a_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 16, CLK_COOPERATIVE_MATRIX_A_QCOM)));
typedef half mat_b_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 16, 64, CLK_COOPERATIVE_MATRIX_B_QCOM)));
typedef half mat_c_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 64, CLK_COOPERATIVE_MATRIX_ACCUMULATOR_QCOM)));
typedef half half1024 __attribute__((ext_vector_type(1024)));
typedef half half4096 __attribute__((ext_vector_type(4096)));
typedef union {
half1024 v;
half16 h;
} a_row_t;
typedef union {
half4096 v;
struct {
half16 h0;
half16 h1;
half16 h2;
half16 h3;
} s;
} c_row_t;
inline half8 dequant8(uint v, uint h, half d) {
uint4 t;
t.s0 = ( v & 0x000F) | ((v << 12) & 0x000F0000) | ((h << 4) & 0x0010) | ((h << 19) & 0x00100000);
t.s1 = ((v >> 8) & 0x000F) | ((v << 4) & 0x000F0000) | ((h << 2) & 0x0010) | ((h << 17) & 0x00100000);
t.s2 = ((v >> 16) & 0x000F) | ((v >> 4) & 0x000F0000) | ( h & 0x0010) | ((h << 15) & 0x00100000);
t.s3 = ((v >> 24) & 0x000F) | ((v >> 12) & 0x000F0000) | ((h >> 2) & 0x0010) | ((h << 13) & 0x00100000);
t |= (uint4)(0x64006400);
return (as_half8(t) - (half8)(1040.0h)) * d;
}
inline void store_c16(global float * p, half16 h, int j0, int nv, int M) {
float16 f = convert_float16(h);
if (j0 + 0 < nv) p[(ulong)(j0 + 0) * M] = f.s0;
if (j0 + 1 < nv) p[(ulong)(j0 + 1) * M] = f.s1;
if (j0 + 2 < nv) p[(ulong)(j0 + 2) * M] = f.s2;
if (j0 + 3 < nv) p[(ulong)(j0 + 3) * M] = f.s3;
if (j0 + 4 < nv) p[(ulong)(j0 + 4) * M] = f.s4;
if (j0 + 5 < nv) p[(ulong)(j0 + 5) * M] = f.s5;
if (j0 + 6 < nv) p[(ulong)(j0 + 6) * M] = f.s6;
if (j0 + 7 < nv) p[(ulong)(j0 + 7) * M] = f.s7;
if (j0 + 8 < nv) p[(ulong)(j0 + 8) * M] = f.s8;
if (j0 + 9 < nv) p[(ulong)(j0 + 9) * M] = f.s9;
if (j0 + 10 < nv) p[(ulong)(j0 + 10) * M] = f.sa;
if (j0 + 11 < nv) p[(ulong)(j0 + 11) * M] = f.sb;
if (j0 + 12 < nv) p[(ulong)(j0 + 12) * M] = f.sc;
if (j0 + 13 < nv) p[(ulong)(j0 + 13) * M] = f.sd;
if (j0 + 14 < nv) p[(ulong)(j0 + 14) * M] = f.se;
if (j0 + 15 < nv) p[(ulong)(j0 + 15) * M] = f.sf;
}
#define LOAD_Q(qv, hv, dd, o) { \
qv.s0 = (uint)qp[(8 * (o) + 0) * M] | ((uint)qp[(8 * (o) + 1) * M] << 16); \
qv.s1 = (uint)qp[(8 * (o) + 2) * M] | ((uint)qp[(8 * (o) + 3) * M] << 16); \
qv.s2 = (uint)qp[(8 * (o) + 4) * M] | ((uint)qp[(8 * (o) + 5) * M] << 16); \
qv.s3 = (uint)qp[(8 * (o) + 6) * M] | ((uint)qp[(8 * (o) + 7) * M] << 16); \
hv.s0 = hp[(4 * (o) + 0) * M]; \
hv.s1 = hp[(4 * (o) + 1) * M]; \
hv.s2 = hp[(4 * (o) + 2) * M]; \
hv.s3 = hp[(4 * (o) + 3) * M]; \
dd = dp[(o) * M]; }
#define MMA_K32(qv, hv, dd, b) { \
a_row_t ua0; \
a_row_t ua1; \
ua0.h = (half16)(dequant8(qv.s0, hv.s0, dd), dequant8(qv.s1, hv.s1, dd)); \
ua1.h = (half16)(dequant8(qv.s2, hv.s2, dd), dequant8(qv.s3, hv.s3, dd)); \
mat_a_t a0 = __builtin_astype(ua0.v, mat_a_t); \
mat_a_t a1 = __builtin_astype(ua1.v, mat_a_t); \
mat_b_t b0 = qcom_coop_mat_load((b), CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
mat_b_t b1 = qcom_coop_mat_load((b) + 1024, CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
acc = qcom_coop_mat_mulAdd(a0, b0, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); \
acc = qcom_coop_mat_mulAdd(a1, b1, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); }
kernel void kernel_gemm_noshuffle_q5_0_f32_coopmat(
global const ushort * src0_q, // A: q5_0 quants, [K/4][M] u16
global const uchar * src0_qh, // A: q5_0 high bits, [K/8][M] u8
global const half * src0_d, // A: q5_0 scales, [K/32][M] f16
global const half * src1, // B: packed f16 tiles
global char * dst, // C: [N][M] f32
ulong offsetd,
int K,
int M,
int N
) {
const int lane = get_local_id(0);
const int row = get_global_id(1) * 64;
const int tn = get_global_id(2);
const int n_tiles = (N + 63) / 64;
if (tn >= n_tiles) {
return;
}
global const ushort * qp = src0_q + row + lane;
global const uchar * hp = src0_qh + row + lane;
global const half * dp = src0_d + row + lane;
global const half * bp = src1 + (ulong)tn * (K / 16) * 1024;
mat_c_t acc = qcom_coop_mat_init(0.0);
{
const int odd = (K / 32) & 1;
const int ob = odd ? 0 : 1;
uint4 qa;
uint4 qb;
uint4 ha;
uint4 hb;
half da;
half db;
LOAD_Q(qa, ha, da, 0)
LOAD_Q(qb, hb, db, ob)
da = odd ? 0.0h : da;
MMA_K32(qa, ha, da, bp)
MMA_K32(qb, hb, db, bp + ob * 2048)
qp += (ob + 1) * 8 * M;
hp += (ob + 1) * 4 * M;
dp += (ob + 1) * M;
bp += (ob + 1) * 2048;
}
#pragma unroll 1
for (int k0 = (K / 32) & 1 ? 32 : 64; k0 < K; k0 += 64) {
uint4 qa;
uint4 qb;
uint4 ha;
uint4 hb;
half da;
half db;
LOAD_Q(qa, ha, da, 0)
LOAD_Q(qb, hb, db, 1)
MMA_K32(qa, ha, da, bp)
MMA_K32(qb, hb, db, bp + 2048)
qp += 16 * M;
hp += 8 * M;
dp += 2 * M;
bp += 4096;
}
c_row_t uc;
uc.v = __builtin_astype(acc, half4096);
global float * p = (global float *)(dst + offsetd) + (ulong)tn * 64 * M + row + lane;
const int nv = N - tn * 64;
store_c16(p, uc.s.h0, 0, nv, M);
store_c16(p, uc.s.h1, 16, nv, M);
store_c16(p, uc.s.h2, 32, nv, M);
store_c16(p, uc.s.h3, 48, nv, M);
}
@@ -1,164 +0,0 @@
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
#pragma OPENCL EXTENSION cl_qcom_kernel_cooperative_matrix : enable
// Q5_K x F32 GEMM with cl_qcom_kernel_cooperative_matrix: one 64x64 f16 C tile per wave,
// A dequantized in registers, B packed by kernel_pack_b_f16_coopmat.
#define K_SCALE_SIZE 12
typedef half mat_a_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 16, CLK_COOPERATIVE_MATRIX_A_QCOM)));
typedef half mat_b_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 16, 64, CLK_COOPERATIVE_MATRIX_B_QCOM)));
typedef half mat_c_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 64, CLK_COOPERATIVE_MATRIX_ACCUMULATOR_QCOM)));
typedef half half1024 __attribute__((ext_vector_type(1024)));
typedef half half4096 __attribute__((ext_vector_type(4096)));
typedef union {
half1024 v;
half16 h;
} a_row_t;
typedef union {
half4096 v;
struct {
half16 h0;
half16 h1;
half16 h2;
half16 h3;
} s;
} c_row_t;
inline half8 dequant8(uint v, uint h, half scale, half nminv) {
uint4 t;
t.s0 = ( v & 0x000F) | ((v << 12) & 0x000F0000);
t.s1 = ((v >> 8) & 0x000F) | ((v << 4) & 0x000F0000);
t.s2 = ((v >> 16) & 0x000F) | ((v >> 4) & 0x000F0000);
t.s3 = ((v >> 24) & 0x000F) | ((v >> 12) & 0x000F0000);
t.s0 |= ((h & 0x01u) << 4) | ((h & 0x02u) << 19);
t.s1 |= ((h & 0x04u) << 2) | ((h & 0x08u) << 17);
t.s2 |= ( h & 0x10u ) | ((h & 0x20u) << 15);
t.s3 |= ((h & 0x40u) >> 2) | ((h & 0x80u) << 13);
t |= (uint4)(0x64006400);
return (as_half8(t) - (half8)(1024.0h)) * scale + nminv;
}
inline void store_c16(global float * p, half16 h, int j0, int nv, int M) {
float16 f = convert_float16(h);
if (j0 + 0 < nv) p[(ulong)(j0 + 0) * M] = f.s0;
if (j0 + 1 < nv) p[(ulong)(j0 + 1) * M] = f.s1;
if (j0 + 2 < nv) p[(ulong)(j0 + 2) * M] = f.s2;
if (j0 + 3 < nv) p[(ulong)(j0 + 3) * M] = f.s3;
if (j0 + 4 < nv) p[(ulong)(j0 + 4) * M] = f.s4;
if (j0 + 5 < nv) p[(ulong)(j0 + 5) * M] = f.s5;
if (j0 + 6 < nv) p[(ulong)(j0 + 6) * M] = f.s6;
if (j0 + 7 < nv) p[(ulong)(j0 + 7) * M] = f.s7;
if (j0 + 8 < nv) p[(ulong)(j0 + 8) * M] = f.s8;
if (j0 + 9 < nv) p[(ulong)(j0 + 9) * M] = f.s9;
if (j0 + 10 < nv) p[(ulong)(j0 + 10) * M] = f.sa;
if (j0 + 11 < nv) p[(ulong)(j0 + 11) * M] = f.sb;
if (j0 + 12 < nv) p[(ulong)(j0 + 12) * M] = f.sc;
if (j0 + 13 < nv) p[(ulong)(j0 + 13) * M] = f.sd;
if (j0 + 14 < nv) p[(ulong)(j0 + 14) * M] = f.se;
if (j0 + 15 < nv) p[(ulong)(j0 + 15) * M] = f.sf;
}
#define MMA_K32(q0, q1, q2, q3, h0_, h1_, h2_, h3_, sc, nm, b) { \
a_row_t ua0; \
a_row_t ua1; \
ua0.h = (half16)(dequant8(q0, h0_, sc, nm), dequant8(q1, h1_, sc, nm)); \
ua1.h = (half16)(dequant8(q2, h2_, sc, nm), dequant8(q3, h3_, sc, nm)); \
mat_a_t a0 = __builtin_astype(ua0.v, mat_a_t); \
mat_a_t a1 = __builtin_astype(ua1.v, mat_a_t); \
mat_b_t b0 = qcom_coop_mat_load((b), CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
mat_b_t b1 = qcom_coop_mat_load((b) + 1024, CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
acc = qcom_coop_mat_mulAdd(a0, b0, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); \
acc = qcom_coop_mat_mulAdd(a1, b1, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); }
#define STEP64(sa, na, sb_, nb_) { \
uint8 qv; \
qv.s0 = qp[0]; \
qv.s1 = qp[M]; \
qv.s2 = qp[2 * M]; \
qv.s3 = qp[3 * M]; \
qv.s4 = qp[4 * M]; \
qv.s5 = qp[5 * M]; \
qv.s6 = qp[6 * M]; \
qv.s7 = qp[7 * M]; \
qp += 8 * M; \
uint8 hv; \
hv.s0 = qhp[0]; \
hv.s1 = qhp[M]; \
hv.s2 = qhp[2 * M]; \
hv.s3 = qhp[3 * M]; \
hv.s4 = qhp[4 * M]; \
hv.s5 = qhp[5 * M]; \
hv.s6 = qhp[6 * M]; \
hv.s7 = qhp[7 * M]; \
qhp += 8 * M; \
MMA_K32(qv.s0, qv.s1, qv.s2, qv.s3, hv.s0, hv.s1, hv.s2, hv.s3, sa, na, bp) \
MMA_K32(qv.s4, qv.s5, qv.s6, qv.s7, hv.s4, hv.s5, hv.s6, hv.s7, sb_, nb_, bp + 2048) \
bp += 4096; }
kernel void kernel_gemm_noshuffle_q5_k_f32_coopmat(
global const uint * src0_q, // A: q5_K low nibbles, [K/8][M] u32
global const uchar * src0_qh, // A: q5_K high bits, [K/8][M] u8
global const uchar * src0_s, // A: 6-bit scale/min codes, [K/256*12][M] u8
global const half * src0_d, // A: per-superblock scale, [K/256][M] f16
global const half * src0_dm, // A: per-superblock min, [K/256][M] f16
global const half * src1, // B: packed f16 tiles
global char * dst, // C: [N][M] f32
ulong offsetd,
int K,
int M,
int N
) {
const int lane = get_local_id(0);
const int row = get_global_id(1) * 64;
const int tn = get_global_id(2);
const int n_tiles = (N + 63) / 64;
if (tn >= n_tiles) {
return;
}
const int lrow = row + lane;
global const uint * qp = src0_q + lrow;
global const uchar * qhp = src0_qh + lrow;
global const half * bp = src1 + (ulong)tn * (K / 16) * 1024;
mat_c_t acc = qcom_coop_mat_init(0.0);
#pragma unroll 1
for (int k0 = 0; k0 < K; k0 += 128) {
const int sb = k0 >> 8;
const half dd = src0_d [(ulong)sb * M + lrow];
const half dmm = src0_dm[(ulong)sb * M + lrow];
global const uchar * sp = src0_s + (ulong)(sb * K_SCALE_SIZE) * M + lrow;
const uchar4 c0 = (uchar4)(sp[0], sp[M], sp[2 * M], sp[3 * M]);
const uchar4 c1 = (uchar4)(sp[4 * M], sp[5 * M], sp[6 * M], sp[7 * M]);
const uchar4 c2 = (uchar4)(sp[8 * M], sp[9 * M], sp[10 * M], sp[11 * M]);
const uchar4 sl = c0 & (uchar)0x3F;
const uchar4 ml = c1 & (uchar)0x3F;
const uchar4 sh = (c2 & (uchar)0x0F) | ((c0 >> (uchar)2) & (uchar)0x30);
const uchar4 mh = (c2 >> (uchar)4) | ((c1 >> (uchar)2) & (uchar)0x30);
const int hi = (k0 >> 7) & 1;
const half4 S = dd * convert_half4(hi ? sh : sl);
const half4 Nm = -(dmm * convert_half4(hi ? mh : ml));
STEP64(S.s0, Nm.s0, S.s1, Nm.s1)
STEP64(S.s2, Nm.s2, S.s3, Nm.s3)
}
c_row_t uc;
uc.v = __builtin_astype(acc, half4096);
global float * p = (global float *)(dst + offsetd) + (ulong)tn * 64 * M + row + lane;
const int nv = N - tn * 64;
store_c16(p, uc.s.h0, 0, nv, M);
store_c16(p, uc.s.h1, 16, nv, M);
store_c16(p, uc.s.h2, 32, nv, M);
store_c16(p, uc.s.h3, 48, nv, M);
}
@@ -1,145 +0,0 @@
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
#pragma OPENCL EXTENSION cl_qcom_kernel_cooperative_matrix : enable
// Q6_K x F32 GEMM with cl_qcom_kernel_cooperative_matrix: one 64x64 f16 C tile per wave,
// A dequantized in registers, B packed by kernel_pack_b_f16_coopmat.
typedef half mat_a_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 16, CLK_COOPERATIVE_MATRIX_A_QCOM)));
typedef half mat_b_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 16, 64, CLK_COOPERATIVE_MATRIX_B_QCOM)));
typedef half mat_c_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 64, CLK_COOPERATIVE_MATRIX_ACCUMULATOR_QCOM)));
typedef half half1024 __attribute__((ext_vector_type(1024)));
typedef half half4096 __attribute__((ext_vector_type(4096)));
typedef union {
half1024 v;
half16 h;
} a_row_t;
typedef union {
half4096 v;
struct {
half16 h0;
half16 h1;
half16 h2;
half16 h3;
} s;
} c_row_t;
inline half8 dequant8(uint v, uint hh, half scale) {
uint4 t;
t.s0 = ( v & 0x000F) | ((v << 12) & 0x000F0000);
t.s1 = ((v >> 8) & 0x000F) | ((v << 4) & 0x000F0000);
t.s2 = ((v >> 16) & 0x000F) | ((v >> 4) & 0x000F0000);
t.s3 = ((v >> 24) & 0x000F) | ((v >> 12) & 0x000F0000);
t.s0 |= ((hh & 0x0003u) << 4) | ((hh & 0x000Cu) << 18);
t.s1 |= ( hh & 0x0030u ) | ((hh & 0x00C0u) << 14);
t.s2 |= ((hh & 0x0300u) >> 4) | ((hh & 0x0C00u) << 10);
t.s3 |= ((hh & 0x3000u) >> 8) | ((hh & 0xC000u) << 6);
t |= (uint4)(0x64006400);
return (as_half8(t) - (half8)(1056.0h)) * scale;
}
inline void store_c16(global float * p, half16 h, int j0, int nv, int M) {
float16 f = convert_float16(h);
if (j0 + 0 < nv) p[(ulong)(j0 + 0) * M] = f.s0;
if (j0 + 1 < nv) p[(ulong)(j0 + 1) * M] = f.s1;
if (j0 + 2 < nv) p[(ulong)(j0 + 2) * M] = f.s2;
if (j0 + 3 < nv) p[(ulong)(j0 + 3) * M] = f.s3;
if (j0 + 4 < nv) p[(ulong)(j0 + 4) * M] = f.s4;
if (j0 + 5 < nv) p[(ulong)(j0 + 5) * M] = f.s5;
if (j0 + 6 < nv) p[(ulong)(j0 + 6) * M] = f.s6;
if (j0 + 7 < nv) p[(ulong)(j0 + 7) * M] = f.s7;
if (j0 + 8 < nv) p[(ulong)(j0 + 8) * M] = f.s8;
if (j0 + 9 < nv) p[(ulong)(j0 + 9) * M] = f.s9;
if (j0 + 10 < nv) p[(ulong)(j0 + 10) * M] = f.sa;
if (j0 + 11 < nv) p[(ulong)(j0 + 11) * M] = f.sb;
if (j0 + 12 < nv) p[(ulong)(j0 + 12) * M] = f.sc;
if (j0 + 13 < nv) p[(ulong)(j0 + 13) * M] = f.sd;
if (j0 + 14 < nv) p[(ulong)(j0 + 14) * M] = f.se;
if (j0 + 15 < nv) p[(ulong)(j0 + 15) * M] = f.sf;
}
#define MMA_K32(q0, q1, q2, q3, hA, hB, sc0, sc1, b) { \
a_row_t ua0; \
a_row_t ua1; \
ua0.h = (half16)(dequant8(q0, (hA), sc0), dequant8(q1, (hA) >> 16, sc0)); \
ua1.h = (half16)(dequant8(q2, (hB), sc1), dequant8(q3, (hB) >> 16, sc1)); \
mat_a_t a0 = __builtin_astype(ua0.v, mat_a_t); \
mat_a_t a1 = __builtin_astype(ua1.v, mat_a_t); \
mat_b_t b0 = qcom_coop_mat_load((b), CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
mat_b_t b1 = qcom_coop_mat_load((b) + 1024, CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
acc = qcom_coop_mat_mulAdd(a0, b0, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); \
acc = qcom_coop_mat_mulAdd(a1, b1, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); }
kernel void kernel_gemm_noshuffle_q6_k_f32_coopmat(
global const uint * src0_ql, // A: q6_K low nibbles, [K/8][M] u32
global const uint * src0_qh, // A: q6_K high 2-bit, [K/16][M] u32
global const uint * src0_s, // A: int8 sub-scales, [M][K/16] (4 per u32)
global const half * src0_d, // A: per-superblock scale, [K/256][M] f16
global const half * src1, // B: packed f16 tiles
global char * dst, // C: [N][M] f32
ulong offsetd,
int K,
int M,
int N
) {
const int lane = get_local_id(0);
const int row = get_global_id(1) * 64;
const int tn = get_global_id(2);
const int n_tiles = (N + 63) / 64;
if (tn >= n_tiles) {
return;
}
const int lrow = row + lane;
global const uint * qp = src0_ql + lrow;
global const uint * qhp = src0_qh + lrow;
global const half * bp = src1 + (ulong)tn * (K / 16) * 1024;
global const uint * sp = src0_s + (ulong)lrow * (K / 64);
mat_c_t acc = qcom_coop_mat_init(0.0);
#pragma unroll 1
for (int k0 = 0; k0 < K; k0 += 64) {
const half dd = src0_d[(ulong)(k0 >> 8) * M + lrow];
const half4 g = convert_half4(as_char4(sp[0])) * dd;
sp += 1;
uint8 qv;
qv.s0 = qp[0];
qv.s1 = qp[M];
qv.s2 = qp[2 * M];
qv.s3 = qp[3 * M];
qv.s4 = qp[4 * M];
qv.s5 = qp[5 * M];
qv.s6 = qp[6 * M];
qv.s7 = qp[7 * M];
qp += 8 * M;
uint4 hv;
hv.s0 = qhp[0];
hv.s1 = qhp[M];
hv.s2 = qhp[2 * M];
hv.s3 = qhp[3 * M];
qhp += 4 * M;
MMA_K32(qv.s0, qv.s1, qv.s2, qv.s3, hv.s0, hv.s1, g.s0, g.s1, bp)
MMA_K32(qv.s4, qv.s5, qv.s6, qv.s7, hv.s2, hv.s3, g.s2, g.s3, bp + 2048)
bp += 4096;
}
c_row_t uc;
uc.v = __builtin_astype(acc, half4096);
global float * p = (global float *)(dst + offsetd) + (ulong)tn * 64 * M + row + lane;
const int nv = N - tn * 64;
store_c16(p, uc.s.h0, 0, nv, M);
store_c16(p, uc.s.h1, 16, nv, M);
store_c16(p, uc.s.h2, 32, nv, M);
store_c16(p, uc.s.h3, 48, nv, M);
}
@@ -1,157 +0,0 @@
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
#pragma OPENCL EXTENSION cl_qcom_kernel_cooperative_matrix : enable
// Q8_0 x F32 GEMM with cl_qcom_kernel_cooperative_matrix: one 64x64 f16 C tile per wave,
// A dequantized in registers, B packed by kernel_pack_b_f16_coopmat.
typedef half mat_a_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 16, CLK_COOPERATIVE_MATRIX_A_QCOM)));
typedef half mat_b_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 16, 64, CLK_COOPERATIVE_MATRIX_B_QCOM)));
typedef half mat_c_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 64, CLK_COOPERATIVE_MATRIX_ACCUMULATOR_QCOM)));
typedef half half1024 __attribute__((ext_vector_type(1024)));
typedef half half4096 __attribute__((ext_vector_type(4096)));
typedef union {
half1024 v;
half16 h;
} a_row_t;
typedef union {
half4096 v;
struct {
half16 h0;
half16 h1;
half16 h2;
half16 h3;
} s;
} c_row_t;
inline half4 dequant4(uint v, half d) {
return convert_half4(as_char4(v)) * d;
}
inline void store_c16(global float * p, half16 h, int j0, int nv, int M) {
float16 f = convert_float16(h);
if (j0 + 0 < nv) p[(ulong)(j0 + 0) * M] = f.s0;
if (j0 + 1 < nv) p[(ulong)(j0 + 1) * M] = f.s1;
if (j0 + 2 < nv) p[(ulong)(j0 + 2) * M] = f.s2;
if (j0 + 3 < nv) p[(ulong)(j0 + 3) * M] = f.s3;
if (j0 + 4 < nv) p[(ulong)(j0 + 4) * M] = f.s4;
if (j0 + 5 < nv) p[(ulong)(j0 + 5) * M] = f.s5;
if (j0 + 6 < nv) p[(ulong)(j0 + 6) * M] = f.s6;
if (j0 + 7 < nv) p[(ulong)(j0 + 7) * M] = f.s7;
if (j0 + 8 < nv) p[(ulong)(j0 + 8) * M] = f.s8;
if (j0 + 9 < nv) p[(ulong)(j0 + 9) * M] = f.s9;
if (j0 + 10 < nv) p[(ulong)(j0 + 10) * M] = f.sa;
if (j0 + 11 < nv) p[(ulong)(j0 + 11) * M] = f.sb;
if (j0 + 12 < nv) p[(ulong)(j0 + 12) * M] = f.sc;
if (j0 + 13 < nv) p[(ulong)(j0 + 13) * M] = f.sd;
if (j0 + 14 < nv) p[(ulong)(j0 + 14) * M] = f.se;
if (j0 + 15 < nv) p[(ulong)(j0 + 15) * M] = f.sf;
}
#define LOAD_Q(qv, dd) { \
qv.s0 = qp[0]; \
qv.s1 = qp[M]; \
qv.s2 = qp[2 * M]; \
qv.s3 = qp[3 * M]; \
qv.s4 = qp[4 * M]; \
qv.s5 = qp[5 * M]; \
qv.s6 = qp[6 * M]; \
qv.s7 = qp[7 * M]; \
dd = dp[0]; \
qp += 8 * M; \
dp += M; }
#define MMA_K32(qv, dd, b) { \
a_row_t ua0; \
a_row_t ua1; \
ua0.h = (half16)(dequant4(qv.s0, dd), dequant4(qv.s1, dd), dequant4(qv.s2, dd), dequant4(qv.s3, dd)); \
ua1.h = (half16)(dequant4(qv.s4, dd), dequant4(qv.s5, dd), dequant4(qv.s6, dd), dequant4(qv.s7, dd)); \
mat_a_t a0 = __builtin_astype(ua0.v, mat_a_t); \
mat_a_t a1 = __builtin_astype(ua1.v, mat_a_t); \
mat_b_t b0 = qcom_coop_mat_load((b), CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
mat_b_t b1 = qcom_coop_mat_load((b) + 1024, CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
acc = qcom_coop_mat_mulAdd(a0, b0, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); \
acc = qcom_coop_mat_mulAdd(a1, b1, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); }
kernel void kernel_gemm_noshuffle_q8_0_f32_coopmat(
global const uint * src0_q, // A: q8_0 quants, [K/4][M] u32
global const half * src0_d, // A: q8_0 scales, [K/32][M] f16
global const half * src1, // B: packed f16 tiles
global char * dst, // C: [N][M] f32
ulong offsetd,
int K,
int M,
int N
) {
const int lane = get_local_id(0);
const int row = get_global_id(1) * 64;
const int tn = get_global_id(2);
const int n_tiles = (N + 63) / 64;
if (tn >= n_tiles) {
return;
}
global const uint * qp = src0_q + row + lane;
global const half * dp = src0_d + row + lane;
global const half * bp = src1 + (ulong)tn * (K / 16) * 1024;
mat_c_t acc = qcom_coop_mat_init(0.0);
{
const int odd = (K / 32) & 1;
const int kb = odd ? 0 : 32;
uint8 qa;
uint8 qb;
half da;
half db;
qa.s0 = qp[0];
qa.s1 = qp[M];
qa.s2 = qp[2 * M];
qa.s3 = qp[3 * M];
qa.s4 = qp[4 * M];
qa.s5 = qp[5 * M];
qa.s6 = qp[6 * M];
qa.s7 = qp[7 * M];
da = odd ? 0.0h : dp[0];
qb.s0 = qp[(kb / 4 + 0) * M];
qb.s1 = qp[(kb / 4 + 1) * M];
qb.s2 = qp[(kb / 4 + 2) * M];
qb.s3 = qp[(kb / 4 + 3) * M];
qb.s4 = qp[(kb / 4 + 4) * M];
qb.s5 = qp[(kb / 4 + 5) * M];
qb.s6 = qp[(kb / 4 + 6) * M];
qb.s7 = qp[(kb / 4 + 7) * M];
db = dp[(kb / 32) * M];
MMA_K32(qa, da, bp)
MMA_K32(qb, db, bp + (kb / 16) * 1024)
qp += (kb / 4 + 8) * M;
dp += (kb / 32 + 1) * M;
bp += (kb / 16 + 2) * 1024;
}
#pragma unroll 1
for (int k0 = (K / 32) & 1 ? 32 : 64; k0 < K; k0 += 64) {
uint8 qa;
uint8 qb;
half da;
half db;
LOAD_Q(qa, da)
LOAD_Q(qb, db)
MMA_K32(qa, da, bp)
MMA_K32(qb, db, bp + 2048)
bp += 4096;
}
c_row_t uc;
uc.v = __builtin_astype(acc, half4096);
global float * p = (global float *)(dst + offsetd) + (ulong)tn * 64 * M + row + lane;
const int nv = N - tn * 64;
store_c16(p, uc.s.h0, 0, nv, M);
store_c16(p, uc.s.h1, 16, nv, M);
store_c16(p, uc.s.h2, 32, nv, M);
store_c16(p, uc.s.h3, 48, nv, M);
}
@@ -1,48 +0,0 @@
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
// Pack B f32 [N][K] into f16 64x16 tiles (column-major, ld 16) at (tn * (K/16) + ks) * 1024.
// Columns >= N are zero. A workgroup (64, 4) packs 64 columns x 64 k.
#define PACK_LDS (64 + 8)
kernel void kernel_pack_b_f16_coopmat(
global const char * src1,
ulong offset1,
global half * dst,
int K,
int N
) {
local half tile[64 * PACK_LDS];
const int l = get_local_id(1) * 64 + get_local_id(0);
const int tn = get_group_id(0);
const int kt = get_group_id(1);
const int k0 = kt * 64;
global const float * src = (global const float *)(src1 + offset1);
for (int j = 0; j < 4; j++) {
const int f = j * 256 + l;
const int r = f >> 4;
const int c = (f & 15) * 4;
const int n = tn * 64 + r;
float4 v = (float4)(0.0f);
if (n < N && k0 + c < K) {
v = vload4(0, src + (ulong)n * K + k0 + c);
}
vstore_half4(v, 0, tile + r * PACK_LDS + c);
}
barrier(CLK_LOCAL_MEM_FENCE);
global half * out = dst + ((ulong)tn * (K / 16) + kt * 4) * 1024;
for (int j = 0; j < 2; j++) {
const int q = j * 256 + l;
const int ks = q >> 7;
const int n = (q >> 1) & 63;
const int h = q & 1;
if (kt * 4 + ks < K / 16) {
half8 v = vload8(0, tile + n * PACK_LDS + ks * 16 + h * 8);
vstore8(v, 0, out + q * 8);
}
}
}
+1 -17
View File
@@ -809,25 +809,9 @@ void ggml_vk_buffer_memset(vk_buffer& dst, size_t offset, uint32_t c, size_t siz
return;
}
// vkCmdFillBuffer requires a 4-byte aligned offset and size, so write the unaligned ends from the matching bytes of c
const size_t end = offset + size;
const size_t begin_aligned = std::min(GGML_PAD(offset, 4), end);
const size_t end_aligned = std::max(end - end % 4, begin_aligned);
const uint8_t * c_bytes = (const uint8_t *) &c;
if (begin_aligned > offset) {
ggml_vk_buffer_write(dst, offset, c_bytes + offset % 4, begin_aligned - offset);
}
if (end > end_aligned) {
ggml_vk_buffer_write(dst, end_aligned, c_bytes, end - end_aligned);
}
if (end_aligned == begin_aligned) {
return;
}
std::lock_guard<std::recursive_mutex> guard(dst->device->mutex);
vk_context subctx = ggml_vk_begin_sync_ctx(dst->device, dst->device->transfer_queue->cmd_pool);
subctx->s->buffer->buf.fillBuffer(dst->buffer, begin_aligned, end_aligned - begin_aligned, c);
subctx->s->buffer->buf.fillBuffer(dst->buffer, offset, size, c);
ggml_vk_submit_sync(dst->device, subctx);
}
+1 -3
View File
@@ -14435,9 +14435,7 @@ bool ggml_backend_vk_buffer_cpy_tensor_n(ggml_backend_buffer_t buffer, const ggm
void ggml_backend_vk_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) {
ggml_backend_vk_buffer_context * ctx = (ggml_backend_vk_buffer_context *)buffer->context;
// set all bytes of a u32 to the same 1 byte value
uint32_t val32 = (uint32_t)value * 0x01010101;
ggml_vk_buffer_memset(ctx->dev_buffer, 0, val32, buffer->size);
ggml_vk_buffer_memset(ctx->dev_buffer, 0, value, buffer->size);
}
const char * ggml_backend_vk_buffer_type_name(ggml_backend_buffer_type_t buft) {
+3 -7
View File
@@ -4066,7 +4066,7 @@ static void ggml_backend_webgpu_buffer_memset_tensor(ggml_backend_buffer_t buffe
size_t total_offset = ggml_webgpu_tensor_offset(tensor) + offset;
// set all bytes of a u32 to the same 1 byte value
// This is a trick to set all bytes of a u32 to the same 1 byte value.
uint32_t val32 = (uint32_t) value * 0x01010101;
ggml_backend_webgpu_buffer_memset(buf_ctx->global_ctx, buf_ctx->buffer, val32, total_offset, size);
WEBGPU_CPU_PROFILE_TOTAL_END(memset_tensor, buf_ctx->global_ctx);
@@ -4406,10 +4406,7 @@ static void ggml_backend_webgpu_buffer_clear(ggml_backend_buffer_t buffer, uint8
WEBGPU_LOG_DEBUG("ggml_backend_webgpu_buffer_clear(" << buffer << ", " << (uint32_t) value << ")");
WEBGPU_CPU_PROFILE_TOTAL_START(clear);
ggml_backend_webgpu_buffer_context * buf_ctx = (ggml_backend_webgpu_buffer_context *) buffer->context;
// set all bytes of a u32 to the same 1 byte value
uint32_t val32 = (uint32_t) value * 0x01010101;
ggml_backend_webgpu_buffer_memset(buf_ctx->global_ctx, buf_ctx->buffer, val32, 0, buffer->size);
ggml_backend_webgpu_buffer_memset(buf_ctx->global_ctx, buf_ctx->buffer, value, 0, buffer->size);
WEBGPU_CPU_PROFILE_TOTAL_END(clear, buf_ctx->global_ctx);
}
@@ -5266,8 +5263,7 @@ static bool ggml_backend_webgpu_device_supports_op(ggml_backend_dev_t dev, const
supports_op = op->type == GGML_TYPE_F32 && src0->type == GGML_TYPE_F32 && src1->type == GGML_TYPE_F32 &&
src2->type == GGML_TYPE_F32 && op->src[3]->type == GGML_TYPE_F32 &&
op->src[4]->type == GGML_TYPE_F32 && op->src[5]->type == GGML_TYPE_F32 &&
s_v <= ctx->webgpu_global_ctx->capabilities.limits.maxComputeInvocationsPerWorkgroup &&
ggml_are_same_stride(src0, src1);
s_v <= ctx->webgpu_global_ctx->capabilities.limits.maxComputeInvocationsPerWorkgroup;
}
break;
case GGML_OP_CLAMP:
+2 -26
View File
@@ -1116,11 +1116,9 @@ static const char * GGML_OP_NAME[GGML_OP_COUNT] = {
"GLU",
"SLEEP",
"SIGMOID_BACK",
};
static_assert(GGML_OP_COUNT == 101, "GGML_OP_COUNT != 101");
static_assert(GGML_OP_COUNT == 100, "GGML_OP_COUNT != 100");
static const char * GGML_OP_SYMBOL[GGML_OP_COUNT] = {
"none",
@@ -1233,11 +1231,9 @@ static const char * GGML_OP_SYMBOL[GGML_OP_COUNT] = {
"glu(x)",
"sleep(x)",
"sigmoid_back(x)",
};
static_assert(GGML_OP_COUNT == 101, "GGML_OP_COUNT != 101");
static_assert(GGML_OP_COUNT == 100, "GGML_OP_COUNT != 100");
static_assert(GGML_OP_POOL_COUNT == 2, "GGML_OP_POOL_COUNT != 2");
@@ -2870,21 +2866,6 @@ struct ggml_tensor * ggml_silu_back(
return result;
}
// ggml_sigmoid_back
struct ggml_tensor * ggml_sigmoid_back(
struct ggml_context * ctx,
struct ggml_tensor * a,
struct ggml_tensor * b) {
struct ggml_tensor * result = ggml_dup_tensor(ctx, a);
result->op = GGML_OP_SIGMOID_BACK;
result->src[0] = a;
result->src[1] = b;
return result;
}
// ggml hardswish
struct ggml_tensor * ggml_hardswish(
@@ -7195,11 +7176,6 @@ static void ggml_compute_backward(
ggml_add_or_set(ctx, cgraph, isrc0, ggml_silu_back(ctx, grad, src0));
}
} break;
case GGML_UNARY_OP_SIGMOID: {
if (src0_needs_grads) {
ggml_add_or_set(ctx, cgraph, isrc0, ggml_sigmoid_back(ctx, grad, src0));
}
} break;
case GGML_UNARY_OP_EXP: {
if (src0_needs_grads) {
ggml_add_or_set(ctx, cgraph, isrc0, ggml_mul(ctx, tensor, grad));
+22 -9
View File
@@ -322,6 +322,24 @@ struct gguf_reader {
return true;
}
bool read(enum ggml_type & dst) const {
int32_t tmp = -1;
if (!read(tmp)) {
return false;
}
dst = ggml_type(tmp);
return true;
}
bool read(enum gguf_type & dst) const {
int32_t tmp = -1;
if (!read(tmp)) {
return false;
}
dst = gguf_type(tmp);
return true;
}
bool read(std::string & dst) const {
uint64_t size = 0;
if (!read(size)) {
@@ -537,7 +555,7 @@ static struct gguf_context * gguf_init_from_reader(const struct gguf_reader & gr
for (int64_t i = 0; ok && i < n_kv; ++i) {
std::string key;
int32_t type = -1;
gguf_type type = gguf_type(-1);
bool is_array = false;
uint64_t n = 1;
@@ -706,20 +724,15 @@ static struct gguf_context * gguf_init_from_reader(const struct gguf_reader & gr
// tensor type
{
int32_t type = -1;
ok = ok && gr.read(type);
if (!ok) {
break;
}
ok = ok && gr.read(info.t.type);
// check that tensor type is within defined range
if (type < 0 || type >= GGML_TYPE_COUNT) {
if (info.t.type < 0 || info.t.type >= GGML_TYPE_COUNT) {
GGML_LOG_ERROR("%s: tensor '%s' has invalid ggml type %d. should be in [0, %d)\n",
__func__, info.t.name, type, GGML_TYPE_COUNT);
__func__, info.t.name, info.t.type, GGML_TYPE_COUNT);
ok = false;
break;
}
info.t.type = ggml_type(type);
const size_t type_size = ggml_type_size(info.t.type);
const int64_t blck_size = ggml_blck_size(info.t.type);
+2 -4
View File
@@ -73,13 +73,11 @@ Tokens allow grammars to match specific tokenizer tokens rather than character s
Tokens can be specified in two ways:
1. **Token ID**: Use angle brackets with the token ID in square brackets: `<[token-id]>`. For example, `<[1000]>` matches the token with ID 1000. Multiple IDs and inclusive ranges can be combined: `<[1000,1005-1010]>` matches token 1000 or any token from 1005 to 1010.
1. **Token ID**: Use angle brackets with the token ID in square brackets: `<[token-id]>`. For example, `<[1000]>` matches the token with ID 1000.
2. **Token string**: Use angle brackets with the token text directly: `<token>`. For example, `<think>` will match the token whose text is exactly `<think>`. This only works if the string tokenizes to exactly one token in the vocabulary, otherwise the grammar will fail to parse.
You can negate token matches using the `!` prefix: `!<[1000]>` or `!<think>` matches any token *except* the specified one, and `!<[1000,1005-1010]>` matches any token outside the set.
Tokens are matched whole, so a token rule can only match at the start of a token, never after a character rule has consumed part of its text. Any token named by a token rule is also excluded from character rules: once `<think>` appears as a token in a grammar, `.*` in that grammar will not match the `<think>` token.
You can negate token matches using the `!` prefix: `!<[1000]>` or `!<think>` matches any token *except* the specified one.
```
# Match a thinking block: <think>...</think>
+2 -6
View File
@@ -149,10 +149,7 @@ now_ms() { date +%s%3N; }
# ---------------------------------------------------------------- scratch + cleanup
S=$(mktemp -d /tmp/fork-validate.XXXXXX)
# the arena lives on tmpfs, like production's /dev/shm/llama-vram: on overlayfs (the pod's rootfs,
# so /tmp, since 2026-11-22) stat reports the overlay device while /proc/locks prints the upper
# filesystem's, so lock_id never matched and every token check failed
ARENA=/dev/shm/fork-validate.${S##*.}
ARENA=$S/arena
mkdir -p "$ARENA"
cd "$S" || exit 2 # anything written to a relative path lands in the scratch dir
PIDS=() # "pid:starttime" of every background process we start (starttime guards PID reuse)
@@ -186,7 +183,6 @@ finish() {
local rc=$?
trap - EXIT INT TERM HUP
cleanup
rm -rf "$ARENA" "$ARENA-moec"
echo
echo "==================== fork-validate summary ===================="
echo "candidate: $BUILD ($(git -C "$BUILD/.." rev-parse --short HEAD 2>/dev/null || echo '?'))"
@@ -826,7 +822,7 @@ run_moecache() {
[ -f "$MOECACHE_MODEL" ] || { fail moecache "model missing: $MOECACHE_MODEL"; return; }
local free_mb=$(( ( $(cat "$VRAM_TOTAL") - $(cat "$VRAM_USED") ) / 1048576 ))
log "moecache: $free_mb MiB VRAM free; server on :$PC with --moe-cache-mib $MOECACHE_MIB"
local arena=$ARENA-moec; mkdir -p "$arena"
local arena=$S/arena-moec; mkdir -p "$arena"
# -lv 4: the cache size and hit/miss stats are only logged at that verbosity
# shellcheck disable=SC2086
start_server M "$BUILD" "$PC" LLAMA_SLEEP_VRAM_ONLY=1 "LLAMA_VRAM_ARENA=$arena" LLAMA_SLEEP_EVICT_KV=1 -- \
+6 -8
View File
@@ -69,7 +69,6 @@ These recur often enough in review comments on past add-model PRs that they're w
- Before writing a dedicated tool-call/output parser, check whether the existing autoparser already handles the template (`test-chat-auto-parser <jinja>` shows what it detects).
- Marking a custom EOS/closing-tag token as `eot` at conversion time isn't always sufficient - in long/agentic generations a model can emit the closing sequence as literal text instead of the token, so generation never stops on EOG and raw text leaks past the parser. Verify this case, not just the token path.
- If reusing or aliasing an existing pre-tokenizer for convenience, justify and test that choice explicitly - silent reuse is an easy source of subtle tokenizer bugs.
- The GGUF only stores the pre-tokenizer name (`tokenizer.pre`), the behavior flags are hard-coded per name in `src/llama-vocab.cpp`. When adding a new name, set each flag from the HF `tokenizer.json`, not from the case you copied. Examples: `model.ignore_merges` -> `ignore_merges`, the BOS/EOS of the post-processor -> `special_bos_id` / `add_bos`, prefix space and space cleanup -> `add_space_prefix` / `clean_spaces`. The `chkhsh` check in the converter only identifies the pre-tokenizer on a short text, it may not catch a wrong flag.
- Watch for excessive graph splits caused by building per-layer view/index tensors inside the layer loop - hoist tensors that don't vary per layer out of the loop (relevant if you hit `GGML_SCHED_MAX_SPLIT_INPUTS`).
- A custom KQ mask fed into flash attention must match FA's expected dtype - cast it to F16 before passing it to `build_attn_mha` when FA is enabled.
- When padding a custom KV-cache size to an alignment (e.g. `GGML_PAD(..., 256)`), apply the padding after all other size adjustments, not before - otherwise later logic can un-align it again.
@@ -84,13 +83,12 @@ These recur often enough in review comments on past add-model PRs that they're w
Reference: `examples/model-conversion/README.md`.
1. Convert to GGUF, then inspect/run both the original and converted tensors.
2. If the model adds a new pre-tokenizer, compare the token ids of `llama-tokenize -f <file> --ids --no-escape` with the HF tokenizer on whole files of prose, code and multilingual text (e.g. wikitext-2 and a few source files).
3. Run logits verification (original vs converted). If this model is a new version of an already-supported family, verify the *previous* version still passes logits verification first - numerical differences may be pre-existing, not caused by the new work. The tools to perform full logits validation are available in `examples/model-conversion`.
4. Quantize (including QAT variants if relevant) and re-verify.
5. Run perplexity evaluation (simple and full).
6. Sanity-check across `tools/cli`, `tools/completion`, `tools/imatrix`, `tools/quantize`, and `tools/server`.
7. CPU backend first; other backends (CUDA, Metal, ...) can be separate follow-up PRs per `CONTRIBUTING.md`.
8. Re-review every changed file against the coding/naming guidelines in `AGENTS.md` (and `CONTRIBUTING.md`'s "Coding guidelines"/"Naming guidelines" sections) - this is a separate pass from functional testing and is just as important: no forced line-wrapping, no unicode punctuation, minimal/non-redundant comments, `snake_case` naming (`kebab-case` for file names), matching indentation/brace style, etc.
2. Run logits verification (original vs converted). If this model is a new version of an already-supported family, verify the *previous* version still passes logits verification first - numerical differences may be pre-existing, not caused by the new work. The tools to perform full logits validation are available in `examples/model-conversion`.
3. Quantize (including QAT variants if relevant) and re-verify.
4. Run perplexity evaluation (simple and full).
5. Sanity-check across `tools/cli`, `tools/completion`, `tools/imatrix`, `tools/quantize`, and `tools/server`.
6. CPU backend first; other backends (CUDA, Metal, ...) can be separate follow-up PRs per `CONTRIBUTING.md`.
7. Re-review every changed file against the coding/naming guidelines in `AGENTS.md` (and `CONTRIBUTING.md`'s "Coding guidelines"/"Naming guidelines" sections) - this is a separate pass from functional testing and is just as important: no forced line-wrapping, no unicode punctuation, minimal/non-redundant comments, `snake_case` naming (`kebab-case` for file names), matching indentation/brace style, etc.
## Before opening a PR
-13
View File
@@ -3230,19 +3230,6 @@ bool llama_context::sched_copy_experts(ggml_backend_t backend, const ggml_tensor
return false;
}
// Later loop passes can select different experts from the same device copy.
// TODO: copy only the experts needed by each pass (see #32807 and #32809).
// [fork F13] with the per-pass host-weight views every pass has its own copy, so the partial
// copy is correct again; LLAMA_LOOP_FULL_COPY=0 skips this guard (A/B knob, only correct with
// the views on, LLAMA_LOOP_HOST_VIEWS unset or 1)
static const bool loop_full_copy = [] {
const char * env = getenv("LLAMA_LOOP_FULL_COPY");
return env == nullptr || strcmp(env, "0") != 0;
}();
if (loop_full_copy && lctx->model.arch == LLM_ARCH_INKLING2_LOOP) {
return false;
}
const ggml_tensor * ids = node->src[2];
if (ggml_nelements(ids) == 0) {
return true;
-3
View File
@@ -112,9 +112,6 @@ using llama_memory_breakdown = std::map<ggml_backend_buffer_type_t, llama_memory
LLAMA_API int32_t llama_model_n_expert (const struct llama_model * model);
LLAMA_API int32_t llama_model_n_devices(const struct llama_model * model);
// number of logical passes that execute each layer's weights (0 for weight-sharing alias slots)
LLAMA_API std::vector<uint32_t> llama_model_layer_visits(const struct llama_model * model);
LLAMA_API ggml_backend_dev_t llama_model_get_device(const struct llama_model * model, int i);
LLAMA_API llama_memory_breakdown llama_get_memory_breakdown(const struct llama_context * ctx);
+51 -165
View File
@@ -7,7 +7,6 @@
#include <cmath>
#include <algorithm>
#include <cstdint>
#include <limits>
#include <set>
#include <numeric>
#include <stdexcept>
@@ -34,9 +33,9 @@ static std::pair<uint32_t, const char *> decode_utf8(const char * src) {
return std::make_pair(value, pos);
}
std::pair<std::vector<uint32_t>, llama_partial_utf8> llama_grammar_decode_utf8(
const std::string & src,
llama_partial_utf8 partial_start) {
static std::pair<std::vector<uint32_t>, llama_partial_utf8> decode_utf8(
const std::string & src,
llama_partial_utf8 partial_start) {
static const int lookup[] = { 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 2, 2, 3, 4 };
const char * pos = src.c_str();
std::vector<uint32_t> code_points;
@@ -186,45 +185,23 @@ static std::pair<uint32_t, const char *> parse_char(const char * src) {
throw std::runtime_error("unexpected end of input");
}
static std::pair<uint32_t, const char *> parse_token_id(const char * src) {
const char * int_end = parse_int(src);
unsigned long id = std::stoul(std::string(src, int_end - src));
if (id > std::numeric_limits<uint32_t>::max()) {
throw std::runtime_error(std::string("parsed token id is too big at ") + src);
}
return std::make_pair(static_cast<uint32_t>(id), int_end);
}
static const char * parse_token(const llama_vocab * vocab, const char * src, llama_gretype type, llama_grammar_rule & rule) {
static std::pair<uint32_t, const char *> parse_token(const llama_vocab * vocab, const char * src) {
const char * pos = src;
if (*pos != '<') {
throw std::runtime_error(std::string("expecting '<' at ") + pos);
}
pos++;
// Parse <[id,id-id,...]>
// Parse <[id]>
if (*pos == '[') {
pos++;
const size_t start = rule.size();
while (true) {
auto id_pair = parse_token_id(pos);
uint32_t token_id = id_pair.first;
pos = id_pair.second;
rule.push_back({rule.size() == start ? type : LLAMA_GRETYPE_TOKEN_ALT, token_id});
if (*pos == '-') {
pos++;
auto end_pair = parse_token_id(pos);
if (end_pair.first < token_id) {
throw std::runtime_error(std::string("invalid token range at ") + pos);
}
pos = end_pair.second;
rule.push_back({LLAMA_GRETYPE_TOKEN_RNG_UPPER, end_pair.first});
}
if (*pos != ',') {
break;
}
pos++;
const char * int_end = parse_int(pos);
unsigned long id = std::stoul(std::string(pos, int_end - pos));
if (id > std::numeric_limits<uint32_t>::max()) {
throw std::runtime_error(std::string("parsed token id is too big at ") + pos);
}
uint32_t token_id = static_cast<uint32_t>(id);
pos = int_end;
if (*pos != ']') {
throw std::runtime_error(std::string("expecting ']' at ") + pos);
}
@@ -233,7 +210,7 @@ static const char * parse_token(const llama_vocab * vocab, const char * src, lla
throw std::runtime_error(std::string("expecting '>' at ") + pos);
}
pos++;
return pos;
return std::make_pair(token_id, pos);
}
if (vocab == nullptr) {
@@ -255,8 +232,7 @@ static const char * parse_token(const llama_vocab * vocab, const char * src, lla
// must tokenize to exactly 1 token
throw std::runtime_error("invalid token '" + std::string(src, pos - src) + "'");
}
rule.push_back({type, static_cast<uint32_t>(tokens[0])});
return pos;
return std::make_pair(tokens[0], pos);
}
static void print_grammar_char(FILE * file, uint32_t c) {
@@ -268,16 +244,6 @@ static void print_grammar_char(FILE * file, uint32_t c) {
}
}
static bool is_token_element(llama_grammar_element elem) {
switch (elem.type) {
case LLAMA_GRETYPE_TOKEN: return true;
case LLAMA_GRETYPE_TOKEN_NOT: return true;
case LLAMA_GRETYPE_TOKEN_ALT: return true;
case LLAMA_GRETYPE_TOKEN_RNG_UPPER: return true;
default: return false;
}
}
static bool is_char_element(llama_grammar_element elem) {
switch (elem.type) {
case LLAMA_GRETYPE_CHAR: return true;
@@ -292,18 +258,16 @@ static bool is_char_element(llama_grammar_element elem) {
static void print_rule_binary(FILE * file, const llama_grammar_rule & rule) {
for (auto elem : rule) {
switch (elem.type) {
case LLAMA_GRETYPE_END: fprintf(file, "END"); break;
case LLAMA_GRETYPE_ALT: fprintf(file, "ALT"); break;
case LLAMA_GRETYPE_RULE_REF: fprintf(file, "RULE_REF"); break;
case LLAMA_GRETYPE_CHAR: fprintf(file, "CHAR"); break;
case LLAMA_GRETYPE_CHAR_NOT: fprintf(file, "CHAR_NOT"); break;
case LLAMA_GRETYPE_CHAR_RNG_UPPER: fprintf(file, "CHAR_RNG_UPPER"); break;
case LLAMA_GRETYPE_CHAR_ALT: fprintf(file, "CHAR_ALT"); break;
case LLAMA_GRETYPE_CHAR_ANY: fprintf(file, "CHAR_ANY"); break;
case LLAMA_GRETYPE_TOKEN: fprintf(file, "TOKEN"); break;
case LLAMA_GRETYPE_TOKEN_NOT: fprintf(file, "TOKEN_NOT"); break;
case LLAMA_GRETYPE_TOKEN_RNG_UPPER: fprintf(file, "TOKEN_RNG_UPPER"); break;
case LLAMA_GRETYPE_TOKEN_ALT: fprintf(file, "TOKEN_ALT"); break;
case LLAMA_GRETYPE_END: fprintf(file, "END"); break;
case LLAMA_GRETYPE_ALT: fprintf(file, "ALT"); break;
case LLAMA_GRETYPE_RULE_REF: fprintf(file, "RULE_REF"); break;
case LLAMA_GRETYPE_CHAR: fprintf(file, "CHAR"); break;
case LLAMA_GRETYPE_CHAR_NOT: fprintf(file, "CHAR_NOT"); break;
case LLAMA_GRETYPE_CHAR_RNG_UPPER: fprintf(file, "CHAR_RNG_UPPER"); break;
case LLAMA_GRETYPE_CHAR_ALT: fprintf(file, "CHAR_ALT"); break;
case LLAMA_GRETYPE_CHAR_ANY: fprintf(file, "CHAR_ANY"); break;
case LLAMA_GRETYPE_TOKEN: fprintf(file, "TOKEN"); break;
case LLAMA_GRETYPE_TOKEN_NOT: fprintf(file, "TOKEN_NOT"); break;
}
switch (elem.type) {
case LLAMA_GRETYPE_END:
@@ -331,10 +295,6 @@ static void print_rule_binary(FILE * file, const llama_grammar_rule & rule) {
fprintf(file, "%u", elem.value);
fprintf(file, "]> ");
break;
case LLAMA_GRETYPE_TOKEN_RNG_UPPER:
case LLAMA_GRETYPE_TOKEN_ALT:
fprintf(file, "(%u) ", elem.value);
break;
}
}
fprintf(file, "\n");
@@ -394,32 +354,14 @@ static void print_rule(
case LLAMA_GRETYPE_TOKEN:
fprintf(file, "<[");
fprintf(file, "%u", elem.value);
fprintf(file, "]> ");
break;
case LLAMA_GRETYPE_TOKEN_NOT:
fprintf(file, "!");
fprintf(file, "<[");
fprintf(file, "%u", elem.value);
fprintf(file, "]> ");
break;
case LLAMA_GRETYPE_TOKEN_RNG_UPPER:
if (i == 0 || !is_token_element(rule[i - 1])) {
throw std::runtime_error(
"LLAMA_GRETYPE_TOKEN_RNG_UPPER without preceding token: " +
std::to_string(rule_id) + "," + std::to_string(i));
}
fprintf(file, "-%u", elem.value);
break;
case LLAMA_GRETYPE_TOKEN_ALT:
if (i == 0 || !is_token_element(rule[i - 1])) {
throw std::runtime_error(
"LLAMA_GRETYPE_TOKEN_ALT without preceding token: " +
std::to_string(rule_id) + "," + std::to_string(i));
}
fprintf(file, ",%u", elem.value);
break;
}
if (is_token_element(elem) &&
rule[i + 1].type != LLAMA_GRETYPE_TOKEN_ALT && rule[i + 1].type != LLAMA_GRETYPE_TOKEN_RNG_UPPER) {
fprintf(file, "]> ");
}
if (is_char_element(elem)) {
switch (rule[i + 1].type) {
@@ -645,9 +587,11 @@ const char * llama_grammar_parser::parse_sequence(
type = LLAMA_GRETYPE_TOKEN_NOT;
pos++;
}
auto token_pair = parse_token(vocab, pos);
const char * token_end = token_pair.second;
last_sym_start = rule.size();
n_prev_rules = 1;
const char * token_end = parse_token(vocab, pos, type, rule);
rule.push_back({type, token_pair.first});
pos = parse_space(token_end, is_nested);
} else if (is_word_char(*pos)) { // rule reference
const char * name_end = parse_name(pos);
@@ -910,58 +854,17 @@ static bool llama_grammar_match_partial_char(
// returns true iff token matches the rule at pos (regular or inverse)
// asserts that pos is pointing to a token element
static std::pair<bool, const llama_grammar_element *> llama_grammar_match_token(
static bool llama_grammar_match_token(
const llama_grammar_element * pos,
const llama_token token) {
bool is_positive_token = pos->type == LLAMA_GRETYPE_TOKEN;
GGML_ASSERT(is_positive_token || pos->type == LLAMA_GRETYPE_TOKEN_NOT);
const uint32_t id = static_cast<uint32_t>(token);
bool found = false;
do {
if (pos[1].type == LLAMA_GRETYPE_TOKEN_RNG_UPPER) {
// inclusive range, e.g. <[1-5]>
found = found || (pos->value <= id && id <= pos[1].value);
pos += 2;
} else {
found = found || pos->value == id;
pos += 1;
}
} while (pos->type == LLAMA_GRETYPE_TOKEN_ALT);
return std::make_pair(found == is_positive_token, pos);
}
// every token id a token rule names, merged into sorted inclusive ranges
static std::vector<std::pair<uint32_t, uint32_t>> llama_grammar_opaque_tokens(const llama_grammar_rules & rules) {
std::vector<std::pair<uint32_t, uint32_t>> ranges;
for (const auto & rule : rules) {
for (size_t i = 0; i < rule.size(); i++) {
const auto type = rule[i].type;
if (type == LLAMA_GRETYPE_TOKEN || type == LLAMA_GRETYPE_TOKEN_NOT || type == LLAMA_GRETYPE_TOKEN_ALT) {
const bool is_range = i + 1 < rule.size() && rule[i + 1].type == LLAMA_GRETYPE_TOKEN_RNG_UPPER;
ranges.emplace_back(rule[i].value, is_range ? rule[i + 1].value : rule[i].value);
}
}
GGML_ASSERT(pos->type == LLAMA_GRETYPE_TOKEN || pos->type == LLAMA_GRETYPE_TOKEN_NOT);
if (pos->type == LLAMA_GRETYPE_TOKEN) {
return pos->value == static_cast<uint32_t>(token);
}
std::sort(ranges.begin(), ranges.end());
std::vector<std::pair<uint32_t, uint32_t>> merged;
for (const auto & r : ranges) {
if (!merged.empty() && r.first <= (uint64_t) merged.back().second + 1) {
merged.back().second = std::max(merged.back().second, r.second);
} else {
merged.push_back(r);
}
if (pos->type == LLAMA_GRETYPE_TOKEN_NOT) {
return pos->value != static_cast<uint32_t>(token);
}
return merged;
}
bool llama_grammar_is_opaque(const llama_grammar & grammar, llama_token token) {
const auto id = static_cast<uint32_t>(token);
auto it = std::upper_bound(grammar.opaque_tokens.begin(), grammar.opaque_tokens.end(), std::make_pair(id, UINT32_MAX));
return it != grammar.opaque_tokens.begin() && std::prev(it)->second >= id;
return false;
}
// transforms a grammar pushdown stack into N possible stacks, all ending
@@ -1043,7 +946,8 @@ static void llama_grammar_advance_stack(
break;
default:
// end of alternate (LLAMA_GRETYPE_END, LLAMA_GRETYPE_ALT) or middle of char range
// (LLAMA_GRETYPE_CHAR_ALT, LLAMA_GRETYPE_CHAR_RNG_UPPER) or token set (LLAMA_GRETYPE_TOKEN_ALT, LLAMA_GRETYPE_TOKEN_RNG_UPPER); stack should never be left on those
// (LLAMA_GRETYPE_CHAR_ALT, LLAMA_GRETYPE_CHAR_RNG_UPPER); stack should never be left on
// those
GGML_ABORT("fatal error");
}
}
@@ -1334,13 +1238,13 @@ static bool llama_grammar_reject_step(
if (stack_pos->type == LLAMA_GRETYPE_TOKEN || stack_pos->type == LLAMA_GRETYPE_TOKEN_NOT) {
for (const uint32_t i : cur) {
const auto & tok = candidates[i];
if (depth > 0) {
// char rules already consumed part of this token, so it cannot match by id.
// accept it only if they consumed all of it and did not end in a partial sequence.
if (tok.code_points[depth] != 0 || tok.partial_utf8.n_remain != 0) {
if (tok.code_points[depth] == 0) {
// reached the end of a token consumed by char rules, reject iff it ended
// in a partial response
if (tok.partial_utf8.n_remain != 0) {
rejects.push_back(i);
}
} else if (tok.partial_utf8.n_remain < 0 || !llama_grammar_match_token(stack_pos, tok.id).first) {
} else if (!llama_grammar_match_token(stack_pos, tok.id)) {
rejects.push_back(i);
}
}
@@ -1351,10 +1255,7 @@ static bool llama_grammar_reject_step(
for (const uint32_t i : cur) {
const auto & tok = candidates[i];
if (tok.opaque) {
// a token named by a token rule only matches token rules
rejects.push_back(i);
} else if (tok.code_points[depth] == 0) {
if (tok.code_points[depth] == 0) {
// reached end of full codepoints in token, reject iff it ended in a partial sequence
// that cannot satisfy this position in grammar
if (tok.partial_utf8.n_remain != 0 &&
@@ -1524,8 +1425,6 @@ struct llama_grammar * llama_grammar_init_impl(
}
} while (true);
auto opaque_tokens = llama_grammar_opaque_tokens(vec_rules);
// Important: vec_rules has to be moved here, not copied, because stacks contains
// pointers to elements of vec_rules. If vec_rules were copied into llama_grammar
// then the pointers would be invalidated when the local vec_rules goes out of scope.
@@ -1540,7 +1439,6 @@ struct llama_grammar * llama_grammar_init_impl(
/* .trigger_buffer_positions = */ {},
/* .trigger_tokens = */ {},
/* .trigger_patterns = */ {},
/* .opaque_tokens = */ std::move(opaque_tokens),
};
}
@@ -1642,8 +1540,6 @@ struct llama_grammar * llama_grammar_init_impl(
trigger.regex = std::regex(trigger.pattern);
}
auto opaque_tokens = llama_grammar_opaque_tokens(vec_rules);
// Important: vec_rules has to be moved here, not copied, because stacks contains
// pointers to elements of vec_rules. If vec_rules were copied into llama_grammar
// then the pointers would be invalidated when the local vec_rules goes out of scope.
@@ -1658,7 +1554,6 @@ struct llama_grammar * llama_grammar_init_impl(
/* .trigger_buffer_positions = */ {},
std::move(vec_trigger_tokens),
std::move(vec_trigger_patterns),
std::move(opaque_tokens),
};
}
@@ -1706,9 +1601,8 @@ void llama_grammar_apply_impl(const struct llama_grammar & grammar, llama_token_
} else if (piece.empty() || piece[0] == 0) {
cur_p->data[i].logit = -INFINITY;
} else {
candidates_decoded.push_back(llama_grammar_decode_utf8(piece, grammar.partial_utf8));
candidates_grammar.push_back({ i, candidates_decoded.back().first.data(), candidates_decoded.back().second, id,
llama_grammar_is_opaque(grammar, id) });
candidates_decoded.push_back(decode_utf8(piece, grammar.partial_utf8));
candidates_grammar.push_back({ i, candidates_decoded.back().first.data(), candidates_decoded.back().second, id });
}
}
@@ -1779,7 +1673,7 @@ void llama_grammar_accept_impl(struct llama_grammar & grammar, llama_token token
void llama_grammar_accept_str(struct llama_grammar & grammar, const std::string & piece) {
// Note terminating 0 in decoded string
const auto decoded = llama_grammar_decode_utf8(piece, grammar.partial_utf8);
const auto decoded = decode_utf8(piece, grammar.partial_utf8);
const auto & code_points = decoded.first;
for (auto it = code_points.begin(), end = code_points.end() - 1; it != end; ++it) {
@@ -1794,16 +1688,9 @@ void llama_grammar_accept_str(struct llama_grammar & grammar, const std::string
void llama_grammar_accept_token(struct llama_grammar & grammar, llama_token token, const std::string & piece) {
// Note terminating 0 in decoded string
const auto decoded = llama_grammar_decode_utf8(piece, grammar.partial_utf8);
const auto decoded = decode_utf8(piece, grammar.partial_utf8);
const auto & code_points = decoded.first;
// the mask rejects a piece that does not continue a partial character, so accepting one has to fail too
if (decoded.second.n_remain < 0) {
throw std::runtime_error("Invalid UTF-8 continuation in piece: " + piece + " (" + std::to_string(token) + ")");
}
const bool opaque = llama_grammar_is_opaque(grammar, token);
llama_grammar_stacks stacks_new;
stacks_new.reserve(grammar.stacks.size());
@@ -1815,15 +1702,14 @@ void llama_grammar_accept_token(struct llama_grammar & grammar, llama_token toke
const llama_grammar_element * pos = stack.back();
if (pos->type == LLAMA_GRETYPE_TOKEN || pos->type == LLAMA_GRETYPE_TOKEN_NOT) {
auto match = llama_grammar_match_token(pos, token);
if (match.first) {
if (llama_grammar_match_token(pos, token)) {
llama_grammar_stack new_stack(stack.begin(), stack.end() - 1);
if (!llama_grammar_is_end_of_sequence(match.second)) {
new_stack.push_back(match.second);
if (!llama_grammar_is_end_of_sequence(pos + 1)) {
new_stack.push_back(pos + 1);
}
llama_grammar_advance_stack(*grammar.rules, new_stack, stacks_new);
}
} else if (!opaque) {
} else {
llama_grammar_stacks current_stacks = {stack};
for (auto it = code_points.begin(), end = code_points.end() - 1; it != end; ++it) {
+2 -18
View File
@@ -38,17 +38,11 @@ enum llama_gretype {
// any character (.)
LLAMA_GRETYPE_CHAR_ANY = 7,
// terminal element: token (<[token-id]>, <[token-id,token-id-token-id]>)
// terminal element: token (<[token-id]>)
LLAMA_GRETYPE_TOKEN = 8,
// inverse token (!<[token-id]>, !<[token-id,token-id-token-id]>)
// inverse token (!<[token-id]>)
LLAMA_GRETYPE_TOKEN_NOT = 9,
// modifies a preceding LLAMA_GRETYPE_TOKEN, LLAMA_GRETYPE_TOKEN_NOT, or LLAMA_GRETYPE_TOKEN_ALT to be an inclusive range (<[1-5]>)
LLAMA_GRETYPE_TOKEN_RNG_UPPER = 10,
// modifies a preceding LLAMA_GRETYPE_TOKEN, LLAMA_GRETYPE_TOKEN_NOT, or LLAMA_GRETYPE_TOKEN_RNG_UPPER to add an alternate token to match (<[1,5]>, <[1-5,9]>)
LLAMA_GRETYPE_TOKEN_ALT = 11,
};
typedef struct llama_grammar_element {
@@ -66,7 +60,6 @@ struct llama_grammar_candidate {
const uint32_t * code_points;
llama_partial_utf8 partial_utf8;
llama_token id;
bool opaque;
};
using llama_grammar_rule = std::vector< llama_grammar_element>;
@@ -91,10 +84,6 @@ std::vector<llama_grammar_candidate> llama_grammar_reject_candidates_for_stack(
const llama_grammar_stack & stack,
const llama_grammar_candidates & candidates);
std::pair<std::vector<uint32_t>, llama_partial_utf8> llama_grammar_decode_utf8(
const std::string & src,
llama_partial_utf8 partial_start);
struct llama_grammar_parser {
const llama_vocab * vocab;
std::map<std::string, uint32_t> symbol_ids;
@@ -162,8 +151,6 @@ struct llama_grammar {
trigger_patterns; // Regular expressions that trigger a lazy grammar. Must be a full match of the entire generated
// string, and the grammar will be given the string from the first match group onwards.
// token ids named by token rules, as sorted inclusive ranges. char rules never match these tokens.
std::vector<std::pair<uint32_t, uint32_t>> opaque_tokens;
};
//
@@ -208,6 +195,3 @@ void llama_grammar_accept_token(
struct llama_grammar & grammar,
llama_token token,
const std::string & piece);
// true if a token rule names the token, char rules do not match it
bool llama_grammar_is_opaque(const llama_grammar & grammar, llama_token token);
-33
View File
@@ -38,7 +38,6 @@
#include <sstream>
#include <stdexcept>
#include <string>
#include <unordered_map>
#include <vector>
static llama_model * llama_model_mapping(llm_arch arch, const llama_model_params & params) {
@@ -4282,38 +4281,6 @@ int32_t llama_model_n_expert(const struct llama_model * model) {
return model->hparams.n_expert;
}
std::vector<uint32_t> llama_model_layer_visits(const struct llama_model * model) {
const size_t n_layer = model->layers.size();
std::vector<uint32_t> visits(n_layer, 0);
std::unordered_map<const ggml_tensor *, size_t> owner;
owner.reserve(n_layer);
for (size_t il = 0; il < n_layer; ++il) {
const auto & l = model->layers[il];
const ggml_tensor * key =
l.ffn_up_exps ? l.ffn_up_exps :
l.ffn_gate_up_exps ? l.ffn_gate_up_exps :
l.ffn_gate_exps ? l.ffn_gate_exps :
l.ffn_down_exps ? l.ffn_down_exps :
l.attn_norm ? l.attn_norm :
l.ffn_norm ? l.ffn_norm :
l.wq ? l.wq :
l.wqkv ? l.wqkv :
l.ffn_up ? l.ffn_up :
l.ffn_gate ? l.ffn_gate :
l.ffn_down ? l.ffn_down :
nullptr;
if (!key) {
visits[il] = 1;
continue;
}
auto [it, inserted] = owner.emplace(key, il);
visits[it->second]++;
}
return visits;
}
int32_t llama_model_n_devices(const struct llama_model * model) {
return (int32_t)model->devices.size();
}
-1
View File
@@ -2673,7 +2673,6 @@ void llama_vocab::impl::load(llama_model_loader & ml, const LLM_KV & kv) {
pre_type = LLAMA_VOCAB_PRE_TYPE_GPT4O;
special_bos_id = LLAMA_TOKEN_NULL;
clean_spaces = false;
ignore_merges = true;
} else if (
tokenizer_pre == "granite-embed-multi-97m") {
pre_type = LLAMA_VOCAB_PRE_TYPE_GRANITE_EMB_MULTI;
+16 -32
View File
@@ -173,35 +173,22 @@ static clef_spans clef_get_spans(const llama_ubatch & ubatch) {
return res;
}
// the head always has the max number of spans, so its graph depends only on n_tokens
// each span has 1 token or more and each question has an option, so there is always a spare question for the padding options
static size_t clef_n_questions(int64_t n_tokens) { return n_tokens / 2 + 1; }
static size_t clef_n_options (int64_t n_tokens) { return std::max<int64_t>(n_tokens - 1, 1); }
// pooling matrices and indices computed from the spans
class llama_model_clef::input_decision : public llm_graph_input_i {
public:
input_decision(const llama_ubatch & ubatch) :
n_tokens(ubatch.n_tokens), n_questions(clef_n_questions(n_tokens)), n_options(clef_n_options(n_tokens)) {}
input_decision(const llama_ubatch & ubatch) : n_tokens(ubatch.n_tokens) {
const auto spans = clef_get_spans(ubatch);
n_questions = spans.questions.size();
n_options = spans.options.size();
}
void set_input(const llama_ubatch * ubatch) override {
// a batch of embeddings has no token ids, and no usable spans
const std::vector<llama_token> no_tokens(ubatch->token ? 0 : n_tokens, 0);
ggml_backend_tensor_set(tokens, ubatch->token ? ubatch->token : no_tokens.data(), 0, n_tokens * sizeof(llama_token));
auto spans = clef_get_spans(*ubatch);
const int32_t n_q_used = spans.questions.size();
GGML_ASSERT((size_t) n_q_used < n_questions && spans.options.size() <= n_options);
// the padding options belong to the first padding question, the questions do not read the padding questions
spans.questions.resize(n_questions, { 0, 0, 0 });
spans.options.resize(n_options, { n_q_used, 0, 0 });
std::vector<float> q_mask(n_questions * n_questions, 0.0f);
for (size_t i = 0; i < n_questions; i++) {
std::fill(q_mask.begin() + i * n_questions + n_q_used, q_mask.begin() + (i + 1) * n_questions, -INFINITY);
}
ggml_backend_tensor_set(question_mask, q_mask.data(), 0, ggml_nbytes(question_mask));
const auto spans = clef_get_spans(*ubatch);
GGML_ASSERT(spans.questions.size() == n_questions && spans.options.size() == n_options);
// the scores are NaN if the batch has a decision order that cannot be used
const float status_data = spans.valid || ubatch->decision_order == nullptr ? 0.0f : NAN;
@@ -228,8 +215,6 @@ public:
owner[i] = o.question;
mask[o.question * n_options + i] = 0.0f;
}
// a padding question reads all options, a row of -inf gives NaN
std::fill(mask.begin() + n_q_used * n_options, mask.end(), 0.0f);
ggml_backend_tensor_set(pool_q, pool_q_data.data(), 0, ggml_nbytes(pool_q));
ggml_backend_tensor_set(pool_o, pool_o_data.data(), 0, ggml_nbytes(pool_o));
@@ -240,7 +225,8 @@ public:
bool can_reuse(const llm_graph_params & params) override {
// the values are computed again in set_input(), only the shapes must match
return params.ubatch.n_tokens == (uint32_t) n_tokens;
const auto spans = clef_get_spans(params.ubatch);
return spans.questions.size() == n_questions && spans.options.size() == n_options;
}
ggml_tensor * tokens = nullptr; // I32 [n_tokens]
@@ -249,12 +235,11 @@ public:
ggml_tensor * question_type = nullptr; // I32 [n_questions]
ggml_tensor * option_question = nullptr; // I32 [n_options]
ggml_tensor * option_mask = nullptr; // F32 [n_options, n_questions], 0 if the option belongs to the question, else -inf
ggml_tensor * question_mask = nullptr; // F32 [n_questions, n_questions], -inf for the padding questions
ggml_tensor * status = nullptr; // F32 [1], added to the scores: 0, or NaN on invalid input
const int64_t n_tokens;
const size_t n_questions;
const size_t n_options;
size_t n_questions;
size_t n_options;
};
// the backbone is copied from llama_model_qwen35::graph, without the memory module
@@ -494,8 +479,8 @@ ggml_tensor * llama_model_clef::graph::build_head_norm(ggml_tensor * cur, const
return build_norm(cur, n.w, n.b, LLM_NORM, -1);
}
// q: [n_embd_h, n_q], kv: [n_embd_h, n_kv], kq_mask: [n_kv, n_q] or null
ggml_tensor * llama_model_clef::graph::build_head_attn(ggml_tensor * q, ggml_tensor * kv, const attn & a, ggml_tensor * kq_mask) {
// q: [n_embd_h, n_q], kv: [n_embd_h, n_kv], no mask
ggml_tensor * llama_model_clef::graph::build_head_attn(ggml_tensor * q, ggml_tensor * kv, const attn & a) {
const int64_t n_embd_h = q->ne[0];
const int64_t n_head_h = model.n_head_decision;
const int64_t d_head = n_embd_h / n_head_h;
@@ -512,7 +497,7 @@ ggml_tensor * llama_model_clef::graph::build_head_attn(ggml_tensor * q, ggml_ten
Vcur = ggml_cont(ctx0, Vcur);
ggml_tensor * kq = ggml_mul_mat(ctx0, Kcur, Qcur); // [n_kv, n_q, n_head]
kq = ggml_soft_max_ext(ctx0, kq, kq_mask, 1.0f / sqrtf(float(d_head)), 0.0f);
kq = ggml_soft_max_ext(ctx0, kq, nullptr, 1.0f / sqrtf(float(d_head)), 0.0f);
ggml_tensor * cur = ggml_mul_mat(ctx0, Vcur, kq); // [d_head, n_q, n_head]
cur = ggml_cont_2d(ctx0, ggml_permute(ctx0, cur, 0, 2, 1, 3), n_embd_h, n_q);
@@ -538,9 +523,8 @@ ggml_tensor * llama_model_clef::graph::build_head(ggml_tensor * hidden, input_de
inp->question_type = ggml_new_tensor_1d(ctx0, GGML_TYPE_I32, n_questions);
inp->option_question = ggml_new_tensor_1d(ctx0, GGML_TYPE_I32, n_options);
inp->option_mask = ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, n_options, n_questions);
inp->question_mask = ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, n_questions, n_questions);
inp->status = ggml_new_tensor_1d(ctx0, GGML_TYPE_F32, 1);
for (ggml_tensor * t : { inp->tokens, inp->pool_q, inp->pool_o, inp->question_type, inp->option_question, inp->option_mask, inp->question_mask, inp->status }) {
for (ggml_tensor * t : { inp->tokens, inp->pool_q, inp->pool_o, inp->question_type, inp->option_question, inp->option_mask, inp->status }) {
ggml_set_input(t);
}
@@ -600,7 +584,7 @@ ggml_tensor * llama_model_clef::graph::build_head(ggml_tensor * hidden, input_de
const auto & layer = model.head_layers[il];
ggml_tensor * cur = build_head_norm(fields, layer.self_norm);
fields = ggml_add(ctx0, fields, build_head_attn(cur, cur, layer.self_attn, inp->question_mask));
fields = ggml_add(ctx0, fields, build_head_attn(cur, cur, layer.self_attn));
cur = build_head_norm(fields, layer.cross_norm);
fields = ggml_add(ctx0, fields, build_head_attn(cur, memory, layer.cross_attn));
+1 -1
View File
@@ -2930,7 +2930,7 @@ struct llama_model_clef : public llama_model_qwen35 {
input_decision * inp);
ggml_tensor * build_head_norm(ggml_tensor * cur, const norm & n);
ggml_tensor * build_head_attn(ggml_tensor * q, ggml_tensor * kv, const attn & a, ggml_tensor * kq_mask = nullptr);
ggml_tensor * build_head_attn(ggml_tensor * q, ggml_tensor * kv, const attn & a);
ggml_tensor * build_head_ffn (ggml_tensor * cur, const head_layer & layer);
const llama_model_clef & model;
+5 -11
View File
@@ -99,19 +99,13 @@ static decision2_rows decision2_get_rows(const llama_ubatch & ubatch) {
return res;
}
// no decision order (e.g. reserve): use the max number of options
static size_t decision2_n_options(const llama_ubatch & ubatch, uint32_t n_outputs) {
return ubatch.decision_order ? decision2_get_rows(ubatch).options.size() : std::max<uint32_t>(n_outputs, 1);
}
class llama_model_vllm_decision2::input_decision : public llm_graph_input_i {
public:
input_decision(const llama_ubatch & ubatch, uint32_t n_outputs) : n_options(decision2_n_options(ubatch, n_outputs)) {}
input_decision(const llama_ubatch & ubatch) : n_options(decision2_get_rows(ubatch).options.size()) {}
void set_input(const llama_ubatch * ubatch) override {
auto rows = decision2_get_rows(*ubatch);
GGML_ASSERT(rows.options.size() <= n_options);
rows.options.resize(n_options, 0);
const auto rows = decision2_get_rows(*ubatch);
GGML_ASSERT(rows.options.size() == n_options);
// the scores are NaN if the batch has a decision order that cannot be used
const float status_data = rows.valid || ubatch->decision_order == nullptr ? 0.0f : NAN;
@@ -123,7 +117,7 @@ public:
bool can_reuse(const llm_graph_params & params) override {
// the values are computed again in set_input(), only the shapes must match
return decision2_n_options(params.ubatch, params.n_outputs) == n_options;
return decision2_get_rows(params.ubatch).options.size() == n_options;
}
ggml_tensor * options = nullptr; // I32 [n_options]
@@ -135,7 +129,7 @@ public:
llama_model_vllm_decision2::graph::graph(const llm_graph_params & params) :
llama_model_qwen3::graph(params, no_build()), model(static_cast<const llama_model_vllm_decision2 &>(*params.model)) {
auto inp_decision_ptr = std::make_unique<input_decision>(ubatch, n_outputs);
auto inp_decision_ptr = std::make_unique<input_decision>(ubatch);
auto * inp_decision = inp_decision_ptr.get();
res->add_input(std::move(inp_decision_ptr));
-43
View File
@@ -367,49 +367,6 @@ void test_basic(testing & t) {
});
t.test("through", [](testing & t) {
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
return p.through(p.tag("body", p.until_one_of({ "</a>", "</b>" })), { { p.literal("</a>"), p.literal("x") }, { p.literal("</b>") } }, true);
});
t.test("consumes the delimiter and its rest", [&](testing & t) {
auto ctx = common_peg_parse_context("hello</a>x");
auto result = parser.parse(ctx);
t.assert_equal("success", true, result.success());
t.assert_equal("end", (size_t) 10, result.end);
t.assert_equal("body", std::string("hello"), std::string(ctx.ast.get(result.nodes[0]).text));
});
t.test("picks the branch of the delimiter found", [&](testing & t) {
auto ctx = common_peg_parse_context("hello</b>x");
auto result = parser.parse(ctx);
t.assert_equal("success", true, result.success());
t.assert_equal("end", (size_t) 9, result.end);
});
t.test("waits for the rest of a partial delimiter", [&](testing & t) {
auto ctx = common_peg_parse_context("hello</", COMMON_PEG_PARSE_FLAG_LENIENT);
auto result = parser.parse(ctx);
t.assert_equal("need_more_input", true, result.need_more_input());
});
t.test("optional may end before a delimiter", [&](testing & t) {
auto ctx = common_peg_parse_context("hello");
auto result = parser.parse(ctx);
t.assert_equal("success", true, result.success());
t.assert_equal("end", (size_t) 5, result.end);
});
t.test("fails when content stops off a delimiter", [&](testing & t) {
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
return p.through(p.literal("he"), "</a>");
});
auto ctx = common_peg_parse_context("hello</a>");
auto result = parser.parse(ctx);
t.assert_equal("fail", true, result.fail());
});
});
t.test("recursive rules", [](testing &t) {
// Test simple number
t.test("simple_number", [](testing &t) {
+68 -294
View File
@@ -129,14 +129,14 @@ void test_gbnf_generation(testing &t) {
});
assert_gbnf_equal(t, R"""(
root ::= until-1
root ::= until-0
space ::= | " " | "\n"{1,2} [ \t]{0,20}
until-1 ::= | [<] until-1-01 | [^<] until-1
until-1-01 ::= | [<] until-1-01 | [/] until-1-02 | [^/<] until-1
until-1-02 ::= | [<] until-1-01 | [t] until-1-03 | [^<t] until-1
until-1-03 ::= | [<] until-1-01 | [a] until-1-04 | [^<a] until-1
until-1-04 ::= | [<] until-1-01 | [g] until-1-05 | [^<g] until-1
until-1-05 ::= | [<] until-1-01 | [^<>] until-1
until-0 ::= | [<] until-0-01 | [^<] until-0
until-0-01 ::= | [<] until-0-01 | [/] until-0-02 | [^/<] until-0
until-0-02 ::= | [<] until-0-01 | [t] until-0-03 | [^<t] until-0
until-0-03 ::= | [<] until-0-01 | [a] until-0-04 | [^<a] until-0
until-0-04 ::= | [<] until-0-01 | [g] until-0-05 | [^<g] until-0
until-0-05 ::= | [<] until-0-01 | [^<>] until-0
)""", gbnf);
});
@@ -150,22 +150,22 @@ void test_gbnf_generation(testing &t) {
});
assert_gbnf_equal(t, R"""(
root ::= until-1
root ::= until-0
space ::= | " " | "\n"{1,2} [ \t]{0,20}
until-1 ::= | [\n] until-1-01 | [^\n] until-1
until-1-01 ::= | [\n] until-1-01 | [<] until-1-02 | [^\n<] until-1
until-1-02 ::= | [\n] until-1-01 | [/] until-1-03 | [^\n/] until-1
until-1-03 ::= | [\n] until-1-01 | [p] until-1-04 | [^\np] until-1
until-1-04 ::= | [\n] until-1-01 | [a] until-1-05 | [^\na] until-1
until-1-05 ::= | [\n] until-1-01 | [r] until-1-06 | [^\nr] until-1
until-1-06 ::= | [\n] until-1-01 | [a] until-1-07 | [^\na] until-1
until-1-07 ::= | [\n] until-1-01 | [m] until-1-08 | [^\nm] until-1
until-1-08 ::= | [\n] until-1-01 | [e] until-1-09 | [^\ne] until-1
until-1-09 ::= | [\n] until-1-01 | [t] until-1-10 | [^\nt] until-1
until-1-10 ::= | [\n] until-1-01 | [e] until-1-11 | [^\ne] until-1
until-1-11 ::= | [\n] until-1-01 | [r] until-1-12 | [^\nr] until-1
until-1-12 ::= | [\n] until-1-01 | [>] until-1-13 | [^\n>] until-1
until-1-13 ::= | [^\n] until-1
until-0 ::= | [\n] until-0-01 | [^\n] until-0
until-0-01 ::= | [\n] until-0-01 | [<] until-0-02 | [^\n<] until-0
until-0-02 ::= | [\n] until-0-01 | [/] until-0-03 | [^\n/] until-0
until-0-03 ::= | [\n] until-0-01 | [p] until-0-04 | [^\np] until-0
until-0-04 ::= | [\n] until-0-01 | [a] until-0-05 | [^\na] until-0
until-0-05 ::= | [\n] until-0-01 | [r] until-0-06 | [^\nr] until-0
until-0-06 ::= | [\n] until-0-01 | [a] until-0-07 | [^\na] until-0
until-0-07 ::= | [\n] until-0-01 | [m] until-0-08 | [^\nm] until-0
until-0-08 ::= | [\n] until-0-01 | [e] until-0-09 | [^\ne] until-0
until-0-09 ::= | [\n] until-0-01 | [t] until-0-10 | [^\nt] until-0
until-0-10 ::= | [\n] until-0-01 | [e] until-0-11 | [^\ne] until-0
until-0-11 ::= | [\n] until-0-01 | [r] until-0-12 | [^\nr] until-0
until-0-12 ::= | [\n] until-0-01 | [>] until-0-13 | [^\n>] until-0
until-0-13 ::= | [^\n] until-0
)""", gbnf);
});
@@ -181,15 +181,15 @@ void test_gbnf_generation(testing &t) {
});
assert_gbnf_equal(t, R"""(
root ::= until-1
root ::= until-0
space ::= | " " | "\n"{1,2} [ \t]{0,20}
until-1 ::= | [<] until-1-01 | [^<] until-1
until-1-01 ::= | [<] until-1-01 | [\uFF5C] until-1-02 | [^<\uFF5C] until-1
until-1-02 ::= | [<] until-1-01 | [D] until-1-03 | [^<D] until-1
until-1-03 ::= | [<] until-1-01 | [S] until-1-04 | [^<S] until-1
until-1-04 ::= | [<] until-1-01 | [M] until-1-05 | [^<M] until-1
until-1-05 ::= | [<] until-1-01 | [L] until-1-06 | [^<L] until-1
until-1-06 ::= | [<] until-1-01 | [^<\uFF5C] until-1
until-0 ::= | [<] until-0-01 | [^<] until-0
until-0-01 ::= | [<] until-0-01 | [\uFF5C] until-0-02 | [^<\uFF5C] until-0
until-0-02 ::= | [<] until-0-01 | [D] until-0-03 | [^<D] until-0
until-0-03 ::= | [<] until-0-01 | [S] until-0-04 | [^<S] until-0
until-0-04 ::= | [<] until-0-01 | [M] until-0-05 | [^<M] until-0
until-0-05 ::= | [<] until-0-01 | [L] until-0-06 | [^<L] until-0
until-0-06 ::= | [<] until-0-01 | [^<\uFF5C] until-0
)""", gbnf);
});
@@ -203,18 +203,18 @@ void test_gbnf_generation(testing &t) {
});
assert_gbnf_equal(t, R"""(
root ::= until-3
root ::= until-0
space ::= | " " | "\n"{1,2} [ \t]{0,20}
until-3 ::= | [a] until-3-01 | [c] until-3-03 | [e] until-3-05 | [^ace] until-3
until-3-01 ::= | [a] until-3-01 | [c] until-3-03 | [e] until-3-05 | [^abce] until-3
until-3-03 ::= | [a] until-3-01 | [c] until-3-03 | [e] until-3-05 | [^acde] until-3
until-3-05 ::= | [a] until-3-01 | [c] until-3-03 | [e] until-3-05 | [^acef] until-3
until-0 ::= | [a] until-0-01 | [c] until-0-03 | [e] until-0-05 | [^ace] until-0
until-0-01 ::= | [a] until-0-01 | [c] until-0-03 | [e] until-0-05 | [^abce] until-0
until-0-03 ::= | [a] until-0-01 | [c] until-0-03 | [e] until-0-05 | [^acde] until-0
until-0-05 ::= | [a] until-0-01 | [c] until-0-03 | [e] until-0-05 | [^acef] until-0
)""", gbnf);
});
t.test("through grammar", [](testing &t) {
t.test("ac grammar", [](testing &t) {
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
return p.through(p.until("</tag>"), "</tag>");
return p.ac(p.until("</tag>") + p.literal("</tag>"), "</tag>");
});
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
@@ -222,20 +222,20 @@ void test_gbnf_generation(testing &t) {
});
assert_gbnf_equal(t, R"""(
root ::= until-3
ac-3 ::= [<] ac-3-01 | [^<] ac-3
ac-3-01 ::= [<] ac-3-01 | [/] ac-3-02 | [^/<] ac-3
ac-3-02 ::= [<] ac-3-01 | [t] ac-3-03 | [^<t] ac-3
ac-3-03 ::= [<] ac-3-01 | [a] ac-3-04 | [^<a] ac-3
ac-3-04 ::= [<] ac-3-01 | [g] ac-3-05 | [^<g] ac-3
ac-3-05 ::= [>] | [<] ac-3-01 | [^<>] ac-3
root ::= ac-3
space ::= | " " | "\n"{1,2} [ \t]{0,20}
until-3 ::= [<] until-3-01 | [^<] until-3
until-3-01 ::= [<] until-3-01 | [/] until-3-02 | [^/<] until-3
until-3-02 ::= [<] until-3-01 | [t] until-3-03 | [^<t] until-3
until-3-03 ::= [<] until-3-01 | [a] until-3-04 | [^<a] until-3
until-3-04 ::= [<] until-3-01 | [g] until-3-05 | [^<g] until-3
until-3-05 ::= [>] | [<] until-3-01 | [^<>] until-3
)""", gbnf);
});
t.test("through grammar terminates at first delimiter", [](testing &t) {
t.test("ac grammar terminates at first delimiter", [](testing &t) {
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
return p.through(p.until("\n</parameter>\n"), "\n</parameter>\n");
return p.ac(p.until("\n</parameter>\n") + p.literal("\n</parameter>\n"), "\n</parameter>\n");
});
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
@@ -243,28 +243,28 @@ void test_gbnf_generation(testing &t) {
});
assert_gbnf_equal(t, R"""(
root ::= until-3
ac-3 ::= [\n] ac-3-01 | [^\n] ac-3
ac-3-01 ::= [\n] ac-3-01 | [<] ac-3-02 | [^\n<] ac-3
ac-3-02 ::= [\n] ac-3-01 | [/] ac-3-03 | [^\n/] ac-3
ac-3-03 ::= [\n] ac-3-01 | [p] ac-3-04 | [^\np] ac-3
ac-3-04 ::= [\n] ac-3-01 | [a] ac-3-05 | [^\na] ac-3
ac-3-05 ::= [\n] ac-3-01 | [r] ac-3-06 | [^\nr] ac-3
ac-3-06 ::= [\n] ac-3-01 | [a] ac-3-07 | [^\na] ac-3
ac-3-07 ::= [\n] ac-3-01 | [m] ac-3-08 | [^\nm] ac-3
ac-3-08 ::= [\n] ac-3-01 | [e] ac-3-09 | [^\ne] ac-3
ac-3-09 ::= [\n] ac-3-01 | [t] ac-3-10 | [^\nt] ac-3
ac-3-10 ::= [\n] ac-3-01 | [e] ac-3-11 | [^\ne] ac-3
ac-3-11 ::= [\n] ac-3-01 | [r] ac-3-12 | [^\nr] ac-3
ac-3-12 ::= [\n] ac-3-01 | [>] ac-3-13 | [^\n>] ac-3
ac-3-13 ::= [\n] | [^\n] ac-3
root ::= ac-3
space ::= | " " | "\n"{1,2} [ \t]{0,20}
until-3 ::= [\n] until-3-01 | [^\n] until-3
until-3-01 ::= [\n] until-3-01 | [<] until-3-02 | [^\n<] until-3
until-3-02 ::= [\n] until-3-01 | [/] until-3-03 | [^\n/] until-3
until-3-03 ::= [\n] until-3-01 | [p] until-3-04 | [^\np] until-3
until-3-04 ::= [\n] until-3-01 | [a] until-3-05 | [^\na] until-3
until-3-05 ::= [\n] until-3-01 | [r] until-3-06 | [^\nr] until-3
until-3-06 ::= [\n] until-3-01 | [a] until-3-07 | [^\na] until-3
until-3-07 ::= [\n] until-3-01 | [m] until-3-08 | [^\nm] until-3
until-3-08 ::= [\n] until-3-01 | [e] until-3-09 | [^\ne] until-3
until-3-09 ::= [\n] until-3-01 | [t] until-3-10 | [^\nt] until-3
until-3-10 ::= [\n] until-3-01 | [e] until-3-11 | [^\ne] until-3
until-3-11 ::= [\n] until-3-01 | [r] until-3-12 | [^\nr] until-3
until-3-12 ::= [\n] until-3-01 | [>] until-3-13 | [^\n>] until-3
until-3-13 ::= [\n] | [^\n] until-3
)""", gbnf);
});
t.test("through grammar multiple delimiters", [](testing &t) {
t.test("ac grammar multiple delimiters", [](testing &t) {
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
return p.through(p.eps(), { { p.literal("ab") }, { p.literal("cd") }, { p.literal("ef") } });
return p.ac(p.eps(), std::vector<std::string>{"ab", "cd", "ef"});
});
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
@@ -272,238 +272,12 @@ void test_gbnf_generation(testing &t) {
});
assert_gbnf_equal(t, R"""(
root ::= until-4
ac-1 ::= [a] ac-1-01 | [c] ac-1-03 | [e] ac-1-05 | [^ace] ac-1
ac-1-01 ::= [b] | [a] ac-1-01 | [c] ac-1-03 | [e] ac-1-05 | [^abce] ac-1
ac-1-03 ::= [d] | [a] ac-1-01 | [c] ac-1-03 | [e] ac-1-05 | [^acde] ac-1
ac-1-05 ::= [f] | [a] ac-1-01 | [c] ac-1-03 | [e] ac-1-05 | [^acef] ac-1
root ::= ac-1
space ::= | " " | "\n"{1,2} [ \t]{0,20}
until-4 ::= [a] until-4-01 | [c] until-4-03 | [e] until-4-05 | [^ace] until-4
until-4-01 ::= [b] | [a] until-4-01 | [c] until-4-03 | [e] until-4-05 | [^abce] until-4
until-4-03 ::= [d] | [a] until-4-01 | [c] until-4-03 | [e] until-4-05 | [^acde] until-4
until-4-05 ::= [f] | [a] until-4-01 | [c] until-4-03 | [e] until-4-05 | [^acef] until-4
)""", gbnf);
});
t.test("until grammar with parser delimiters", [](testing &t) {
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
return p.until((p.literal("a") + p.literal("b")) | p.literal("cd"));
});
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
parser.build_grammar(builder);
});
assert_gbnf_equal(t, R"""(
root ::= until-5
space ::= | " " | "\n"{1,2} [ \t]{0,20}
until-5 ::= | [a] until-5-01 | [c] until-5-03 | [^ac] until-5
until-5-01 ::= | [a] until-5-01 | [c] until-5-03 | [^abc] until-5
until-5-03 ::= | [a] until-5-01 | [c] until-5-03 | [^acd] until-5
)""", gbnf);
});
t.test("delimiters see through tags and atomics", [](testing &t) {
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
return p.until(p.tag("open", p.literal("a") + p.atomic(p.literal("b"))));
});
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
parser.build_grammar(builder);
});
assert_gbnf_equal(t, R"""(
root ::= until-5
space ::= | " " | "\n"{1,2} [ \t]{0,20}
until-5 ::= | [a] until-5-01 | [^a] until-5
until-5-01 ::= | [a] until-5-01 | [^ab] until-5
)""", gbnf);
});
t.test("delimiters expand a choice in a sequence", [](testing &t) {
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
return p.until(p.literal("a") + (p.literal("b") | p.literal("d")));
});
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
parser.build_grammar(builder);
});
assert_gbnf_equal(t, R"""(
root ::= until-5
space ::= | " " | "\n"{1,2} [ \t]{0,20}
until-5 ::= | [a] until-5-01 | [^a] until-5
until-5-01 ::= | [a] until-5-01 | [^abd] until-5
)""", gbnf);
});
t.test("delimiters reject anything but sequences, choices, literals, and tokens", [](testing &t) {
bool threw = false;
try {
build_peg_parser([](common_peg_parser_builder & p) {
return p.until(p.literal("a") + p.space());
});
} catch (const std::invalid_argument &) {
threw = true;
}
t.assert_equal("space_rejected", true, threw);
});
t.test("through branches reject a shared delimiter", [](testing &t) {
bool threw = false;
try {
build_peg_parser([](common_peg_parser_builder & p) {
return p.through(p.eps(), { { p.literal("ab"), p.literal("x") }, { p.literal("ab"), p.literal("y") } });
});
} catch (const std::invalid_argument &) {
threw = true;
}
t.assert_equal("duplicate_rejected", true, threw);
});
t.test("through grammar branches continue with their rest", [](testing &t) {
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
return p.through(p.eps(), { { p.literal("ab"), p.literal("x") }, { p.literal("cd") } });
});
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
parser.build_grammar(builder);
});
assert_gbnf_equal(t, R"""(
root ::= until-4
space ::= | " " | "\n"{1,2} [ \t]{0,20}
until-4 ::= [a] until-4-01 | [c] until-4-03 | [^ac] until-4
until-4-01 ::= [b] until-4-rest-0 | [a] until-4-01 | [c] until-4-03 | [^abc] until-4
until-4-03 ::= [d] | [a] until-4-01 | [c] until-4-03 | [^acd] until-4
until-4-rest-0 ::= "x"
)""", gbnf);
});
t.test("through grammar optional may end before a delimiter", [](testing &t) {
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
return p.through(p.eps(), { { p.literal("ab"), p.literal("x") } }, true);
});
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
parser.build_grammar(builder);
});
assert_gbnf_equal(t, R"""(
root ::= until-3
space ::= | " " | "\n"{1,2} [ \t]{0,20}
until-3 ::= | [a] until-3-01 | [^a] until-3
until-3-01 ::= | [b] until-3-rest-0 | [a] until-3-01 | [^ab] until-3
until-3-rest-0 ::= "x"
)""", gbnf);
});
t.test("trigger rules emit their start and rest when not lazy", [](testing &t) {
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
auto ab = p.literal("ab");
auto cd = p.literal("cd");
auto scan = p.until(ab | cd);
return scan + (p.trigger_rule("one", ab, p.literal("x")) | p.trigger_rule("two", cd, p.literal("y")));
});
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
parser.build_grammar(builder, false);
});
assert_gbnf_equal(t, R"""(
one ::= "ab" "x"
root ::= until-3 (one | two)
space ::= | " " | "\n"{1,2} [ \t]{0,20}
two ::= "cd" "y"
until-3 ::= | [a] until-3-01 | [c] until-3-03 | [^ac] until-3
until-3-01 ::= | [a] until-3-01 | [c] until-3-03 | [^abc] until-3
until-3-03 ::= | [a] until-3-01 | [c] until-3-03 | [^acd] until-3
)""", gbnf);
});
t.test("trigger rules make the lazy grammar scan for their starts", [](testing &t) {
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
auto ab = p.literal("ab");
auto cd = p.literal("cd");
auto scan = p.until(ab | cd);
return scan + (p.trigger_rule("one", ab, p.literal("x")) | p.trigger_rule("two", cd, p.literal("y")));
});
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
parser.build_grammar(builder, true);
});
assert_gbnf_equal(t, R"""(
one-rest ::= "x"
root ::= trigger
space ::= | " " | "\n"{1,2} [ \t]{0,20}
trigger ::= | [a] trigger-01 | [c] trigger-03 | [^ac] trigger
trigger-01 ::= | [b] one-rest | [a] trigger-01 | [c] trigger-03 | [^abc] trigger
trigger-03 ::= | [d] two-rest | [a] trigger-01 | [c] trigger-03 | [^acd] trigger
two-rest ::= "y"
)""", gbnf);
});
t.test("scanning grammar lets other named tokens through", [](testing &t) {
common_peg_tokens tokens;
tokens.ids.emplace("<a>", 1);
tokens.ids.emplace("<b>", 2);
tokens.tokens.emplace(1, common_peg_token{ "<a>", LLAMA_TOKEN_ATTR_CONTROL });
tokens.tokens.emplace(2, common_peg_token{ "<b>", LLAMA_TOKEN_ATTR_CONTROL });
common_peg_parser_builder p(tokens);
auto start = p.token("<a>") + p.literal("x");
p.set_root(p.until(start) + p.trigger_rule("tool", start, p.token("<b>")));
auto parser = p.build();
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
parser.build_grammar(builder, true);
});
assert_gbnf_equal(t, R"""(
root ::= trigger
space ::= | " " | "\n"{1,2} [ \t]{0,20}
tool-rest ::= <[2]>
trigger ::= | <[1]> trigger-01 | (. | <[2]>) trigger
trigger-01 ::= | [x] tool-rest | <[1]> trigger-01 | ([^x] | <[2]>) trigger
)""", gbnf);
});
t.test("token sets", [](testing &t) {
common_peg_tokens tokens;
tokens.ids.emplace("<a>", 1);
tokens.ids.emplace("<b>", 2);
tokens.tokens.emplace(1, common_peg_token{ "<a>", LLAMA_TOKEN_ATTR_CONTROL });
tokens.tokens.emplace(2, common_peg_token{ "<b>", LLAMA_TOKEN_ATTR_CONTROL });
common_peg_parser_builder p(tokens);
p.set_root(p.token({ "<a>", "<b>" }) + p.token({ "<a>", "<b>" }, true));
auto parser = p.build();
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
parser.build_grammar(builder);
});
assert_gbnf_equal(t, R"""(
root ::= (<[1]> | <[2]>) !<[1-2]>
space ::= | " " | "\n"{1,2} [ \t]{0,20}
)""", gbnf);
});
t.test("through grammar suffix delimiter offers both rests", [](testing &t) {
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
return p.through(p.eps(), { { p.literal("xab"), p.literal("1") }, { p.literal("ab"), p.literal("2") } });
});
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
parser.build_grammar(builder);
});
assert_gbnf_equal(t, R"""(
root ::= until-5
space ::= | " " | "\n"{1,2} [ \t]{0,20}
until-5 ::= [x] until-5-01 | [a] until-5-04 | [^ax] until-5
until-5-01 ::= [x] until-5-01 | [a] until-5-02 | [^ax] until-5
until-5-02 ::= [b] (until-5-rest-0 | until-5-rest-1) | [x] until-5-01 | [a] until-5-04 | [^abx] until-5
until-5-04 ::= [b] until-5-rest-1 | [x] until-5-01 | [a] until-5-04 | [^abx] until-5
until-5-rest-0 ::= "1"
until-5-rest-1 ::= "2"
)""", gbnf);
});
+2 -50
View File
@@ -2176,7 +2176,7 @@ struct test_unary : public test_case {
ggml_tensor * build_graph(ggml_context * ctx) override {
const bool grad_supported = op == GGML_UNARY_OP_ABS || op == GGML_UNARY_OP_SGN || op == GGML_UNARY_OP_NEG ||
op == GGML_UNARY_OP_STEP || op == GGML_UNARY_OP_RELU || op == GGML_UNARY_OP_SILU ||
op == GGML_UNARY_OP_EXPM1 || op == GGML_UNARY_OP_SOFTPLUS || op == GGML_UNARY_OP_SIGMOID;
op == GGML_UNARY_OP_EXPM1 || op == GGML_UNARY_OP_SOFTPLUS;
ggml_tensor * a;
if (v & 1) {
@@ -2218,12 +2218,6 @@ struct test_unary : public test_case {
max = 10.f;
}
// saturated sigmoid inputs give zero analytic and numeric gradients, which the grad check cannot compare
if (mode == MODE_GRAD && op == GGML_UNARY_OP_SIGMOID) {
min = -2.f;
max = 2.f;
}
for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != NULL; t = ggml_get_next_tensor(ctx, t)) {
// test extended range of values to check for NaNs in GELU
init_tensor_uniform(t, min, max);
@@ -2231,16 +2225,7 @@ struct test_unary : public test_case {
}
float grad_eps() override {
return op == GGML_UNARY_OP_SIGMOID ? 0.25f : 15.0f;
}
bool grad_precise() override {
return op == GGML_UNARY_OP_SIGMOID;
}
// f32 rounding of a large summed loss exceeds the tolerance, so only the small sigmoid case is grad checked
int64_t grad_nmax() override {
return op == GGML_UNARY_OP_SIGMOID ? 64 : test_case::grad_nmax();
return 15.0f;
}
std::vector<float> grad_expect() override {
@@ -3913,37 +3898,6 @@ struct test_silu_back : public test_case {
}
};
// GGML_OP_SIGMOID_BACK
struct test_sigmoid_back : public test_case {
const ggml_type type;
const std::array<int64_t, 4> ne;
std::string vars() override {
return VARS_TO_STR2(type, ne);
}
test_sigmoid_back(ggml_type type = GGML_TYPE_F32,
std::array<int64_t, 4> ne = {64, 5, 4, 3})
: type(type), ne(ne) {}
ggml_tensor * build_graph(ggml_context * ctx) override {
ggml_tensor * a = ggml_new_tensor(ctx, type, 4, ne.data());
ggml_set_name(a, "a");
ggml_tensor * grad = ggml_new_tensor(ctx, type, 4, ne.data());
ggml_set_name(grad, "grad");
ggml_tensor * out = ggml_sigmoid_back(ctx, a, grad);
ggml_set_name(out, "out");
return out;
}
bool grad_precise() override {
return true;
}
};
// GGML_OP_NORM
struct test_norm : public test_case {
const ggml_type type;
@@ -10827,7 +10781,6 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
}
}
}
test_cases.emplace_back(new test_unary(GGML_UNARY_OP_SIGMOID, GGML_TYPE_F32, { 16, 4, 1, 1 }));
// fused relu + sqr (squared ReLU)
for (ggml_type type : {GGML_TYPE_F16, GGML_TYPE_F32}) {
@@ -11808,7 +11761,6 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
test_cases.emplace_back(new test_scale(GGML_TYPE_F16, {10, 10, 10, 10}, 2.0f, 1.0f, false, 1)); // view test
test_cases.emplace_back(new test_softcap(GGML_TYPE_F32, {10, 10, 10, 10}, 50.0f));
test_cases.emplace_back(new test_silu_back());
test_cases.emplace_back(new test_sigmoid_back());
for (float eps : { 0.0f, 1e-6f, 1e-4f, 1e-1f, 10.f }) {
for (uint32_t n : { 64, 1025 }) {
-32
View File
@@ -170,25 +170,6 @@ static bool test_tensor_n(ggml_backend_buffer_type_t buft) {
}
}
{
uint8_t expected[24], out[24] = {};
memset(expected, 0x11, sizeof(expected));
ggml_backend_tensor_set(t_big, expected, 0, sizeof(expected));
// Verify memset of unaligned ranges, such as single F16 elements.
const size_t ranges[][2] = { { 2, 2 }, { 1, 1 }, { 5, 2 }, { 9, 13 }, { 22, 2 } };
for (const auto & range : ranges) {
const uint8_t value = (uint8_t) (0xA0 + range[0]);
memset(expected + range[0], value, range[1]);
ggml_backend_tensor_memset(t_big, value, range[0], range[1]);
}
ggml_backend_tensor_get(t_big, out, 0, sizeof(out));
if (memcmp(out, expected, sizeof(out)) != 0) {
return false;
}
}
{
std::vector<ggml_tensor *> batch;
ggml_backend_buffer_ptr scatter_buf = alloc_tensors(ctx.get(), buft, { 64 }, batch);
@@ -223,19 +204,6 @@ static bool test_tensor_n(ggml_backend_buffer_type_t buft) {
}
}
{
uint8_t expected[24], out[24] = {};
memset(expected, 0x5A, sizeof(expected));
// Verify that clear fills every byte with a nonzero value.
ggml_backend_buffer_clear(buf.get(), 0x5A);
ggml_backend_tensor_get(t_big, out, 0, sizeof(out));
if (memcmp(out, expected, sizeof(out)) != 0) {
return false;
}
}
return true;
}
+50 -251
View File
@@ -305,66 +305,6 @@ static std::string get_expected_description(const llama_grammar_rules & rules, c
return result;
}
// Every text token shares one ID, the parsers and grammars only tell special tokens apart
static constexpr llama_token FAKE_TEXT_TOKEN = 0;
struct fake_token {
llama_token id;
std::string piece;
};
// Splits text into the special tokens it spells, longest first, and one text token per UTF-8 character.
static std::vector<fake_token> fake_tokenize(const common_peg_tokens & specials, const std::string & text) {
std::vector<fake_token> result;
size_t pos = 0;
while (pos < text.size()) {
llama_token best = LLAMA_TOKEN_NULL;
size_t best_len = 0;
for (const auto & [piece, id] : specials.ids) {
if (piece.size() > best_len && text.compare(pos, piece.size(), piece) == 0) {
best = id;
best_len = piece.size();
}
}
if (best != LLAMA_TOKEN_NULL) {
result.push_back({ best, text.substr(pos, best_len) });
pos += best_len;
continue;
}
size_t len = 1;
while (pos + len < text.size() && (static_cast<unsigned char>(text[pos + len]) & 0xC0) == 0x80) {
len++;
}
result.push_back({ FAKE_TEXT_TOKEN, text.substr(pos, len) });
pos += len;
}
return result;
}
// Returns the tokens from byte offset pos. A token split by pos keeps the rest of its piece as text.
static std::vector<fake_token> fake_tokens_from(const std::vector<fake_token> & tokens, size_t pos) {
std::vector<fake_token> result;
size_t start = 0;
for (const auto & token : tokens) {
const size_t end = start + token.piece.size();
if (start >= pos) {
result.push_back(token);
} else if (end > pos) {
result.push_back({ FAKE_TEXT_TOKEN, token.piece.substr(pos - start) });
}
start = end;
}
return result;
}
static common_chat_input fake_input(const std::vector<fake_token> & tokens) {
common_chat_input input;
for (const auto & token : tokens) {
input.append(token.piece, token.id);
}
return input;
}
// Result of a detailed grammar match attempt
struct grammar_match_result {
bool success = false; // Did the string fully match the grammar?
@@ -378,49 +318,56 @@ struct grammar_match_result {
bool incomplete = false; // True if matched all input but grammar expects more
};
// Detailed version of match_string that returns failure information, accepting the input token by token
static grammar_match_result match_string_detailed(const std::vector<fake_token> & tokens, llama_grammar * grammar) {
std::string input;
for (const auto & token : tokens) {
input += token.piece;
}
// Detailed version of match_string that returns failure information
static grammar_match_result match_string_detailed(const std::string & input, llama_grammar * grammar) {
grammar_match_result result;
result.total_bytes = input.size();
result.total_codepoints = unicode_cpts_from_utf8(input).size();
result.total_bytes = input.size();
const auto cpts = unicode_cpts_from_utf8(input);
result.total_codepoints = cpts.size();
auto & stacks_cur = llama_grammar_get_stacks(grammar);
const auto & rules = llama_grammar_get_rules(grammar);
size_t byte_pos = 0;
size_t cpt_pos = 0;
for (const auto & token : tokens) {
for (size_t i = 0; i < cpts.size(); i++) {
const auto & cpt = cpts[i];
// Get expected before accepting (for error reporting)
std::string expected_before = get_expected_description(rules, stacks_cur);
try {
llama_grammar_accept_token(*grammar, token.id, token.piece);
} catch (const std::runtime_error & /*e*/) {
llama_grammar_accept(grammar, cpt);
// Calculate byte position for this codepoint
size_t cpt_bytes = 0;
if (cpt < 0x80) {
cpt_bytes = 1;
} else if (cpt < 0x800) {
cpt_bytes = 2;
} else if (cpt < 0x10000) {
cpt_bytes = 3;
} else {
cpt_bytes = 4;
}
if (stacks_cur.empty()) {
// Grammar failed to match at this point
result.matched_bytes = byte_pos;
result.matched_codepoints = cpt_pos;
result.matched_codepoints = i;
result.matched_prefix = input.substr(0, byte_pos);
result.failing_char = token.id == FAKE_TEXT_TOKEN
? format_codepoint(unicode_cpts_from_utf8(token.piece).at(0))
: token.piece + " (token " + std::to_string(token.id) + ")";
result.failing_char = format_codepoint(cpt);
result.expected_description = expected_before;
result.incomplete = false;
return result;
}
byte_pos += token.piece.size();
cpt_pos += unicode_cpts_from_utf8(token.piece).size();
byte_pos += cpt_bytes;
}
// All input matched - check if grammar is complete
result.matched_bytes = input.size();
result.matched_codepoints = result.total_codepoints;
result.matched_codepoints = cpts.size();
result.matched_prefix = input;
if (std::any_of(stacks_cur.begin(), stacks_cur.end(), [](const auto & stack) { return stack.empty(); })) {
@@ -1171,7 +1118,6 @@ struct peg_test_case {
common_chat_msg expect;
bool is_partial = false;
bool expect_reconstruction = false;
bool expect_grammar = false;
std::vector<std::string> expect_rules;
};
@@ -1182,7 +1128,6 @@ static std::string g_template_filter;
static bool g_force_reconstruction_test = false;
static void test_peg_parser(common_chat_templates * tmpls,
const common_peg_tokens & specials,
const std::function<void(peg_test_case &)> & init,
bool detailed_debug) {
// UTF-8-safe truncation helper (same as in test_parser_with_streaming)
@@ -1227,15 +1172,8 @@ static void test_peg_parser(common_chat_templates * tmpls,
tc.expect.role = "assistant";
}
const bool tokenized = !specials.ids.empty();
common_chat_session_params session_params;
session_params.debug = detailed_debug;
if (tokenized) {
session_params.tokenize = [&](const std::string & text) {
return fake_input(fake_tokenize(specials, text));
};
}
common_chat_session session(tmpls, nullptr, tc.params, session_params);
const auto & parser = session.parser();
common_params_sampling sampling;
@@ -1253,35 +1191,18 @@ static void test_peg_parser(common_chat_templates * tmpls,
}
}
if (tc.expect_grammar && session.grammar().empty()) {
throw std::runtime_error("expected a grammar constraining the tool calls, but none was produced");
}
common_chat_msg msg_accum;
common_chat_msg msg_prev;
msg_accum.role = msg_prev.role = "assistant";
const auto input_tokens = fake_tokenize(specials, tc.input);
// A token arrives whole, so a special token is never split across chunks
std::vector<common_chat_input> chunks;
if (tokenized) {
for (const auto & token : input_tokens) {
chunks.push_back(fake_input({ token }));
}
} else {
size_t fed = 0;
for (size_t i = 1; i <= tc.input.size(); ++i) {
// Use UTF-8 safe truncation to avoid corrupting multi-byte characters
size_t safe_len = utf8_truncate_safe_len(std::string_view(tc.input).substr(0, i));
chunks.emplace_back(tc.input.substr(fed, safe_len - fed));
fed = safe_len;
}
}
for (size_t i = 0; i < chunks.size(); ++i) {
auto is_partial = i + 1 < chunks.size() || tc.is_partial;
const common_chat_msg & msg_current = is_partial ? session.feed(chunks[i]) : session.finish(chunks[i]);
size_t fed = 0;
for (size_t i = 1; i <= tc.input.size(); ++i) {
auto is_partial = i < tc.input.size() || tc.is_partial;
// Use UTF-8 safe truncation to avoid corrupting multi-byte characters
size_t safe_len = utf8_truncate_safe_len(std::string_view(tc.input).substr(0, i));
common_chat_input chunk(tc.input.substr(fed, safe_len - fed));
fed = safe_len;
const common_chat_msg & msg_current = is_partial ? session.feed(chunk) : session.finish(chunk);
for (const auto & diff : common_chat_msg_diff::compute_diffs(msg_prev, msg_current)) {
if (!diff.reasoning_content_delta.empty()) {
@@ -1296,15 +1217,10 @@ static void test_peg_parser(common_chat_templates * tmpls,
while (msg_accum.tool_calls.size() <= diff.tool_call_index) {
msg_accum.tool_calls.push_back({ "", "", "" });
}
// The name and id are set once the delta carries them.
// A name that is already set must not change, clients concatenate name deltas.
// Always update name and id from diff (may change during incremental parsing), but only if the delta
// actually contains them
if (!diff.tool_call_delta.name.empty()) {
auto & name = msg_accum.tool_calls[diff.tool_call_index].name;
if (!name.empty() && name != diff.tool_call_delta.name) {
throw std::runtime_error("Tool call " + std::to_string(diff.tool_call_index) + " name changed from " +
name + " to " + diff.tool_call_delta.name + " during incremental parsing");
}
name = diff.tool_call_delta.name;
msg_accum.tool_calls[diff.tool_call_index].name = diff.tool_call_delta.name;
}
if (!diff.tool_call_delta.id.empty()) {
msg_accum.tool_calls[diff.tool_call_index].id = diff.tool_call_delta.id;
@@ -1492,10 +1408,9 @@ static void test_peg_parser(common_chat_templates * tmpls,
// sampler was already active before model output began — constrain from the
// start of the model output (i.e. tc.input).
std::string constrained = full_input;
size_t constrain_from = 0;
bool grammar_triggered = false;
if (earliest_trigger_pos != std::string::npos) {
constrain_from = std::max(earliest_trigger_pos, gen_prompt.size());
auto constrain_from = std::max(earliest_trigger_pos, gen_prompt.size());
constrained = full_input.substr(constrain_from);
grammar_triggered = true;
} else if (!sampling.grammar_lazy) {
@@ -1505,9 +1420,7 @@ static void test_peg_parser(common_chat_templates * tmpls,
// Test the constrained portion against the grammar
if (grammar_triggered && !tc.is_partial) {
auto full_tokens = fake_tokenize(specials, gen_prompt);
full_tokens.insert(full_tokens.end(), input_tokens.begin(), input_tokens.end());
auto result = match_string_detailed(fake_tokens_from(full_tokens, constrain_from), grammar.get());
auto result = match_string_detailed(constrained, grammar.get());
if (!result.success) {
std::string error_msg;
if (result.incomplete) {
@@ -1596,7 +1509,6 @@ class peg_tester {
common_chat_templates_ptr tmpls_;
std::string template_path_;
bool detailed_debug_;
common_peg_tokens specials_;
friend class peg_test_builder;
public:
@@ -1612,18 +1524,6 @@ class peg_tester {
const std::string & template_path() const { return template_path_; }
// Run the tests tokenized with these pieces as special tokens, as if the vocab defined them
peg_tester & specials(const std::vector<std::string> & pieces) {
llama_token id = FAKE_TEXT_TOKEN + 1;
for (const auto & piece : pieces) {
specials_.ids.emplace(piece, id);
specials_.tokens.emplace(id, common_peg_token{ piece, LLAMA_TOKEN_ATTR_CONTROL });
id++;
}
common_chat_templates_set_specials(tmpls_.get(), specials_);
return *this;
}
peg_test_builder test(const std::string & input);
};
@@ -1714,11 +1614,6 @@ class peg_test_builder {
return *this;
}
peg_test_builder & expect_grammar() {
tc_.expect_grammar = true;
return *this;
}
peg_test_builder & tool_choice(common_chat_tool_choice choice) {
tc_.params.tool_choice = choice;
return *this;
@@ -1744,10 +1639,8 @@ class peg_test_builder {
return;
}
}
LOG_INF("\n\x1b[38;5;126m[%s%s]\x1b[0m\n%s\n\n", tester_.template_path().c_str(),
tester_.specials_.ids.empty() ? "" : ", tokenized", tc_.input.c_str());
test_peg_parser(tester_.tmpls_.get(), tester_.specials_, [this](peg_test_case & t) { t = tc_; },
tester_.detailed_debug_);
LOG_INF("\n\x1b[38;5;126m[%s]\x1b[0m\n%s\n\n", tester_.template_path().c_str(), tc_.input.c_str());
test_peg_parser(tester_.tmpls_.get(), [this](peg_test_case & t) { t = tc_; }, tester_.detailed_debug_);
}
};
@@ -2366,17 +2259,9 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
"const": "42"
})";
// Special tokens of the templates that use the Qwen3-Coder format.
// Check that the vocab of a model defines them before reuse. A model can share the syntax without the tokens.
const std::vector<std::string> qwen3_coder_specials = {
"<|im_start|>", "<|im_end|>", "<think>", "</think>",
"<tool_call>", "</tool_call>", "<tool_response>", "</tool_response>",
};
{
// Qwen3.5 (basically same as Nemotron, but keeping separate tests just in case)
auto tst = peg_tester("models/templates/Qwen3.5-4B.jinja", detailed_debug);
tst.specials(qwen3_coder_specials);
tst.test("I'm\nthinking\n</think>\n\nHello, world!\nWhat's up?")
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
@@ -2948,7 +2833,6 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
{
// NVIDIA Nemotron-3 Nano
auto tst = peg_tester("models/templates/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.jinja", detailed_debug);
tst.specials(qwen3_coder_specials);
tst.test("Hello, world!\nWhat's up?").
enable_thinking(false).
@@ -3244,8 +3128,6 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
{
// Google Gemma 4 (tool calling with Gemma4 dict format)
auto tst = peg_tester("models/templates/google-gemma-4-31B-it.jinja");
tst.specials({ "<|turn>", "<turn|>", "<|channel>", "<channel|>", "<|tool_call>", "<tool_call|>",
"<|tool_response>", "<tool_response|>", "<|tool>", "<tool|>", "<|\"|>", "<|think|>" });
tst.test("Hello, world!").expect(simple_assist_msg("Hello, world!")).run();
@@ -3874,7 +3756,6 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
{
// Qwen3-Coder (tool calling with XML-style format)
auto tst = peg_tester("models/templates/Qwen3-Coder.jinja", detailed_debug);
tst.specials(qwen3_coder_specials);
tst.test("Hello, world!\nWhat's up?").expect(message_assist).expect_reconstruction().run();
@@ -3903,20 +3784,6 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
.expect(message_assist_call)
.run();
// Required tool choice constrains from the first token instead of gating on a trigger
tst.test(
"<tool_call>\n"
"<function=special_function>\n"
"<parameter=arg1>\n"
"1\n"
"</parameter>\n"
"</function>\n"
"</tool_call>")
.tools({ special_function_tool })
.tool_choice(COMMON_CHAT_TOOL_CHOICE_REQUIRED)
.expect(message_assist_call)
.run();
tst.test(
"Let me call it.\n"
"<function=special_function>\n"
@@ -6018,7 +5885,6 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
{
auto tst = peg_tester("models/templates/LFM3.jinja", detailed_debug);
tst.specials(qwen3_coder_specials);
tst.test("Hello, world!\nWhat's up?")
.enable_thinking(false)
@@ -7242,19 +7108,6 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
auto tst = peg_tester("models/templates/meta-llama-Llama-3.2-3B-Instruct.jinja", detailed_debug);
tst.test("Hello, world!\nWhat's up?").tools({ special_function_tool }).expect(message_assist).expect_reconstruction().expect_rules({ "tool-0" }).run();
// bare JSON calls have no start marker, auto mode must still produce a lazy grammar
tst.test("{\"name\": \"special_function\", \"parameters\": {\"arg1\": 1}}")
.tools({ special_function_tool })
.expect(message_assist_call)
.expect_grammar()
.run();
// same without spaces
tst.test("{\"name\":\"special_function\",\"parameters\":{\"arg1\":1}}")
.tools({ special_function_tool })
.expect(message_assist_call)
.expect_grammar()
.run();
// Continuation tests
tst.test("world!\nWhat's up?")
.messages({ message_user, message_assist_prefill_content })
@@ -7281,7 +7134,6 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
// Muse Glimmer format tests
{
auto tst = peg_tester("models/templates/muse-glimmer.jinja", detailed_debug);
tst.specials({ "<|start|>", "<|message|>", "<|eom|>" });
const std::string call_markup =
"<atem:function_calls>\n"
@@ -7291,7 +7143,7 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
"</atem:function_calls>";
// A plain answer is unaffected
tst.test(" to=user<|message|>Hello, world!\nWhat's up?")
tst.test(" to=user<|message|>Hello, world!\nWhat's up?<|eot|>")
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
.expect(message_assist)
.run();
@@ -7302,34 +7154,7 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
.tools({ special_function_tool })
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
.expect(message_assist_call)
.expect_rules({ "tool-0", "tool-calls" })
.run();
// Parallel calls, one message per call
tst.test(" to=special_function<|message|>" + call_markup + "<|eom|>"
"<|start|>assistant to=special_function_with_opt<|message|>"
"<atem:function_calls>\n"
"<atem:invoke name=\"special_function_with_opt\">\n"
"<atem:parameter name=\"arg1\">1</atem:parameter>\n"
"<atem:parameter name=\"arg2\">2</atem:parameter>\n"
"</atem:invoke>\n"
"</atem:function_calls>")
.tools({ special_function_tool, special_function_tool_with_optional_param })
.parallel_tool_calls(true)
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
.expect_tool_calls({
{ "special_function", R"({"arg1": 1})", {} },
{ "special_function_with_opt", R"({"arg1": 1, "arg2": 2})", {} },
})
.run();
// Required tool choice after a reasoning message
tst.test(" to=self<|message|>I'm\nthinking<|eom|>"
"<|start|>assistant to=special_function<|message|>" + call_markup)
.tools({ special_function_tool })
.tool_choice(COMMON_CHAT_TOOL_CHOICE_REQUIRED)
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
.expect(message_assist_call_thoughts)
.expect_rules({ "tool-0" })
.run();
// "Inform then act": the model answers the user and calls a tool in ONE generation,
@@ -7345,7 +7170,7 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
// Markup quoted in an answer has no preceding <|eom|>, so it stays content instead of
// becoming an invocation the user never asked for
tst.test(" to=user<|message|>You invoke it like this:\n" + call_markup)
tst.test(" to=user<|message|>You invoke it like this:\n" + call_markup + "<|eot|>")
.tools({ special_function_tool })
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
.expect_content("You invoke it like this:\n" + call_markup)
@@ -7369,7 +7194,7 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
// Tool markup inside the analysis channel is reasoning, not a call
tst.test(" to=self<|message|>I could use " + call_markup + " here<|eom|>"
"<|start|>assistant to=user<|message|>Hello!")
"<|start|>assistant to=user<|message|>Hello!<|eot|>")
.tools({ special_function_tool })
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
.expect_reasoning("I could use " + call_markup + " here")
@@ -7440,7 +7265,6 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
// GPT-OSS format tests
{
auto tst = peg_tester("models/templates/openai-gpt-oss-120b.jinja", detailed_debug);
tst.specials({ "<|start|>", "<|end|>", "<|message|>", "<|channel|>", "<|constrain|>", "<|return|>", "<|call|>" });
// Basic content only - final channel
tst.test("<|channel|>final<|message|>Hello, world!\nWhat's up?").expect(message_assist).run();
@@ -7480,30 +7304,7 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
.tools({ special_function_tool })
.expect(message_assist_call)
.expect_rules({ "tool-0-in-role", "tool-0-in-channel" })
.run();
// A tool whose name prefixes another's must not be reported while the header is still open
tst.test(" to=functions.special_function_with_opt<|channel|>analysis<|message|>{\"arg1\": 1}")
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
.tools({ special_function_tool, special_function_tool_with_optional_param })
.expect(simple_assist_msg("", "", "special_function_with_opt", "{\"arg1\": 1}"))
.run();
tst.test("<|channel|>commentary to=functions.special_function_with_opt<|message|>{\"arg1\": 1}")
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
.tools({ special_function_tool, special_function_tool_with_optional_param })
.expect(simple_assist_msg("", "", "special_function_with_opt", "{\"arg1\": 1}"))
.run();
// Required tool choice constrains from the first token instead of gating on a trigger
tst.test(
"<|channel|>analysis<|message|>I'm\nthinking<|end|>"
"<|start|>assistant to=functions.special_function<|channel|>analysis<|message|>{\"arg1\": 1}")
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
.tools({ special_function_tool })
.tool_choice(COMMON_CHAT_TOOL_CHOICE_REQUIRED)
.expect(message_assist_call_thoughts)
.expect_rules({ "tool-0" })
.run();
// Tool call with recipient in channel header: "<|channel|>analysis to=functions.NAME<|message|>JSON"
@@ -7720,7 +7521,6 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
{
auto tst = peg_tester("models/templates/StepFun3.5-Flash.jinja", detailed_debug);
tst.specials(qwen3_coder_specials);
tst.test("I was thinking\n</think>\nNow I'm not.").
enable_thinking(true).
@@ -8920,7 +8720,7 @@ static void test_reasoning_effort_caps() {
assert_equals(json(expected).dump(), common_chat_templates_get_caps(tmpls.get()).at("reasoning_effort_levels").dump());
};
const std::vector<std::string> all_levels = { "none", "minimal", "low", "medium", "high", "xhigh", "max" };
const std::vector<std::string> all_levels = { "minimal", "low", "medium", "high", "xhigh", "max" };
assert_effort_levels("models/templates/deepseek-ai-DeepSeek-V4.jinja", { "max" });
assert_effort_levels("models/templates/deepseek-ai-DeepSeek-V4-Flash-0731.jinja", { "high", "max" });
@@ -8930,8 +8730,7 @@ static void test_reasoning_effort_caps() {
assert_effort_levels("models/templates/upstage-Solar-Open-100B.jinja", { "low" });
assert_effort_levels("models/templates/muse-glimmer.jinja", all_levels);
assert_effort_levels("models/templates/openai-gpt-oss-120b.jinja", all_levels);
assert_effort_levels("models/templates/Cohere2MoE.jinja", { "none" });
assert_effort_levels("models/templates/thinkingmachines-Inkling-2-Small.jinja", { "none", "minimal", "low", "medium", "high", "max" });
assert_effort_levels("models/templates/Cohere2MoE.jinja", {});
assert_effort_levels("models/templates/Qwen-Qwen3-0.6B.jinja", {});
auto assert_effort_rejected = [](const std::string & path, const std::vector<std::string> & expected) {
+12 -177
View File
@@ -41,21 +41,15 @@ struct token_and_piece {
std::string piece;
};
// token() encodes a token and its piece as a 0xff marker, the ID in big-endian order, the piece length, and the piece.
static std::string token(llama_token id, const std::string & piece) {
// token() encodes a 32-bit ID as 5 bytes: a 0xff marker followed by the ID in big-endian order.
static std::string token(llama_token id) {
return std::string{
static_cast<char>(0xff),
static_cast<char>((id >> 24) & 0xff),
static_cast<char>((id >> 16) & 0xff),
static_cast<char>((id >> 8) & 0xff),
static_cast<char>(id & 0xff),
static_cast<char>(piece.size())
} + piece;
}
// token() without a piece encodes a token whose piece is its <[id]> spelling.
static std::string token(llama_token id) {
return token(id, "<[" + std::to_string(id) + "]>");
static_cast<char>(id & 0xff)
};
}
// parse_tokens() parses the token encodes above and UTF-8 text.
@@ -66,7 +60,7 @@ static std::vector<token_and_piece> parse_tokens(const std::string & input) {
while (offset < input.size()) {
try {
if (static_cast<unsigned char>(input[offset]) == 0xff) {
if (offset + 6 > input.size()) {
if (offset + 5 > input.size()) {
throw std::runtime_error("not enough bytes for token id");
}
uint32_t val =
@@ -74,12 +68,9 @@ static std::vector<token_and_piece> parse_tokens(const std::string & input) {
(static_cast<unsigned char>(input[offset + 2]) << 16) |
(static_cast<unsigned char>(input[offset + 3]) << 8) |
(static_cast<unsigned char>(input[offset + 4]));
size_t len = static_cast<unsigned char>(input[offset + 5]);
if (offset + 6 + len > input.size()) {
throw std::runtime_error("not enough bytes for token piece");
}
result.push_back({static_cast<llama_token>(val), input.substr(offset + 6, len)});
offset += 6 + len;
auto piece = "<[" + std::to_string(val) + "]>";
result.push_back({static_cast<llama_token>(val), piece});
offset += 5;
} else {
uint32_t cpt = unicode_cpt_from_utf8(input, offset);
result.push_back({0, unicode_cpt_to_utf8(cpt)});
@@ -93,42 +84,23 @@ static std::vector<token_and_piece> parse_tokens(const std::string & input) {
return result;
}
// Returns true if the sampler mask allows the token. The mask rejects a token only when every stack rejects it.
static bool mask_allows(llama_grammar * grammar, const token_and_piece & in) {
const auto decoded = llama_grammar_decode_utf8(in.piece, grammar->partial_utf8);
llama_grammar_candidates rejects = { { 0, decoded.first.data(), decoded.second, in.token, llama_grammar_is_opaque(*grammar, in.token) } };
for (const auto & stack : llama_grammar_get_stacks(grammar)) {
rejects = llama_grammar_reject_candidates_for_stack(llama_grammar_get_rules(grammar), stack, rejects);
}
return rejects.empty();
}
static bool match_string(const std::string & input, llama_grammar * grammar) {
const auto parsed = parse_tokens(input);
auto & stacks_cur = llama_grammar_get_stacks(grammar);
for (const auto & in : parsed) {
const bool allowed = mask_allows(grammar, in);
// the mask decides what can be sampled, so it has to agree with what accepting the token does
try {
llama_grammar_accept_token(*grammar, in.token, in.piece);
} catch (const std::runtime_error & /*e*/) {
if (allowed) {
fprintf(stderr, "❌ (token %d \"%s\" is allowed by the mask but not accepted)\n", in.token, in.piece.c_str());
}
assert(!allowed);
// no stacks means that the grammar failed to match at this point
// normally this shouldn't get hit because of llama_grammar_apply
return false;
}
if (!allowed) {
fprintf(stderr, "❌ (token %d \"%s\" is masked but accepted)\n", in.token, in.piece.c_str());
if (stacks_cur.empty()) {
// no stacks means that the grammar failed to match at this point
return false;
}
assert(allowed);
}
for (const auto & stack : stacks_cur) {
@@ -149,7 +121,6 @@ static void test(const std::string & test_desc, const std::string & grammar_str,
// Save the original grammar stacks so that we can reset after every new string we want to test
const llama_grammar_stacks stacks_org = llama_grammar_get_stacks(grammar); // copy
const llama_partial_utf8 partial_org = grammar->partial_utf8;
llama_grammar_stacks & stacks_cur = llama_grammar_get_stacks(grammar);
@@ -189,7 +160,6 @@ static void test(const std::string & test_desc, const std::string & grammar_str,
// Reset the grammar stacks
stacks_cur = stacks_org;
grammar->partial_utf8 = partial_org;
}
fprintf(stderr, " 🟠 Invalid strings:\n");
@@ -210,7 +180,6 @@ static void test(const std::string & test_desc, const std::string & grammar_str,
// Reset the grammar stacks
stacks_cur = stacks_org;
grammar->partial_utf8 = partial_org;
}
// Clean up allocated memory
@@ -553,137 +522,6 @@ static void test_simple_grammar() {
);
}
static void test_token_rule_utf8() {
// A byte-level token can hold part of a multibyte character, a token rule still takes it as a whole token
test_grammar(
"byte-split characters under token rules",
R"""(
root ::= <[10]> content <[11]>
content ::= (!<[11]>)*)""",
// Passing strings
{
token(10) + token(20, "\xF0") + token(21, "\x9F\xA6\x99") + token(11),
token(10) + token(22, "\xE2") + token(23, "\x98\x80") + token(11),
token(10) + token(24, "a\xF0") + token(21, "\x9F\xA6\x99") + token(11),
},
// Failing strings
{
token(10) + token(20, "\xF0"),
token(10) + token(20, "\xF0") + token(11),
}
);
// A token rule can only start where a token starts, not after char rules consumed part of it
test_grammar(
"token rule after a char rule",
R"""(
root ::= "a" <[11]>)""",
// Passing strings
{
"a" + token(11),
token(12, "a") + token(11),
},
// Failing strings
{
token(11, "ab"),
token(12, "ab") + token(11),
}
);
// A token partly matched by a char rule stays partly matched for every alternative, whichever is checked first
test_grammar(
"token rule alternative after a char rule",
R"""(
root ::= "a" (!<[7]> | "bx"))""",
// Passing strings
{
"abx",
token(12, "abx"),
"a" + token(5),
},
// Failing strings
{
token(5, "aby"),
token(5, "ab"),
}
);
}
static void test_token_sets() {
test_grammar(
"token set with alternatives and ranges",
R"""(
root ::= <[1,5-9,12]>+)""",
// Passing strings
{
token(1),
token(5) + token(9),
token(7) + token(12) + token(1),
},
// Failing strings
{
token(2),
token(4),
token(10),
token(13),
token(5) + token(11),
}
);
test_grammar(
"negated token set",
R"""(
root ::= <[10]> (!<[11,20-29]>)* <[11]>)""",
// Passing strings
{
token(10) + token(11),
token(10) + "text" + token(12) + token(19) + token(30) + token(11),
},
// Failing strings
{
token(10) + token(20) + token(11),
token(10) + token(29) + token(11),
token(10) + token(11) + token(11),
}
);
}
static void test_opaque_tokens() {
// Char rules do not match a token that a token rule names, other tokens still match as text
test_grammar(
"tokens named by token rules are opaque to char rules",
R"""(
root ::= [^x]* <[11]> "done")""",
// Passing strings
{
"abc" + token(11) + "done",
token(11) + "done",
token(12) + token(11) + "done",
},
// Failing strings
{
"abc" + token(11) + "more" + token(11) + "done",
token(11) + token(11) + "done",
}
);
test_grammar(
"tokens named by a negated set are opaque to char rules",
R"""(
root ::= [^x]* !<[11,20-29]>)""",
// Passing strings
{
"abc" + token(30),
token(12) + token(30),
},
// Failing strings
{
token(25) + token(30),
token(11) + token(30),
}
);
}
static void test_complex_grammar() {
// Test case for a more complex grammar, with both failure strings and success strings
test_grammar(
@@ -1701,9 +1539,6 @@ int main() {
fprintf(stdout, "Running grammar integration tests...\n");
test_simple_grammar();
test_complex_grammar();
test_token_rule_utf8();
test_token_sets();
test_opaque_tokens();
test_special_chars();
test_quantifiers();
test_failure_missing_root();
-24
View File
@@ -179,14 +179,6 @@ int main()
verify_failure(deep_grammar.c_str());
}
verify_failure(R"""(
root ::= <[9-5]>
)""");
verify_failure(R"""(
root ::= <[4294967296]>
)""");
verify_parsing(R"""(
root ::= "a"{0,5000}
)""", {
@@ -602,21 +594,5 @@ int main()
{LLAMA_GRETYPE_END, 0},
});
verify_parsing(R"""(
root ::= <[1,5-9,12]> !<[11,20-29]>
)""", {
{"root", 0}
}, {
// root (index 0)
{LLAMA_GRETYPE_TOKEN, 1},
{LLAMA_GRETYPE_TOKEN_ALT, 5},
{LLAMA_GRETYPE_TOKEN_RNG_UPPER, 9},
{LLAMA_GRETYPE_TOKEN_ALT, 12},
{LLAMA_GRETYPE_TOKEN_NOT, 11},
{LLAMA_GRETYPE_TOKEN_ALT, 20},
{LLAMA_GRETYPE_TOKEN_RNG_UPPER, 29},
{LLAMA_GRETYPE_END, 0},
});
return 0;
}
+1 -1
View File
@@ -202,7 +202,7 @@ int main()
uint32_t *cp = new uint32_t[2]; // dynamically allocate memory for code_point
cp[0] = 37 + i;
cp[1] = 0;
next_candidates[i] = {i, cp, {}, 0, false};
next_candidates[i] = {i, cp, {}, 0};
}
std::vector<std::vector<std::pair<uint32_t, uint16_t>>> expected_reject = {
-2
View File
@@ -2435,8 +2435,6 @@ Response:
### GET `/models/sse`: Real-time events
This endpoint is also available in single-model mode (`-m`, `-hf`, ...), where it is served before the model is loaded, to follow the download and loading progress. As `GET /models` is unavailable until the model is loaded, each new client first receives the last event sent. If the download or the loading fails, a `status_change` event with status `unloaded` is sent before the server exits.
Example events:
```js
+2 -24
View File
@@ -1,4 +1,3 @@
#include "arg.h"
#include "common.h"
#include "download.h"
#include "log.h"
@@ -1642,18 +1641,13 @@ json oaicompat_chat_params_parse(
// Parse the OAI "reasoning_effort" field; "none" disables reasoning.
if (body.contains("reasoning_effort")) {
auto reasoning_effort = json_value(body, "reasoning_effort", std::string(""));
const auto levels = caps.at("reasoning_effort_levels").get<std::vector<std::string>>();
if (reasoning_effort == "none") {
inputs.enable_thinking = false;
// Some models use reasoning_effort = none and some throw an exception, so check the model's list of effort levels first
if (std::find(levels.begin(), levels.end(), reasoning_effort) != levels.end()) {
inputs.chat_template_kwargs["reasoning_effort"] = json(reasoning_effort).dump();
} else {
inputs.chat_template_kwargs.erase("reasoning_effort");
}
inputs.chat_template_kwargs.erase("reasoning_effort");
} else if (!reasoning_effort.empty()) {
const auto rejected = caps.at("reasoning_effort_rejected").get<std::vector<std::string>>();
if (std::find(rejected.begin(), rejected.end(), reasoning_effort) != rejected.end()) {
const auto levels = caps.at("reasoning_effort_levels").get<std::vector<std::string>>();
throw std::invalid_argument("reasoning_effort \"" + reasoning_effort + "\" is not supported by the chat template, supported levels: " + string_join(levels, ", "));
}
inputs.chat_template_kwargs["reasoning_effort"] = json(reasoning_effort).dump();
@@ -2387,22 +2381,6 @@ server_role::~server_role() {
}
}
std::string server_get_model_name(const common_params & params) {
if (!params.model_alias.empty()) {
// backward compat: use first alias as model name
return *params.model_alias.begin();
}
if (!params.model.get_name().empty()) {
return params.model.get_name();
}
if (!params.model.url.empty()) {
// -mu before the download: the path the model will be stored at
return common_get_default_local_path(params.model.url);
}
// fallback: derive model name from file name
return std::filesystem::path(params.model.path).filename().string();
}
bool server_role::is_managed_by_router() {
return std::getenv(SERVER_ENV_ROUTER_PORT) != nullptr;
}
-3
View File
@@ -837,9 +837,6 @@ static inline server_state server_state_from_str(const std::string & str) {
GGML_ASSERT(false && "invalid server_state string");
}
// the model name reported by the server: the first alias, else the model's repo or path
std::string server_get_model_name(const common_params & params);
// the role of this process and, when it is a child of a router, its side of the protocol
struct server_role {
// serializes the notify_to_router writes
+13 -27
View File
@@ -2170,7 +2170,16 @@ private:
SRV_TRC("%s", "context checkpoints disabled\n");
}
model_name = server_get_model_name(params_base);
if (!params_base.model_alias.empty()) {
// backward compat: use first alias as model name
model_name = *params_base.model_alias.begin();
} else if (!params_base.model.get_name().empty()) {
model_name = params_base.model.get_name();
} else {
// fallback: derive model name from file name
auto model_path = std::filesystem::path(params_base.model.path);
model_name = model_path.filename().string();
}
model_aliases = params_base.model_alias;
model_tags = params_base.model_tags;
@@ -4813,9 +4822,7 @@ private:
if (!slot.can_split()) {
// cannot fit the prompt in the current batch - will try next iter
// with non-causal attention the whole batch must fit in one ubatch, see llama_decode
const int32_t n_batch_max = llama_get_causal_attn(ctx_tgt) ? n_batch : std::min(n_batch, n_ubatch);
if (batch.size() + slot.task->n_tokens() > n_batch_max) {
if (batch.size() + slot.task->n_tokens() > n_batch) {
return;
}
}
@@ -4948,23 +4955,6 @@ private:
const bool merge_user_starts = userstart_merge && !has_mtmd &&
slot.task->n_tokens() - slot.prompt.n_tokens() <= n_ubatch;
// [fork F14] merged or not, keep the break at the first user message, the end of
// the system prompt: the checkpoint there is what a new chat with the same system
// prompt restores. The batch only reaches it when the prompt is processed from
// before it (a fresh prompt; checkpoints past the restore point are erased), so
// follow-up turns are not affected. Below 8 tokens there is no system prompt
// worth a checkpoint (only a BOS before the first user message).
int32_t first_user_pos = -1;
for (const auto & s : spans.spans) {
if (s.role == COMMON_CHAT_ROLE_USER) {
first_user_pos = (int32_t) s.pos;
break;
}
}
if (first_user_pos < 8) {
first_user_pos = -1;
}
// add prompt tokens for processing in the current batch
while (slot.prompt.n_tokens() < slot.task->n_tokens() && batch.size() < n_batch) {
// get next token to process
@@ -5008,15 +4998,11 @@ private:
slot.prompt.tokens.push_back(cur_tok);
// break at the last user message, or at user messages at least min step past the last checkpoint
if (do_checkpoint && spans.is_user_start(slot.prompt.n_tokens())) {
if (do_checkpoint && !merge_user_starts && spans.is_user_start(slot.prompt.n_tokens())) {
const auto pos = slot.prompt.n_tokens();
const auto & checkpoints = slot.prompt.checkpoints;
if (merge_user_starts) {
if (pos == first_user_pos) {
break;
}
} else if (pos == last_user_pos || checkpoints.empty() || pos > checkpoints.back()->n_tokens + params_base.checkpoint_min_step) {
if (pos == last_user_pos || checkpoints.empty() || pos > checkpoints.back()->n_tokens + params_base.checkpoint_min_step) {
break;
}
}
-3
View File
@@ -396,9 +396,6 @@ bool server_http_context::init_listener(const common_params & params) {
if (frontend_paths.count(req.path)) {
return true; // frontend asset, allow it to load and show "loading"
}
if (req.path == "/models/sse") {
return true; // reports the download and loading progress
}
// no endpoints are allowed to be accessed when the server is not ready
// this is to prevent any data races or inconsistent states
res.status = 503;
+28 -60
View File
@@ -734,7 +734,16 @@ bool server_models::remove_sidecar(const std::string & name, const std::string &
}
void server_models::notify_sse(const std::string & event, const std::string & model_id, const json & data) {
sse.notify(event, model_id, data);
std::unique_ptr<server_task_result_router> result = std::make_unique<server_task_result_router>();
result->data = {
{"model", model_id},
{"event", event},
};
if (!data.is_null()) {
result->data["data"] = data;
}
SRV_DBG("notifying SSE clients about event '%s' for model '%s': %s\n", event.c_str(), model_id.c_str(), safe_json_to_str(result->data).c_str());
sse.broadcast(std::move(result));
}
void server_models::load_models() {
@@ -1922,7 +1931,7 @@ void server_models::handle_child_state(const std::string & name, const std::stri
}
//
// server_models_sse
// server_models_routes
//
// RAII wrapper similar to server_response_reader, but doesn't use server_queue
@@ -1961,61 +1970,6 @@ struct server_models_sse_client {
}
};
void server_models_sse::notify(const std::string & event, const std::string & model_id, const json & data) {
std::unique_ptr<server_task_result_router> result = std::make_unique<server_task_result_router>();
result->data = {
{"model", model_id},
{"event", event},
};
if (!data.is_null()) {
result->data["data"] = data;
}
SRV_DBG("notifying SSE clients about event '%s' for model '%s': %s\n", event.c_str(), model_id.c_str(), safe_json_to_str(result->data).c_str());
// under the lock, so that a client connecting meanwhile gets this event either replayed or queued, not both
std::lock_guard<std::mutex> lk(mutex);
if (replay_last) {
last_event = result->data;
}
queue.broadcast(std::move(result));
}
server_http_context::handler_t server_models_sse::make_handler(std::function<bool()> should_stop) {
return [this, should_stop = std::move(should_stop)](const server_http_req & req) {
auto res = std::make_unique<server_http_res>();
res->status = 200;
res->content_type = "text/event-stream";
std::shared_ptr<server_models_sse_client> sse_client;
std::string pending; // the replayed event, sent first
{
std::lock_guard<std::mutex> lk(mutex);
sse_client = std::make_shared<server_models_sse_client>(queue);
if (replay_last && !last_event.is_null()) {
pending = "data: " + safe_json_to_str(last_event) + "\n\n";
}
}
res->next = [should_stop, sse_client, pending = std::move(pending), &req](std::string & output) mutable -> bool {
if (!pending.empty()) {
output = std::move(pending);
pending.clear();
return true;
}
auto result = sse_client->next([&]() {
return should_stop() || req.should_stop();
});
if (result == nullptr) {
return false; // client disconnected or should_stop
}
output = "data: " + safe_json_to_str(result->to_json()) + "\n\n";
return true; // listen for the next event
};
return res;
};
}
//
// server_models_routes
//
static void res_ok(std::unique_ptr<server_http_res> & res, const json & response_data) {
res->status = 200;
res->data = safe_json_to_str(response_data);
@@ -2358,9 +2312,23 @@ void server_models_routes::init_routes() {
return res;
};
this->get_router_models_sse = models.sse.make_handler([this]() {
return stopping.load(std::memory_order_relaxed);
});
this->get_router_models_sse = [this](const server_http_req & req) {
auto res = std::make_unique<server_http_res>();
res->status = 200;
res->content_type = "text/event-stream";
auto sse_client = std::make_shared<server_models_sse_client>(models.sse);
res->next = [this, sse_client, &req](std::string & output) -> bool {
auto result = sse_client->next([&]() {
return stopping.load(std::memory_order_relaxed) || req.should_stop();
});
if (result == nullptr) {
return false; // client disconnected or should_stop
}
output = "data: " + safe_json_to_str(result->to_json()) + "\n\n";
return true; // listen for the next event
};
return res;
};
this->post_router_models = [this](const server_http_req & req) {
auto res = std::make_unique<server_http_res>();
+1 -20
View File
@@ -157,25 +157,6 @@ private:
void delete_partials(const std::vector<common_download_partial> & parts);
};
// broadcasts model events to the clients of GET /models/sse, used by both router and single-model mode
struct server_models_sse {
// send the last event to each new client, for single-model mode where GET /models is unavailable while loading
const bool replay_last;
server_models_sse(bool replay_last = false) : replay_last(replay_last) {}
// thread-safe
void notify(const std::string & event, const std::string & model_id, const json & data = nullptr);
// handler for GET /models/sse, a client is disconnected once should_stop() returns true
server_http_context::handler_t make_handler(std::function<bool()> should_stop);
private:
std::mutex mutex;
json last_event;
server_response queue;
};
struct server_models_routes;
struct server_monitor; // defined in server-models.cpp
@@ -308,7 +289,7 @@ public:
server_models(const common_params & params, int argc, char ** argv);
~server_models();
server_models_sse sse; // for real-time updates via SSE endpoint
server_response sse; // for real-time updates via SSE endpoint
// (re-)load the list of models from various sources and prepare the metadata mapping
// - if this is called the first time, simply populate the metadata
+8 -21
View File
@@ -1121,7 +1121,6 @@ struct server_tool_grep_search : server_tool {
{"literal", {{"type", "boolean"}, {"description", "Treat pattern as a literal string instead of a regular expression (default: false)"}}},
{"ignore_case", {{"type", "boolean"}, {"description", "Case-insensitive search (default: false)"}}},
{"context_lines", {{"type", "integer"}, {"description", "Number of lines of context to show before and after each match (default: 0)"}}},
{"offset", {{"type", "integer"}, {"description", "Number of matching lines to skip (default: 0)"}}},
}},
{"required", json::array({"path", "pattern"})},
}},
@@ -1138,7 +1137,6 @@ struct server_tool_grep_search : server_tool {
bool literal = json_value(params, "literal", false);
bool ignore_case = json_value(params, "ignore_case", false);
int ctx_lines = std::max(0, json_value(params, "context_lines", 0));
uint64_t offset = std::max<int64_t>(0, json_value(params, "offset", int64_t{0}));
std::string pattern_src = pat_str;
if (literal) {
@@ -1166,7 +1164,6 @@ struct server_tool_grep_search : server_tool {
// collect (absolute_path, display_path) pairs to search
std::vector<std::pair<std::string, std::string>> files;
bool listing_truncated = false;
const std::string abs_path = io->resolve(path);
if (io->is_regular_file(abs_path)) {
files.emplace_back(abs_path, path);
@@ -1175,7 +1172,6 @@ struct server_tool_grep_search : server_tool {
if (!listing.err.empty()) {
return {{"error", listing.err + ": " + path}};
}
listing_truncated = listing.truncated;
for (const auto & entry : listing.entries) {
if (!path_glob_match(include, entry.rel)) continue;
if (!exclude.empty() && path_glob_match(exclude, entry.rel)) continue;
@@ -1184,18 +1180,14 @@ struct server_tool_grep_search : server_tool {
} else {
return {{"error", "path does not exist: " + path}};
}
std::sort(files.begin(), files.end(), [](const auto & a, const auto & b) {
return a.second < b.second;
});
std::ostringstream output_text;
uint64_t skipped = 0;
size_t total = 0;
bool has_more = false;
bool limit_reached = false;
bool show_num = show_lineno || ctx_lines > 0;
for (const auto & file_entry : files) {
if (has_more) break;
if (limit_reached) break;
const std::string & fpath = file_entry.first;
const std::string & display_path = file_entry.second;
@@ -1209,15 +1201,11 @@ struct server_tool_grep_search : server_tool {
}
for (size_t i = 0; i < lines.size(); i++) {
if (!std::regex_search(lines[i], pattern)) continue;
if (skipped < offset) {
skipped++;
continue;
}
if (total >= SERVER_TOOL_GREP_SEARCH_MAX_RESULTS) {
has_more = true;
limit_reached = true;
break;
}
if (!std::regex_search(lines[i], pattern)) continue;
long ctx_start = ctx_lines > 0 ? std::max<long>(0, (long) i - ctx_lines) : (long) i;
long ctx_end = ctx_lines > 0 ? std::min<long>((long) lines.size() - 1, (long) i + ctx_lines) : (long) i;
@@ -1238,11 +1226,10 @@ struct server_tool_grep_search : server_tool {
}
output_text << "\n---\nTotal matches: " << total << "\n";
if (has_more) {
output_text << "Continue with offset=" << offset + total << " and the same search arguments.\n";
}
if (listing_truncated) {
output_text << "[results truncated: time budget or unreadable directory]\n";
if (limit_reached) {
output_text << string_format(
"[%zu matches limit reached. Narrow the path/pattern/include to see more.]\n",
SERVER_TOOL_GREP_SEARCH_MAX_RESULTS);
}
return {{"plain_text_response", output_text.str()}};
+65 -125
View File
@@ -160,7 +160,6 @@ static server_role_type server_role_detect(const common_params & params, server_
}
// a server given no model is a router
const bool has_model = !params.model.path.empty()
|| !params.model.url.empty()
|| !params.model.hf_repo.empty()
|| !params.model.docker_repo.empty();
return has_model ? SERVER_ROLE_INSTANCE : SERVER_ROLE_ROUTER;
@@ -415,45 +414,66 @@ static bool server_on_listening(server_http_context & ctx_http, const common_par
return true;
}
// throttled download progress, shared by the download worker and single-model mode
struct server_download_progress : public common_download_callback {
// called under the lock, the files are downloaded in parallel threads
std::function<void(const common_download_progress &)> on_progress;
std::function<bool()> should_stop;
// throttle each file on its own, a shared timer lets a small, fast file (e.g. a draft) starve the updates of the others
std::mutex mutex;
std::unordered_map<std::string, int64_t> last_progress_time;
//
// download worker: a child of the router that pulls a model into the cache and reports the progress
//
void notify(const common_download_progress & p) {
std::lock_guard<std::mutex> lk(mutex);
on_progress(p);
struct server_download_state : public common_download_callback {
server_role * role;
std::function<bool()> should_stop;
// files download in parallel threads: throttle each file on its own, a shared
// timer lets a small, fast file (e.g. a draft) starve the updates of the others
std::mutex progress_mutex;
std::unordered_map<std::string, int64_t> last_progress_time;
bool is_ok = false;
server_download_state(server_role * r) : role(r) {}
bool run(common_params & params) {
try {
common_models_handler handler = common_models_handler_init(params, LLAMA_EXAMPLE_SERVER);
common_models_handler_apply(handler, params, this);
is_ok = true;
} catch (const std::exception & e) {
auto model_name = params.model.get_name();
SRV_ERR("download failed for model name=%s: %s\n", model_name.c_str(), e.what());
is_ok = false;
}
return is_ok;
}
void on_progress(const common_download_progress & p) {
json data = {
{"url", p.url},
{"downloaded", p.downloaded},
{"total", p.total},
};
role->notify_to_router(server_state_to_str(SERVER_STATE_DOWNLOADING), data);
}
void on_start(const common_download_progress & p) override {
notify(p);
on_progress(p);
}
void on_update(const common_download_progress & p) override {
int64_t now = ggml_time_ms();
// the first update of a file always passes, it carries the file's total size
std::lock_guard<std::mutex> lk(mutex);
auto it = last_progress_time.find(p.url);
if (it != last_progress_time.end() && now - it->second < 100) {
return;
// throttle progress updates to avoid flooding logs; the first update of a file
// always passes, it carries the file's total size
{
std::lock_guard<std::mutex> lk(progress_mutex);
auto it = last_progress_time.find(p.url);
if (it != last_progress_time.end() && now - it->second < 100) {
return;
}
last_progress_time[p.url] = now;
}
last_progress_time[p.url] = now;
on_progress(p);
}
void on_done(const common_download_progress & p, bool) override {
notify(p);
on_progress(p);
}
bool is_cancelled() const override {
return should_stop ? should_stop() : false;
}
};
//
// download worker: a child of the router that pulls a model into the cache and reports the progress
//
static int server_run_download(common_params & params, server_role & role) {
auto cancelled = std::make_shared<std::atomic<bool>>(false);
@@ -462,27 +482,12 @@ static int server_run_download(common_params & params, server_role & role) {
cancelled->store(true, std::memory_order_relaxed);
});
server_download_progress dl;
dl.on_progress = [&role](const common_download_progress & p) {
role.notify_to_router(server_state_to_str(SERVER_STATE_DOWNLOADING), {
{"url", p.url},
{"downloaded", p.downloaded},
{"total", p.total},
});
};
server_download_state dl(&role);
dl.should_stop = [cancelled]() {
return cancelled->load(std::memory_order_relaxed);
};
bool ok = true;
try {
common_models_handler handler = common_models_handler_init(params, LLAMA_EXAMPLE_SERVER);
common_models_handler_apply(handler, params, &dl);
} catch (const std::exception & e) {
auto model_name = params.model.get_name();
SRV_ERR("download failed for model name=%s: %s\n", model_name.c_str(), e.what());
ok = false;
}
bool ok = dl.run(params);
role.notify_to_router(server_state_to_str(SERVER_STATE_DOWNLOADING), {
{"result", ok ? "download_finished" : "download_failed"},
@@ -684,21 +689,15 @@ static int server_run_router(common_params & params, int argc, char ** argv, con
//
static int server_run_instance(common_params & params, server_role & role, common_models_handler & models_handler, const server_run_opts & opts) {
// a router child reports its state to the router, a single-model server to its own /models/sse clients
const bool managed = role.is_managed_by_router();
const std::string model_name = server_get_model_name(params);
server_models_sse sse(/* replay_last */ true);
std::atomic<bool> sse_stopping = false;
auto notify = [&](const std::string & event, const json & data = nullptr) {
if (!managed) {
sse.notify(event, model_name, data);
// -hf / docker: download if needed and resolve the local path. a child resolves what the router downloaded
if (opts.resolve_model) {
try {
common_models_handler_apply(models_handler, params);
} catch (const std::exception & e) {
SRV_ERR("failed to download model: %s\n", e.what());
return 1;
}
};
auto notify_status = [&](server_model_status status, json data = json::object()) {
data["status"] = server_model_status_to_string(status);
notify("status_change", data);
};
}
// note: this is guaranteed to out-live ctx_http and tools
server_mcp mcp_mgr;
@@ -719,82 +718,37 @@ static int server_run_instance(common_params & params, server_role & role, commo
server_stream_make_get_handler(),
server_stream_make_lookup_handler(),
server_stream_make_delete_handler());
if (!managed) {
ctx_http.get("/models/sse", ex_wrapper(sse.make_handler([&sse_stopping]() {
return sse_stopping.load(std::memory_order_relaxed);
})));
}
if (!server_setup_features(ctx_http, params, mcp_mgr, tools, SERVER_ROLE_INSTANCE)) {
return 1;
}
// setup clean up function, to be called before exit
auto clean_up = [&ctx_http, &ctx_server, &mcp_mgr, &sse_stopping]() {
auto clean_up = [&ctx_http, &ctx_server, &mcp_mgr]() {
SRV_INF("%s: cleaning up before exit...\n", __func__);
// stop the session GC first, it finalizes live sessions and wakes pending readers
server_stream_session_manager_stop();
sse_stopping.store(true); // disconnect the SSE clients
ctx_http.stop();
ctx_server.terminate();
mcp_mgr.shutdown();
llama_backend_free();
};
// start the HTTP server before downloading and loading the model to be able to serve /health and /models/sse requests
// start the HTTP server before loading the model to be able to serve /health requests
if (!ctx_http.start()) {
clean_up();
SRV_ERR("%s", "exiting due to HTTP server error\n");
return 1;
}
// -hf / docker: download if needed and resolve the local path. a child resolves what the router downloaded
if (opts.resolve_model) {
json progress = json::object(); // url -> {done, total}, the same shape as the router sends
server_download_progress dl;
dl.on_progress = [&](const common_download_progress & p) {
if (p.total == 0) {
return; // not started yet, or found in the cache
}
progress[p.url] = {
{"done", p.downloaded},
{"total", p.total},
};
notify("download_progress", {{"progress", progress}});
};
try {
common_models_handler_apply(models_handler, params, managed ? nullptr : &dl);
if (!progress.empty()) {
notify("download_finished");
}
} catch (const std::exception & e) {
SRV_ERR("failed to download model: %s\n", e.what());
notify("download_failed");
notify_status(SERVER_MODEL_STATUS_UNLOADED, {{"exit_code", 1}});
clean_up();
ctx_http.join();
return 1;
}
// setup communication child --> router if necessary
if (role.is_managed_by_router()) {
ctx_server.set_state_callback([&](server_state state, json payload) {
role.notify_to_router(server_state_to_str(state), payload);
});
}
// setup communication child --> router, or to the SSE clients
ctx_server.set_state_callback([&](server_state state, json payload) {
if (managed) {
role.notify_to_router(server_state_to_str(state), payload);
return;
}
switch (state) {
case SERVER_STATE_LOADING: notify_status(SERVER_MODEL_STATUS_LOADING, {{"progress", payload}}); break;
case SERVER_STATE_READY: notify_status(SERVER_MODEL_STATUS_LOADED); break; // wakeup from sleeping
case SERVER_STATE_SLEEPING: notify_status(SERVER_MODEL_STATUS_SLEEPING); break;
default: break;
}
});
notify_status(SERVER_MODEL_STATUS_LOADING);
if (!ctx_server.load_model(params)) {
notify_status(SERVER_MODEL_STATUS_UNLOADED, {{"exit_code", 1}});
clean_up();
ctx_http.join();
SRV_ERR("%s", "exiting due to model loading error\n");
@@ -806,22 +760,7 @@ static int server_run_instance(common_params & params, server_role & role, commo
SRV_INF("%s", "model loaded\n");
json info = routes.get_model_info();
json memory;
if (const auto * ll_ctx = ctx_server.get_llama_context()) {
memory = server_model_memory(llama_n_ctx(ll_ctx), common_memory_breakdown_rows(ll_ctx)).to_json();
}
{
json data = {{"info", info}};
if (!memory.is_null()) {
data["memory"] = memory;
}
notify_status(SERVER_MODEL_STATUS_LOADED, data);
}
shutdown_handler = [&](int) {
sse_stopping.store(true);
mcp_mgr.shutdown();
// this will unblock start_loop()
ctx_server.terminate();
@@ -847,11 +786,12 @@ static int server_run_instance(common_params & params, server_role & role, commo
// optionally, notify router server that this instance is ready
std::thread monitor_thread;
if (managed) {
if (role.is_managed_by_router()) {
monitor_thread = role.setup(shutdown_handler);
json info = routes.get_model_info();
// the router tracks device memory from these reports
if (!memory.is_null()) {
info["memory"] = memory;
if (const auto * ll_ctx = ctx_server.get_llama_context()) {
info["memory"] = server_model_memory(llama_n_ctx(ll_ctx), common_memory_breakdown_rows(ll_ctx)).to_json();
}
role.notify_to_router(server_state_to_str(SERVER_STATE_READY), info);
}
-16
View File
@@ -143,19 +143,3 @@ def test_server_model_aliases_and_tags():
assert set(model["tags"]) == {"chat", "fim", "small"}
# id is derived from first alias (alphabetical order from std::set)
assert model["id"] == "code"
def test_server_models_sse_single_model():
global server
server.start()
# a client connecting after the load gets the last status replayed
with requests.get(server.make_url("/models/sse"), stream=True, timeout=10) as res:
assert res.status_code == 200
assert res.headers["Content-Type"].startswith("text/event-stream")
line = next(x for x in res.iter_lines() if x)
event = json.loads(line.decode("utf-8").removeprefix("data: "))
model_id = server.make_request("GET", "/models").body["data"][0]["id"]
assert event["model"] == model_id
assert event["event"] == "status_change"
assert event["data"]["status"] == "loaded"
assert event["data"]["info"]["id"] == model_id
-24
View File
@@ -375,30 +375,6 @@ def test_systemone_d1_parallel():
assert_answers_close(a[qid], b[qid], 1e-6)
def test_systemone_noncausal_parallel():
global server
server = ServerPreset.tinydecision("vllm-d3")
# n_batch > n_ubatch: each prompt fits in one ubatch, but several of them do not
server.n_batch = 2048
server.n_ubatch = 512
server.n_slots = 4
server.start()
state = " ".join(f"The order {i} arrived damaged and the customer wants a replacement." for i in range(14))
questions = {"angry": TEST_QUESTIONS["angry"]}
def decide():
res = server.make_request("POST", "/v1/systemone", data={"state": state, "questions": questions})
assert res.status_code == 200
assert res.body["usage"]["input_tokens"] > 512 // 2
return res.body["answers"]
# a non-causal batch is evaluated in one ubatch, the concurrent prompts are spread over several batches
single = decide()
parallel = parallel_function_calls([(decide, ()) for _ in range(4)])
for answers in parallel:
assert_answers_close(single["angry"], answers["angry"], 1e-6)
def test_systemone_d1_special_tokens():
global server
server = ServerPreset.tinydecision("lfm3-d1-omni")
@@ -1,5 +1,4 @@
import os
import re
import shutil
import subprocess
@@ -54,85 +53,6 @@ def test_tools_builtin_grep_search():
assert "Total matches: 1" in text
def _grep_matches(text: str) -> list[tuple[str, int, str]]:
matches = []
for line in text.splitlines():
match = re.match(r"^(.*):(\d+):(match-\d+)$", line)
if match:
path, line_number, identity = match.groups()
matches.append((path, int(line_number), identity))
return matches
def test_tools_builtin_grep_search_pagination(tmp_path):
global server
server.start()
path = tmp_path / "matches.txt"
expected = [f"match-{i:03d}" for i in range(231)]
path.write_text("\n".join(expected) + "\n")
params = {"path": str(path), "pattern": "match-", "return_line_numbers": True}
pages = []
next_offset = 0
for page_index, expected_size in enumerate([100, 100, 31]):
page_params = params if not pages else {**params, "offset": next_offset}
text = call_tool("grep_search", page_params)["plain_text_response"]
matches = _grep_matches(text)
pages.append(matches)
assert len(matches) == expected_size
assert f"\n---\nTotal matches: {expected_size}\n" in text
continuation = re.search(r"Continue with offset=(\d+) and the same search arguments\.", text)
if page_index < 2:
assert continuation is not None
next_offset = int(continuation.group(1))
assert next_offset == (page_index + 1) * 100
else:
assert continuation is None
assert [identity for page in pages for _, _, identity in page] == expected
def test_tools_builtin_grep_search_exact_page_has_no_continuation(tmp_path):
global server
server.start()
path = tmp_path / "matches.txt"
path.write_text("".join(f"match-{i:03d}\n" for i in range(100)) + "unrelated\n")
res = call_tool("grep_search", {"path": str(path), "pattern": "match-", "return_line_numbers": True})
text = res["plain_text_response"]
assert len(_grep_matches(text)) == 100
assert "Total matches: 100" in text
assert "Continue with offset=" not in text
assert "matches limit reached" not in text
def test_tools_builtin_grep_search_page_order_with_context(tmp_path):
global server
server.start()
(tmp_path / "z.txt").write_text("".join(f"match-{i:03d}\n" for i in range(50)))
(tmp_path / "a.txt").write_text("".join(f"match-{i:03d}\n" for i in range(50, 120)))
params = {"path": str(tmp_path), "pattern": "match-", "context_lines": 1}
first = call_tool("grep_search", {**params, "offset": 0})["plain_text_response"]
second = call_tool("grep_search", {**params, "offset": 100})["plain_text_response"]
past_end = [call_tool("grep_search", {**params, "offset": offset})["plain_text_response"] for offset in (121, 1 << 32)]
expected = [("a.txt", line, f"match-{line + 49:03d}") for line in range(1, 71)]
expected += [("z.txt", line, f"match-{line - 1:03d}") for line in range(1, 51)]
assert _grep_matches(first) == expected[:100]
assert _grep_matches(second) == expected[100:]
assert "Continue with offset=100 and the same search arguments." in first
assert "Continue with offset=" not in second
for text in past_end:
assert _grep_matches(text) == []
assert "Total matches: 0" in text
assert "Continue with offset=" not in text
def test_tools_builtin_read_file(tmp_path):
global server
server.start()