Compare commits
65
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2caae7e113 | ||
|
|
f3397cded0 | ||
|
|
510aa6e090 | ||
|
|
166ec22e2b | ||
|
|
e8386267ec | ||
|
|
b13c413555 | ||
|
|
312c29c2b1 | ||
|
|
24b605331c | ||
|
|
3621025e8f | ||
|
|
72da62cea0 | ||
|
|
d2396e4e36 | ||
|
|
a27217e278 | ||
|
|
3bf6b55b39 | ||
|
|
3ab97b345f | ||
|
|
1df5ced65e | ||
|
|
cf217b0318 | ||
|
|
383dbf4e6c | ||
|
|
060e46516e | ||
|
|
35d58d62cd | ||
|
|
da244dc458 | ||
|
|
062a025714 | ||
|
|
7a233df171 | ||
|
|
7dbe7fdafa | ||
|
|
2766d7977b | ||
|
|
4b29f38b92 | ||
|
|
dbae58b893 | ||
|
|
f58520aa4f | ||
|
|
263d75cba3 | ||
|
|
5f753f7a0c | ||
|
|
4e455b79df | ||
|
|
9537f885e0 | ||
|
|
12a2add2c5 | ||
|
|
deb09fb305 | ||
|
|
3b2c7188fb | ||
|
|
3141ad2a9e | ||
|
|
0ecd1a43cc | ||
|
|
bf3bda9f65 | ||
|
|
bdf2b81b81 | ||
|
|
a213514855 | ||
|
|
8742446dba | ||
|
|
79e6ade8b3 | ||
|
|
3b068bdb3a | ||
|
|
3d4e7bc065 | ||
|
|
a620add3b7 | ||
|
|
9c042626b3 | ||
|
|
2c6aa3b6e7 | ||
|
|
c61db474d8 | ||
|
|
7194d47311 | ||
|
|
04552507fe | ||
|
|
d773ab0ffc | ||
|
|
58e693b794 | ||
|
|
b7cb284e54 | ||
|
|
fa24747925 | ||
|
|
98c7e109b7 | ||
|
|
bc55fb9990 | ||
|
|
25c12e56ca | ||
|
|
8e575c9e54 | ||
|
|
1ee9806b3d | ||
|
|
558f7f65a2 | ||
|
|
cb471f9b2f | ||
|
|
56a1010bbc | ||
|
|
cfda297cbb | ||
|
|
e1bc1f7398 | ||
|
|
b5085151f2 | ||
|
|
ab7e03ec6c |
No files matched your search
@@ -72,7 +72,7 @@ jobs:
|
||||
|
||||
- name: Fetch emdawnwebgpu
|
||||
run: |
|
||||
DAWN_TAG="v20261120.121236"
|
||||
DAWN_TAG="v20260908.214631"
|
||||
EMDAWN_PKG="emdawnwebgpu_pkg-${DAWN_TAG}.zip"
|
||||
echo "Downloading ${EMDAWN_PKG}"
|
||||
curl -L -o emdawn.zip \
|
||||
|
||||
@@ -1,7 +1,5 @@
|
||||
# Instructions for llama.cpp
|
||||
|
||||
> **Private fork: read [FORK.md](FORK.md) and [docs/fork-rebase-runbook.md](docs/fork-rebase-runbook.md) before any work.**
|
||||
|
||||
> [!IMPORTANT]
|
||||
>
|
||||
> AI-generated code is allowed. What is **not** allowed is submitting code you do not understand. You are 100% responsible for every line, however it was produced.
|
||||
|
||||
@@ -6,12 +6,6 @@ which also holds the operations reference (pod access, files, recovery).
|
||||
|
||||
Status at the time of writing (2026-11-22):
|
||||
|
||||
- Candidate, not deployed: branch `fable5/vram-ondemand-rebase-20261122` on upstream `5bacb7c86`
|
||||
(19 upstream commits past `e6d624b9a`, including upstream's #32755 fix #32807 and the merge of
|
||||
#32735), with the F14 first-user-message break, the `LLAMA_LOOP_FULL_COPY` knob (F13) and the
|
||||
carry of #32777 (F12). Pod build `/root/arms/reb20261122/build`. Its validation and the Inkling
|
||||
A/Bs are in runbook section 3 ("2026-11-22") and in F12, F13 and F14 below; production stays on
|
||||
`f3397cded` until they pass.
|
||||
- Production runs `f3397cded` on branch `fable5/vram-ondemand-rebase-20261121` (upstream `e6d624b9a`,
|
||||
757 upstream commits, plus the temporary upstream carries #32735 and #32740, F12, F13, the
|
||||
per-pass loop views for Inkling, and F14, the user-message batch-break merge), deployed 2026-11-22
|
||||
@@ -147,13 +141,6 @@ Oldest first. "Feature" points at the section below.
|
||||
| 72da62cea | upstream-carry #32740: ggml: compact bias on the existing FLASH_ATTN_EXT for CUDA, ROCm and Vulkan (open upstream PR, temporary) | F12 |
|
||||
| 312c29c2b | llama: inkling2_loop - per-pass views of host weights for the later loop passes (deployed 2026-11-22 as `b13c41355`) | F13 |
|
||||
| 510aa6e09 | server: merge user-message batch breaks when the rest of the prompt fits in one ubatch (deployed 2026-11-22 as `f3397cded`) | F14 |
|
||||
| e7e907b7c | server: F14 - keep the batch break at the first user message (system-prompt checkpoint) (branch `-20261122` only) | F14 |
|
||||
| 5f88124b3 | llama: env knob LLAMA_LOOP_FULL_COPY=0 to skip upstream's inkling2_loop full-copy guard (#32807) (branch `-20261122` only) | F13 |
|
||||
| 6c895a491 | upstream-carry #32777: fit : prioritize looped MoE layers in device memory (branch `-20261122` only) | F12 |
|
||||
| 70d39bdbc | scripts: fork-validate.sh - scratch arenas on /dev/shm, lock_id failed on overlayfs (branch `-20261122` only) | F9 |
|
||||
|
||||
On `fable5/vram-ondemand-rebase-20261122` the #32735 carry (d2396e4e3) is gone (merged upstream) and
|
||||
the other hashes in this table differ; the last four rows are hashes of that branch.
|
||||
|
||||
Later documentation-only commits (FORK.md, README, runbook) are not listed; they carry no code.
|
||||
Everything down to `cce0ac1f0` is in production (`2fc9f9c35`, the documentation commit after it);
|
||||
@@ -1133,69 +1120,6 @@ Carried since 2026-11-22 (both on upstream `e6d624b9a`, for Inkling-2-Mini, sect
|
||||
`!bias` (F2). Vulkan supports it only for F16 K/V, head size 64 or 128, with subgroup shuffle and
|
||||
vote (RX 580 RADV has both; Inkling is f16 KV, head size 128).
|
||||
|
||||
State after the 2026-11-22 rebase onto `5bacb7c86` (branch `fable5/vram-ondemand-rebase-20261122`):
|
||||
- #32735: merged upstream as `8a97fa98a`, patch identical to the carried head. Carry dropped.
|
||||
- #32740: still open, but the PR was split on maintainer request (one backend per PR). Its head
|
||||
`8c6f0c0ca` (8 commits) is now only the GGML contract: `ggml_flash_attn_ext_add_indexed_bias`,
|
||||
a CPU reference (the CPU backend now accepts indexed bias) and test-backend-ops cases (the
|
||||
standalone `test-flash-attn-indexed-bias` is gone). `be0bf1fbd` removed the Vulkan support
|
||||
(supports_op returns false for `src[5]` or indexed bias), `9cd195304` moved the CUDA/HIP kernels
|
||||
and the Inkling graph changes (`llama-graph.cpp`, `llama-kv-cache.cpp`, `inkling2.cpp`) out of
|
||||
it. No open PR carries the Vulkan part (Metal is #32808). The contract also changed in wording:
|
||||
the bias is added "in F32" (before: "before its F16 rounding"); the API and the `src[6]`/`src[7]`
|
||||
layout are the same. **Decision: the carry of head `808fd1c4e` is kept unchanged.** Carrying the
|
||||
new head would remove the Vulkan kernel and the graph use, so Inkling would fall back to the
|
||||
dense mask: compute buffer about 15.8 GiB at ctx 65536 instead of 7 GiB, and the k=4 placement of
|
||||
F13 would no longer fit. Owner: decide whether to keep carrying `808fd1c4e` until a Vulkan
|
||||
follow-up PR appears, or to port the Vulkan part onto the new contract. Once the contract merges
|
||||
upstream, the carry will conflict with it in `ggml.h`, `ggml.c`, `ggml-cpu/ops.cpp` and every
|
||||
backend's supports_op, and has to be reduced to the Vulkan and graph parts by hand.
|
||||
- #32777 (new carry, owner's wish): `fit : prioritize looped MoE layers in device memory`, head
|
||||
`01945dcb0`, one commit, applied cleanly. See "#32777 and `--fit` for Inkling" below.
|
||||
|
||||
**#32777 and `--fit` for Inkling.** `llama_model_layer_visits()` counts how many slots share each
|
||||
layer's tensors (by pointer: `ffn_up_exps`, else the next present weight), so inkling2_loop gets 3
|
||||
for the loop layers 6..17, 1 for the others, 0 for the 24 repeat slots. `--fit` then (a) probes
|
||||
"only the non-loop experts on the CPU" against "all experts on the CPU" and keeps the loop experts
|
||||
on the GPU if that leaves more free memory, and (b) when it moves experts back to the GPU layer by
|
||||
layer, ranks the layers by visit count, so loop layers go first (the order of the F13 placement
|
||||
guide).
|
||||
- F13 interaction, checked by reading: fit's first probe, the only one that computes the visits,
|
||||
runs with fit's own defaults (fit refuses to run with a user `-ngl`, `-ot`, `-ncmoe` or `-ts`), so
|
||||
every weight is in a device (dummy) buffer, no F13 view exists, and the pointer sharing is
|
||||
intact. Views appear only in the later probes with CPU experts, which do not recompute the
|
||||
visits. If the visits were ever computed with the loop experts on the host, the views would make
|
||||
every slot count 1 and the loop ranking would silently turn off (fix then: key on
|
||||
`t->view_src ? t->view_src : t`); not reachable through `--fit` today, so not changed.
|
||||
- Memory estimate: fit does not model the per-pass copies; it measures every candidate placement
|
||||
with a no_alloc load and a graph reserve (`common_get_device_memory_data_impl`). The reserve sees
|
||||
the F13 views, so its compute buffer is the F13 one (about 1.4 GiB at ctx 16384, not 7 GiB). The
|
||||
estimate therefore already accounts for F13, and the PR's comment ("CPU-offloading repeated core
|
||||
layers can inflate the compute buffer") describes the build without views; with views step (a)
|
||||
will usually pick "all experts on the CPU" as the base and let step (b) add loop layers back.
|
||||
Not changed.
|
||||
- The mmproj: llama-server adds the mmproj estimate (about 0.8 GiB) to the fit margin of its
|
||||
device before fitting, on top of `--fit-target` (default 1024 MiB). F5 releases the backbone
|
||||
while the encoder runs, so that headroom is conservative.
|
||||
- Fit plans against the VRAM free when it runs. Under llama-swap it runs inside
|
||||
`common_init_from_params`, after `vram_acquire_for_load()` took the GPU token, so the previous
|
||||
holder has released its VRAM; a process outside the arbiter still counts against it.
|
||||
|
||||
How to use it for Inkling (production command line `inkling_args` in `/root/config.yaml`):
|
||||
- Remove `-ngl 99` and the `-ot blk[.](...)[.]ffn_.*_exps=CPU` pattern (either makes fit abort with
|
||||
"already set by user"), keep `--ctx-size 65536` (a user context is not changed by fit), and add
|
||||
`--fit on` (the default, but explicit) and `-lv 4` (fit's log and the memory breakdown are info
|
||||
level). Optionally `--fit-target 512` to give back part of the 1024 MiB margin (the mmproj
|
||||
estimate is still added).
|
||||
- Dry run without serving: `build/bin/llama-fit-params` with the same arguments prints the chosen
|
||||
`-c ... -ngl ... -ot "..."` line on stdout. Compare its `-ot` list with production's k=4 (CPU
|
||||
experts for layers 0-5 and 10-23, GPU for 6-9).
|
||||
- A/B in the window (GPU to itself): the k=4 `-ot` line against the `--fit` line, with F13's
|
||||
drivers (`/root/arms/overnight/sweep.sh` for llama-bench pp2048 @ d8192/d16384 and tg256 @
|
||||
d4096; give llama-bench the explicit `-ot` that fit-params printed rather than its own `-fitt`,
|
||||
which is off by default and fits for the bench context, not 65536), plus the process VRAM/GTT after load and the `-lv 4` breakdown (the free column must keep
|
||||
about 0.8 GiB for the mmproj).
|
||||
|
||||
Measured on the RX 580 (2026-11-21, the previous PR head: `1fa93f902`, `d29f9d489`, `ff91e4f84`,
|
||||
without `808fd1c4e`; base `3bf6b55b3` vs base + both PRs; logs `/root/arms/pr-out/` on the pod):
|
||||
- Inkling pp2048 at ub1024: 91.6 -> 110.5 t/s. pp2048 at depth 16384 (ub512): 48.4 -> 73.0 t/s.
|
||||
@@ -1437,35 +1361,6 @@ only for the first split; not investigated.
|
||||
- Model saving is not supported for `inkling2_loop` (`llama_arch_supports_save`), so duplicate data
|
||||
in a saved file cannot happen.
|
||||
|
||||
**Upstream #32807 (in `5bacb7c86`, branch `-20261122`).** Upstream's fix for #32755: in
|
||||
`sched_copy_experts`, after the first-node check and before F1's logic, `if (arch ==
|
||||
LLM_ARCH_INKLING2_LOOP) return false;`, so the scheduler copies every expert of a shared host
|
||||
weight (a TODO points at per-pass copies; the scheduler-side alternative #32809, "copy host experts
|
||||
again for each split that reads them", was closed unmerged). That fixes correctness but keeps one
|
||||
device copy per (host tensor, backend) per graph, alive until the last pass: the 6.5 GiB of held
|
||||
copies and the ~7 GiB compute buffer stay. So F13 is still needed for memory; for correctness
|
||||
it is now redundant with upstream's guard.
|
||||
- With both, each pass uploads its own copy (F13) and every copy is a whole tensor (the guard):
|
||||
in the 32-63-token window the readback path, which would upload only the selected experts, no
|
||||
longer runs. Ubatches of 64+ tokens are unchanged (F1 copies every expert anyway), below 32 the
|
||||
experts run on the CPU.
|
||||
- Knob (fork, commit "env knob LLAMA_LOOP_FULL_COPY=0"): `LLAMA_LOOP_FULL_COPY=0` skips upstream's
|
||||
guard. Only correct with the views on: with `LLAMA_LOOP_HOST_VIEWS=0` it brings #32755 back.
|
||||
- On this branch, "upstream's fix alone" is `LLAMA_LOOP_HOST_VIEWS=0` (shared tensors plus the
|
||||
guard), "F13 plus upstream" is the default, "F13 alone" is `LLAMA_LOOP_FULL_COPY=0`.
|
||||
|
||||
Window A/B to run (GPU to itself, candidate build, Inkling production command line):
|
||||
1. Memory: process VRAM/GTT after load and the `-lv 4` breakdown at ctx 65536 with production's
|
||||
k=4 `-ot`, default vs `LLAMA_LOOP_HOST_VIEWS=0`. Expected: upstream-only spills (compute about
|
||||
7 GiB, as before F13); F13 fits with about 0.8 GiB free.
|
||||
2. Correctness: `/root/arms/overnight/kld48.sh` (48-token ubatches, KLD against the knob-64 base)
|
||||
for default, `LLAMA_LOOP_HOST_VIEWS=0` and `LLAMA_LOOP_FULL_COPY=0`. Expected: all three at the
|
||||
GPU-vs-CPU noise level (mean KLD about 0.003); before #32807 views off gave 0.032.
|
||||
3. Speed: llama-bench pp2048 @ d8192 and tg256 @ d4096 (`sweep.sh` settings) and a 48-token-tail
|
||||
server prompt (the window where the guard costs uploads), default vs `LLAMA_LOOP_FULL_COPY=0`,
|
||||
arms interleaved in one invocation (section 6). Keep the guard unless the knob wins clearly.
|
||||
Then decide whether F13 stays (expected: yes, for memory).
|
||||
|
||||
**How to drop it.** If upstream fixes #32755 (for example per-reader copies or a copy-callback call for
|
||||
every reader in the scheduler, or real per-pass tensors in the loader), revert 312c29c2b; the hook has
|
||||
no other user. Check that upstream's fix also bounds the copy lifetime, otherwise the 6.5 GiB comes back.
|
||||
@@ -1527,8 +1422,7 @@ which splits off the system prompt. Tool results count as user messages (Qwen op
|
||||
|
||||
**Where.** `tools/server/server-context.cpp`, `update_slots`, the prompt-batch loop:
|
||||
`merge_user_starts` before the loop and the `!merge_user_starts` term of the user-start break.
|
||||
Commit 510aa6e09. On branch `-20261122` the refinement adds `first_user_pos` before the loop, and the
|
||||
user-start break checks `merge_user_starts` first (merged: break only at `first_user_pos`).
|
||||
Commit 510aa6e09.
|
||||
|
||||
**How it works.** When the rest of the prompt from the current batch start fits in one ubatch
|
||||
(`task.n_tokens() - prompt.n_tokens() <= n_ubatch`) and no media chunk was processed in this
|
||||
@@ -1543,38 +1437,6 @@ whose rest is longer than one ubatch keep the upstream breaks (there the break d
|
||||
whole of a ubatch, and the fallback would not be bounded). `LLAMA_USERSTART_MERGE=0` restores the
|
||||
upstream behaviour everywhere.
|
||||
|
||||
**Refinement: the system-prompt checkpoint (owner's wish, commit "F14 - keep the batch break at the
|
||||
first user message", branch `-20261122`, not deployed).** The merged path keeps one user-start
|
||||
break: the one at the first user message, which is the end of the system prompt. Upstream breaks
|
||||
there on a fresh prompt (no checkpoints yet), and the plain F14 had dropped it, so a new chat with
|
||||
the same system prompt found no checkpoint at the system-prompt end and re-processed the system
|
||||
prompt (a hybrid/recurrent state cannot be cut back without one). Now, merged or not, a prompt
|
||||
processed from before its first user message ends a batch there; the next batch starts at the
|
||||
user message and gets a checkpoint (user-start batch, near-end rule). A later new chat with the same
|
||||
system prompt diverges inside the first user message, restores that checkpoint and processes only
|
||||
its own user message and the generation prompt.
|
||||
- Only fresh prompts are affected: checkpoints past the restore point are erased on reuse, so the
|
||||
batch reaches the first user message only when the prompt is processed from before it (a new
|
||||
chat, or a changed system prompt). Follow-up turns start past it and keep the full merge,
|
||||
including no break at the last user message (the edit cost below is unchanged).
|
||||
- First user messages below 8 tokens are skipped (no system prompt, only a BOS before it).
|
||||
- Cost: turn 1 of every new chat runs the system prompt as its own batch again. That was part of
|
||||
the U arm's turn-1 layout below (TTFT 6.08 vs 3.99 s for M, which also merged the last-user
|
||||
break; the system-prompt split alone was not measured). A system prompt of 32+ tokens runs on the
|
||||
GPU with one extra expert upload, below 32 tokens on the CPU.
|
||||
- Lifetime of that checkpoint: it is the oldest one of the slot. When the list is full
|
||||
(`--ctx-checkpoints`, default 32), the min-step thinning keeps the oldest checkpoint and erases
|
||||
later ones within `--checkpoint-min-step` (8192) of it, and only then the FIFO drops the oldest. A
|
||||
very long chat can still lose it; the RAM prompt cache (and F11 on disk) keeps the earlier
|
||||
conversation's copy.
|
||||
- Disk cache (F11), read: `disk_cache_load` replaces the slot prompt only with an entry that
|
||||
reuses at least `DISK_CACHE_MIN_TOKENS` (256) more of the new prompt, and `n_reuse` counts a
|
||||
saved prompt that diverges inside it as its newest checkpoint at or below the divergence. A saved
|
||||
chat therefore offers the system-prompt checkpoint to a new chat after a restart, but only for a
|
||||
system prompt of 256+ tokens; it never replaces a better slot prompt. The checkpoints are saved in
|
||||
the file appendix and trimmed to `--ctx-checkpoints` from the front on load (only if the file has
|
||||
more). Nothing in it removes the break; check it in the window anyway (validation plan item 5).
|
||||
|
||||
**Invariants that must hold.**
|
||||
- The merge only applies when the rest fits in one ubatch: that bounds the extra re-processing on a
|
||||
divergence inside the last user message to less than `n_ubatch` tokens.
|
||||
@@ -1671,24 +1533,6 @@ measured above.
|
||||
restore after a restart, `prompt_n` and output identical to the reference built from the same
|
||||
commit.
|
||||
|
||||
Additions for the refinement (branch `-20261122`, to run in the window with the 35B think entry's
|
||||
exact command line plus `-lv 4`, scratch port, arena and cache dir):
|
||||
5. New chat #2 with the same system prompt. Chat 1: a system prompt of about 300 tokens (above the
|
||||
256-token disk-cache minimum), then 3 turns as in item 2. Expected at trace level on chat 1 turn
|
||||
1: a batch `[0, first user start)`, then a `created context checkpoint ... n_tokens = <first
|
||||
user start>`, then the merged batch to end minus 4. Then chat 2: the same system prompt, a new
|
||||
first user message. Expected: `restored context checkpoint ... n_tokens = <first user start>`
|
||||
and `timings.prompt_n` = chat 2's user message plus the generation prompt only (not the system
|
||||
prompt); no `forcing full prompt re-processing`. Follow-up turns of chat 1 must still run as one
|
||||
batch (no break at the last user message): the M layout of the table above.
|
||||
6. The same after a restart (disk cache): run chat 1, stop the server with SIGTERM (`disk cache:
|
||||
saved the prompt`), start it again, send chat 2. Expected: `disk cache: restored`, then the
|
||||
restore of the system-prompt checkpoint and the same `prompt_n` as in item 5. Also once with a
|
||||
system prompt below 256 tokens: no disk restore is expected there (the minimum), only a fresh
|
||||
prompt; that is the known limit, not a failure.
|
||||
7. The 6-turn TTFT table of item 2 with the candidate, M only: turn 1 may be slower than 3.99 s
|
||||
(the system-prompt split is back), turns 2-6 must match the M column within noise.
|
||||
|
||||
---
|
||||
|
||||
## 4. Dead ends
|
||||
|
||||
+5
-5
@@ -326,7 +326,7 @@ static bool parse_bool_value(const std::string & value) {
|
||||
// common_models_handler
|
||||
//
|
||||
|
||||
std::string common_get_default_local_path(const std::string & url) {
|
||||
static std::string get_default_local_path(const std::string & url) {
|
||||
auto f = string_split<std::string>(url, '#').front();
|
||||
f = string_split<std::string>(f, '?').front();
|
||||
return fs_get_cache_file(string_split<std::string>(f, '/').back()).string();
|
||||
@@ -436,7 +436,7 @@ static std::vector<common_download_task> build_url_tasks(const common_params_mod
|
||||
if (parts.size() == 1) {
|
||||
common_download_task task;
|
||||
task.url = parts[0];
|
||||
task.local_path = model.path.empty() ? common_get_default_local_path(parts[0]) : model.path;
|
||||
task.local_path = model.path.empty() ? get_default_local_path(parts[0]) : model.path;
|
||||
task.opts = opts;
|
||||
tasks.push_back(std::move(task));
|
||||
return tasks;
|
||||
@@ -454,7 +454,7 @@ static std::vector<common_download_task> build_url_tasks(const common_params_mod
|
||||
task.url = part;
|
||||
task.opts = opts;
|
||||
|
||||
std::string local = common_get_default_local_path(part);
|
||||
std::string local = get_default_local_path(part);
|
||||
if (!base_dir.empty()) {
|
||||
auto pos = local.rfind('/');
|
||||
std::string name = pos == std::string::npos ? local : local.substr(pos + 1);
|
||||
@@ -487,7 +487,7 @@ void common_models_handler_apply(common_models_handler & handler, common_params
|
||||
auto handle_url = [&](common_params_model & model) {
|
||||
if (!model.url.empty()) {
|
||||
if (model.path.empty()) {
|
||||
model.path = common_get_default_local_path(model.url);
|
||||
model.path = get_default_local_path(model.url);
|
||||
}
|
||||
}
|
||||
};
|
||||
@@ -498,7 +498,7 @@ void common_models_handler_apply(common_models_handler & handler, common_params
|
||||
// optionally, if docker repo is set, resolve it
|
||||
if (!params.model.docker_repo.empty()) {
|
||||
params.model.url = common_docker_resolve_model(params.model.docker_repo);
|
||||
params.model.path = common_get_default_local_path(params.model.url);
|
||||
params.model.path = get_default_local_path(params.model.url);
|
||||
}
|
||||
|
||||
// handle plain "url" tasks (non-hf)
|
||||
|
||||
@@ -149,9 +149,6 @@ common_models_handler common_models_handler_init(const common_params & params, l
|
||||
// check if the model is a preset repo (i.e. has a preset file)
|
||||
bool common_models_handler_is_preset_repo(const common_models_handler & handler);
|
||||
|
||||
// the cache path a model downloaded from a plain url is stored at
|
||||
std::string common_get_default_local_path(const std::string & url);
|
||||
|
||||
// download and update params with the downloaded model path
|
||||
void common_models_handler_apply(common_models_handler & handler, common_params & params, common_download_callback * callback = nullptr);
|
||||
|
||||
|
||||
@@ -69,21 +69,9 @@ common_chat_params peg_generator::generate_parser(const common_chat_template &
|
||||
std::string trigger_marker = !autoparser.tools.format.section_start.empty() ? autoparser.tools.format.section_start :
|
||||
autoparser.tools.format.per_call_start;
|
||||
|
||||
// bare JSON calls without a start marker (e.g. Llama 3.x): trigger the lazy grammar on the call's opening instead
|
||||
std::string trigger_pattern;
|
||||
{
|
||||
const auto & fmt = autoparser.tools.format;
|
||||
bool top_level_name = (fmt.function_field.empty() || fmt.function_field == "function") &&
|
||||
fmt.name_field.find('.') == std::string::npos;
|
||||
if (trigger_marker.empty() && fmt.mode == tool_format::JSON_NATIVE && !fmt.fun_name_is_key &&
|
||||
!fmt.tools_array_wrapped && top_level_name) {
|
||||
trigger_pattern = "\\{\\s*\"" + regex_escape(fmt.name_field) + "\"\\s*:";
|
||||
}
|
||||
}
|
||||
|
||||
bool has_response_format = !inputs.json_schema.empty() && inputs.json_schema.is_object();
|
||||
bool include_grammar = has_response_format || (has_tools &&
|
||||
((inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO && (!trigger_marker.empty() || !trigger_pattern.empty())) ||
|
||||
((inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO && !trigger_marker.empty()) ||
|
||||
inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
|
||||
|
||||
if (include_grammar) {
|
||||
@@ -94,11 +82,9 @@ common_chat_params peg_generator::generate_parser(const common_chat_template &
|
||||
|
||||
// Set grammar triggers based on tool section markers (fall back to per-call markers)
|
||||
if (data.grammar_lazy) {
|
||||
if (!trigger_marker.empty()) {
|
||||
data.grammar_triggers = { { COMMON_GRAMMAR_TRIGGER_TYPE_WORD, trigger_marker } };
|
||||
} else {
|
||||
data.grammar_triggers = { { COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN, trigger_pattern } };
|
||||
}
|
||||
data.grammar_triggers = {
|
||||
{ COMMON_GRAMMAR_TRIGGER_TYPE_WORD, trigger_marker }
|
||||
};
|
||||
if (autoparser.tools.format.openai_wrapper_trigger) {
|
||||
// model emits the OpenAI function wrapper, trigger on it
|
||||
data.grammar_triggers.push_back({ COMMON_GRAMMAR_TRIGGER_TYPE_WORD, "{\"type\": \"function\"," });
|
||||
@@ -373,9 +359,7 @@ common_peg_parser analyze_tools::build_tool_parser_tag_tagged(parser_build_conte
|
||||
auto & p = ctx.p;
|
||||
const auto & inputs = ctx.inputs;
|
||||
|
||||
// A raw string value needs a suffix to end on, without one it is parsed as JSON
|
||||
bool raw_strings = !arguments.value_suffix.empty();
|
||||
auto until_suffix = raw_strings ? p.rule("until-suffix", p.until(arguments.value_suffix)) : p.eps();
|
||||
auto until_suffix = p.rule("until-suffix", p.until(arguments.value_suffix));
|
||||
|
||||
common_peg_parser tool_choice = p.choice();
|
||||
|
||||
@@ -391,9 +375,9 @@ common_peg_parser analyze_tools::build_tool_parser_tag_tagged(parser_build_conte
|
||||
p.tool_arg(p.tool_arg_open(arguments.name_prefix + p.tool_arg_name(p.literal(param.name)) +
|
||||
arguments.name_suffix) +
|
||||
arguments.value_prefix +
|
||||
(raw_strings && param.schema->may_be_string() ?
|
||||
p.through(p.tool_arg_string_value(until_suffix),
|
||||
p.tool_arg_close(p.literal(arguments.value_suffix))) :
|
||||
(param.schema->may_be_string() ?
|
||||
p.ac(p.tool_arg_string_value(until_suffix) +
|
||||
p.tool_arg_close(p.literal(arguments.value_suffix)), arguments.value_suffix) :
|
||||
(p.tool_arg_json_value(p.schema(
|
||||
p.json(), "tool-" + std::to_string(tool_index) + "-arg-" + std::to_string(param_index) + "-schema", doc, *param.schema)) +
|
||||
p.tool_arg_close(p.literal(arguments.value_suffix)))));
|
||||
|
||||
@@ -71,7 +71,6 @@ struct generation_params {
|
||||
bool is_inference = true;
|
||||
bool add_inference = false;
|
||||
bool mark_input = true; // whether to mark input strings in the jinja context
|
||||
common_peg_tokens specials;
|
||||
|
||||
bool has_continuation() const {
|
||||
return continue_final_message != COMMON_CHAT_CONTINUATION_NONE && !continue_msg.empty();
|
||||
|
||||
@@ -71,9 +71,6 @@ constexpr size_t COMMON_CHAT_MAX_PERMUTE = 6;
|
||||
|
||||
class common_chat_peg_builder : public common_peg_parser_builder {
|
||||
public:
|
||||
common_chat_peg_builder() = default;
|
||||
explicit common_chat_peg_builder(common_peg_tokens tokens) : common_peg_parser_builder(std::move(tokens)) {}
|
||||
|
||||
// Tag constants (from former common_chat_peg_base_builder)
|
||||
static constexpr const char * REASONING_BLOCK = "reasoning-block";
|
||||
static constexpr const char * REASONING = "reasoning";
|
||||
@@ -203,14 +200,6 @@ inline common_peg_arena build_chat_peg_parser(
|
||||
return builder.build();
|
||||
}
|
||||
|
||||
inline common_peg_arena build_chat_peg_parser(
|
||||
const common_peg_tokens & specials,
|
||||
const std::function<common_peg_parser(common_chat_peg_builder & builder)> & fn) {
|
||||
common_chat_peg_builder builder(specials);
|
||||
builder.set_root(fn(builder));
|
||||
return builder.build();
|
||||
}
|
||||
|
||||
class tag_based_peg_mapper {
|
||||
public:
|
||||
std::map<std::string, std::string> tags;
|
||||
|
||||
+2
-18
@@ -315,7 +315,6 @@ struct common_chat_templates {
|
||||
bool add_bos;
|
||||
bool add_eos;
|
||||
bool has_explicit_template; // Model had builtin template or template overridden was specified.
|
||||
common_peg_tokens specials;
|
||||
std::unique_ptr<common_chat_template> template_default; // always set (defaults to chatml)
|
||||
std::unique_ptr<common_chat_template> template_tool_use;
|
||||
std::unique_ptr<common_chat_template> template_tts; // the prompt of a text to speak
|
||||
@@ -803,10 +802,6 @@ void common_chat_templates_free(struct common_chat_templates * tmpls) {
|
||||
delete tmpls;
|
||||
}
|
||||
|
||||
void common_chat_templates_set_specials(struct common_chat_templates * tmpls, common_peg_tokens specials) {
|
||||
tmpls->specials = std::move(specials);
|
||||
}
|
||||
|
||||
bool common_chat_templates_was_explicit(const struct common_chat_templates * tmpls) {
|
||||
return tmpls->has_explicit_template;
|
||||
}
|
||||
@@ -983,7 +978,6 @@ common_chat_templates_ptr common_chat_templates_init(const struct llama_model *
|
||||
std::string token_eos = eos_token_override;
|
||||
bool add_bos = false;
|
||||
bool add_eos = false;
|
||||
common_peg_tokens specials;
|
||||
if (model) {
|
||||
const auto * vocab = llama_model_get_vocab(model);
|
||||
const auto get_token = [&](llama_token token, const char * name, const char * jinja_variable_name) {
|
||||
@@ -1003,13 +997,11 @@ common_chat_templates_ptr common_chat_templates_init(const struct llama_model *
|
||||
token_eos = get_token(llama_vocab_eos(vocab), "EOS", "eos_token");
|
||||
add_bos = llama_vocab_get_add_bos(vocab);
|
||||
add_eos = llama_vocab_get_add_eos(vocab);
|
||||
specials = common_peg_tokens(vocab);
|
||||
}
|
||||
common_chat_templates_ptr tmpls(new common_chat_templates());
|
||||
tmpls->has_explicit_template = has_explicit_template;
|
||||
tmpls->add_bos = add_bos;
|
||||
tmpls->add_eos = add_eos;
|
||||
tmpls->specials = std::move(specials);
|
||||
try {
|
||||
tmpls->template_default = std::make_unique<common_chat_template>(default_template_src, token_bos, token_eos);
|
||||
} catch (const std::exception & e) {
|
||||
@@ -1551,7 +1543,6 @@ static common_chat_params common_chat_templates_apply_jinja(const struct common_
|
||||
params.add_generation_prompt = inputs.add_generation_prompt;
|
||||
params.add_bos = tmpls->add_bos;
|
||||
params.add_eos = tmpls->add_eos;
|
||||
params.specials = tmpls->specials;
|
||||
|
||||
params.continue_final_message = inputs.continue_final_message;
|
||||
if (params.continue_final_message != COMMON_CHAT_CONTINUATION_NONE) {
|
||||
@@ -1834,7 +1825,6 @@ common_chat_input common_chat_input_tokenize(const llama_vocab * vocab, const st
|
||||
}
|
||||
if (input.text != text) {
|
||||
// the pieces do not give back the same text, keep the text without tokens
|
||||
LOG_WRN("%s: token pieces do not match the text, parsing without token ids\n", __func__);
|
||||
return common_chat_input(text);
|
||||
}
|
||||
return input;
|
||||
@@ -1945,17 +1935,11 @@ common_chat_session::common_chat_session(const common_chat_templates * tm
|
||||
thinking_ends = std::move(applied.thinking_end_tags);
|
||||
|
||||
parser_params.format = applied.format;
|
||||
parser_params.generation_prompt = vocab ? common_chat_input_tokenize(vocab, applied.generation_prompt)
|
||||
: common_chat_input(applied.generation_prompt);
|
||||
parser_params.debug = params.debug;
|
||||
parser_params.parser = std::move(applied.parser);
|
||||
|
||||
if (vocab) {
|
||||
parser_params.generation_prompt = common_chat_input_tokenize(vocab, applied.generation_prompt);
|
||||
} else if (params.tokenize) {
|
||||
parser_params.generation_prompt = params.tokenize(applied.generation_prompt);
|
||||
} else {
|
||||
parser_params.generation_prompt = common_chat_input(applied.generation_prompt);
|
||||
}
|
||||
|
||||
delimiters = std::move(applied.message_delimiters);
|
||||
|
||||
if (vocab) {
|
||||
|
||||
@@ -372,9 +372,6 @@ common_chat_templates_ptr common_chat_templates_init(const struct llama_model *
|
||||
const std::string & bos_token_override = "",
|
||||
const std::string & eos_token_override = "");
|
||||
|
||||
// override special tokens for testing
|
||||
void common_chat_templates_set_specials(struct common_chat_templates * tmpls, common_peg_tokens specials);
|
||||
|
||||
bool common_chat_templates_was_explicit(const struct common_chat_templates * tmpls);
|
||||
std::string common_chat_templates_source(const struct common_chat_templates * tmpls, const std::string & variant = "");
|
||||
|
||||
@@ -400,9 +397,6 @@ common_chat_msg common_chat_peg_parse(const common_peg_arena & src_parser, const
|
||||
struct common_chat_session_params {
|
||||
bool echo = false; // include the assistant prefill in the output when continuing a message
|
||||
bool debug = false; // enable debug output for the PEG parser
|
||||
|
||||
// tokenize the generation prompt when there is no vocab for testing
|
||||
std::function<common_chat_input(const std::string &)> tokenize;
|
||||
};
|
||||
|
||||
class common_chat_session {
|
||||
|
||||
+6
-20
@@ -98,9 +98,6 @@ class ProgressBar : public common_download_callback {
|
||||
std::string filename;
|
||||
size_t len = 0;
|
||||
|
||||
// the caller's callback, if any, still receives every event
|
||||
common_download_callback * next;
|
||||
|
||||
static void cleanup(const ProgressBar * line) {
|
||||
lines.erase(line);
|
||||
if (lines.empty()) {
|
||||
@@ -113,12 +110,9 @@ class ProgressBar : public common_download_callback {
|
||||
}
|
||||
|
||||
public:
|
||||
ProgressBar(common_download_callback * next) : next(next) {}
|
||||
ProgressBar() = default;
|
||||
|
||||
void on_start(const common_download_progress & p) override {
|
||||
if (next) {
|
||||
next->on_start(p);
|
||||
}
|
||||
filename = url_get_filename(p.url);
|
||||
|
||||
for (size_t i = 0; i < filename.size(); ++i) {
|
||||
@@ -132,22 +126,12 @@ public:
|
||||
}
|
||||
}
|
||||
|
||||
void on_done(const common_download_progress & p, bool ok) override {
|
||||
if (next) {
|
||||
next->on_done(p, ok);
|
||||
}
|
||||
void on_done(const common_download_progress &, bool) override {
|
||||
std::lock_guard<std::mutex> lock(mutex);
|
||||
cleanup(this);
|
||||
}
|
||||
|
||||
bool is_cancelled() const override {
|
||||
return next && next->is_cancelled();
|
||||
}
|
||||
|
||||
void on_update(const common_download_progress & p) override {
|
||||
if (next) {
|
||||
next->on_update(p);
|
||||
}
|
||||
if (!p.total || !is_output_a_tty()) {
|
||||
return;
|
||||
}
|
||||
@@ -535,9 +519,11 @@ int common_download_file_single(const std::string & url,
|
||||
const std::string & lock_path,
|
||||
const std::string & alt_path) {
|
||||
if (!opts.offline) {
|
||||
ProgressBar tty_cb(opts.callback);
|
||||
ProgressBar tty_cb;
|
||||
common_download_opts online_opts = opts;
|
||||
online_opts.callback = &tty_cb;
|
||||
if (!online_opts.callback) {
|
||||
online_opts.callback = &tty_cb;
|
||||
}
|
||||
return common_download_file_single_online(url, path, online_opts, skip_etag, lock_path, alt_path);
|
||||
}
|
||||
|
||||
|
||||
+16
-144
@@ -7,7 +7,6 @@
|
||||
|
||||
#include "../src/llama-ext.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <array>
|
||||
#include <cassert>
|
||||
#include <stdexcept>
|
||||
@@ -65,8 +64,7 @@ static std::vector<llama_device_memory_data> common_get_device_memory_data_impl(
|
||||
uint32_t & hp_ngl_skip,
|
||||
uint32_t & hp_n_ctx_train,
|
||||
uint32_t & hp_n_expert,
|
||||
ggml_log_level log_level,
|
||||
std::vector<uint32_t> * hp_layer_visits = nullptr) {
|
||||
ggml_log_level log_level) {
|
||||
common_fit_log_scope log_scope(log_level);
|
||||
|
||||
llama_model_params mparams_copy = *mparams;
|
||||
@@ -159,9 +157,6 @@ static std::vector<llama_device_memory_data> common_get_device_memory_data_impl(
|
||||
hp_ngl_skip = mparams->load_mtp ? 0 : llama_model_n_layer_nextn(model);
|
||||
hp_n_ctx_train = llama_model_n_ctx_train(model);
|
||||
hp_n_expert = llama_model_n_expert(model);
|
||||
if (hp_layer_visits != nullptr) {
|
||||
*hp_layer_visits = llama_model_layer_visits(model);
|
||||
}
|
||||
|
||||
common_memory_breakdown_print(ctx);
|
||||
|
||||
@@ -282,7 +277,6 @@ static void common_params_fit_impl(
|
||||
uint32_t hp_nsk = 0; // number of skipped NextN layers, counted by the model loader but not by hp_ngl
|
||||
uint32_t hp_nct = 0; // hparams.n_ctx_train
|
||||
uint32_t hp_nex = 0; // hparams.n_expert
|
||||
std::vector<uint32_t> hp_layer_visits;
|
||||
|
||||
// with non-unified kv, we need to take into account n_streams
|
||||
// for example, if memory can hold more than model's trained context size, we must extend the n_ctx to hold enough n_streams
|
||||
@@ -354,7 +348,7 @@ static void common_params_fit_impl(
|
||||
// step 1: get data for default parameters and check whether any changes are necessary in the first place
|
||||
|
||||
LOG_TRC("%s: getting device memory data for initial parameters:\n", __func__);
|
||||
dmds_t dmds_full = common_get_device_memory_data_impl(paths_model, mparams, cparams, devs, hp_ngl, hp_nsk, hp_nct, hp_nex, log_level, &hp_layer_visits);
|
||||
dmds_t dmds_full = common_get_device_memory_data_impl(paths_model, mparams, cparams, devs, hp_ngl, hp_nsk, hp_nct, hp_nex, log_level);
|
||||
|
||||
// saturate instead of overflowing, this also preserves the UINT32_MAX sentinel of n_ctx_min:
|
||||
const uint32_t n_ctx_max = (uint32_t) std::min<uint64_t>(uint64_t(hp_nct) * n_streams, UINT32_MAX);
|
||||
@@ -637,42 +631,6 @@ static void common_params_fit_impl(
|
||||
|
||||
const size_t ntbo = llama_max_tensor_buft_overrides();
|
||||
|
||||
const bool has_looped_layers = std::any_of(
|
||||
hp_layer_visits.begin(), hp_layer_visits.end(),
|
||||
[](uint32_t v) { return v != 1; });
|
||||
|
||||
// For looped architectures (e.g. inkling2_loop), logical layers share physical weights across passes
|
||||
// and extra KV slots have visits == 0. Rank physical layers by descending visit count so hot looped
|
||||
// MoE layers stay in device memory before single-pass prelude/coda layers.
|
||||
auto get_device_phys_layers = [&](const uint32_t dev_il0, const uint32_t dev_il1) -> std::vector<uint32_t> {
|
||||
std::vector<uint32_t> phys;
|
||||
const uint32_t limit = std::min(dev_il1, (uint32_t) hp_layer_visits.size());
|
||||
for (uint32_t il = dev_il0; il < limit; ++il) {
|
||||
if (hp_layer_visits[il] > 0) {
|
||||
phys.push_back(il);
|
||||
}
|
||||
}
|
||||
std::stable_sort(phys.begin(), phys.end(), [&](uint32_t a, uint32_t b) {
|
||||
return hp_layer_visits[a] > hp_layer_visits[b];
|
||||
});
|
||||
return phys;
|
||||
};
|
||||
|
||||
auto device_layer_slice = [&](const std::vector<ngl_t> & cfg, const size_t target_id) -> std::pair<uint32_t, uint32_t> {
|
||||
uint32_t total_ngl = 0;
|
||||
for (size_t id = 0; id < nd; ++id) {
|
||||
total_ngl += cfg[id].n_layer;
|
||||
}
|
||||
uint32_t cur_il0 = hp_ngl + 1 - total_ngl;
|
||||
for (size_t id = 0; id < target_id; ++id) {
|
||||
cur_il0 += cfg[id].n_layer;
|
||||
}
|
||||
const uint32_t dev_n_tf = (target_id == nd - 1 && cfg[target_id].n_layer > 0)
|
||||
? cfg[target_id].n_layer - 1
|
||||
: cfg[target_id].n_layer;
|
||||
return { cur_il0, cur_il0 + dev_n_tf };
|
||||
};
|
||||
|
||||
// utility function to set n_gpu_layers and tensor_split
|
||||
auto set_ngl_tensor_split_tbo = [&](
|
||||
const std::vector<ngl_t> & ngl_per_device,
|
||||
@@ -705,45 +663,22 @@ static void common_params_fit_impl(
|
||||
mparams.tensor_split = tensor_split;
|
||||
|
||||
size_t itbo = 0;
|
||||
auto push_tbo = [&](const char * pattern, ggml_backend_buffer_type_t buft) {
|
||||
if (itbo + 1 >= ntbo) {
|
||||
tensor_buft_overrides[itbo].pattern = nullptr;
|
||||
tensor_buft_overrides[itbo].buft = nullptr;
|
||||
itbo++;
|
||||
mparams.tensor_buft_overrides = tensor_buft_overrides;
|
||||
throw common_params_fit_exception("llama_max_tensor_buft_overrides() == "
|
||||
+ std::to_string(ntbo) + " is insufficient for model");
|
||||
}
|
||||
tensor_buft_overrides[itbo].pattern = pattern;
|
||||
tensor_buft_overrides[itbo].buft = buft;
|
||||
itbo++;
|
||||
};
|
||||
|
||||
for (size_t id = 0; id < nd; id++) {
|
||||
if (!has_looped_layers) {
|
||||
il0 += ngl_per_device[id].n_full();
|
||||
for (uint32_t il = il0; il < il0 + ngl_per_device[id].n_part; il++) {
|
||||
push_tbo(
|
||||
get_overflow_pattern(il, il == il0 ? ngl_per_device[id].overflow_type : LAYER_FRACTION_MOE),
|
||||
il == il0 ? overflow_bufts[id] : ggml_backend_cpu_buffer_type());
|
||||
il0 += ngl_per_device[id].n_full();
|
||||
for (uint32_t il = il0; il < il0 + ngl_per_device[id].n_part; il++) {
|
||||
if (itbo + 1 >= ntbo) {
|
||||
tensor_buft_overrides[itbo].pattern = nullptr;
|
||||
tensor_buft_overrides[itbo].buft = nullptr;
|
||||
itbo++;
|
||||
mparams.tensor_buft_overrides = tensor_buft_overrides;
|
||||
throw common_params_fit_exception("llama_max_tensor_buft_overrides() == "
|
||||
+ std::to_string(ntbo) + " is insufficient for model");
|
||||
}
|
||||
il0 += ngl_per_device[id].n_part;
|
||||
continue;
|
||||
tensor_buft_overrides[itbo].pattern = get_overflow_pattern(il, il == il0 ? ngl_per_device[id].overflow_type : LAYER_FRACTION_MOE);
|
||||
tensor_buft_overrides[itbo].buft = il == il0 ? overflow_bufts[id] : ggml_backend_cpu_buffer_type();
|
||||
itbo++;
|
||||
}
|
||||
|
||||
const uint32_t dev_n_tf = (id == nd - 1 && ngl_per_device[id].n_layer > 0)
|
||||
? ngl_per_device[id].n_layer - 1
|
||||
: ngl_per_device[id].n_layer;
|
||||
const std::vector<uint32_t> phys = get_device_phys_layers(il0, il0 + dev_n_tf);
|
||||
const uint32_t n_part_tf = std::min(dev_n_tf, ngl_per_device[id].n_part);
|
||||
const uint32_t n_full_tf = dev_n_tf - n_part_tf;
|
||||
for (size_t i = n_full_tf; i < phys.size(); ++i) {
|
||||
const bool is_first = i == n_full_tf;
|
||||
push_tbo(
|
||||
get_overflow_pattern(phys[i], is_first ? ngl_per_device[id].overflow_type : LAYER_FRACTION_MOE),
|
||||
is_first ? overflow_bufts[id] : ggml_backend_cpu_buffer_type());
|
||||
}
|
||||
il0 += ngl_per_device[id].n_layer;
|
||||
il0 += ngl_per_device[id].n_part;
|
||||
}
|
||||
tensor_buft_overrides[itbo].pattern = nullptr;
|
||||
tensor_buft_overrides[itbo].buft = nullptr;
|
||||
@@ -780,7 +715,6 @@ static void common_params_fit_impl(
|
||||
};
|
||||
|
||||
int64_t global_surplus_cpu_moe = 0;
|
||||
bool keep_looped_moe_in_step3 = false;
|
||||
if (hp_nex > 0) {
|
||||
const static std::string pattern_moe_all = "blk\\.\\d+\\.ffn_(up|down|gate_up|gate)_(ch|)exps"; // matches all MoE tensors
|
||||
ggml_backend_buffer_type_t cpu_buft = ggml_backend_cpu_buffer_type();
|
||||
@@ -806,43 +740,6 @@ static void common_params_fit_impl(
|
||||
__func__, -global_surplus_cpu_moe/MiB);
|
||||
}
|
||||
|
||||
// For looped MoE models (e.g. inkling2_loop), CPU-offloading repeated core layers can inflate the
|
||||
// compute buffer enough that keeping looped experts on GPU uses less total VRAM than offloading all MoEs.
|
||||
if (has_looped_layers) {
|
||||
size_t itbo = 0;
|
||||
bool overrides_fit = true;
|
||||
for (uint32_t il = 0; il < hp_layer_visits.size(); ++il) {
|
||||
if (hp_layer_visits[il] == 1) {
|
||||
if (itbo + 1 >= ntbo) {
|
||||
overrides_fit = false;
|
||||
break;
|
||||
}
|
||||
tensor_buft_overrides[itbo++] = {get_overflow_pattern(il, LAYER_FRACTION_MOE), cpu_buft};
|
||||
}
|
||||
}
|
||||
if (overrides_fit && itbo > 0) {
|
||||
tensor_buft_overrides[itbo] = {nullptr, nullptr};
|
||||
mparams->tensor_buft_overrides = tensor_buft_overrides;
|
||||
|
||||
LOG_TRC("%s: getting device memory data with only non-looped MoE tensors moved to system memory:\n", __func__);
|
||||
dmds_t dmds_looped_gpu = common_get_device_memory_data_impl(
|
||||
paths_model, mparams, cparams, devs, hp_ngl, hp_nsk, hp_nct, hp_nex, log_level);
|
||||
add_extra_memory(dmds_looped_gpu);
|
||||
|
||||
int64_t surplus_looped_gpu = 0;
|
||||
for (size_t id = 0; id < nd; id++) {
|
||||
surplus_looped_gpu += dmds_looped_gpu[id].free;
|
||||
surplus_looped_gpu -= int64_t(dmds_looped_gpu[id].mb.total()) + margins[id];
|
||||
}
|
||||
if (surplus_looped_gpu > 0 && surplus_looped_gpu >= global_surplus_cpu_moe) {
|
||||
LOG_TRC("%s: keeping looped MoE weights in device memory yields a larger surplus of %" PRId64 " MiB\n",
|
||||
__func__, surplus_looped_gpu/MiB);
|
||||
global_surplus_cpu_moe = surplus_looped_gpu;
|
||||
keep_looped_moe_in_step3 = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// reset
|
||||
tensor_buft_overrides[0] = {nullptr, nullptr};
|
||||
mparams->tensor_buft_overrides = tensor_buft_overrides;
|
||||
@@ -887,17 +784,6 @@ static void common_params_fit_impl(
|
||||
ngl_per_device_high[id].n_layer = n_unassigned;
|
||||
if (hp_nex > 0) {
|
||||
ngl_per_device_high[id].n_part = size_t(id) < nd - 1 ? ngl_per_device_high[id].n_layer : ngl_per_device_high[id].n_layer - 1;
|
||||
if (keep_looped_moe_in_step3) {
|
||||
const auto [dev_il0, dev_il1] = device_layer_slice(ngl_per_device_high, id);
|
||||
uint32_t n_hot = 0;
|
||||
const uint32_t limit = std::min(dev_il1, (uint32_t) hp_layer_visits.size());
|
||||
for (uint32_t il = dev_il0; il < limit; ++il) {
|
||||
if (hp_layer_visits[il] > 1) {
|
||||
n_hot++;
|
||||
}
|
||||
}
|
||||
ngl_per_device_high[id].n_part -= std::min(ngl_per_device_high[id].n_part, n_hot);
|
||||
}
|
||||
}
|
||||
if (ngl_per_device_high[id].n_layer > 0) {
|
||||
std::vector<int64_t> mem_high = get_memory_for_layers(__func__, ngl_per_device_high, overflow_bufts);
|
||||
@@ -972,12 +858,6 @@ static void common_params_fit_impl(
|
||||
ngl_per_device_high[jd].n_part = 0;
|
||||
}
|
||||
size_t id_dense_start_high = nd - 1;
|
||||
if (has_looped_layers && id == id_dense_start && ngl_per_device_high[id].n_layer == ngl_per_device[id].n_layer) {
|
||||
const auto [dev_il0, dev_il1] = device_layer_slice(ngl_per_device_high, id);
|
||||
const uint32_t n_phys_dev = (uint32_t) get_device_phys_layers(dev_il0, dev_il1).size();
|
||||
const uint32_t dev_n_tf = dev_il1 - dev_il0;
|
||||
ngl_per_device_high[id].n_part = dev_n_tf - std::min(dev_n_tf, n_phys_dev);
|
||||
}
|
||||
std::vector<int64_t> mem_high = get_memory_for_layers(__func__, ngl_per_device_high, overflow_bufts);
|
||||
|
||||
if (mem_high[id] > targets[id]) {
|
||||
@@ -1029,15 +909,7 @@ static void common_params_fit_impl(
|
||||
}
|
||||
|
||||
// try to fit at least part of one more layer
|
||||
bool can_fit_partial = ngl_per_device[id_dense_start].n_layer > (id < nd - 1 ? 0 : 1);
|
||||
if (can_fit_partial && has_looped_layers && id == id_dense_start) {
|
||||
const auto [dev_il0, dev_il1] = device_layer_slice(ngl_per_device, id);
|
||||
const uint32_t n_phys_dev = (uint32_t) get_device_phys_layers(dev_il0, dev_il1).size();
|
||||
const uint32_t dev_n_tf = dev_il1 - dev_il0;
|
||||
const uint32_t n_full_tf = dev_n_tf - std::min(dev_n_tf, ngl_per_device[id].n_part);
|
||||
can_fit_partial = n_full_tf < n_phys_dev;
|
||||
}
|
||||
if (can_fit_partial) {
|
||||
if (ngl_per_device[id_dense_start].n_layer > (id < nd - 1 ? 0 : 1)) {
|
||||
std::vector<ngl_t> ngl_per_device_test = ngl_per_device;
|
||||
size_t id_dense_start_test = id_dense_start;
|
||||
ngl_per_device_test[id_dense_start_test].n_layer--;
|
||||
|
||||
@@ -593,7 +593,7 @@ caps caps_get(jinja::program & prog) {
|
||||
if (result.supports_reasoning_effort) {
|
||||
JJ_DEBUG("%s\n", ">>> Running capability check: reasoning effort levels");
|
||||
|
||||
static const std::vector<std::string> candidates = { "none", "minimal", "low", "medium", "high", "xhigh", "max" };
|
||||
static const std::vector<std::string> candidates = { "minimal", "low", "medium", "high", "xhigh", "max" };
|
||||
|
||||
// renders a single user turn with the given effort, nothing when the template raises
|
||||
const auto render = [&](const std::string & effort) {
|
||||
@@ -631,10 +631,7 @@ caps caps_get(jinja::program & prog) {
|
||||
for (const auto & level : candidates) {
|
||||
const auto rendered = render(level);
|
||||
if (!rendered) {
|
||||
// "none" also turns thinking off, so the server does not answer it with an error
|
||||
if (level != "none") {
|
||||
rejected.push_back(level);
|
||||
}
|
||||
rejected.push_back(level);
|
||||
continue;
|
||||
}
|
||||
if (rendered == unknown) {
|
||||
|
||||
@@ -676,8 +676,8 @@ private:
|
||||
const auto & node = trie.nodes[idx];
|
||||
std::string rejects;
|
||||
auto first = true;
|
||||
for (const auto & [sym, child] : node.children) {
|
||||
std::string c = common_unicode_cpt_to_utf8(sym.value);
|
||||
for (const auto & [cpt, child] : node.children) {
|
||||
std::string c = common_unicode_cpt_to_utf8(cpt);
|
||||
rejects += c;
|
||||
if (first) {
|
||||
first = false;
|
||||
|
||||
+26
-29
@@ -24,9 +24,7 @@ struct depth_guard {
|
||||
// Gemma 4 value parsers
|
||||
void gemma4_value_rules(common_chat_peg_builder & p) {
|
||||
// The tags are what gemma4_to_json() in the mapper reads, the rules may wrap them freely.
|
||||
auto string_delim = p.token("<|\"|>");
|
||||
|
||||
p.rule("gemma4-string", p.tag(gemma4::STRING, string_delim + p.tag(gemma4::STRING_CONTENT, p.until(string_delim)) + string_delim));
|
||||
p.rule("gemma4-string", p.tag(gemma4::STRING, p.literal("<|\"|>") + p.tag(gemma4::STRING_CONTENT, p.until("<|\"|>")) + p.literal("<|\"|>")));
|
||||
|
||||
p.rule("gemma4-bool", p.tag(gemma4::BOOL, p.json_bool()));
|
||||
|
||||
@@ -36,7 +34,7 @@ void gemma4_value_rules(common_chat_peg_builder & p) {
|
||||
|
||||
// Keys are bare in the template, but Gemma 4 occasionally wraps them in quotes
|
||||
p.rule("gemma4-dict-key", p.tag(gemma4::DICT_KEY, p.choice({
|
||||
string_delim + p.tag(gemma4::DICT_KEY_NAME, p.until(string_delim)) + string_delim,
|
||||
p.literal("<|\"|>") + p.tag(gemma4::DICT_KEY_NAME, p.until("<|\"|>")) + p.literal("<|\"|>"),
|
||||
p.literal("\"") + p.tag(gemma4::DICT_KEY_NAME, p.chars("[^\"]", 1, -1)) + p.literal("\""),
|
||||
p.literal("'") + p.tag(gemma4::DICT_KEY_NAME, p.chars("[^']", 1, -1)) + p.literal("'"),
|
||||
p.tag(gemma4::DICT_KEY_NAME, p.chars("[^:}]", 1, -1)),
|
||||
@@ -146,7 +144,7 @@ class gemma4_schema_parser {
|
||||
alts |= name();
|
||||
}
|
||||
// Gemma 4 may emit keys in its string delimiters or quotes
|
||||
alts |= p_.token("<|\"|>") + name() + p_.token("<|\"|>");
|
||||
alts |= p_.literal("<|\"|>") + name() + p_.literal("<|\"|>");
|
||||
if (key.find('\'') == std::string::npos) {
|
||||
alts |= p_.literal("'") + name() + p_.literal("'");
|
||||
}
|
||||
@@ -333,23 +331,18 @@ common_chat_params common_chat_params_init_gemma4(const common_chat_template &
|
||||
auto include_grammar = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
|
||||
auto extract_reasoning = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
|
||||
|
||||
data.parser = build_chat_peg_parser(inputs.specials, [&](common_chat_peg_builder & p) {
|
||||
auto start = p.rule("start", p.optional(p.token("<|turn>") + p.literal("model\n")));
|
||||
auto channel_start = p.token("<|channel>");
|
||||
auto channel_end = p.token("<channel|>");
|
||||
auto think_start = channel_start + p.literal("thought");
|
||||
auto tool_call_start = p.token("<|tool_call>");
|
||||
// the model may start a tool call before closing the thought channel
|
||||
auto think_end = channel_end | p.peek(tool_call_start);
|
||||
data.parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
|
||||
auto start = p.rule("start", p.optional(p.literal("<|turn>model\n")));
|
||||
|
||||
if (extract_reasoning) {
|
||||
p.rule("thought", think_start + p.space() + p.reasoning(p.until(channel_end | tool_call_start)) + think_end);
|
||||
// the model may start a tool call before closing the thought channel
|
||||
p.rule("thought", p.literal("<|channel>thought") + p.space() + p.reasoning(p.until_one_of({"<channel|>", "<|tool_call>"})) + (p.literal("<channel|>") | p.peek(p.literal("<|tool_call>"))));
|
||||
} else {
|
||||
p.rule("thought", p.content(think_start + p.space() + p.until(channel_end | tool_call_start) + think_end));
|
||||
p.rule("thought", p.content(p.literal("<|channel>thought") + p.space() + p.until_one_of({"<channel|>", "<|tool_call>"}) + (p.literal("<channel|>") | p.peek(p.literal("<|tool_call>")))));
|
||||
}
|
||||
|
||||
auto consume_empty_channels = p.gbnf(p.zero_or_more(channel_start + p.negate(p.literal("thought"))), "");
|
||||
auto thought = (p.peek(channel_start) + consume_empty_channels + p.ref("thought")) | p.negate(channel_start);
|
||||
auto consume_empty_channels = p.gbnf(p.zero_or_more(p.literal("<|channel>") + p.negate(p.literal("thought"))), "");
|
||||
auto thought = (p.peek(p.literal("<|channel>")) + consume_empty_channels + p.ref("thought")) | p.negate(p.literal("<|channel>"));
|
||||
|
||||
if (has_response_format) {
|
||||
auto response_format = p.literal("```json") <<
|
||||
@@ -385,35 +378,39 @@ common_chat_params common_chat_params_init_gemma4(const common_chat_template &
|
||||
})));
|
||||
});
|
||||
|
||||
auto tool_call_rest = p.literal("call:") + tool_choice + p.token("<tool_call|>");
|
||||
auto tool_call = p.rule("tool-call", tool_call_start + tool_call_rest);
|
||||
auto more = inputs.parallel_tool_calls ? p.zero_or_more(tool_call) : p.eps();
|
||||
|
||||
auto tool_calls = p.trigger_rule("tool-calls", tool_call_start, tool_call_rest + more);
|
||||
auto tool_call = p.trigger_rule("tool-call", p.repeat(
|
||||
"<|tool_call>call:" + tool_choice + "<tool_call|>",
|
||||
/* min = */ inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED ? 1 : 0,
|
||||
/* max = */ inputs.parallel_tool_calls ? -1 : 1
|
||||
));
|
||||
|
||||
if (inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED) {
|
||||
return start + thought + tool_calls;
|
||||
return start + thought + tool_call;
|
||||
}
|
||||
|
||||
auto scan_to_toolcall = p.rule("scan-to-toolcall", p.until(tool_call_start));
|
||||
auto content = p.rule("content", p.content(p.until(channel_start | channel_end | tool_call_start)));
|
||||
auto scan_to_toolcall = p.rule("scan-to-toolcall", p.until("<|tool_call>"));
|
||||
auto content = p.rule("content", p.content(p.until_one_of({"<|channel>", "<channel|>", "<|tool_call>"})));
|
||||
auto message = p.rule("message", thought + content);
|
||||
return start + p.zero_or_more(message) + scan_to_toolcall + p.optional(tool_calls);
|
||||
return start + p.zero_or_more(message) + scan_to_toolcall + tool_call;
|
||||
}
|
||||
|
||||
// Gemma 4 may emit an extra <|channel>thought\n<channel|> at the end of the content. It may
|
||||
// also emit a single trailing <channel|> token. Consume all complete reasoning blocks and
|
||||
// then stop at the first unmatched <channel|> token.
|
||||
auto content = p.rule("content", p.content(p.until(channel_start | channel_end)));
|
||||
auto content = p.rule("content", p.content(p.until_one_of({"<|channel>", "<channel|>"})));
|
||||
auto message = p.rule("message", thought + content);
|
||||
return start + p.one_or_more(message);
|
||||
});
|
||||
|
||||
if (include_grammar) {
|
||||
data.grammar_lazy = false;
|
||||
data.grammar_lazy = !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
|
||||
data.grammar = build_grammar([&](const common_grammar_builder & builder) {
|
||||
data.parser.build_grammar(builder, !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED)));
|
||||
data.parser.build_grammar(builder, data.grammar_lazy);
|
||||
});
|
||||
|
||||
data.grammar_triggers = {
|
||||
{ COMMON_GRAMMAR_TRIGGER_TYPE_WORD, "<|tool_call>" },
|
||||
};
|
||||
}
|
||||
|
||||
return data;
|
||||
|
||||
+30
-31
@@ -65,18 +65,17 @@ common_chat_params common_chat_params_init_gpt_oss(const common_chat_template &
|
||||
auto include_grammar = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
|
||||
auto extract_reasoning = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
|
||||
|
||||
data.parser = build_chat_peg_parser(inputs.specials, [&](common_chat_peg_builder & p) {
|
||||
auto start_assistant = p.token("<|start|>") + p.literal("assistant");
|
||||
auto start = p.rule("start", start_assistant);
|
||||
auto end = p.rule("end", p.token("<|end|>"));
|
||||
auto content = p.rule("message-content", p.until(p.token("<|end|>")));
|
||||
auto channel = p.token("<|channel|>") + (p.literal("commentary") | p.literal("analysis"));
|
||||
data.parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
|
||||
auto start = p.rule("start", p.literal("<|start|>assistant"));
|
||||
auto end = p.rule("end", p.literal("<|end|>"));
|
||||
auto content = p.rule("message-content", p.until("<|end|>"));
|
||||
auto channel = p.literal("<|channel|>") + (p.literal("commentary") | p.literal("analysis"));
|
||||
auto constrain_type = p.chars("[A-Za-z0-9_-]", 1, -1);
|
||||
auto constraint = p.optional(p.chars("[ ]", 1, -1) + p.optional(p.token("<|constrain|>")) + constrain_type);
|
||||
auto constraint = p.optional(p.space() + p.optional(p.literal("<|constrain|>")) + constrain_type);
|
||||
|
||||
// Occasionally, gpt-oss-20b will prefix channels with this commentary
|
||||
auto stray_commentary = p.optional(p.token("<|channel|>") + p.literal("commentary") + p.optional(p.literal(" to=assistant")));
|
||||
auto start_analysis = stray_commentary + p.token("<|channel|>") + p.literal("analysis") + p.token("<|message|>");
|
||||
auto stray_commentary = p.optional(p.literal("<|channel|>commentary") + p.optional(p.literal(" to=assistant")));
|
||||
auto start_analysis = stray_commentary + p.literal("<|channel|>analysis<|message|>");
|
||||
|
||||
if (extract_reasoning) {
|
||||
p.rule("analysis", start_analysis + p.reasoning(content) + end);
|
||||
@@ -85,8 +84,8 @@ common_chat_params common_chat_params_init_gpt_oss(const common_chat_template &
|
||||
}
|
||||
|
||||
auto analysis = p.ref("analysis");
|
||||
auto preamble = p.rule("preamble", p.token("<|channel|>") + p.literal("commentary") + p.token("<|message|>") + p.content(content) + end);
|
||||
auto final_msg = p.rule("final", stray_commentary + p.token("<|channel|>") + p.literal("final") + constraint + p.token("<|message|>") + p.content(content));
|
||||
auto preamble = p.rule("preamble", p.literal("<|channel|>commentary<|message|>") + p.content(content) + end);
|
||||
auto final_msg = p.rule("final", stray_commentary + p.literal("<|channel|>final") + constraint + p.literal("<|message|>") + p.content(content));
|
||||
|
||||
// Consume any unsolicited tool calls, e.g. builtin functions
|
||||
auto unsolicited = p.rule("unsolicited", p.atomic(p.optional(channel) + p.literal(" to=") + content + end));
|
||||
@@ -95,58 +94,58 @@ common_chat_params common_chat_params_init_gpt_oss(const common_chat_template &
|
||||
|
||||
if (has_response_format) {
|
||||
auto response_format = p.rule("response-format",
|
||||
p.token("<|channel|>") + p.literal("final") + constraint + p.token("<|message|>") +
|
||||
p.literal("<|channel|>final") + constraint + p.literal("<|message|>") +
|
||||
p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema)));
|
||||
|
||||
return p.zero_or_more(start + analysis) + start + response_format;
|
||||
}
|
||||
|
||||
if (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE) {
|
||||
auto in_role = p.choice();
|
||||
auto in_channel = p.choice();
|
||||
auto tool_choice = p.choice();
|
||||
|
||||
foreach_function(inputs.tools, [&](size_t tool_index, const json & tool) {
|
||||
const auto & function = tool.at("function");
|
||||
std::string name = function.at("name");
|
||||
const auto params = common_chat_tool_parameters(function);
|
||||
|
||||
auto func_name = p.tool_name(p.literal(name));
|
||||
auto func_name = p.literal(" to=functions.") + p.tool_name(p.literal(name));
|
||||
auto args = p.tool_args(p.schema(p.json(), "tool-" + std::to_string(tool_index) + "-schema", params));
|
||||
|
||||
// recipient in role header
|
||||
// <|start|>assistant to=functions.NAME<|channel|>(commentary|analysis)[constraint]<|message|>ARGS
|
||||
in_role |= p.rule("tool-" + std::to_string(tool_index) + "-in-role",
|
||||
p.atomic(func_name + channel + constraint + p.token("<|message|>")) + args);
|
||||
auto tool_in_role = p.tool(p.tool_open(func_name + channel + constraint + p.literal("<|message|>")) + args);
|
||||
|
||||
// recipient in channel header
|
||||
// <|start|>assistant<|channel|>(commentary|analysis) to=functions.NAME[constraint]<|message|>ARGS
|
||||
in_channel |= p.rule("tool-" + std::to_string(tool_index) + "-in-channel",
|
||||
p.atomic(func_name + constraint + p.token("<|message|>")) + args);
|
||||
// <|channel|>(commentary|analysis) to=functions.NAME[constraint]<|message|>ARGS
|
||||
auto tool_in_channel = p.tool(p.tool_open(channel + func_name + constraint + p.literal("<|message|>")) + args);
|
||||
|
||||
tool_choice |= p.rule("tool-" + std::to_string(tool_index), tool_in_role | tool_in_channel);
|
||||
});
|
||||
|
||||
// The recipient prefix is the trigger, so the grammar constrains the name that follows it
|
||||
auto to_functions = p.tool_open(p.literal(" to=functions."));
|
||||
auto tool_call = p.choice({
|
||||
p.trigger_rule("tool-call-in-role", start_assistant + to_functions, in_role),
|
||||
p.trigger_rule("tool-call-in-channel", start_assistant + channel + to_functions, in_channel)
|
||||
});
|
||||
auto tool_call = p.trigger_rule("tool-call", tool_choice);
|
||||
|
||||
if (inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED) {
|
||||
return p.zero_or_more(start + any) + tool_call;
|
||||
return p.zero_or_more(start + any) + start + tool_call;
|
||||
}
|
||||
|
||||
return p.zero_or_more(start + any) + (tool_call | (start + final_msg));
|
||||
return p.zero_or_more(start + any) + start + (tool_call | final_msg);
|
||||
}
|
||||
|
||||
return p.zero_or_more(start + any) + start + (final_msg | unsolicited);
|
||||
});
|
||||
|
||||
if (include_grammar) {
|
||||
// The grammar scans for the triggers itself, so the sampler runs it from the first token
|
||||
data.grammar_lazy = false;
|
||||
data.grammar_lazy = !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
|
||||
data.grammar = build_grammar([&](const common_grammar_builder & builder) {
|
||||
data.parser.build_grammar(builder, !has_response_format && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO);
|
||||
data.parser.build_grammar(builder, data.grammar_lazy);
|
||||
});
|
||||
|
||||
data.grammar_triggers = {
|
||||
{ COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN, "^\\s+to$" },
|
||||
{ COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN, "^<\\|channel\\|>(?:commentary|analysis)\\s+to=functions$" },
|
||||
{ COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN, "<\\|start\\|>assistant(\\s+to)" },
|
||||
{ COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN, "<\\|start\\|>assistant(<\\|channel\\|>(?:commentary|analysis)\\s+to)" }
|
||||
};
|
||||
}
|
||||
|
||||
return data;
|
||||
|
||||
@@ -99,11 +99,8 @@ common_chat_params common_chat_params_init_harmony_xml(const common_chat_templat
|
||||
auto arg_close = p.tool_arg_close(p.gbnf(
|
||||
p.literal(PARAM_CLOSE) | (p.literal("\n") + p.peek(p.optional(p.literal("\n")) + next_open)),
|
||||
gbnf_format_literal("\n") + " " + gbnf_format_literal(PARAM_CLOSE.substr(1)) + "?"));
|
||||
auto string_value = p.tool_arg_string_value(p.until_one_of(string_ends));
|
||||
auto arg_string = p.rule("rfl-arg-string", p.through(string_value, p.tool_arg_close(p.literal(PARAM_CLOSE))));
|
||||
// the parser also accepts a value with no closing tag, the grammar still requires the closing tag
|
||||
auto missing_close = string_value + p.tool_arg_close(p.literal("\n") + p.peek(p.optional(p.literal("\n")) + next_open));
|
||||
arg_string = p.gbnf(arg_string | missing_close, "rfl-arg-string");
|
||||
auto arg_string = p.rule("rfl-arg-string",
|
||||
p.ac(p.tool_arg_string_value(p.until_one_of(string_ends)) + arg_close, PARAM_CLOSE));
|
||||
auto arg_sep = p.optional(p.literal("\n"));
|
||||
|
||||
std::string call_close_gbnf = gbnf_format_literal("<|end|>");
|
||||
|
||||
@@ -91,7 +91,7 @@ common_chat_params common_chat_params_init_k2_horizon(const common_chat_template
|
||||
}
|
||||
auto think_body = p.until_one_of(data.thinking_end_tags);
|
||||
auto think_block = [&](const common_peg_parser & body) {
|
||||
return p.optional(THINK_START + p.space() + p.through(body, think_end));
|
||||
return p.optional(THINK_START + p.space() + p.ac(body + think_end, data.thinking_end_tags));
|
||||
};
|
||||
auto reasoning = extract_reasoning ? think_block(p.reasoning(think_body)) : p.eps();
|
||||
|
||||
@@ -110,7 +110,7 @@ common_chat_params common_chat_params_init_k2_horizon(const common_chat_template
|
||||
tool_choice = p.standard_json_tools(CALL_START, CALL_END, inputs.tools, false, true);
|
||||
} else {
|
||||
auto arg_close = p.tool_arg_close(p.literal(ARG_VAL_END));
|
||||
auto arg_string = p.rule("xml-arg-string", p.through(p.tool_arg_string_value(p.until(ARG_VAL_END)), arg_close));
|
||||
auto arg_string = p.rule("xml-arg-string", p.ac(p.tool_arg_string_value(p.until(ARG_VAL_END)) + arg_close, ARG_VAL_END));
|
||||
|
||||
// The models leave out <ifm|arg_type> even when asked for xml_typed
|
||||
auto arg_type = call_format == "xml_typed" ? p.optional(ARG_TYPE + p.until(ARG_TYPE_END) + ARG_TYPE_END + p.space()) : p.eps();
|
||||
|
||||
@@ -63,8 +63,8 @@ common_chat_params common_chat_params_init_minicpm5(const common_chat_template &
|
||||
// CDATA lets a value carry characters that would otherwise close the tag (e.g.
|
||||
// </param>); capture the inner text only, excluding the CDATA markers.
|
||||
auto string_value = p.choice({
|
||||
p.literal("<![CDATA[") + p.through(p.tool_arg_string_value(p.until("]]>")), "]]>") + p.tool_arg_close(p.literal("</param>")),
|
||||
p.negate(p.literal("<![CDATA[")) + p.through(p.tool_arg_string_value(p.until("</param>")), p.tool_arg_close(p.literal("</param>")))
|
||||
p.literal("<![CDATA[") + p.ac(p.tool_arg_string_value(p.until("]]>")) + p.literal("]]>"), "]]>") + p.tool_arg_close(p.literal("</param>")),
|
||||
p.negate(p.literal("< {
|
||||
auto start_token = p.token("<|start|>");
|
||||
auto message = p.token("<|message|>");
|
||||
auto eom = p.token("<|eom|>");
|
||||
auto start = p.rule("start", start_token + p.literal("assistant"));
|
||||
data.parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
|
||||
auto start = p.rule("start", p.literal("<|start|>assistant"));
|
||||
|
||||
if (!extract_reasoning && !include_grammar) {
|
||||
return start + p.content(p.rest());
|
||||
}
|
||||
|
||||
if (extract_reasoning) {
|
||||
p.rule("analysis", p.literal(" to=self") + message + p.reasoning(p.until(eom)) + eom);
|
||||
p.rule("analysis", p.literal(" to=self<|message|>") + p.reasoning(p.until("<|eom|>")) + p.literal("<|eom|>"));
|
||||
} else {
|
||||
p.rule("analysis", p.literal(" to=self") + message + p.content(p.until(eom)) + eom);
|
||||
p.rule("analysis", p.literal(" to=self<|message|>") + p.content(p.until("<|eom|>")) + p.literal("<|eom|>"));
|
||||
}
|
||||
auto analysis = p.ref("analysis");
|
||||
|
||||
auto recipient = p.optional(p.literal(" to=user"));
|
||||
auto final_msg = p.rule("final", recipient + message + p.content(p.until(eom)));
|
||||
auto final_msg = p.rule("final", recipient + p.literal("<|message|>") +
|
||||
p.content(p.until_one_of({ "<|eot|>", "<|eom|>" })));
|
||||
|
||||
if (has_response_format) {
|
||||
auto response_json = p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema));
|
||||
auto response_format = p.rule("response-format",
|
||||
recipient + message +
|
||||
recipient + p.literal("<|message|>") +
|
||||
((p.literal("```json") + p.space() + response_json + p.space() + p.literal("```")) | response_json));
|
||||
|
||||
return p.zero_or_more(start + analysis) + start + response_format;
|
||||
}
|
||||
|
||||
if (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE) {
|
||||
auto string_value = p.through(p.tool_arg_string_value(p.until("</atem:parameter>")),
|
||||
p.tool_arg_close(p.literal("</atem:parameter>")));
|
||||
auto string_value = p.ac(
|
||||
p.tool_arg_string_value(p.until("</atem:parameter>")) + p.tool_arg_close(p.literal("</atem:parameter>")),
|
||||
"</atem:parameter>");
|
||||
|
||||
auto tool_choice = p.choice();
|
||||
foreach_function(inputs.tools, [&](size_t tool_index, const json & tool) {
|
||||
@@ -107,35 +106,40 @@ common_chat_params common_chat_params_init_muse_glimmer(const common_chat_templa
|
||||
args = p.zero_or_more(p.choice(arg_rules) + p.space());
|
||||
}
|
||||
|
||||
auto open = p.tool_open(start_token + p.literal("assistant to=") + p.tool_name(p.literal(name)) + message);
|
||||
auto body = p.rule("tool-" + std::to_string(tool_index),
|
||||
p.literal("<atem:function_calls>") + p.space() + p.literal("<atem:invoke name=\"" + name + "\">") + p.space()
|
||||
auto tool_parser = p.tool(
|
||||
p.tool_open(p.literal(" to=") + p.until("<|message|>") +
|
||||
p.literal("<|message|><atem:function_calls>") + p.space() +
|
||||
p.literal("<atem:invoke name=\"") + p.tool_name(p.literal(name)) + p.literal("\">") + p.space())
|
||||
<< p.tool_args(args)
|
||||
<< p.tool_close(p.literal("</atem:invoke>") + p.space() + p.literal("</atem:function_calls>")));
|
||||
|
||||
tool_choice |= open + body;
|
||||
tool_choice |= p.rule("tool-" + std::to_string(tool_index), tool_parser);
|
||||
});
|
||||
|
||||
auto more = inputs.parallel_tool_calls ? p.zero_or_more(eom + tool_choice) : p.eps();
|
||||
auto tool_calls = p.rule("tool-calls", tool_choice + more);
|
||||
auto tool_calls = inputs.parallel_tool_calls
|
||||
? p.trigger_rule("tool-call", tool_choice + p.zero_or_more(p.literal("<|eom|>") + start + tool_choice))
|
||||
: p.trigger_rule("tool-call", tool_choice);
|
||||
|
||||
|
||||
if (inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED) {
|
||||
return p.zero_or_more(start + analysis) + tool_calls;
|
||||
return p.zero_or_more(start + analysis) + start + tool_calls;
|
||||
}
|
||||
auto trailing_calls = p.optional(eom + tool_calls);
|
||||
return p.zero_or_more(start + analysis) + (tool_calls | (start + final_msg + trailing_calls));
|
||||
auto trailing_calls = p.optional(p.literal("<|eom|>") + start + tool_calls);
|
||||
return p.zero_or_more(start + analysis) + start + (tool_calls | (final_msg + trailing_calls));
|
||||
}
|
||||
|
||||
return p.zero_or_more(start + analysis) + start + final_msg;
|
||||
});
|
||||
|
||||
if (include_grammar) {
|
||||
// Every message starts with a recipient header, so the grammar is not lazy.
|
||||
// The model cannot generate a recipient that is not self, user, or a tool.
|
||||
data.grammar_lazy = false;
|
||||
data.grammar_lazy = !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
|
||||
data.grammar = build_grammar([&](const common_grammar_builder & builder) {
|
||||
data.parser.build_grammar(builder);
|
||||
data.parser.build_grammar(builder, data.grammar_lazy);
|
||||
});
|
||||
data.grammar_triggers = {
|
||||
{ COMMON_GRAMMAR_TRIGGER_TYPE_PATTERN,
|
||||
"(?:^|<\\|start\\|>assistant)( to=(?!self<\\|message\\|>)(?!user<\\|message\\|>)[^<]*?<\\|message\\|>)" },
|
||||
};
|
||||
}
|
||||
|
||||
return data;
|
||||
|
||||
@@ -86,8 +86,9 @@ common_chat_params common_chat_params_init_muse_spark(const common_chat_template
|
||||
}
|
||||
|
||||
if (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE) {
|
||||
auto string_value = p.through(p.tool_arg_string_value(p.until("</atem:parameter>")),
|
||||
p.tool_arg_close(p.literal("</atem:parameter>")));
|
||||
auto string_value = p.ac(
|
||||
p.tool_arg_string_value(p.until("</atem:parameter>")) + p.tool_arg_close(p.literal("</atem:parameter>")),
|
||||
"</atem:parameter>");
|
||||
|
||||
auto tool_choice = p.choice();
|
||||
foreach_function(inputs.tools, [&](size_t tool_index, const json & tool) {
|
||||
|
||||
@@ -59,16 +59,27 @@ common_chat_params common_chat_params_init_qwen3_coder(const common_chat_templat
|
||||
data.prompt.append(data.generation_prompt);
|
||||
}
|
||||
|
||||
data.parser = build_chat_peg_parser(inputs.specials, [&](common_chat_peg_builder & p) {
|
||||
std::vector<std::string> tool_call_starts = { "<tool_call>" };
|
||||
|
||||
if (is_qwen3_coder) {
|
||||
// Match complete <function=name> opener for Qwen3-Coder models that occasionally omit the
|
||||
// starting <tool_call>. The model may hallucinate a tool name, but it is preferable over
|
||||
// constraining on <function which may occur in valid content generation, e.g. #include <functional>
|
||||
foreach_function(inputs.tools, [&](size_t, const json & tool) {
|
||||
const std::string name = tool.at("function").at("name");
|
||||
tool_call_starts.push_back("<function=" + name + ">");
|
||||
});
|
||||
}
|
||||
|
||||
data.parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
|
||||
auto generation_prompt = p.literal(GEN_PREFIX);
|
||||
|
||||
// The template strips the newline before </think>, so leave it out of the reasoning
|
||||
auto think_end = (p.literal("\n") + p.token("</think>")) | p.token("</think>");
|
||||
auto think_body = p.until(think_end | p.token("<tool_call>"));
|
||||
auto think_close = think_end | p.peek(p.token("<tool_call>"));
|
||||
auto think_body = p.until_one_of({ "\n</think>", "</think>", "<tool_call>" });
|
||||
auto think_close = p.literal("\n</think>") | p.literal("</think>") | p.peek(p.literal("<tool_call>"));
|
||||
|
||||
auto reasoning_extracted = p.optional(p.token("<think>") + p.space() + p.reasoning(think_body) + think_close);
|
||||
auto reasoning_skipped = p.optional(p.token("<think>") + p.space() + think_body + think_close);
|
||||
auto reasoning_extracted = p.optional("<think>" + p.space() + p.reasoning(think_body) + think_close);
|
||||
auto reasoning_skipped = p.optional("<think>" + p.space() + think_body + think_close);
|
||||
|
||||
auto reasoning = p.eps();
|
||||
if (supports_reasoning && extract_reasoning) {
|
||||
@@ -88,13 +99,7 @@ common_chat_params common_chat_params_init_qwen3_coder(const common_chat_templat
|
||||
if (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE) {
|
||||
auto arg_close = p.tool_arg_close(p.literal("\n</parameter>\n"));
|
||||
auto arg_string = p.rule("xml-arg-string",
|
||||
p.through(p.tool_arg_string_value(p.until("\n</parameter>\n")), arg_close));
|
||||
|
||||
auto min_calls = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED ? 1 : 0;
|
||||
|
||||
auto more = inputs.parallel_tool_calls ? p.zero_or_more(p.ref("tool-call")) : p.eps();
|
||||
auto starts = p.token("<tool_call>");
|
||||
auto tool_calls = p.choice();
|
||||
p.ac(p.tool_arg_string_value(p.until("\n</parameter>\n")) + arg_close, "\n</parameter>\n"));
|
||||
|
||||
auto tool_choice = p.choice();
|
||||
foreach_function(inputs.tools, [&](size_t tool_index, const json & tool) {
|
||||
@@ -150,29 +155,35 @@ common_chat_params common_chat_params_init_qwen3_coder(const common_chat_templat
|
||||
args = args + p.zero_or_more(p.choice(optional_args));
|
||||
}
|
||||
|
||||
auto opener = p.tool_open("<function=" + p.tool_name(p.literal(name)) + ">");
|
||||
auto body = p.literal("\n") + p.tool_args(args) + p.tool_close(p.literal("</function>\n"));
|
||||
auto func = p.tool(p.tool_open("<function=" + p.tool_name(p.literal(name)) + ">\n") +
|
||||
p.tool_args(args) +
|
||||
p.tool_close(p.literal("</function>\n")));
|
||||
|
||||
tool_choice |= p.rule("tool-" + std::to_string(tool_index), p.tool(opener + body));
|
||||
|
||||
if (is_qwen3_coder) {
|
||||
// Qwen3-Coder models sometimes omit <tool_call>, so the full <function=name> opener is also a trigger.
|
||||
// Do not trigger on <function alone, it occurs in normal content, e.g. #include <functional>.
|
||||
tool_calls |= p.trigger_rule("tool-calls-" + std::to_string(tool_index), opener, body + p.token("</tool_call>") + p.space() + more);
|
||||
starts |= opener;
|
||||
}
|
||||
tool_choice |= p.rule("tool-" + std::to_string(tool_index), func);
|
||||
});
|
||||
|
||||
auto tool_call_body = tool_choice + p.token("</tool_call>") + p.space();
|
||||
p.rule("tool-call", p.token("<tool_call>") + p.literal("\n") + tool_call_body);
|
||||
auto min_calls = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED ? 1 : 0;
|
||||
|
||||
tool_calls |= p.trigger_rule("tool-calls", p.token("<tool_call>"), p.literal("\n") + tool_call_body + more);
|
||||
auto tool_call_body = tool_choice + "</tool_call>" + p.space();
|
||||
auto tool_call = p.rule("tool-call", "<tool_call>\n" + tool_call_body);
|
||||
|
||||
// Qwen3-Coder models may occasionally omit the <tool_call> token.
|
||||
auto tool_call_first = is_qwen3_coder ?
|
||||
p.rule("tool-call-first", p.optional(p.literal("<tool_call>\n")) + tool_call_body) :
|
||||
tool_call;
|
||||
|
||||
auto calls = inputs.parallel_tool_calls ? tool_call_first + p.zero_or_more(tool_call) : tool_call_first;
|
||||
auto tool_calls = p.trigger_rule("tool-call-root", p.repeat(calls, min_calls, 1));
|
||||
|
||||
// The templates separate content from the tool calls with one or two newlines,
|
||||
// stop the content before them
|
||||
auto content_end = (p.literal("\n\n") + starts) | (p.literal("\n") + starts) | starts;
|
||||
std::vector<std::string> content_ends;
|
||||
for (const auto & start : tool_call_starts) {
|
||||
content_ends.insert(content_ends.end(), { "\n\n" + start, "\n" + start, start });
|
||||
}
|
||||
|
||||
return generation_prompt + (reasoning << p.content(p.until(content_end)) << p.repeat(tool_calls, min_calls, 1));
|
||||
return generation_prompt +
|
||||
(reasoning << p.content(p.until_one_of(content_ends)) << tool_calls);
|
||||
}
|
||||
|
||||
// Content only parser
|
||||
@@ -180,11 +191,17 @@ common_chat_params common_chat_params_init_qwen3_coder(const common_chat_templat
|
||||
});
|
||||
|
||||
if (include_grammar) {
|
||||
// The grammar scans for the triggers itself, so the sampler runs it from the first token
|
||||
data.grammar_lazy = false;
|
||||
data.grammar_lazy = !has_response_format && has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO;
|
||||
|
||||
data.grammar = build_grammar([&](const common_grammar_builder & builder) {
|
||||
data.parser.build_grammar(builder, !has_response_format && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_AUTO);
|
||||
data.parser.build_grammar(builder, data.grammar_lazy);
|
||||
});
|
||||
|
||||
if (data.grammar_lazy) {
|
||||
for (const auto & start : tool_call_starts) {
|
||||
data.grammar_triggers.push_back({ COMMON_GRAMMAR_TRIGGER_TYPE_WORD, start });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return data;
|
||||
|
||||
+115
-543
@@ -208,89 +208,6 @@ void common_peg_ast_arena::visit(const common_peg_parse_result & result, const c
|
||||
}
|
||||
}
|
||||
|
||||
common_peg_tokens::common_peg_tokens(const llama_vocab * vocab) {
|
||||
const llama_token n_tokens = llama_vocab_n_tokens(vocab);
|
||||
for (llama_token id = 0; id < n_tokens; ++id) {
|
||||
const auto attr = llama_vocab_get_attr(vocab, id);
|
||||
if (attr & (LLAMA_TOKEN_ATTR_CONTROL | LLAMA_TOKEN_ATTR_USER_DEFINED)) {
|
||||
std::string text = llama_vocab_get_text(vocab, id);
|
||||
ids.emplace(text, id);
|
||||
tokens.emplace(id, common_peg_token{ std::move(text), attr });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
llama_token common_peg_tokens::token_id(const std::string & text) const {
|
||||
auto it = ids.find(text);
|
||||
return it != ids.end() ? it->second : LLAMA_TOKEN_NULL;
|
||||
}
|
||||
|
||||
// Insert every delimiter the parser matches into the trie, starting from each node in from. Returns the end nodes.
|
||||
// A choice adds the delimiters of each branch. A sequence joins every combination of its children.
|
||||
static std::set<size_t> insert_delimiter(common_trie & trie, const common_peg_arena & arena, common_peg_parser_id id,
|
||||
std::set<size_t> from = { 0 }) {
|
||||
return std::visit([&](const auto & p) -> std::set<size_t> {
|
||||
using T = std::decay_t<decltype(p)>;
|
||||
if constexpr (std::is_same_v<T, common_peg_tag_parser> || std::is_same_v<T, common_peg_atomic_parser>) {
|
||||
return insert_delimiter(trie, arena, p.child, std::move(from));
|
||||
} else if constexpr (std::is_same_v<T, common_peg_literal_parser>) {
|
||||
std::set<size_t> ends;
|
||||
for (size_t node : from) {
|
||||
size_t pos = 0;
|
||||
while (pos < p.literal.size()) {
|
||||
auto result = common_parse_utf8_codepoint(p.literal, pos);
|
||||
if (result.status != utf8_parse_result::SUCCESS) {
|
||||
break;
|
||||
}
|
||||
node = trie.extend(node, common_trie::symbol::codepoint(result.codepoint));
|
||||
pos += result.bytes_consumed;
|
||||
}
|
||||
ends.insert(node);
|
||||
}
|
||||
return ends;
|
||||
} else if constexpr (std::is_same_v<T, common_peg_token_parser>) {
|
||||
std::set<size_t> ends;
|
||||
for (size_t node : from) {
|
||||
ends.insert(trie.extend(node, common_trie::symbol::token(p.token)));
|
||||
}
|
||||
return ends;
|
||||
} else if constexpr (std::is_same_v<T, common_peg_choice_parser>) {
|
||||
std::set<size_t> ends;
|
||||
for (auto child : p.children) {
|
||||
ends.merge(insert_delimiter(trie, arena, child, from));
|
||||
}
|
||||
return ends;
|
||||
} else if constexpr (std::is_same_v<T, common_peg_sequence_parser>) {
|
||||
for (auto child : p.children) {
|
||||
from = insert_delimiter(trie, arena, child, std::move(from));
|
||||
}
|
||||
return from;
|
||||
} else {
|
||||
throw std::invalid_argument("a delimiter may only be a sequence of literals, tokens, and choices of them, possibly wrapped in tags and atomics");
|
||||
}
|
||||
}, arena.get(id));
|
||||
}
|
||||
|
||||
// Build the matcher of an until parser, each pattern valued by the branch it belongs to
|
||||
static common_trie build_until_matcher(const common_peg_arena & arena,
|
||||
const std::vector<common_peg_until_branch> & branches,
|
||||
bool include) {
|
||||
common_trie matcher;
|
||||
for (size_t b = 0; b < branches.size(); b++) {
|
||||
for (size_t end : insert_delimiter(matcher, arena, branches[b].delimiter)) {
|
||||
if (end == 0) {
|
||||
throw std::invalid_argument("delimiter must not match the empty string");
|
||||
}
|
||||
// nothing follows a delimiter without include, so the first branch to claim it wins
|
||||
auto owner = matcher.mark(end, (int32_t) b);
|
||||
if (owner != (int32_t) b && include) {
|
||||
throw std::invalid_argument("until branches " + std::to_string(owner) + " and " + std::to_string(b) + " must not share a delimiter");
|
||||
}
|
||||
}
|
||||
}
|
||||
return matcher;
|
||||
}
|
||||
|
||||
struct parser_executor;
|
||||
|
||||
common_peg_parser_id common_peg_arena::add_parser(common_peg_parser_variant parser) {
|
||||
@@ -381,49 +298,6 @@ struct parser_executor {
|
||||
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, pos);
|
||||
}
|
||||
|
||||
common_peg_parse_result operator()(const common_peg_token_parser & p) {
|
||||
if (start_pos >= ctx.input.size()) {
|
||||
if (!ctx.is_lenient()) {
|
||||
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_FAIL, start_pos);
|
||||
}
|
||||
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_NEED_MORE_INPUT, start_pos);
|
||||
}
|
||||
|
||||
if (start_pos >= ctx.tokens.size() || ctx.tokens[start_pos] != p.token) {
|
||||
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_FAIL, start_pos);
|
||||
}
|
||||
|
||||
// Skip the rest of the piece, up to the next token or the end of the input
|
||||
auto pos = start_pos + 1;
|
||||
while (pos < ctx.input.size() && ctx.tokens[pos] == LLAMA_TOKEN_NULL) {
|
||||
++pos;
|
||||
}
|
||||
|
||||
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, pos);
|
||||
}
|
||||
|
||||
common_peg_parse_result operator()(const common_peg_token_not_parser & p) {
|
||||
if (start_pos >= ctx.input.size()) {
|
||||
if (!ctx.is_lenient()) {
|
||||
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_FAIL, start_pos);
|
||||
}
|
||||
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_NEED_MORE_INPUT, start_pos);
|
||||
}
|
||||
|
||||
if (start_pos >= ctx.tokens.size() || ctx.tokens[start_pos] == LLAMA_TOKEN_NULL ||
|
||||
std::find(p.tokens.begin(), p.tokens.end(), ctx.tokens[start_pos]) != p.tokens.end()) {
|
||||
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_FAIL, start_pos);
|
||||
}
|
||||
|
||||
// Skip the rest of the piece, up to the next token or the end of the input
|
||||
auto pos = start_pos + 1;
|
||||
while (pos < ctx.input.size() && ctx.tokens[pos] == LLAMA_TOKEN_NULL) {
|
||||
++pos;
|
||||
}
|
||||
|
||||
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, pos);
|
||||
}
|
||||
|
||||
common_peg_parse_result operator()(const common_peg_sequence_parser & p) {
|
||||
if (ctx.is_debug()) {
|
||||
LOG_DBG("%sSEQ start at %zu '%s' (%zu children)\n", debug_indent().c_str(), start_pos,
|
||||
@@ -815,11 +689,7 @@ struct parser_executor {
|
||||
}
|
||||
|
||||
common_peg_parse_result operator()(const common_peg_until_parser & p) const {
|
||||
if (p.include) {
|
||||
return parse_through(p);
|
||||
}
|
||||
|
||||
const auto & matcher = p.matcher;
|
||||
common_trie matcher(p.delimiters);
|
||||
|
||||
// Scan input and check for delimiters
|
||||
size_t pos = start_pos;
|
||||
@@ -827,24 +697,24 @@ struct parser_executor {
|
||||
std::vector<common_peg_invalid_utf8> invalid_utf8;
|
||||
|
||||
while (pos < ctx.input.size()) {
|
||||
auto step = matcher.next(ctx.input, ctx.tokens, pos);
|
||||
auto utf8_result = common_parse_utf8_codepoint(ctx.input, pos);
|
||||
|
||||
if (step.status == utf8_parse_result::INCOMPLETE && ctx.is_lenient()) {
|
||||
if (utf8_result.status == utf8_parse_result::INCOMPLETE && ctx.is_lenient()) {
|
||||
// The rest of the sequence may still arrive, return what we have so far
|
||||
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_NEED_MORE_INPUT, start_pos, last_valid_pos, {}, std::move(invalid_utf8));
|
||||
}
|
||||
|
||||
if (step.status != utf8_parse_result::SUCCESS) {
|
||||
if (utf8_result.status != utf8_parse_result::SUCCESS) {
|
||||
// Malformed UTF-8, or a sequence truncated by the end of a complete input.
|
||||
// A delimiter cannot start inside bytes that fail to decode, so consume them and move on
|
||||
invalid_utf8.push_back({pos, step.bytes_consumed});
|
||||
pos += step.bytes_consumed;
|
||||
invalid_utf8.push_back({pos, utf8_result.bytes_consumed});
|
||||
pos += utf8_result.bytes_consumed;
|
||||
last_valid_pos = pos;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Check if a delimiter starts at this position
|
||||
auto match = matcher.check_at(ctx.input, ctx.tokens, pos);
|
||||
auto match = matcher.check_at(ctx.input, pos);
|
||||
|
||||
if (match == common_trie::COMPLETE_MATCH) {
|
||||
// Found a complete delimiter, return everything before it
|
||||
@@ -857,7 +727,7 @@ struct parser_executor {
|
||||
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, pos, {}, std::move(invalid_utf8));
|
||||
}
|
||||
|
||||
pos += step.bytes_consumed;
|
||||
pos += utf8_result.bytes_consumed;
|
||||
last_valid_pos = pos;
|
||||
}
|
||||
|
||||
@@ -868,48 +738,6 @@ struct parser_executor {
|
||||
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, last_valid_pos, {}, std::move(invalid_utf8));
|
||||
}
|
||||
|
||||
// Parse content, then the first branch whose delimiter and rest follow it, like C (D1 R1 | D2 R2 | ...)
|
||||
common_peg_parse_result parse_through(const common_peg_until_parser & p) const {
|
||||
auto content = arena.parse(p.content, ctx, start_pos);
|
||||
if (content.fail() || content.need_more_input()) {
|
||||
return content;
|
||||
}
|
||||
|
||||
auto pos = content.end;
|
||||
for (const auto & branch : p.branches) {
|
||||
auto nodes = content.nodes;
|
||||
auto invalid_utf8 = content.invalid_utf8;
|
||||
auto end = pos;
|
||||
bool matched = true;
|
||||
|
||||
for (auto id : { branch.delimiter, branch.rest }) {
|
||||
if (id == COMMON_PEG_INVALID_PARSER_ID) {
|
||||
continue;
|
||||
}
|
||||
auto result = arena.parse(id, ctx, end);
|
||||
if (result.fail()) {
|
||||
matched = false;
|
||||
break;
|
||||
}
|
||||
nodes.insert(nodes.end(), result.nodes.begin(), result.nodes.end());
|
||||
invalid_utf8.insert(invalid_utf8.end(), result.invalid_utf8.begin(), result.invalid_utf8.end());
|
||||
end = result.end;
|
||||
if (result.need_more_input()) {
|
||||
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_NEED_MORE_INPUT, start_pos, end, std::move(nodes), std::move(invalid_utf8));
|
||||
}
|
||||
}
|
||||
if (matched) {
|
||||
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_SUCCESS, start_pos, end, std::move(nodes), std::move(invalid_utf8));
|
||||
}
|
||||
}
|
||||
|
||||
if (p.optional && pos == ctx.input.size()) {
|
||||
auto type = ctx.is_lenient() ? COMMON_PEG_PARSE_RESULT_NEED_MORE_INPUT : COMMON_PEG_PARSE_RESULT_SUCCESS;
|
||||
return common_peg_parse_result(type, start_pos, pos, std::move(content.nodes), std::move(content.invalid_utf8));
|
||||
}
|
||||
return common_peg_parse_result(COMMON_PEG_PARSE_RESULT_FAIL, start_pos, pos);
|
||||
}
|
||||
|
||||
common_peg_parse_result operator()(const common_peg_schema_parser & p) {
|
||||
return arena.parse(p.child, ctx, start_pos);
|
||||
}
|
||||
@@ -995,6 +823,10 @@ struct parser_executor {
|
||||
common_peg_parse_result operator()(const common_peg_gbnf_parser & p) {
|
||||
return arena.parse(p.child, ctx, start_pos);
|
||||
}
|
||||
|
||||
common_peg_parse_result operator()(const common_peg_ac_parser & p) {
|
||||
return arena.parse(p.child, ctx, start_pos);
|
||||
}
|
||||
};
|
||||
|
||||
common_peg_parse_result common_peg_arena::parse(common_peg_parse_context & ctx, size_t start) const {
|
||||
@@ -1067,33 +899,19 @@ void common_peg_arena::resolve_refs() {
|
||||
std::is_same_v<T, common_peg_not_parser> ||
|
||||
std::is_same_v<T, common_peg_tag_parser> ||
|
||||
std::is_same_v<T, common_peg_atomic_parser> ||
|
||||
std::is_same_v<T, common_peg_gbnf_parser>) {
|
||||
std::is_same_v<T, common_peg_gbnf_parser> ||
|
||||
std::is_same_v<T, common_peg_ac_parser>) {
|
||||
p.child = resolve_ref(p.child);
|
||||
} else if constexpr (std::is_same_v<T, common_peg_until_parser>) {
|
||||
if (p.content != COMMON_PEG_INVALID_PARSER_ID) {
|
||||
p.content = resolve_ref(p.content);
|
||||
}
|
||||
for (auto & branch : p.branches) {
|
||||
branch.delimiter = resolve_ref(branch.delimiter);
|
||||
if (branch.rest != COMMON_PEG_INVALID_PARSER_ID) {
|
||||
branch.rest = resolve_ref(branch.rest);
|
||||
}
|
||||
}
|
||||
} else if constexpr (std::is_same_v<T, common_peg_rule_parser>) {
|
||||
p.child = resolve_ref(p.child);
|
||||
if (p.start != COMMON_PEG_INVALID_PARSER_ID) {
|
||||
p.start = resolve_ref(p.start);
|
||||
p.rest = resolve_ref(p.rest);
|
||||
}
|
||||
} else if constexpr (std::is_same_v<T, common_peg_schema_parser>) {
|
||||
p.child = resolve_ref(p.child);
|
||||
} else if constexpr (std::is_same_v<T, common_peg_epsilon_parser> ||
|
||||
std::is_same_v<T, common_peg_start_parser> ||
|
||||
std::is_same_v<T, common_peg_end_parser> ||
|
||||
std::is_same_v<T, common_peg_ref_parser> ||
|
||||
std::is_same_v<T, common_peg_until_parser> ||
|
||||
std::is_same_v<T, common_peg_literal_parser> ||
|
||||
std::is_same_v<T, common_peg_token_parser> ||
|
||||
std::is_same_v<T, common_peg_token_not_parser> ||
|
||||
std::is_same_v<T, common_peg_string_parser> ||
|
||||
std::is_same_v<T, common_peg_chars_parser> ||
|
||||
std::is_same_v<T, common_peg_any_parser> ||
|
||||
@@ -1137,14 +955,6 @@ std::string common_peg_arena::dump_impl(common_peg_parser_id
|
||||
return "End";
|
||||
} else if constexpr (std::is_same_v<T, common_peg_literal_parser>) {
|
||||
return "Literal(" + p.literal + ")";
|
||||
} else if constexpr (std::is_same_v<T, common_peg_token_parser>) {
|
||||
return "Token(" + p.piece + ", " + std::to_string(p.token) + ")";
|
||||
} else if constexpr (std::is_same_v<T, common_peg_token_not_parser>) {
|
||||
std::vector<std::string> parts;
|
||||
for (auto token : p.tokens) {
|
||||
parts.push_back(std::to_string(token));
|
||||
}
|
||||
return "TokenNot(" + string_join(parts, ", ") + ")";
|
||||
} else if constexpr (std::is_same_v<T, common_peg_sequence_parser>) {
|
||||
std::vector<std::string> parts;
|
||||
for (const auto & child : p.children) {
|
||||
@@ -1171,6 +981,8 @@ std::string common_peg_arena::dump_impl(common_peg_parser_id
|
||||
return "Atomic(" + dump_impl(p.child, visited) + ")";
|
||||
} else if constexpr (std::is_same_v<T, common_peg_gbnf_parser>) {
|
||||
return "Gbnf(" + p.grammar + ", " + dump_impl(p.child, visited) + ")";
|
||||
} else if constexpr (std::is_same_v<T, common_peg_ac_parser>) {
|
||||
return "Ac(" + string_join(p.delimiters, " | ") + ", " + dump_impl(p.child, visited) + ")";
|
||||
} else if constexpr (std::is_same_v<T, common_peg_any_parser>) {
|
||||
return "Any";
|
||||
} else if constexpr (std::is_same_v<T, common_peg_space_parser>) {
|
||||
@@ -1183,18 +995,7 @@ std::string common_peg_arena::dump_impl(common_peg_parser_id
|
||||
} else if constexpr (std::is_same_v<T, common_peg_string_parser>) {
|
||||
return "String(" + std::string(1, p.delimiter) + ")";
|
||||
} else if constexpr (std::is_same_v<T, common_peg_until_parser>) {
|
||||
std::vector<std::string> branches;
|
||||
for (const auto & branch : p.branches) {
|
||||
auto text = dump_impl(branch.delimiter, visited);
|
||||
if (branch.rest != COMMON_PEG_INVALID_PARSER_ID) {
|
||||
text += " " + dump_impl(branch.rest, visited);
|
||||
}
|
||||
branches.push_back(std::move(text));
|
||||
}
|
||||
if (p.include) {
|
||||
return "Until(" + dump_impl(p.content, visited) + ", " + string_join(branches, " | ") + (p.optional ? ", optional" : "") + ")";
|
||||
}
|
||||
return "Until(" + string_join(branches, " | ") + ")";
|
||||
return "Until(" + string_join(p.delimiters, " | ") + ")";
|
||||
} else if constexpr (std::is_same_v<T, common_peg_schema_parser>) {
|
||||
return "Schema(" + dump_impl(p.child, visited) + ", " + (p.node ? common_chat_schema::kind_name(p.node->kind()) : "null") + ")";
|
||||
} else if constexpr (std::is_same_v<T, common_peg_rule_parser>) {
|
||||
@@ -1288,67 +1089,6 @@ common_peg_parser operator|(const std::string & str, const common_peg_parser & p
|
||||
|
||||
common_peg_parser_builder::common_peg_parser_builder() {}
|
||||
|
||||
common_peg_parser_builder::common_peg_parser_builder(common_peg_tokens tokens) {
|
||||
arena_.tokens_ = std::move(tokens);
|
||||
}
|
||||
|
||||
common_peg_parser common_peg_parser_builder::until_one_of(const std::vector<std::string> & delimiters) {
|
||||
common_peg_until_parser p;
|
||||
for (const auto & d : delimiters) {
|
||||
p.branches.push_back({ literal(d).id() });
|
||||
}
|
||||
p.matcher = build_until_matcher(arena_, p.branches, p.include);
|
||||
return add(std::move(p));
|
||||
}
|
||||
|
||||
common_peg_parser common_peg_parser_builder::until(const common_peg_parser & delimiter) {
|
||||
common_peg_until_parser p;
|
||||
p.branches = { { delimiter.id() } };
|
||||
p.matcher = build_until_matcher(arena_, p.branches, p.include);
|
||||
return add(std::move(p));
|
||||
}
|
||||
|
||||
common_peg_parser common_peg_parser_builder::through(const common_peg_parser & content, const std::vector<common_peg_until_branch> & branches, bool optional) {
|
||||
if (branches.empty()) {
|
||||
throw std::runtime_error("through requires at least one branch");
|
||||
}
|
||||
common_peg_until_parser p;
|
||||
p.branches = branches;
|
||||
p.include = true;
|
||||
p.optional = optional;
|
||||
p.content = content.id();
|
||||
p.matcher = build_until_matcher(arena_, p.branches, p.include);
|
||||
return add(std::move(p));
|
||||
}
|
||||
|
||||
common_peg_parser common_peg_parser_builder::token(const std::string & piece) {
|
||||
auto token = arena_.tokens_.token_id(piece);
|
||||
if (token == LLAMA_TOKEN_NULL) {
|
||||
return literal(piece);
|
||||
}
|
||||
return add(common_peg_token_parser{token, piece});
|
||||
}
|
||||
|
||||
common_peg_parser common_peg_parser_builder::token(const std::vector<std::string> & pieces, bool negate) {
|
||||
if (!negate) {
|
||||
std::vector<common_peg_parser> alts;
|
||||
for (const auto & piece : pieces) {
|
||||
alts.push_back(token(piece));
|
||||
}
|
||||
return choice(alts);
|
||||
}
|
||||
|
||||
common_peg_token_not_parser p;
|
||||
for (const auto & piece : pieces) {
|
||||
auto token = arena_.tokens_.token_id(piece);
|
||||
if (token == LLAMA_TOKEN_NULL) {
|
||||
throw std::invalid_argument("token is not registered with the builder: " + piece);
|
||||
}
|
||||
p.tokens.push_back(token);
|
||||
}
|
||||
return add(p);
|
||||
}
|
||||
|
||||
common_peg_parser common_peg_parser_builder::sequence(const std::vector<common_peg_parser_id> & parsers) {
|
||||
// Flatten nested sequences
|
||||
std::vector<common_peg_parser_id> flattened;
|
||||
@@ -1455,17 +1195,6 @@ common_peg_parser common_peg_parser_builder::rule(const std::string & name, cons
|
||||
return ref(clean_name);
|
||||
}
|
||||
|
||||
common_peg_parser common_peg_parser_builder::trigger_rule(const std::string & name, const common_peg_parser & start, const common_peg_parser & rest) {
|
||||
common_trie scratch;
|
||||
if (insert_delimiter(scratch, arena_, start.id()).count(0)) {
|
||||
throw std::invalid_argument("trigger start must not match the empty string");
|
||||
}
|
||||
auto clean_name = gbnf_format_rule_name(name);
|
||||
auto rule_id = arena_.add_parser(common_peg_rule_parser{clean_name, (start + rest).id(), true, start.id(), rest.id()});
|
||||
arena_.add_rule(clean_name, rule_id);
|
||||
return ref(clean_name);
|
||||
}
|
||||
|
||||
void common_peg_parser_builder::set_root(const common_peg_parser & p) {
|
||||
arena_.set_root(p.id());
|
||||
}
|
||||
@@ -1663,6 +1392,13 @@ common_peg_parser common_peg_parser_builder::json_member(const std::string & key
|
||||
});
|
||||
}
|
||||
|
||||
common_peg_parser common_peg_parser_builder::ac(const common_peg_parser & p, const std::vector<std::string> & delimiters) {
|
||||
if (delimiters.empty()) {
|
||||
throw std::runtime_error("ac parser requires at least one delimiter");
|
||||
}
|
||||
return add(common_peg_ac_parser{p, delimiters});
|
||||
}
|
||||
|
||||
static std::string gbnf_escape_char_class(uint32_t c) {
|
||||
if (c == '-' || c == ']' || c == '[' || c == '\\') {
|
||||
return "\\" + std::string(1, (char) c);
|
||||
@@ -1721,63 +1457,15 @@ static std::string gbnf_char_class(const std::vector<uint32_t> & chars, bool neg
|
||||
return s + "]";
|
||||
}
|
||||
|
||||
// Token set such as <[1,5-9]>, with consecutive ids collapsed into ranges
|
||||
static std::string gbnf_token_set(const std::vector<uint32_t> & tokens, bool negate) {
|
||||
std::string s = negate ? "!<[" : "<[";
|
||||
for (size_t i = 0; i < tokens.size();) {
|
||||
size_t j = i;
|
||||
while (j + 1 < tokens.size() && tokens[j + 1] == tokens[j] + 1) {
|
||||
j++;
|
||||
}
|
||||
s += (i == 0 ? "" : ",") + std::to_string(tokens[i]);
|
||||
if (j > i) {
|
||||
s += "-" + std::to_string(tokens[j]);
|
||||
}
|
||||
i = j + 1;
|
||||
}
|
||||
return s + "]>";
|
||||
}
|
||||
|
||||
// Symbols on the transitions out of an automaton state, codepoints and tokens kept apart
|
||||
struct gbnf_ac_symbols {
|
||||
std::vector<uint32_t> chars;
|
||||
std::vector<uint32_t> tokens;
|
||||
|
||||
void add(const common_trie::symbol & sym) {
|
||||
(sym.is_token() ? tokens : chars).push_back(sym.value);
|
||||
}
|
||||
|
||||
// One alternative for the codepoints and one for the tokens, each followed by next if given
|
||||
std::vector<std::string> alternatives(const std::string & next = "") const {
|
||||
std::vector<std::string> alts;
|
||||
const std::string tail = next.empty() ? "" : " " + next;
|
||||
if (!chars.empty()) {
|
||||
alts.push_back(gbnf_char_class(chars, false) + tail);
|
||||
}
|
||||
if (!tokens.empty()) {
|
||||
alts.push_back(gbnf_token_set(tokens, false) + tail);
|
||||
}
|
||||
return alts;
|
||||
}
|
||||
};
|
||||
|
||||
static std::string gbnf_ac_grammar(
|
||||
const common_grammar_builder & builder,
|
||||
const std::string & prefix,
|
||||
const common_trie & trie,
|
||||
const std::set<uint32_t> & opaque,
|
||||
const std::function<std::string(const gbnf_ac_symbols &,
|
||||
const std::map<size_t, gbnf_ac_symbols> &,
|
||||
const std::map<size_t, gbnf_ac_symbols> &,
|
||||
const std::string &,
|
||||
const common_grammar_builder & builder,
|
||||
const std::string & prefix,
|
||||
const std::vector<std::string> & strings,
|
||||
const std::function<std::string(const std::vector<uint32_t> &,
|
||||
const std::map<size_t, std::vector<uint32_t>> &,
|
||||
const std::vector<uint32_t> &,
|
||||
const std::function<std::string(size_t)> &)> & build_rule) {
|
||||
common_aho_corasick ac(trie);
|
||||
|
||||
// With only tokens in the alphabet, the grammar can step a whole token at a time
|
||||
bool token_only = true;
|
||||
for (const auto & sym : ac.alphabet) {
|
||||
token_only = token_only && sym.is_token();
|
||||
}
|
||||
common_aho_corasick ac(strings);
|
||||
|
||||
auto state_name = [&](size_t s) -> std::string {
|
||||
if (s == 0) {
|
||||
@@ -1793,43 +1481,27 @@ static std::string gbnf_ac_grammar(
|
||||
continue; // match states
|
||||
}
|
||||
|
||||
std::map<size_t, gbnf_ac_symbols> buckets;
|
||||
std::map<size_t, gbnf_ac_symbols> completing_to; // completing symbols by the match state they reach
|
||||
gbnf_ac_symbols completing; // symbols that complete a delimiter
|
||||
gbnf_ac_symbols specific; // symbols with an explicit transition
|
||||
for (const auto & sym : ac.alphabet) {
|
||||
size_t d = ac.next(q, sym);
|
||||
std::map<size_t, std::vector<uint32_t>> buckets;
|
||||
std::vector<uint32_t> completing; // chars that complete a delimiter
|
||||
std::vector<uint32_t> specific; // chars with an explicit transition
|
||||
for (uint32_t c : ac.alphabet) {
|
||||
size_t d = ac.next(q, c);
|
||||
if (ac.is_terminal(d)) {
|
||||
completing.add(sym);
|
||||
completing_to[d].add(sym);
|
||||
specific.add(sym);
|
||||
completing.push_back(c);
|
||||
specific.push_back(c);
|
||||
} else if (d != 0) {
|
||||
buckets[d].add(sym); // specific non-root destination
|
||||
specific.add(sym);
|
||||
buckets[d].push_back(c); // specific non-root destination
|
||||
specific.push_back(c);
|
||||
}
|
||||
}
|
||||
|
||||
// every other symbol goes back to the start
|
||||
std::string fallback;
|
||||
if (token_only) {
|
||||
fallback = gbnf_token_set(specific.tokens, true);
|
||||
} else {
|
||||
fallback = specific.chars.empty() ? "." : gbnf_char_class(specific.chars, true);
|
||||
builder.add_rule(state_name(q), build_rule(completing, buckets, specific, state_name));
|
||||
}
|
||||
|
||||
// Char rules never match a token named by a token rule, so allow the other named tokens by name
|
||||
std::vector<uint32_t> others;
|
||||
for (auto token : opaque) {
|
||||
if (std::find(specific.tokens.begin(), specific.tokens.end(), token) == specific.tokens.end()) {
|
||||
others.push_back(token);
|
||||
}
|
||||
}
|
||||
if (!others.empty()) {
|
||||
fallback = "(" + fallback + " | " + gbnf_token_set(others, false) + ")";
|
||||
}
|
||||
}
|
||||
fallback += " " + state_name(0);
|
||||
|
||||
builder.add_rule(state_name(q), build_rule(completing, completing_to, buckets, fallback, state_name));
|
||||
// An empty delimiter makes the start state terminal. Emit an entry rule
|
||||
// that matches the empty string so the returned reference stays valid.
|
||||
if (ac.is_terminal(0)) {
|
||||
builder.add_rule(prefix, "|");
|
||||
}
|
||||
|
||||
return state_name(0);
|
||||
@@ -1842,104 +1514,52 @@ static std::string gbnf_ac_grammar(
|
||||
// ref: https://github.com/ggml-org/llama.cpp/pull/24839
|
||||
static std::string gbnf_excluding_grammar(const common_grammar_builder & builder,
|
||||
const std::string & prefix,
|
||||
const common_trie & trie,
|
||||
const std::set<uint32_t> & opaque) {
|
||||
return gbnf_ac_grammar(builder, prefix, trie, opaque,
|
||||
[](const gbnf_ac_symbols & /*completing*/,
|
||||
const std::map<size_t, gbnf_ac_symbols> & /*completing_to*/,
|
||||
const std::map<size_t, gbnf_ac_symbols> & buckets,
|
||||
const std::string & fallback,
|
||||
const std::vector<std::string> & strings) {
|
||||
return gbnf_ac_grammar(builder, prefix, strings,
|
||||
[](const std::vector<uint32_t> & /*completing*/,
|
||||
const std::map<size_t, std::vector<uint32_t>> & buckets,
|
||||
const std::vector<uint32_t> & specific,
|
||||
const std::function<std::string(size_t)> & state_name) {
|
||||
// every state accepts and completing symbols get no alternative, so a forbidden string never matches
|
||||
// every state is accepting and completing chars get no
|
||||
// alternative, so a forbidden string can never be matched
|
||||
std::string rhs = "|";
|
||||
for (const auto & [d, symbols] : buckets) {
|
||||
for (const auto & alt : symbols.alternatives(state_name(d))) {
|
||||
rhs += " " + alt + " |";
|
||||
}
|
||||
for (const auto & [d, chars] : buckets) {
|
||||
rhs += " " + gbnf_char_class(chars, false) + " " + state_name(d) + " |";
|
||||
}
|
||||
rhs += " " + fallback;
|
||||
rhs += " " + gbnf_char_class(specific, true) + " " + state_name(0);
|
||||
return rhs;
|
||||
});
|
||||
}
|
||||
|
||||
// GBNF grammar matching up to the first pattern in the trie, then the rest rule of that pattern's value.
|
||||
// For the pattern "ab" with rest "x":
|
||||
//
|
||||
// prefix ::= [a] prefix-01 | [^a] prefix
|
||||
// prefix-01 ::= [b] rest | [a] prefix-01 | [^ab] prefix
|
||||
// rest ::= "x"
|
||||
static std::string gbnf_including_grammar(const common_grammar_builder & builder,
|
||||
const std::string & prefix,
|
||||
const common_trie & trie,
|
||||
const std::set<uint32_t> & opaque,
|
||||
const std::vector<std::string> & rests,
|
||||
bool optional) {
|
||||
common_aho_corasick ac(trie);
|
||||
|
||||
// every pattern ending at a match state, the longest and the ones reached by suffix links
|
||||
auto rests_at = [&](size_t s, bool & terminal) {
|
||||
std::vector<std::string> alts;
|
||||
terminal = false;
|
||||
for (; s != 0; s = ac.fail[s]) {
|
||||
auto pattern = ac.t.nodes[s].pattern;
|
||||
if (pattern < 0) {
|
||||
continue;
|
||||
}
|
||||
if (rests[pattern].empty()) {
|
||||
terminal = true;
|
||||
} else if (std::find(alts.begin(), alts.end(), rests[pattern]) == alts.end()) {
|
||||
alts.push_back(rests[pattern]);
|
||||
}
|
||||
}
|
||||
return alts;
|
||||
};
|
||||
|
||||
return gbnf_ac_grammar(builder, prefix, trie, opaque,
|
||||
[&](const gbnf_ac_symbols & /*completing*/,
|
||||
const std::map<size_t, gbnf_ac_symbols> & completing_to,
|
||||
const std::map<size_t, gbnf_ac_symbols> & buckets,
|
||||
const std::string & fallback,
|
||||
const std::function<std::string(size_t)> & state_name) {
|
||||
// GBNF grammar matching everything up to and including the first occurrence of
|
||||
// any string in `strings`. Emits the Aho-Corasick automaton DFA and returns
|
||||
// the start state rule name.
|
||||
static std::string gbnf_including_grammar(const common_grammar_builder & builder,
|
||||
const std::string & prefix,
|
||||
const std::vector<std::string> & strings) {
|
||||
return gbnf_ac_grammar(builder, prefix, strings,
|
||||
[](const std::vector<uint32_t> & completing,
|
||||
const std::map<size_t, std::vector<uint32_t>> & buckets,
|
||||
const std::vector<uint32_t> & specific,
|
||||
const std::function<std::string(size_t)> & state_name) {
|
||||
std::vector<std::string> alts;
|
||||
gbnf_ac_symbols terminating; // symbols that complete a pattern with no rest
|
||||
for (const auto & [d, symbols] : completing_to) {
|
||||
bool terminal = false;
|
||||
auto rest = rests_at(d, terminal);
|
||||
if (!rest.empty()) {
|
||||
for (auto & alt : symbols.alternatives(rest.size() == 1 ? rest[0] : "(" + string_join(rest, " | ") + ")")) {
|
||||
alts.push_back(std::move(alt));
|
||||
}
|
||||
}
|
||||
if (terminal) {
|
||||
terminating.chars.insert(terminating.chars.end(), symbols.chars.begin(), symbols.chars.end());
|
||||
terminating.tokens.insert(terminating.tokens.end(), symbols.tokens.begin(), symbols.tokens.end());
|
||||
}
|
||||
if (!completing.empty()) {
|
||||
alts.push_back(gbnf_char_class(completing, false)); // terminate on match
|
||||
}
|
||||
for (auto & alt : terminating.alternatives()) {
|
||||
alts.push_back(std::move(alt));
|
||||
for (const auto & [d, chars] : buckets) {
|
||||
alts.push_back(gbnf_char_class(chars, false) + " " + state_name(d));
|
||||
}
|
||||
for (const auto & [d, symbols] : buckets) {
|
||||
for (auto & alt : symbols.alternatives(state_name(d))) {
|
||||
alts.push_back(std::move(alt));
|
||||
}
|
||||
}
|
||||
// every other symbol keeps scanning from the start state
|
||||
alts.push_back(fallback);
|
||||
return (optional ? "| " : "") + string_join(alts, " | ");
|
||||
// every other character keeps scanning from the start state
|
||||
alts.push_back(gbnf_char_class(specific, true) + " " + state_name(0));
|
||||
return string_join(alts, " | ");
|
||||
});
|
||||
}
|
||||
|
||||
// The rules and the tokens named by token rules that the grammar of a parser reaches
|
||||
struct common_peg_reachable {
|
||||
std::set<std::string> rules;
|
||||
std::set<uint32_t> tokens;
|
||||
};
|
||||
|
||||
static void collect_reachable(
|
||||
static std::set<std::string> collect_reachable_rules(
|
||||
const common_peg_arena & arena,
|
||||
const common_peg_parser_id & root,
|
||||
common_peg_reachable & reachable
|
||||
const common_peg_parser_id & rule
|
||||
) {
|
||||
std::set<std::string> reachable;
|
||||
std::set<std::string> visited;
|
||||
|
||||
std::function<void(common_peg_parser_id)> visit = [&](common_peg_parser_id id) {
|
||||
@@ -1951,27 +1571,13 @@ static void collect_reachable(
|
||||
if constexpr (std::is_same_v<T, common_peg_epsilon_parser> ||
|
||||
std::is_same_v<T, common_peg_start_parser> ||
|
||||
std::is_same_v<T, common_peg_end_parser> ||
|
||||
std::is_same_v<T, common_peg_until_parser> ||
|
||||
std::is_same_v<T, common_peg_literal_parser> ||
|
||||
std::is_same_v<T, common_peg_chars_parser> ||
|
||||
std::is_same_v<T, common_peg_space_parser> ||
|
||||
std::is_same_v<T, common_peg_any_parser> ||
|
||||
std::is_same_v<T, common_peg_string_parser>) {
|
||||
// These parsers do not have any children
|
||||
} else if constexpr (std::is_same_v<T, common_peg_token_parser>) {
|
||||
reachable.tokens.insert(p.token);
|
||||
} else if constexpr (std::is_same_v<T, common_peg_token_not_parser>) {
|
||||
reachable.tokens.insert(p.tokens.begin(), p.tokens.end());
|
||||
} else if constexpr (std::is_same_v<T, common_peg_until_parser>) {
|
||||
// The delimiters name their tokens in the grammar
|
||||
if (p.content != COMMON_PEG_INVALID_PARSER_ID) {
|
||||
visit(p.content);
|
||||
}
|
||||
for (const auto & branch : p.branches) {
|
||||
visit(branch.delimiter);
|
||||
if (branch.rest != COMMON_PEG_INVALID_PARSER_ID) {
|
||||
visit(branch.rest);
|
||||
}
|
||||
}
|
||||
} else if constexpr (std::is_same_v<T, common_peg_sequence_parser>) {
|
||||
for (auto child : p.children) {
|
||||
visit(child);
|
||||
@@ -1986,12 +1592,13 @@ static void collect_reachable(
|
||||
std::is_same_v<T, common_peg_tag_parser> ||
|
||||
std::is_same_v<T, common_peg_atomic_parser> ||
|
||||
std::is_same_v<T, common_peg_gbnf_parser> ||
|
||||
std::is_same_v<T, common_peg_ac_parser> ||
|
||||
std::is_same_v<T, common_peg_schema_parser>) {
|
||||
visit(p.child);
|
||||
} else if constexpr (std::is_same_v<T, common_peg_rule_parser>) {
|
||||
if (visited.find(p.name) == visited.end()) {
|
||||
visited.insert(p.name);
|
||||
reachable.rules.insert(p.name);
|
||||
reachable.insert(p.name);
|
||||
visit(p.child);
|
||||
}
|
||||
} else if constexpr (std::is_same_v<T, common_peg_ref_parser>) {
|
||||
@@ -2004,13 +1611,12 @@ static void collect_reachable(
|
||||
}, parser);
|
||||
};
|
||||
|
||||
visit(root);
|
||||
visit(rule);
|
||||
return reachable;
|
||||
}
|
||||
|
||||
// GBNF generation implementation
|
||||
void common_peg_arena::build_grammar(const common_grammar_builder & builder, bool lazy) const {
|
||||
common_peg_reachable reachable;
|
||||
|
||||
// A raw string value is parsed by the child rather than constrained by the schema
|
||||
auto schema_delegates = [](const common_peg_schema_parser & s) -> bool {
|
||||
return !s.node || (s.raw && s.node->may_be_string());
|
||||
@@ -2049,10 +1655,6 @@ void common_peg_arena::build_grammar(const common_grammar_builder & builder, boo
|
||||
return "";
|
||||
} else if constexpr (std::is_same_v<T, common_peg_literal_parser>) {
|
||||
return gbnf_format_literal(p.literal);
|
||||
} else if constexpr (std::is_same_v<T, common_peg_token_parser>) {
|
||||
return "<[" + std::to_string(p.token) + "]>";
|
||||
} else if constexpr (std::is_same_v<T, common_peg_token_not_parser>) {
|
||||
return gbnf_token_set(std::vector<uint32_t>(p.tokens.begin(), p.tokens.end()), true);
|
||||
} else if constexpr (std::is_same_v<T, common_peg_sequence_parser>) {
|
||||
std::string s;
|
||||
for (const auto & child : p.children) {
|
||||
@@ -2144,19 +1746,10 @@ void common_peg_arena::build_grammar(const common_grammar_builder & builder, boo
|
||||
const std::string delim(1, p.delimiter);
|
||||
return R"(( [^)" + delim + R"(\\] | "\\" ( [)" + delim + R"(\\/ bfnrt] | "u" [0-9a-fA-F]{4} ) )*)";
|
||||
} else if constexpr (std::is_same_v<T, common_peg_until_parser>) {
|
||||
auto prefix = "until-" + std::to_string(id);
|
||||
if (!p.include) {
|
||||
if (p.branches.empty()) {
|
||||
return ".*";
|
||||
}
|
||||
return gbnf_excluding_grammar(builder, prefix, p.matcher, reachable.tokens);
|
||||
if (p.delimiters.empty()) {
|
||||
return ".*";
|
||||
}
|
||||
std::vector<std::string> branch_rests;
|
||||
for (size_t b = 0; b < p.branches.size(); b++) {
|
||||
auto rest = p.branches[b].rest;
|
||||
branch_rests.push_back(rest == COMMON_PEG_INVALID_PARSER_ID ? "" : builder.add_rule(prefix + "-rest-" + std::to_string(b), to_gbnf(rest)));
|
||||
}
|
||||
return gbnf_including_grammar(builder, prefix, p.matcher, reachable.tokens, branch_rests, p.optional);
|
||||
return gbnf_excluding_grammar(builder, "until-" + std::to_string(id), p.delimiters);
|
||||
} else if constexpr (std::is_same_v<T, common_peg_schema_parser>) {
|
||||
if (schema_delegates(p)) {
|
||||
return to_gbnf(p.child);
|
||||
@@ -2173,52 +1766,38 @@ void common_peg_arena::build_grammar(const common_grammar_builder & builder, boo
|
||||
return to_gbnf(p.child);
|
||||
} else if constexpr (std::is_same_v<T, common_peg_gbnf_parser>) {
|
||||
return p.grammar;
|
||||
} else if constexpr (std::is_same_v<T, common_peg_ac_parser>) {
|
||||
return gbnf_including_grammar(builder, "ac-" + std::to_string(id), p.delimiters);
|
||||
} else {
|
||||
static_assert(is_always_false_v<T>);
|
||||
}
|
||||
}, parser);
|
||||
};
|
||||
|
||||
// Trigger rules, sorted for a predictable order
|
||||
std::vector<const common_peg_rule_parser *> trigger_rules;
|
||||
for (const auto & [name, id] : rules_) {
|
||||
if (auto rule = std::get_if<common_peg_rule_parser>(&parsers_.at(id))) {
|
||||
if (rule->trigger) {
|
||||
trigger_rules.push_back(rule);
|
||||
}
|
||||
}
|
||||
}
|
||||
std::sort(trigger_rules.begin(), trigger_rules.end(), [](const auto * a, const auto * b) { return a->name < b->name; });
|
||||
// Collect reachable rules
|
||||
std::set<std::string> reachable_rules;
|
||||
|
||||
// A lazy grammar scans for the starts when the trigger rules were built from a start and rest
|
||||
bool scans = false;
|
||||
for (const auto * rule : trigger_rules) {
|
||||
scans = scans || rule->start != COMMON_PEG_INVALID_PARSER_ID;
|
||||
}
|
||||
if (scans) {
|
||||
for (const auto * rule : trigger_rules) {
|
||||
if (rule->start == COMMON_PEG_INVALID_PARSER_ID) {
|
||||
throw std::runtime_error("trigger rule without a start in a scanning grammar: " + rule->name);
|
||||
if (lazy) {
|
||||
// Collect rules reachable from trigger rules
|
||||
for (const auto & [name, id] : rules_) {
|
||||
const auto & parser = parsers_.at(id);
|
||||
if (auto rule = std::get_if<common_peg_rule_parser>(&parser)) {
|
||||
if (rule->trigger) {
|
||||
// Mark trigger as reachable and visit it
|
||||
reachable_rules.insert(name);
|
||||
auto add_rules = collect_reachable_rules(*this, id);
|
||||
reachable_rules.insert(add_rules.begin(), add_rules.end());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (lazy && scans) {
|
||||
for (const auto * rule : trigger_rules) {
|
||||
collect_reachable(*this, rule->start, reachable);
|
||||
collect_reachable(*this, rule->rest, reachable);
|
||||
}
|
||||
} else if (lazy) {
|
||||
for (const auto * rule : trigger_rules) {
|
||||
collect_reachable(*this, rules_.at(rule->name), reachable);
|
||||
}
|
||||
} else {
|
||||
collect_reachable(*this, root_, reachable);
|
||||
// Collect rules reachable from root
|
||||
reachable_rules = collect_reachable_rules(*this, root_);
|
||||
}
|
||||
|
||||
// Create GBNF rules for all reachable rules
|
||||
for (const auto & [name, rule_id] : rules_) {
|
||||
if (reachable.rules.find(name) == reachable.rules.end()) {
|
||||
if (reachable_rules.find(name) == reachable_rules.end()) {
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -2228,27 +1807,20 @@ void common_peg_arena::build_grammar(const common_grammar_builder & builder, boo
|
||||
}
|
||||
}
|
||||
|
||||
if (lazy && scans) {
|
||||
// The root scans for a trigger start and then constrains its rest
|
||||
common_trie starts;
|
||||
std::vector<std::string> rests;
|
||||
for (const auto * rule : trigger_rules) {
|
||||
auto value = (int32_t) rests.size();
|
||||
rests.push_back(builder.add_rule(rule->name + "-rest", to_gbnf(rule->rest)));
|
||||
for (size_t end : insert_delimiter(starts, *this, rule->start)) {
|
||||
auto owner = starts.mark(end, value);
|
||||
if (owner != value) {
|
||||
throw std::runtime_error("trigger rules " + trigger_rules[owner]->name + " and " + rule->name + " must not share a start");
|
||||
if (lazy) {
|
||||
// Generate root rule from trigger rules only
|
||||
std::vector<std::string> trigger_names;
|
||||
for (const auto & [name, rule_id] : rules_) {
|
||||
const auto & parser = parsers_.at(rule_id);
|
||||
if (auto rule = std::get_if<common_peg_rule_parser>(&parser)) {
|
||||
if (rule->trigger) {
|
||||
trigger_names.push_back(rule->name);
|
||||
}
|
||||
}
|
||||
}
|
||||
builder.add_rule("root", gbnf_including_grammar(builder, "trigger", starts, reachable.tokens, rests, /* optional = */ true));
|
||||
} else if (lazy) {
|
||||
// Generate root rule from trigger rules only
|
||||
std::vector<std::string> trigger_names;
|
||||
for (const auto * rule : trigger_rules) {
|
||||
trigger_names.push_back(rule->name);
|
||||
}
|
||||
|
||||
// Sort for predictable order
|
||||
std::sort(trigger_names.begin(), trigger_names.end());
|
||||
builder.add_rule("root", string_join(trigger_names, " | "));
|
||||
} else if (root_ != COMMON_PEG_INVALID_PARSER_ID) {
|
||||
builder.add_rule("root", to_gbnf(root_));
|
||||
|
||||
+17
-77
@@ -3,7 +3,6 @@
|
||||
#include "json-schema.h"
|
||||
#include "json.h"
|
||||
#include "llama.h"
|
||||
#include "trie.h"
|
||||
|
||||
#include <memory>
|
||||
#include <set>
|
||||
@@ -189,22 +188,6 @@ struct common_peg_recursion_error : std::invalid_argument {
|
||||
using std::invalid_argument::invalid_argument;
|
||||
};
|
||||
|
||||
struct common_peg_token {
|
||||
std::string text;
|
||||
llama_token_attr attr;
|
||||
};
|
||||
|
||||
struct common_peg_tokens {
|
||||
std::unordered_map<llama_token, common_peg_token> tokens;
|
||||
std::unordered_map<std::string, llama_token> ids;
|
||||
|
||||
common_peg_tokens() = default;
|
||||
explicit common_peg_tokens(const llama_vocab * vocab);
|
||||
|
||||
// LLAMA_TOKEN_NULL if text is not a special token
|
||||
llama_token token_id(const std::string & text) const;
|
||||
};
|
||||
|
||||
struct common_peg_parse_context {
|
||||
std::string input; // [h, e, l, l, o, _, w, o, r, l, d]
|
||||
std::vector<llama_token> tokens; // [id, -1, -1, -1, -1, id, -1, -1, -1, -1, -1]
|
||||
@@ -243,16 +226,6 @@ struct common_peg_literal_parser {
|
||||
std::string literal;
|
||||
};
|
||||
|
||||
struct common_peg_token_parser {
|
||||
llama_token token;
|
||||
std::string piece;
|
||||
};
|
||||
|
||||
// Matches one token that is not one of the tokens
|
||||
struct common_peg_token_not_parser {
|
||||
std::vector<llama_token> tokens;
|
||||
};
|
||||
|
||||
struct common_peg_sequence_parser {
|
||||
std::vector<common_peg_parser_id> children;
|
||||
};
|
||||
@@ -297,18 +270,8 @@ struct common_peg_string_parser {
|
||||
char delimiter;
|
||||
};
|
||||
|
||||
// A delimiter an until parser stops at, and the parser that follows it when the delimiter is consumed
|
||||
struct common_peg_until_branch {
|
||||
common_peg_parser_id delimiter;
|
||||
common_peg_parser_id rest = COMMON_PEG_INVALID_PARSER_ID;
|
||||
};
|
||||
|
||||
struct common_peg_until_parser {
|
||||
std::vector<common_peg_until_branch> branches;
|
||||
bool include = false; // consume the first delimiter and parse the rest of its branch
|
||||
bool optional = false; // with include, the input may end before any delimiter
|
||||
common_peg_parser_id content = COMMON_PEG_INVALID_PARSER_ID; // with include, parsed up to the delimiter
|
||||
common_trie matcher; // each pattern valued by its branch
|
||||
std::vector<std::string> delimiters;
|
||||
};
|
||||
|
||||
struct common_peg_schema_parser {
|
||||
@@ -325,9 +288,6 @@ struct common_peg_rule_parser {
|
||||
std::string name;
|
||||
common_peg_parser_id child;
|
||||
bool trigger;
|
||||
// set when the rule has a start and rest, the lazy grammar scans for the start
|
||||
common_peg_parser_id start = COMMON_PEG_INVALID_PARSER_ID;
|
||||
common_peg_parser_id rest = COMMON_PEG_INVALID_PARSER_ID;
|
||||
};
|
||||
|
||||
struct common_peg_ref_parser {
|
||||
@@ -348,14 +308,17 @@ struct common_peg_gbnf_parser {
|
||||
std::string grammar;
|
||||
};
|
||||
|
||||
struct common_peg_ac_parser {
|
||||
common_peg_parser_id child;
|
||||
std::vector<std::string> delimiters;
|
||||
};
|
||||
|
||||
// Variant holding all parser types
|
||||
using common_peg_parser_variant = std::variant<
|
||||
common_peg_epsilon_parser,
|
||||
common_peg_start_parser,
|
||||
common_peg_end_parser,
|
||||
common_peg_literal_parser,
|
||||
common_peg_token_parser,
|
||||
common_peg_token_not_parser,
|
||||
common_peg_sequence_parser,
|
||||
common_peg_choice_parser,
|
||||
common_peg_repetition_parser,
|
||||
@@ -371,14 +334,14 @@ using common_peg_parser_variant = std::variant<
|
||||
common_peg_ref_parser,
|
||||
common_peg_atomic_parser,
|
||||
common_peg_tag_parser,
|
||||
common_peg_gbnf_parser
|
||||
common_peg_gbnf_parser,
|
||||
common_peg_ac_parser
|
||||
>;
|
||||
|
||||
class common_peg_arena {
|
||||
std::vector<common_peg_parser_variant> parsers_;
|
||||
std::unordered_map<std::string, common_peg_parser_id> rules_;
|
||||
common_peg_parser_id root_ = COMMON_PEG_INVALID_PARSER_ID;
|
||||
common_peg_tokens tokens_;
|
||||
|
||||
public:
|
||||
const common_peg_parser_variant & get(common_peg_parser_id id) const { return parsers_.at(id); }
|
||||
@@ -393,8 +356,6 @@ class common_peg_arena {
|
||||
common_peg_parser_id root() const { return root_; }
|
||||
void set_root(common_peg_parser_id id) { root_ = id; }
|
||||
|
||||
const common_peg_tokens & tokens() const { return tokens_; }
|
||||
|
||||
common_peg_parse_result parse(common_peg_parse_context & ctx, size_t start = 0) const;
|
||||
common_peg_parse_result parse(common_peg_parser_id id, common_peg_parse_context & ctx, size_t start) const;
|
||||
|
||||
@@ -423,7 +384,6 @@ class common_peg_parser_builder {
|
||||
|
||||
public:
|
||||
common_peg_parser_builder();
|
||||
explicit common_peg_parser_builder(common_peg_tokens tokens);
|
||||
|
||||
// Match nothing, always succeed.
|
||||
// S -> ε
|
||||
@@ -441,16 +401,6 @@ class common_peg_parser_builder {
|
||||
// S -> "hello"
|
||||
common_peg_parser literal(const std::string & literal) { return add(common_peg_literal_parser{literal}); }
|
||||
|
||||
// Matches a token. Falls back to a literal if the token is not registered with the builder.
|
||||
// S -> <[token-id]>
|
||||
common_peg_parser token(const std::string & piece);
|
||||
|
||||
// Matches one of the tokens. With negate, matches one token that is not in the set.
|
||||
// A negated set may only name tokens registered with the builder.
|
||||
// S -> <[id1]> | <[id2]> | ... or !<[id1,id2,...]>
|
||||
common_peg_parser token(const std::vector<std::string> & pieces, bool negate = false);
|
||||
common_peg_parser token(std::initializer_list<std::string> pieces, bool negate = false) { return token(std::vector<std::string>(pieces), negate); }
|
||||
|
||||
// Matches a sequence of parsers in order, all must succeed.
|
||||
// S -> A B C
|
||||
common_peg_parser sequence() { return add(common_peg_sequence_parser{}); }
|
||||
@@ -507,29 +457,16 @@ class common_peg_parser_builder {
|
||||
// Matches all characters until a delimiter is found (delimiter not consumed).
|
||||
// Invalid UTF-8 is consumed and recorded on the AST nodes.
|
||||
// S -> (!delim .)*
|
||||
common_peg_parser until(const std::string & delimiter) { return until_one_of({delimiter}); }
|
||||
common_peg_parser until(const std::string & delimiter) { return add(common_peg_until_parser{{delimiter}}); }
|
||||
|
||||
// Matches all characters until one of the delimiters in the list is found (delimiter not consumed).
|
||||
// S -> (!delim .)*
|
||||
common_peg_parser until_one_of(const std::vector<std::string> & delimiters);
|
||||
|
||||
// Matches all characters until the delimiter is found (delimiter not consumed).
|
||||
// The delimiter may only be a sequence of literals, tokens, and choices of them. Tags and atomics may wrap them.
|
||||
// S -> (!delim .)*
|
||||
common_peg_parser until(const common_peg_parser & delimiter);
|
||||
common_peg_parser until_one_of(const std::vector<std::string> & delimiters) { return add(common_peg_until_parser{delimiters}); }
|
||||
|
||||
// Matches everything
|
||||
// S -> .*
|
||||
common_peg_parser rest() { return until_one_of({}); }
|
||||
|
||||
// Matches content, then the first delimiter, then the rest of the branch of that delimiter.
|
||||
// Branches must not share a delimiter. With optional, the input may end before any delimiter.
|
||||
// S -> C (D1 R1 | D2 R2 | ...)
|
||||
common_peg_parser through(const common_peg_parser & content, const std::vector<common_peg_until_branch> & branches, bool optional = false);
|
||||
common_peg_parser through(const common_peg_parser & content, std::initializer_list<common_peg_until_branch> branches, bool optional = false) { return through(content, std::vector<common_peg_until_branch>(branches), optional); }
|
||||
common_peg_parser through(const common_peg_parser & content, const common_peg_parser & delimiter) { return through(content, { { delimiter } }); }
|
||||
common_peg_parser through(const common_peg_parser & content, const std::string & delimiter) { return through(content, literal(delimiter)); }
|
||||
|
||||
// Matches between min and max repetitions of a parser (inclusive).
|
||||
// S -> A{m,n}
|
||||
// Use -1 for max to represent unbounded repetition (equivalent to {m,})
|
||||
@@ -600,10 +537,6 @@ class common_peg_parser_builder {
|
||||
common_peg_parser trigger_rule(const std::string & name, const common_peg_parser & p) { return rule(name, p, true); }
|
||||
common_peg_parser trigger_rule(const std::string & name, const std::function<common_peg_parser()> & builder) { return rule(name, builder, true); }
|
||||
|
||||
// Creates a trigger rule that matches start followed by rest. A lazy grammar scans for the start of any such rule, then only allows its rest.
|
||||
// The start has the same limits as an until() delimiter.
|
||||
common_peg_parser trigger_rule(const std::string & name, const common_peg_parser & start, const common_peg_parser & rest);
|
||||
|
||||
// Creates an atomic parser. Atomic parsers do not create an AST node if
|
||||
// the child results in a partial parse, i.e. NEEDS_MORE_INPUT. This is
|
||||
// intended for situations where partial output is undesirable.
|
||||
@@ -617,6 +550,13 @@ class common_peg_parser_builder {
|
||||
// the child's grammar. Parsing delegates entirely to the child.
|
||||
common_peg_parser gbnf(const common_peg_parser & p, const std::string & grammar) { return add(common_peg_gbnf_parser{p, grammar}); }
|
||||
|
||||
// Wraps a child parser but emits a GBNF grammar built from the Aho-Corasick
|
||||
// automaton of `delimiters`, matching everything up to and including the
|
||||
// first delimiter. Parsing delegates entirely to the child, which is
|
||||
// responsible for consuming the delimiter (e.g. until(D) + literal(D)).
|
||||
common_peg_parser ac(const common_peg_parser & p, const std::vector<std::string> & delimiters);
|
||||
common_peg_parser ac(const common_peg_parser & p, const std::string & delimiter) { return ac(p, std::vector<std::string>{delimiter}); }
|
||||
|
||||
void set_root(const common_peg_parser & p);
|
||||
|
||||
common_peg_arena build();
|
||||
|
||||
@@ -31,18 +31,14 @@ struct token_matcher {
|
||||
static common_trie build_trie(const std::vector<llama_tokens> & seqs) {
|
||||
common_trie t;
|
||||
for (const auto & seq : seqs) {
|
||||
std::vector<common_trie::symbol> symbols;
|
||||
for (llama_token token : seq) {
|
||||
symbols.push_back(common_trie::symbol::token(token));
|
||||
}
|
||||
t.insert(symbols);
|
||||
t.insert(std::vector<uint32_t>(seq.begin(), seq.end()));
|
||||
}
|
||||
return t;
|
||||
}
|
||||
|
||||
// returns the index into seqs of the longest sequence ending at this token, or -1
|
||||
int32_t advance(llama_token token) {
|
||||
state = ac.next(state, common_trie::symbol::token(token));
|
||||
state = ac.next(state, (uint32_t) token);
|
||||
const int32_t p = ac.match_pattern(state);
|
||||
if (p >= 0) {
|
||||
state = 0;
|
||||
|
||||
+20
-54
@@ -4,46 +4,26 @@
|
||||
|
||||
#include <deque>
|
||||
|
||||
common_trie::step common_trie::next(std::string_view sv, const std::vector<int32_t> & token_map, size_t pos) const {
|
||||
if (!tokens.empty() && pos < token_map.size()) {
|
||||
const int32_t id = token_map[pos];
|
||||
if (id >= 0 && tokens.find(id) != tokens.end()) {
|
||||
size_t end = pos + 1;
|
||||
while (end < token_map.size() && token_map[end] < 0) {
|
||||
++end;
|
||||
}
|
||||
return { utf8_parse_result::SUCCESS, symbol::token(id), end - pos };
|
||||
}
|
||||
}
|
||||
|
||||
auto result = common_parse_utf8_codepoint(sv, pos);
|
||||
return { result.status, symbol::codepoint(result.codepoint), result.bytes_consumed };
|
||||
}
|
||||
|
||||
common_trie::match_result common_trie::check_at(std::string_view sv, size_t start_pos) const {
|
||||
return check_at(sv, {}, start_pos);
|
||||
}
|
||||
|
||||
common_trie::match_result common_trie::check_at(std::string_view sv, const std::vector<int32_t> & token_map, size_t start_pos) const {
|
||||
size_t current = 0; // Start at root
|
||||
size_t pos = start_pos;
|
||||
|
||||
// LOG_DBG("%s: checking at pos %zu, sv='%s'\n", __func__, start_pos, std::string(sv).c_str());
|
||||
|
||||
while (pos < sv.size()) {
|
||||
auto step = next(sv, token_map, pos);
|
||||
if (step.status != utf8_parse_result::SUCCESS) {
|
||||
auto result = common_parse_utf8_codepoint(sv, pos);
|
||||
if (result.status != utf8_parse_result::SUCCESS) {
|
||||
break;
|
||||
}
|
||||
|
||||
auto it = nodes[current].children.find(step.symbol);
|
||||
auto it = nodes[current].children.find(result.codepoint);
|
||||
if (it == nodes[current].children.end()) {
|
||||
// Can't continue matching
|
||||
return match_result{match_result::NO_MATCH};
|
||||
}
|
||||
|
||||
current = it->second;
|
||||
pos += step.bytes_consumed;
|
||||
pos += result.bytes_consumed;
|
||||
|
||||
// Check if we've matched a complete word
|
||||
if (nodes[current].pattern >= 0) {
|
||||
@@ -62,7 +42,7 @@ common_trie::match_result common_trie::check_at(std::string_view sv, const std::
|
||||
}
|
||||
|
||||
int32_t common_trie::insert(const std::string & word) {
|
||||
std::vector<symbol> symbols;
|
||||
std::vector<uint32_t> symbols;
|
||||
size_t pos = 0;
|
||||
while (pos < word.length()) {
|
||||
auto result = common_parse_utf8_codepoint(word, pos);
|
||||
@@ -70,42 +50,28 @@ int32_t common_trie::insert(const std::string & word) {
|
||||
break;
|
||||
}
|
||||
|
||||
symbols.push_back(symbol::codepoint(result.codepoint));
|
||||
symbols.push_back(result.codepoint);
|
||||
pos += result.bytes_consumed;
|
||||
}
|
||||
return insert(symbols);
|
||||
}
|
||||
|
||||
int32_t common_trie::insert(const std::vector<symbol> & symbols) {
|
||||
int32_t common_trie::insert(const std::vector<uint32_t> & symbols) {
|
||||
size_t current = 0;
|
||||
for (symbol ch : symbols) {
|
||||
current = extend(current, ch);
|
||||
for (uint32_t ch : symbols) {
|
||||
auto it = nodes[current].children.find(ch);
|
||||
if (it == nodes[current].children.end()) {
|
||||
size_t child = create_node();
|
||||
nodes[current].children[ch] = child;
|
||||
current = child;
|
||||
} else {
|
||||
current = it->second;
|
||||
}
|
||||
}
|
||||
int32_t pattern = mark(current, n_patterns);
|
||||
if (pattern == n_patterns) {
|
||||
n_patterns++;
|
||||
if (nodes[current].pattern < 0) {
|
||||
nodes[current].pattern = n_patterns++;
|
||||
}
|
||||
return pattern;
|
||||
}
|
||||
|
||||
size_t common_trie::extend(size_t node, symbol ch) {
|
||||
if (ch.is_token()) {
|
||||
tokens.insert((int32_t) ch.value);
|
||||
}
|
||||
auto it = nodes[node].children.find(ch);
|
||||
if (it != nodes[node].children.end()) {
|
||||
return it->second;
|
||||
}
|
||||
size_t child = create_node();
|
||||
nodes[node].children[ch] = child;
|
||||
return child;
|
||||
}
|
||||
|
||||
int32_t common_trie::mark(size_t node, int32_t value) {
|
||||
if (nodes[node].pattern < 0) {
|
||||
nodes[node].pattern = value;
|
||||
}
|
||||
return nodes[node].pattern;
|
||||
return nodes[current].pattern;
|
||||
}
|
||||
|
||||
common_aho_corasick::common_aho_corasick(common_trie trie) : t(std::move(trie)) {
|
||||
@@ -147,7 +113,7 @@ common_aho_corasick::common_aho_corasick(common_trie trie) : t(std::move(trie))
|
||||
}
|
||||
}
|
||||
|
||||
size_t common_aho_corasick::next(size_t state, common_trie::symbol ch) const {
|
||||
size_t common_aho_corasick::next(size_t state, uint32_t ch) const {
|
||||
const auto & nodes = t.nodes;
|
||||
while (state && nodes[state].children.find(ch) == nodes[state].children.end()) {
|
||||
state = fail[state];
|
||||
|
||||
+10
-51
@@ -1,7 +1,5 @@
|
||||
#pragma once
|
||||
|
||||
#include "unicode.h"
|
||||
|
||||
#include <cstdint>
|
||||
#include <map>
|
||||
#include <set>
|
||||
@@ -12,42 +10,13 @@
|
||||
// Trie for matching multiple literals.
|
||||
// This is used in common_peg_until_parser and to build a GBNF exclusion grammar
|
||||
struct common_trie {
|
||||
// A symbol is either a UTF-8 codepoint or a token
|
||||
struct symbol {
|
||||
enum kind_type : uint8_t { CODEPOINT, TOKEN };
|
||||
|
||||
kind_type kind = CODEPOINT;
|
||||
uint32_t value = 0;
|
||||
|
||||
static symbol codepoint(uint32_t cpt) { return { CODEPOINT, cpt }; }
|
||||
static symbol token(int32_t id) { return { TOKEN, (uint32_t) id }; }
|
||||
|
||||
bool is_token() const { return kind == TOKEN; }
|
||||
|
||||
bool operator<(const symbol & other) const {
|
||||
return kind != other.kind ? kind < other.kind : value < other.value;
|
||||
}
|
||||
bool operator==(const symbol & other) const {
|
||||
return kind == other.kind && value == other.value;
|
||||
}
|
||||
};
|
||||
|
||||
struct step {
|
||||
enum utf8_parse_result::status status;
|
||||
common_trie::symbol symbol;
|
||||
size_t bytes_consumed;
|
||||
};
|
||||
|
||||
struct node {
|
||||
std::map<symbol, size_t> children;
|
||||
std::map<uint32_t, size_t> children; // Use uint32_t to store Unicode codepoints
|
||||
int32_t pattern = -1; // index of the pattern ending at this node, -1 if none
|
||||
};
|
||||
|
||||
std::vector<node> nodes;
|
||||
|
||||
// every token id used by a pattern, the input is split so these match as single symbols
|
||||
std::set<int32_t> tokens;
|
||||
|
||||
common_trie() {
|
||||
create_node(); // root node
|
||||
}
|
||||
@@ -60,25 +29,15 @@ struct common_trie {
|
||||
|
||||
enum match_result { NO_MATCH, PARTIAL_MATCH, COMPLETE_MATCH };
|
||||
|
||||
// Read the symbol at pos. It is a token if a pattern uses the token at pos, else a UTF-8 codepoint.
|
||||
step next(std::string_view sv, const std::vector<int32_t> & token_map, size_t pos) const;
|
||||
|
||||
// Check if a delimiter starts at the given position
|
||||
match_result check_at(std::string_view sv, size_t start_pos) const;
|
||||
match_result check_at(std::string_view sv, const std::vector<int32_t> & token_map, size_t start_pos) const;
|
||||
|
||||
// Insert a word as a sequence of Unicode codepoints, returns its pattern index
|
||||
int32_t insert(const std::string & word);
|
||||
|
||||
// Insert a symbol sequence, returns its pattern index (insertion order, duplicates keep the first index)
|
||||
int32_t insert(const std::vector<symbol> & symbols);
|
||||
|
||||
// Child of node on a symbol, created if missing
|
||||
size_t extend(size_t node, symbol ch);
|
||||
|
||||
// Mark node as the end of a pattern with a value from the caller. Returns the value already there, or value if the node had none.
|
||||
// A trie must take its values from mark or from insert, not both.
|
||||
int32_t mark(size_t node, int32_t value);
|
||||
// Insert a raw symbol sequence, returns its pattern index (insertion order,
|
||||
// duplicates keep the first index)
|
||||
int32_t insert(const std::vector<uint32_t> & symbols);
|
||||
|
||||
private:
|
||||
int32_t n_patterns = 0;
|
||||
@@ -92,11 +51,11 @@ struct common_trie {
|
||||
|
||||
// Aho-Corasick automaton
|
||||
struct common_aho_corasick {
|
||||
common_trie t;
|
||||
std::vector<size_t> fail; // failure links
|
||||
std::vector<size_t> order; // states in BFS order
|
||||
std::vector<int32_t> match; // longest pattern ending at each state (directly or via a suffix link), -1 if none
|
||||
std::set<common_trie::symbol> alphabet; // every symbol with a transition
|
||||
common_trie t;
|
||||
std::vector<size_t> fail; // failure links
|
||||
std::vector<size_t> order; // states in BFS order
|
||||
std::vector<int32_t> match; // longest pattern ending at each state (directly or via a suffix link), -1 if none
|
||||
std::set<uint32_t> alphabet; // every character with a transition
|
||||
|
||||
common_aho_corasick(common_trie trie);
|
||||
|
||||
@@ -110,5 +69,5 @@ struct common_aho_corasick {
|
||||
int32_t match_pattern(size_t s) const { return match[s]; }
|
||||
|
||||
// follow failure links until a transition on `ch` exists.
|
||||
size_t next(size_t state, common_trie::symbol ch) const;
|
||||
size_t next(size_t state, uint32_t ch) const;
|
||||
};
|
||||
@@ -67,8 +67,6 @@ For a more complete example, see `test_example_native()` in
|
||||
- **`end()`** - Matches the end of input (anchor `$`)
|
||||
- **`literal(string)`** - Matches an exact literal string
|
||||
- **`any()`** - Matches any single character (`.`)
|
||||
- **`token(piece)`** - Matches a special token, or the literal `piece` if it is not registered with the builder (see [Special Tokens](#special-tokens))
|
||||
- **`token(pieces, negate)`** - Matches one of the tokens, or with `negate` one token that is not one of them
|
||||
|
||||
### Combinators
|
||||
|
||||
@@ -91,8 +89,6 @@ For a more complete example, see `test_example_native()` in
|
||||
- **`space()`** - Matches zero or more whitespace characters (space, tab, newline)
|
||||
- **`until(delimiter)`** - Matches characters until delimiter is found (delimiter not consumed)
|
||||
- **`until_one_of(delimiters)`** - Matches characters until any delimiter in the list is found
|
||||
- **`until(parser)`** - Matches characters until a delimiter built from literals and tokens is found (delimiter not consumed, see [until(parser)](#untilparser))
|
||||
- **`through(content, branches, optional)`** - Matches content, then the first delimiter found and the rest of its branch (delimiter consumed). The grammar scans for the delimiters instead of following content
|
||||
- **`rest()`** - Matches everything remaining (`.*`)
|
||||
|
||||
### JSON Parsers
|
||||
@@ -112,7 +108,6 @@ For a more complete example, see `test_example_native()` in
|
||||
- **`ref(name)`** - Creates a lightweight reference to a named rule (for recursive grammars)
|
||||
- **`rule(name, p, trigger)`** - Creates a named rule and returns a reference
|
||||
- **`trigger_rule(name, p)`** - Creates a trigger rule (entry point for lazy grammar generation)
|
||||
- **`trigger_rule(name, start, rest)`** - Creates a trigger rule matching `start` then `rest`, gating the lazy grammar on the start
|
||||
- **`schema(p, name, schema, raw)`** - Wraps parser with JSON schema metadata for grammar generation
|
||||
- **`schema(p, name, doc, node, raw)`** - Same, for a node of a `common_chat_schema_document` built earlier, e.g. one tool parameter
|
||||
|
||||
@@ -121,110 +116,6 @@ For a more complete example, see `test_example_native()` in
|
||||
- **`atomic(p)`** - Prevents AST node creation for partial parses
|
||||
- **`tag(tag, p)`** - Creates AST nodes with semantic tags (multiple nodes can share tags)
|
||||
|
||||
## Special Tokens
|
||||
|
||||
Matching a special token by its text alone is ambiguous: a model may emit
|
||||
`<|end|>` as a single control token, or spell it out as plain text, such as
|
||||
when quoting a chat template. The parser can tell them apart when the builder
|
||||
knows the special tokens of the vocabulary, and the input carries the ids of
|
||||
the tokens that produced it.
|
||||
|
||||
A `common_peg_tokens` table constructed from a `llama_vocab` registers every
|
||||
control and user-defined token. Pass it to the builder:
|
||||
|
||||
```cpp
|
||||
common_peg_tokens specials(vocab);
|
||||
|
||||
auto parser = build_chat_peg_parser(specials, [&](common_chat_peg_builder & p) {
|
||||
return p.content(p.until(p.token("<|end|>"))) + p.token("<|end|>");
|
||||
});
|
||||
```
|
||||
|
||||
Chat templates initialized with a model build this table from its vocabulary,
|
||||
and format handlers receive it as `inputs.specials`.
|
||||
|
||||
### token()
|
||||
|
||||
`token(piece)` matches only where the input holds the id of that token, and
|
||||
consumes the whole piece. The same characters without the id do not match.
|
||||
If `piece` is not registered with the builder, it falls back to
|
||||
`literal(piece)`.
|
||||
|
||||
`token(pieces)` matches any of the tokens, and `token(pieces, true)` matches one
|
||||
token that is not in the set. Every piece of a negated set must be registered,
|
||||
otherwise it throws `std::invalid_argument`.
|
||||
|
||||
In the grammar, a token emits `<[id]>` and a negated set emits `!<[...]>`:
|
||||
|
||||
```cpp
|
||||
p.token({ "<a>", "<b>" }) + p.token({ "<a>", "<b>" }, true)
|
||||
```
|
||||
|
||||
```
|
||||
root ::= (<[1]> | <[2]>) !<[1-2]>
|
||||
```
|
||||
|
||||
### until(parser)
|
||||
|
||||
`until(delimiter)` with a parser instead of a string stops at a delimiter that
|
||||
may contain tokens, or several literals joined in a sequence:
|
||||
|
||||
```cpp
|
||||
auto end = p.token("<|end|>");
|
||||
auto final = p.token("<|channel|>") + p.literal("final");
|
||||
|
||||
p.reasoning(p.until(end | final))
|
||||
```
|
||||
|
||||
The delimiter may only be built from:
|
||||
|
||||
- `literal()` and `token()`
|
||||
- `sequence()` and `choice()` of the above
|
||||
- `tag()` and `atomic()` wrapping any of the above
|
||||
|
||||
Any other parser throws `std::invalid_argument`. A choice inside a sequence
|
||||
expands to every combination, so `"a" + ("b" | "d")` stops at both `ab` and
|
||||
`ad`.
|
||||
|
||||
The grammar scans for the delimiters the same way as `until_one_of()`:
|
||||
|
||||
```cpp
|
||||
p.until((p.literal("a") + p.literal("b")) | p.literal("cd"))
|
||||
```
|
||||
|
||||
```
|
||||
root ::= until-5
|
||||
until-5 ::= | [a] until-5-01 | [c] until-5-03 | [^ac] until-5
|
||||
until-5-01 ::= | [a] until-5-01 | [c] until-5-03 | [^abc] until-5
|
||||
until-5-03 ::= | [a] until-5-01 | [c] until-5-03 | [^acd] until-5
|
||||
```
|
||||
|
||||
A token never matches a character rule such as `.` or `[^a]`, so the scan lets
|
||||
the other tokens named elsewhere in the grammar through by name. Here `<b>` is
|
||||
token 2:
|
||||
|
||||
```cpp
|
||||
p.until(p.token("<a>") + p.literal("x")) + p.token("<a>") + p.literal("x") + p.token("<b>")
|
||||
```
|
||||
|
||||
```
|
||||
root ::= until-3 <[1]> "x" <[2]>
|
||||
until-3 ::= | <[1]> until-3-01 | (. | <[2]>) until-3
|
||||
until-3-01 ::= | <[1]> until-3-01 | ([^x] | <[2]>) until-3
|
||||
```
|
||||
|
||||
When the delimiters are made of tokens alone, the scan consumes whole tokens
|
||||
with a negated token set instead:
|
||||
|
||||
```cpp
|
||||
p.until(p.token("<a>")) + p.token("<a>")
|
||||
```
|
||||
|
||||
```
|
||||
root ::= until-1 <[1]>
|
||||
until-1 ::= | !<[1]> until-1
|
||||
```
|
||||
|
||||
## GBNF Grammar Generation
|
||||
|
||||
The PEG parser also acts as a convenient DSL for generating GBNF grammars, with
|
||||
@@ -252,28 +143,6 @@ are emitted in the grammar. All trigger rules are added as alternations in the
|
||||
root rule. It is still necessary to define trigger patterns, as the parser has
|
||||
no interaction with the grammar sampling.
|
||||
|
||||
With `trigger_rule(name, start, rest)`, the grammar instead scans for a start and
|
||||
then constrains the rest. It runs from the first token, so `grammar_lazy` is false
|
||||
and no trigger patterns are needed. For example:
|
||||
|
||||
```cpp
|
||||
auto ab = p.literal("ab");
|
||||
auto cd = p.literal("cd");
|
||||
return p.until(ab | cd) + (p.trigger_rule("one", ab, p.literal("x")) | p.trigger_rule("two", cd, p.literal("y")));
|
||||
```
|
||||
|
||||
```
|
||||
root ::= trigger
|
||||
trigger ::= | [a] trigger-01 | [c] trigger-03 | [^ac] trigger
|
||||
trigger-01 ::= | [b] one-rest | [a] trigger-01 | [c] trigger-03 | [^abc] trigger
|
||||
trigger-03 ::= | [d] two-rest | [a] trigger-01 | [c] trigger-03 | [^acd] trigger
|
||||
one-rest ::= "x"
|
||||
two-rest ::= "y"
|
||||
```
|
||||
|
||||
A start may only be a sequence of literals, tokens, and choices of them, possibly
|
||||
wrapped in tags and atomics.
|
||||
|
||||
### JSON Schema
|
||||
|
||||
The `schema(p, name, schema, raw)` parser will use the `json-schema-to-grammar`
|
||||
|
||||
@@ -61,9 +61,7 @@ git log --oneline "$OLD_BASE"..origin/master -- \
|
||||
**Temporary upstream carries (FORK.md F12)** come first: `git log --oneline --grep '^upstream-carry'
|
||||
"$OLD_BASE"..HEAD`. For each: merged into the new upstream -> do not cherry-pick it (check that the
|
||||
merged version is the carried head or newer); still open -> re-fetch the PR head and re-carry it if
|
||||
it moved; closed -> ask the owner. A PR head can also shrink: on 2026-11-22 #32740 dropped the Vulkan part the fork needs, so the old
|
||||
head stayed carried (FORK.md F12). Compare `git diff --stat` of the old and new head before re-carrying.
|
||||
To carry a PR (2026-11-21: #32735, #32740; 2026-11-22: #32777):
|
||||
it moved; closed -> ask the owner. To carry a PR (2026-11-22: #32735, #32740):
|
||||
```sh
|
||||
git fetch https://github.com/ggml-org/llama.cpp pull/<N>/head:pr-<N>
|
||||
git cherry-pick $(git merge-base pr-<N> HEAD)..pr-<N> # the PR's commits, oldest first; resolve by hand
|
||||
@@ -108,10 +106,6 @@ git switch -c "$NEW" origin/master
|
||||
git cherry-pick <first-fork-commit> # one at a time, in the original order
|
||||
```
|
||||
|
||||
**A rebase is not done until `fork-validate.sh` phases 1 and 2 (section 6) pass on a pod build of
|
||||
the new branch.** A rebased branch that is only picked, checked by reading, or built is unfinished
|
||||
work, whatever the cherry-picks looked like. Report it as unfinished until both phases pass.
|
||||
|
||||
Rules:
|
||||
- Cherry-pick in the original order, keep authors and messages. New commits you author get the
|
||||
fork's trailer (`Co-Authored-By: Claude ...` for agent-written commits, FORK.md status block),
|
||||
@@ -181,28 +175,7 @@ built or run locally; every item below is by reading only and needs the pod buil
|
||||
`llama_kv_cache_dsa`, `_dsa_iswa`, `_dsv4`, `_msa`, `llama_memory_hybrid_iswa`. The KV rotation
|
||||
tensors (`attn_rot_bufs`, #31073) are small and are not released either.
|
||||
|
||||
**What the 2026-11-22 rebase (upstream `5bacb7c86`, 19 commits past `e6d624b9a`) changed.** Branch
|
||||
`fable5/vram-ondemand-rebase-20261122`. The 64 picks (65 fork commits minus the #32735 carry) all
|
||||
applied cleanly; the range-diff against the old branch is identical except for that dropped carry and
|
||||
the new commits at the end. Reading checks:
|
||||
- #32735 merged as `8a97fa98a`, patch identical to the carried head `0ea4ffd00`: carry dropped.
|
||||
- #32740 is still open but was split: its head `8c6f0c0ca` is now only the GGML contract (op API,
|
||||
CPU reference, test-backend-ops cases); the Vulkan kernel, the `kv_rows` support and the Inkling
|
||||
graph use were removed from it and are in no open PR (Metal is #32808). Carrying that head would
|
||||
drop Inkling's in-kernel bias (back to the 15.8 GiB compute buffer at ctx 65536, FORK.md F12). The
|
||||
validated carry of head `808fd1c4e` is kept as is (FORK.md F12 has the owner decision).
|
||||
- #32807 (`5bacb7c86`) adds an `LLM_ARCH_INKLING2_LOOP` guard to `sched_copy_experts` (whole-tensor
|
||||
copies), upstream's fix for #32755. It sits beside F1 in the same function; no conflict. It does
|
||||
not remove the held per-pass copies, so F13 stays (FORK.md F13, "Upstream #32807").
|
||||
- #32713 (non-causal prompts fit in one ubatch) touches `update_slots` near the F14 loop but not
|
||||
the user-start break; F14 unchanged by it.
|
||||
- #32799 (`ignore_merges` for inkling2) changes Inkling's tokenization: Inkling prompts cached by
|
||||
the previous build no longer match. The disk cache key holds the build commit, so nothing stale is
|
||||
restored.
|
||||
- New commits: the F14 first-user-message break (FORK.md F14), the `LLAMA_LOOP_FULL_COPY` knob
|
||||
(F13) and the carry of #32777 (F12).
|
||||
|
||||
Open questions from the 2026-11-21 rebase:
|
||||
Open questions from that rebase:
|
||||
- Is the fork's own concat fast path (F3) now redundant with upstream's tiled concat (#30591)?
|
||||
Benchmark `GGML_VK_CONCAT_FASTPATH=0` against the default on pp and decode before deciding.
|
||||
- Does `n_inflight` blocking the want-lock yield delay handoffs? Validation says no: handoff median
|
||||
@@ -375,35 +348,6 @@ setsid nohup sh /root/bench/moe/swapdown.sh /root/arms/fv-perf.sh > /root/arms/f
|
||||
`<scratch>/perf.tsv`, and `fork-validate.sh --perf-report <file>` re-aggregates them offline. A pod
|
||||
restart empties `/tmp`, so copy a scratch dir you want to keep to `/root/arms/` (the 2026-10-24
|
||||
scratch dirs were lost that way).
|
||||
- Overlayfs (since the persistent rootfs, 2026-11-22): `/tmp` is on overlayfs, where `stat`
|
||||
reports another device than `/proc/locks` prints, so a scratch arena under `/tmp` made every
|
||||
token check fail (`arbiter/start: A serves but does not hold token.lock` while A held the lock).
|
||||
`fork-validate.sh` from `70d39bdbc` on keeps its arenas on `/dev/shm`. Use the script of that
|
||||
commit or later, also for a `--ref` run of an older build.
|
||||
- Phase 1 of the 2026-11-22 rebase (candidate build `/root/arms/reb20261122/build`, code
|
||||
`6c895a491`, scripts `70d39bdbc`, vs production `f3397cded`, beside production with 3579 MiB
|
||||
free; logs `/root/arms/reb20261122-p1.log`, `-p1b.log`): `--ops --concat --diskcache` 23 PASS
|
||||
(ops 5/5 with FLASH_ATTN_EXT 5693/5693, concat off/cand 5.3x, diskcache 15/15 including the GPU
|
||||
legs), `--arbiter` 16 PASS on the rerun with the fixed script (handoff median cand 789.5 / ref
|
||||
768 ms; the first run's single FAIL was the overlayfs mismatch above). F12: test-backend-ops
|
||||
SCALE and FLASH_ATTN_EXT OK, `test-flash-attn-indexed-bias` 16 Vulkan0 cases, max error 1.3e-4.
|
||||
Still to run in the window: phase 2 (`--arbiter --perf --moe-exact --moecache`, the runbook
|
||||
command with this build path) and the Inkling/F14 items below.
|
||||
- What the 2026-11-22 window must run, in this order (GPU to itself under swapdown; about 75-90
|
||||
minutes in all):
|
||||
1. Phase 2 as written above with `W=/root/arms/reb20261122` (about 32 minutes).
|
||||
2. F13 vs upstream's #32807 (FORK.md F13, "Window A/B to run": memory, `kld48.sh`, speed with
|
||||
`LLAMA_LOOP_HOST_VIEWS=0` and `LLAMA_LOOP_FULL_COPY=0`).
|
||||
3. `--fit` vs the k=4 `-ot` for Inkling (FORK.md F12, "How to use it for Inkling"). Smoke run
|
||||
beside production (3.5 GiB free, so not a placement result; `/root/arms/reb20261122-fit-smoke.log`):
|
||||
fit's loop path runs on Inkling; the probing took 27.8 s, which every Inkling cold start would
|
||||
pay with `--fit` in the config (freeze the printed `-ot` instead); the projected memory is not
|
||||
monotonic in `n_layer` (5173 MiB at 44, 6736 at 43) and fit still reported success with -3116
|
||||
MiB free. Check both with the full card.
|
||||
4. F14 refinement (FORK.md F14, validation plan items 5-7): the new-chat checkpoint, its disk-cache
|
||||
variant and the M-only TTFT table, with the 35B think entry's exact command line.
|
||||
5. The Inkling post-deploy items of section 8 step 8 against a scratch server of the candidate:
|
||||
#32799 changed Inkling's tokenization, so judge the answer text.
|
||||
- F14 run (2026-11-22, candidate `f3397cded` = `166ec22e2` (docs on `b13c41355`) + F14, server-only change; one
|
||||
swapdown window, `/root/arms/usw-window.sh`, log `/root/arms/usw-window.log`, about 15 minutes):
|
||||
the 35B TTFT A/B with and without `LLAMA_USERSTART_MERGE=0` (FORK.md F14, M U U M, production 35B
|
||||
@@ -634,8 +578,8 @@ A wedged arbiter looks like this: requests hang while `/health` still returns 20
|
||||
shows a process holding WRITE on `token.lock` that is not generating. llama-swap restarts a child
|
||||
that exits (on the next request for its model, FORK.md F7) but never one that wedges. Inspect it with
|
||||
`gdb -p <pid> -batch -ex 'thread apply all bt'` before killing it. gdb is not in
|
||||
the pod image; it was lost in the 2026-10-24 pod restart, but the rootfs persists since 2026-11-22 (section 11), so
|
||||
an apt install now stays. Check `command -v gdb` and install it first if needed.
|
||||
the pod image: it was installed by hand and was gone after the 2026-10-24 pod restart, so check
|
||||
`command -v gdb` and reinstall it first if needed.
|
||||
|
||||
## 9. Rollback
|
||||
|
||||
@@ -726,14 +670,6 @@ kubectl -n llama exec -i deploy/supervisord -- sh -c '<command>'
|
||||
kubectl get pods -n llama -l app=llama-swap -o name # the pod name changes across restarts
|
||||
```
|
||||
|
||||
Alternative (since 2026-11-22): the pod runs sshd, so `ssh root@2001:470:61a3:400::7` works instead
|
||||
of `kubectl exec` (`scp 'root@[2001:470:61a3:400::7]:/path' .` for files). The same rules apply:
|
||||
detach long runs with `setsid nohup ... </dev/null &` and never `pkill -f`.
|
||||
|
||||
The pod's root filesystem now persists across pod restarts: packages installed with apt (gdb,
|
||||
for example) stay installed. `/tmp` and `/run` are still wiped on every boot, so copy a scratch dir
|
||||
you want to keep (fork-validate's `/tmp/fork-validate.*`) to `/root/arms/`.
|
||||
|
||||
Wiring: k8s service `llama:11434` -> pod port 8080 (selector `app=llama-swap`) -> llama-swap `:8080`;
|
||||
a Caddy proxy fronts `llama.lumpiasty.xyz`. llama-swap must listen on `:8080`.
|
||||
|
||||
@@ -768,8 +704,8 @@ Useful commands:
|
||||
- Token holder and waiters: `/proc/locks` filtered by the inode of `token.lock` / `want.lock`
|
||||
(WRITE = holder on token.lock, READ = registered waiters on want.lock).
|
||||
- 120 W cap active: `cat /sys/class/drm/card*/device/hwmon/hwmon*/power1_cap` = `120000000`.
|
||||
- The pod has `curl`, `python3`, `flock`; no `jq`, and `gdb` only if installed with apt (it persists
|
||||
since 2026-11-22; without it `fork-validate.sh` runs its SIGTERM-in-release case by polling the log).
|
||||
- The pod has `curl`, `python3`, `flock`; no `jq`, and no `gdb` unless reinstalled by hand since the
|
||||
last pod restart (`fork-validate.sh` then runs its SIGTERM-in-release case by polling the log).
|
||||
|
||||
Second test box (optional): `ssh user@gaming-pc`, the owner's NixOS desktop with an RX 6600 (RDNA2).
|
||||
Pod-built binaries can run there through the host glibc loader without compiling. Numbers from it do
|
||||
|
||||
@@ -8,10 +8,10 @@ extern "C" {
|
||||
|
||||
#define RPC_PROTO_MAJOR_VERSION 10
|
||||
#define RPC_PROTO_MINOR_VERSION 0
|
||||
#define RPC_PROTO_PATCH_VERSION 2
|
||||
#define RPC_PROTO_PATCH_VERSION 1
|
||||
|
||||
#ifdef __cplusplus
|
||||
static_assert(GGML_OP_COUNT == 101, "GGML_OP_COUNT has changed - update RPC_PROTO_PATCH_VERSION");
|
||||
static_assert(GGML_OP_COUNT == 100, "GGML_OP_COUNT has changed - update RPC_PROTO_PATCH_VERSION");
|
||||
#endif
|
||||
|
||||
#define GGML_RPC_MAX_SERVERS 16
|
||||
|
||||
@@ -433,8 +433,6 @@ extern "C" {
|
||||
|
||||
GGML_OP_SLEEP,
|
||||
|
||||
GGML_OP_SIGMOID_BACK,
|
||||
|
||||
GGML_OP_COUNT,
|
||||
};
|
||||
|
||||
@@ -1040,13 +1038,6 @@ extern "C" {
|
||||
struct ggml_tensor * a,
|
||||
struct ggml_tensor * b);
|
||||
|
||||
// a - dy
|
||||
// b - x
|
||||
GGML_API struct ggml_tensor * ggml_sigmoid_back(
|
||||
struct ggml_context * ctx,
|
||||
struct ggml_tensor * a,
|
||||
struct ggml_tensor * b);
|
||||
|
||||
// hardswish(x) = x * relu6(x + 3) / 6
|
||||
GGML_API struct ggml_tensor * ggml_hardswish(
|
||||
struct ggml_context * ctx,
|
||||
|
||||
@@ -35,7 +35,6 @@ bool ggml_op_can_inplace(enum ggml_op op) {
|
||||
case GGML_OP_ROPE:
|
||||
case GGML_OP_ROPE_BACK:
|
||||
case GGML_OP_SILU_BACK:
|
||||
case GGML_OP_SIGMOID_BACK:
|
||||
case GGML_OP_RMS_NORM:
|
||||
case GGML_OP_RMS_NORM_BACK:
|
||||
case GGML_OP_CLAMP:
|
||||
|
||||
@@ -975,8 +975,7 @@ static struct ggml_backend_meta_split_state ggml_backend_meta_get_split_state(
|
||||
case GGML_OP_CONCAT: {
|
||||
split_state = handle_concat(src_ss);
|
||||
} break;
|
||||
case GGML_OP_SILU_BACK:
|
||||
case GGML_OP_SIGMOID_BACK: {
|
||||
case GGML_OP_SILU_BACK: {
|
||||
split_state = handle_generic(src_ss, /*scalar_only =*/ false);
|
||||
} break;
|
||||
case GGML_OP_NORM:
|
||||
|
||||
@@ -1884,10 +1884,6 @@ static void ggml_compute_forward(struct ggml_compute_params * params, struct ggm
|
||||
{
|
||||
ggml_compute_forward_silu_back(params, tensor);
|
||||
} break;
|
||||
case GGML_OP_SIGMOID_BACK:
|
||||
{
|
||||
ggml_compute_forward_sigmoid_back(params, tensor);
|
||||
} break;
|
||||
case GGML_OP_NORM:
|
||||
{
|
||||
ggml_compute_forward_norm(params, tensor);
|
||||
@@ -2396,7 +2392,6 @@ static int ggml_get_n_tasks(struct ggml_tensor * node, int n_threads) {
|
||||
}
|
||||
break;
|
||||
case GGML_OP_SILU_BACK:
|
||||
case GGML_OP_SIGMOID_BACK:
|
||||
case GGML_OP_MUL:
|
||||
case GGML_OP_DIV:
|
||||
case GGML_OP_NORM:
|
||||
|
||||
@@ -2898,118 +2898,6 @@ void ggml_compute_forward_silu_back(
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
// ggml_compute_forward_sigmoid_back
|
||||
|
||||
static void ggml_compute_forward_sigmoid_back_f32(
|
||||
const ggml_compute_params * params,
|
||||
ggml_tensor * dst) {
|
||||
|
||||
const ggml_tensor * grad = dst->src[0];
|
||||
const ggml_tensor * src1 = dst->src[1];
|
||||
|
||||
GGML_ASSERT(ggml_is_contiguous_1(grad));
|
||||
GGML_ASSERT(ggml_is_contiguous_1(src1));
|
||||
GGML_ASSERT(ggml_is_contiguous_1(dst));
|
||||
GGML_ASSERT(ggml_are_same_shape(src1, dst));
|
||||
GGML_ASSERT(ggml_are_same_shape(src1, grad));
|
||||
|
||||
const int ith = params->ith;
|
||||
const int nth = params->nth;
|
||||
|
||||
const int nc = src1->ne[0];
|
||||
const int nr = ggml_nrows(src1);
|
||||
|
||||
// rows per thread
|
||||
const int dr = (nr + nth - 1)/nth;
|
||||
|
||||
// row range for this thread
|
||||
const int ir0 = dr*ith;
|
||||
const int ir1 = MIN(ir0 + dr, nr);
|
||||
|
||||
for (int i1 = ir0; i1 < ir1; i1++) {
|
||||
ggml_vec_sigmoid_backward_f32(nc,
|
||||
(float *) ((char *) dst->data + i1*( dst->nb[1])),
|
||||
(float *) ((char *) src1->data + i1*(src1->nb[1])),
|
||||
(float *) ((char *) grad->data + i1*(grad->nb[1])));
|
||||
|
||||
#ifndef NDEBUG
|
||||
for (int k = 0; k < nc; k++) {
|
||||
const float x = ((float *) ((char *) dst->data + i1*( dst->nb[1])))[k];
|
||||
GGML_UNUSED(x);
|
||||
assert(!isnan(x));
|
||||
assert(!isinf(x));
|
||||
}
|
||||
#endif // NDEBUG
|
||||
}
|
||||
}
|
||||
|
||||
static void ggml_compute_forward_sigmoid_back_f16(
|
||||
const ggml_compute_params * params,
|
||||
ggml_tensor * dst) {
|
||||
|
||||
const ggml_tensor * grad = dst->src[0];
|
||||
const ggml_tensor * src1 = dst->src[1];
|
||||
|
||||
GGML_ASSERT(ggml_is_contiguous_1(grad));
|
||||
GGML_ASSERT(ggml_is_contiguous_1(src1));
|
||||
GGML_ASSERT(ggml_is_contiguous_1(dst));
|
||||
GGML_ASSERT(ggml_are_same_shape(src1, dst));
|
||||
GGML_ASSERT(ggml_are_same_shape(src1, grad));
|
||||
|
||||
const int ith = params->ith;
|
||||
const int nth = params->nth;
|
||||
|
||||
const int nc = src1->ne[0];
|
||||
const int nr = ggml_nrows(src1);
|
||||
|
||||
// rows per thread
|
||||
const int dr = (nr + nth - 1)/nth;
|
||||
|
||||
// row range for this thread
|
||||
const int ir0 = dr*ith;
|
||||
const int ir1 = MIN(ir0 + dr, nr);
|
||||
|
||||
for (int i1 = ir0; i1 < ir1; i1++) {
|
||||
ggml_vec_sigmoid_backward_f16(nc,
|
||||
(ggml_fp16_t *) ((char *) dst->data + i1*( dst->nb[1])),
|
||||
(ggml_fp16_t *) ((char *) src1->data + i1*(src1->nb[1])),
|
||||
(ggml_fp16_t *) ((char *) grad->data + i1*(grad->nb[1])));
|
||||
|
||||
#ifndef NDEBUG
|
||||
for (int k = 0; k < nc; k++) {
|
||||
const float x = ((ggml_fp16_t *) ((char *) dst->data + i1*( dst->nb[1])))[k];
|
||||
const float v = GGML_CPU_FP16_TO_FP32(x);
|
||||
GGML_UNUSED(v);
|
||||
assert(!isnan(v));
|
||||
assert(!isinf(v));
|
||||
}
|
||||
#endif // NDEBUG
|
||||
}
|
||||
}
|
||||
|
||||
void ggml_compute_forward_sigmoid_back(
|
||||
const ggml_compute_params * params,
|
||||
ggml_tensor * dst) {
|
||||
|
||||
const ggml_tensor * src0 = dst->src[0];
|
||||
|
||||
switch (src0->type) {
|
||||
case GGML_TYPE_F32:
|
||||
{
|
||||
ggml_compute_forward_sigmoid_back_f32(params, dst);
|
||||
} break;
|
||||
case GGML_TYPE_F16:
|
||||
{
|
||||
ggml_compute_forward_sigmoid_back_f16(params, dst);
|
||||
} break;
|
||||
default:
|
||||
{
|
||||
GGML_ABORT("fatal error");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ggml_compute_forward_reglu
|
||||
|
||||
// row loop shared by the BF16 GLU ops
|
||||
|
||||
@@ -41,7 +41,6 @@ void ggml_compute_forward_repeat(const struct ggml_compute_params * params, stru
|
||||
void ggml_compute_forward_repeat_back(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
||||
void ggml_compute_forward_concat(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
||||
void ggml_compute_forward_silu_back(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
||||
void ggml_compute_forward_sigmoid_back(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
||||
void ggml_compute_forward_norm(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
||||
void ggml_compute_forward_rms_norm(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
||||
void ggml_compute_forward_rms_norm_mul_fused(const struct ggml_compute_params * params, struct ggml_tensor * dst_rms_norm, struct ggml_tensor * dst_mul);
|
||||
|
||||
@@ -1536,29 +1536,6 @@ inline static void ggml_vec_silu_backward_f16(const int n, ggml_fp16_t * dx, con
|
||||
}
|
||||
}
|
||||
|
||||
inline static float ggml_sigmoid_backward_f32(float x, float dy) {
|
||||
const float s = 1.0f/(1.0f + expf(-x));
|
||||
return dy*s*(1.0f - s);
|
||||
}
|
||||
|
||||
inline static ggml_fp16_t ggml_sigmoid_backward_f16(ggml_fp16_t x, ggml_fp16_t dy) {
|
||||
const float v = GGML_CPU_FP16_TO_FP32(x);
|
||||
const float s = 1.0f/(1.0f + expf(-v));
|
||||
return GGML_CPU_FP32_TO_FP16(GGML_CPU_FP16_TO_FP32(dy)*s*(1.0f - s));
|
||||
}
|
||||
|
||||
inline static void ggml_vec_sigmoid_backward_f32(const int n, float * dx, const float * x, const float * dy) {
|
||||
for (int i = 0; i < n; ++i) {
|
||||
dx[i] = ggml_sigmoid_backward_f32(x[i], dy[i]);
|
||||
}
|
||||
}
|
||||
|
||||
inline static void ggml_vec_sigmoid_backward_f16(const int n, ggml_fp16_t * dx, const ggml_fp16_t * x, const ggml_fp16_t * dy) {
|
||||
for (int i = 0; i < n; ++i) {
|
||||
dx[i] = ggml_sigmoid_backward_f16(x[i], dy[i]);
|
||||
}
|
||||
}
|
||||
|
||||
inline static void ggml_vec_reglu_f32 (const int n, float * y, const float * x, const float * g) {
|
||||
for (int i = 0; i < n; ++i) {
|
||||
y[i] = (x[i] > 0.f) ? x[i] * g[i] : 0.f;
|
||||
|
||||
@@ -911,7 +911,7 @@ static __device__ __forceinline__ float ggml_cuda_ue4m3_to_fp32(uint8_t x) {
|
||||
// but CDNA3 supports only e4m3_fnuz (no inf).
|
||||
const uint32_t bits = x * (x != 0x7F && x != 0xFF); // Convert NaN to 0.0f to match CPU implementation.
|
||||
const __hip_fp8_e4m3_fnuz xf = *reinterpret_cast<const __hip_fp8_e4m3_fnuz *>(&bits);
|
||||
return static_cast<float>(xf);
|
||||
return static_cast<float>(xf) / 2;
|
||||
#else
|
||||
#if defined(FP8_AVAILABLE) && !defined(GGML_USE_HIP)
|
||||
const uint32_t bits = x * (x != 0x7F && x != 0xFF); // Convert NaN to 0.0f to match CPU implementation.
|
||||
|
||||
@@ -227,13 +227,6 @@ set(GGML_OPENCL_KERNELS
|
||||
gemm_coopk_q4_0_q8_1_dp4a
|
||||
gemm_coopk_q4_k_q8_1_dp4a
|
||||
gemm_coopk_q6_k_q8_1_dp4a
|
||||
pack_b_f16_coopmat
|
||||
gemm_noshuffle_q4_0_f32_coopmat
|
||||
gemm_noshuffle_q5_0_f32_coopmat
|
||||
gemm_noshuffle_q8_0_f32_coopmat
|
||||
gemm_noshuffle_q4_k_f32_coopmat
|
||||
gemm_noshuffle_q5_k_f32_coopmat
|
||||
gemm_noshuffle_q6_k_f32_coopmat
|
||||
quant_a_q8_1
|
||||
gemv_moe_q4_k_f32_ns
|
||||
gemm_moe_q5_k_f32_ns
|
||||
|
||||
@@ -1495,13 +1495,6 @@ struct ggml_backend_opencl_context {
|
||||
cl_kernel kernel_gemm_noshuffle_iq4_nl_q8_1_dp4a = nullptr; // dp4a (int8) dense IQ4_NL prefill GEMM
|
||||
cl_kernel kernel_gemm_noshuffle_q4_0_q8_1_dp4a = nullptr; // dp4a (int8) dense q4_0 prefill GEMM
|
||||
cl_kernel kernel_gemm_noshuffle_q4_0_q8_1_dp4a_act_vload4 = nullptr;
|
||||
cl_kernel kernel_pack_b_f16_coopmat = nullptr;
|
||||
cl_kernel kernel_gemm_noshuffle_q4_0_f32_coopmat = nullptr;
|
||||
cl_kernel kernel_gemm_noshuffle_q5_0_f32_coopmat = nullptr;
|
||||
cl_kernel kernel_gemm_noshuffle_q8_0_f32_coopmat = nullptr;
|
||||
cl_kernel kernel_gemm_noshuffle_q4_k_f32_coopmat = nullptr;
|
||||
cl_kernel kernel_gemm_noshuffle_q5_k_f32_coopmat = nullptr;
|
||||
cl_kernel kernel_gemm_noshuffle_q6_k_f32_coopmat = nullptr;
|
||||
cl_kernel kernel_gemm_coopk_q4_0_q8_1_dp4a_c4 = nullptr;
|
||||
cl_kernel kernel_gemm_coopk_q4_0_q8_1_dp4a_c2 = nullptr;
|
||||
int nsg_coopk_dp4a_q4_0 = 8;
|
||||
@@ -1986,137 +1979,6 @@ static void load_cl_kernels_gemm_coopk(ggml_backend_opencl_context * backend_ctx
|
||||
}
|
||||
}
|
||||
|
||||
static void load_cl_kernels_coopmat(ggml_backend_opencl_context * backend_ctx) {
|
||||
if (backend_ctx->gpu_family != GPU_FAMILY::ADRENO || !strstr(backend_ctx->device_name.c_str(), "850")) {
|
||||
return;
|
||||
}
|
||||
|
||||
cl_int err;
|
||||
const std::string opts = "-cl-std=CL" + std::to_string(backend_ctx->opencl_c_version.major) + "." +
|
||||
std::to_string(backend_ctx->opencl_c_version.minor) +
|
||||
(backend_ctx->adreno_use_large_buffer ? " -qcom-enable-large-buffer" : "");
|
||||
|
||||
// pack_b_f16_coopmat
|
||||
{
|
||||
#ifdef GGML_OPENCL_EMBED_KERNELS
|
||||
const std::string kernel_src {
|
||||
#include "pack_b_f16_coopmat.cl.h"
|
||||
};
|
||||
#else
|
||||
const std::string kernel_src = read_file("pack_b_f16_coopmat.cl");
|
||||
#endif
|
||||
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
|
||||
if (!prog) {
|
||||
return;
|
||||
}
|
||||
CL_CHECK((backend_ctx->kernel_pack_b_f16_coopmat = clCreateKernel(prog, "kernel_pack_b_f16_coopmat", &err), err));
|
||||
CL_CHECK(clReleaseProgram(prog));
|
||||
GGML_LOG_CONT(".");
|
||||
}
|
||||
|
||||
// gemm_noshuffle_q4_0_f32_coopmat
|
||||
{
|
||||
#ifdef GGML_OPENCL_EMBED_KERNELS
|
||||
const std::string kernel_src {
|
||||
#include "gemm_noshuffle_q4_0_f32_coopmat.cl.h"
|
||||
};
|
||||
#else
|
||||
const std::string kernel_src = read_file("gemm_noshuffle_q4_0_f32_coopmat.cl");
|
||||
#endif
|
||||
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
|
||||
if (prog) {
|
||||
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q4_0_f32_coopmat = clCreateKernel(prog, "kernel_gemm_noshuffle_q4_0_f32_coopmat", &err), err));
|
||||
CL_CHECK(clReleaseProgram(prog));
|
||||
GGML_LOG_CONT(".");
|
||||
}
|
||||
}
|
||||
|
||||
// gemm_noshuffle_q5_0_f32_coopmat
|
||||
{
|
||||
#ifdef GGML_OPENCL_EMBED_KERNELS
|
||||
const std::string kernel_src {
|
||||
#include "gemm_noshuffle_q5_0_f32_coopmat.cl.h"
|
||||
};
|
||||
#else
|
||||
const std::string kernel_src = read_file("gemm_noshuffle_q5_0_f32_coopmat.cl");
|
||||
#endif
|
||||
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
|
||||
if (prog) {
|
||||
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q5_0_f32_coopmat = clCreateKernel(prog, "kernel_gemm_noshuffle_q5_0_f32_coopmat", &err), err));
|
||||
CL_CHECK(clReleaseProgram(prog));
|
||||
GGML_LOG_CONT(".");
|
||||
}
|
||||
}
|
||||
|
||||
// gemm_noshuffle_q8_0_f32_coopmat
|
||||
{
|
||||
#ifdef GGML_OPENCL_EMBED_KERNELS
|
||||
const std::string kernel_src {
|
||||
#include "gemm_noshuffle_q8_0_f32_coopmat.cl.h"
|
||||
};
|
||||
#else
|
||||
const std::string kernel_src = read_file("gemm_noshuffle_q8_0_f32_coopmat.cl");
|
||||
#endif
|
||||
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
|
||||
if (prog) {
|
||||
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q8_0_f32_coopmat = clCreateKernel(prog, "kernel_gemm_noshuffle_q8_0_f32_coopmat", &err), err));
|
||||
CL_CHECK(clReleaseProgram(prog));
|
||||
GGML_LOG_CONT(".");
|
||||
}
|
||||
}
|
||||
|
||||
// gemm_noshuffle_q4_k_f32_coopmat
|
||||
{
|
||||
#ifdef GGML_OPENCL_EMBED_KERNELS
|
||||
const std::string kernel_src {
|
||||
#include "gemm_noshuffle_q4_k_f32_coopmat.cl.h"
|
||||
};
|
||||
#else
|
||||
const std::string kernel_src = read_file("gemm_noshuffle_q4_k_f32_coopmat.cl");
|
||||
#endif
|
||||
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
|
||||
if (prog) {
|
||||
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q4_k_f32_coopmat = clCreateKernel(prog, "kernel_gemm_noshuffle_q4_k_f32_coopmat", &err), err));
|
||||
CL_CHECK(clReleaseProgram(prog));
|
||||
GGML_LOG_CONT(".");
|
||||
}
|
||||
}
|
||||
|
||||
// gemm_noshuffle_q5_k_f32_coopmat
|
||||
{
|
||||
#ifdef GGML_OPENCL_EMBED_KERNELS
|
||||
const std::string kernel_src {
|
||||
#include "gemm_noshuffle_q5_k_f32_coopmat.cl.h"
|
||||
};
|
||||
#else
|
||||
const std::string kernel_src = read_file("gemm_noshuffle_q5_k_f32_coopmat.cl");
|
||||
#endif
|
||||
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
|
||||
if (prog) {
|
||||
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q5_k_f32_coopmat = clCreateKernel(prog, "kernel_gemm_noshuffle_q5_k_f32_coopmat", &err), err));
|
||||
CL_CHECK(clReleaseProgram(prog));
|
||||
GGML_LOG_CONT(".");
|
||||
}
|
||||
}
|
||||
|
||||
// gemm_noshuffle_q6_k_f32_coopmat
|
||||
{
|
||||
#ifdef GGML_OPENCL_EMBED_KERNELS
|
||||
const std::string kernel_src {
|
||||
#include "gemm_noshuffle_q6_k_f32_coopmat.cl.h"
|
||||
};
|
||||
#else
|
||||
const std::string kernel_src = read_file("gemm_noshuffle_q6_k_f32_coopmat.cl");
|
||||
#endif
|
||||
cl_program prog = build_program_from_source(backend_ctx, kernel_src.c_str(), opts, false);
|
||||
if (prog) {
|
||||
CL_CHECK((backend_ctx->kernel_gemm_noshuffle_q6_k_f32_coopmat = clCreateKernel(prog, "kernel_gemm_noshuffle_q6_k_f32_coopmat", &err), err));
|
||||
CL_CHECK(clReleaseProgram(prog));
|
||||
GGML_LOG_CONT(".");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#endif
|
||||
|
||||
static std::string ggml_opencl_make_compile_opts(const ggml_backend_opencl_context * backend_ctx) {
|
||||
@@ -7951,7 +7813,6 @@ static void load_cl_kernels(ggml_backend_opencl_context *backend_ctx) {
|
||||
}
|
||||
}
|
||||
load_cl_kernels_gemm_coopk(backend_ctx);
|
||||
load_cl_kernels_coopmat(backend_ctx);
|
||||
#endif // GGML_OPENCL_USE_ADRENO_KERNELS
|
||||
GGML_LOG_CONT("\n");
|
||||
backend_ctx->kernels_loaded = true;
|
||||
@@ -11878,8 +11739,7 @@ inline bool use_q4_0_bin_kernels(const ggml_backend_opencl_context *backend_ctx,
|
||||
#ifdef GGML_OPENCL_USE_ADRENO_KERNELS
|
||||
if (!backend_ctx->kernel_gemv_noshuffle_q4_0_f32_32b_trans ||
|
||||
!(backend_ctx->kernel_gemm_noshuffle_q4_0_f32_32b_trans_ila_a8_bin ||
|
||||
backend_ctx->kernel_gemm_noshuffle_q4_0_f32_32b_trans_ila_a7_bin ||
|
||||
backend_ctx->kernel_gemm_noshuffle_q4_0_f32_coopmat)) {
|
||||
backend_ctx->kernel_gemm_noshuffle_q4_0_f32_32b_trans_ila_a7_bin)) {
|
||||
return false;
|
||||
}
|
||||
|
||||
@@ -12033,8 +11893,7 @@ inline bool use_q6_k_bin_kernels(const ggml_backend_opencl_context *backend_ctx,
|
||||
#ifdef GGML_OPENCL_USE_ADRENO_KERNELS
|
||||
if (!backend_ctx->kernel_gemv_noshuffle_q6_k_f32_32b_trans ||
|
||||
!(backend_ctx->kernel_gemm_noshuffle_q6_k_f32_32b_trans_ila_a8_bin ||
|
||||
backend_ctx->kernel_gemm_noshuffle_q6_k_f32_32b_trans_ila_a7_bin ||
|
||||
backend_ctx->kernel_gemm_noshuffle_q6_k_f32_coopmat)) {
|
||||
backend_ctx->kernel_gemm_noshuffle_q6_k_f32_32b_trans_ila_a7_bin)) {
|
||||
return false;
|
||||
}
|
||||
// bin kernels require ne2 == 1 and ne3 == 1 for weights
|
||||
@@ -12052,8 +11911,7 @@ inline bool use_q4_k_bin_kernels(const ggml_backend_opencl_context *backend_ctx,
|
||||
#ifdef GGML_OPENCL_USE_ADRENO_KERNELS
|
||||
if (!backend_ctx->kernel_gemv_noshuffle_q4_k_f32_32b_trans ||
|
||||
!(backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a8_bin ||
|
||||
backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a7_bin ||
|
||||
backend_ctx->kernel_gemm_noshuffle_q4_k_f32_coopmat)) {
|
||||
backend_ctx->kernel_gemm_noshuffle_q4_k_f32_32b_trans_ila_a7_bin)) {
|
||||
return false;
|
||||
}
|
||||
// bin kernels require ne2 == 1 and ne3 == 1 for weights
|
||||
@@ -12071,8 +11929,7 @@ inline bool use_q5_k_bin_kernels(const ggml_backend_opencl_context *backend_ctx,
|
||||
#ifdef GGML_OPENCL_USE_ADRENO_KERNELS
|
||||
if (!backend_ctx->kernel_gemv_noshuffle_q5_k_f32_32b_trans ||
|
||||
!(backend_ctx->kernel_gemm_noshuffle_q5_k_f32_32b_trans_ila_a8_bin ||
|
||||
backend_ctx->kernel_gemm_noshuffle_q5_k_f32_32b_trans_ila_a7_bin ||
|
||||
backend_ctx->kernel_gemm_noshuffle_q5_k_f32_coopmat)) {
|
||||
backend_ctx->kernel_gemm_noshuffle_q5_k_f32_32b_trans_ila_a7_bin)) {
|
||||
return false;
|
||||
}
|
||||
return (tensor->ne[0] % 256 == 0) && (tensor->ne[1] % 64 == 0);
|
||||
@@ -25762,48 +25619,6 @@ static void ggml_cl_mul_mat_q2_0_f32_adreno(ggml_backend_t backend, const ggml_t
|
||||
CL_CHECK(clReleaseMemObject(b_sub_buf));
|
||||
}
|
||||
|
||||
static bool ggml_cl_mul_mat_coopmat(ggml_backend_opencl_context * backend_ctx, cl_kernel kernel,
|
||||
std::initializer_list<cl_mem> weights,
|
||||
const ggml_tensor * src1, ggml_tensor * dst, int K, int M, int N) {
|
||||
if (kernel == nullptr || M % 64 != 0 || N < 2) {
|
||||
return false;
|
||||
}
|
||||
ggml_tensor_extra_cl * extra1 = (ggml_tensor_extra_cl *)src1->extra;
|
||||
ggml_tensor_extra_cl * extrad = (ggml_tensor_extra_cl *)dst->extra;
|
||||
cl_ulong offset1 = extra1->offset + src1->view_offs;
|
||||
cl_ulong offsetd = extrad->offset + dst->view_offs;
|
||||
|
||||
const int n_tiles = CEIL_DIV(N, 64);
|
||||
|
||||
backend_ctx->prealloc_act_trans.allocate(backend_ctx->context, (size_t)n_tiles * 64 * K * sizeof(cl_half));
|
||||
|
||||
cl_kernel pk = backend_ctx->kernel_pack_b_f16_coopmat;
|
||||
CL_CHECK(clSetKernelArg(pk, 0, sizeof(cl_mem), &extra1->data_device));
|
||||
CL_CHECK(clSetKernelArg(pk, 1, sizeof(cl_ulong), &offset1));
|
||||
CL_CHECK(clSetKernelArg(pk, 2, sizeof(cl_mem), &backend_ctx->prealloc_act_trans.buffer));
|
||||
CL_CHECK(clSetKernelArg(pk, 3, sizeof(int), &K));
|
||||
CL_CHECK(clSetKernelArg(pk, 4, sizeof(int), &N));
|
||||
size_t p_local[2] = { 64, 4 };
|
||||
size_t p_global[2] = { (size_t)n_tiles * 64, (size_t)CEIL_DIV(K, 64) * 4 };
|
||||
backend_ctx->enqueue_ndrange_kernel(pk, 2, p_global, p_local, dst);
|
||||
|
||||
cl_uint ka = 0;
|
||||
for (cl_mem w : weights) {
|
||||
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(cl_mem), &w));
|
||||
}
|
||||
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(cl_mem), &backend_ctx->prealloc_act_trans.buffer));
|
||||
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(cl_mem), &extrad->data_device));
|
||||
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(cl_ulong), &offsetd));
|
||||
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(int), &K));
|
||||
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(int), &M));
|
||||
CL_CHECK(clSetKernelArg(kernel, ka++, sizeof(int), &N));
|
||||
|
||||
size_t c_local[3] = { 64, 1, 4 };
|
||||
size_t c_global[3] = { 64, (size_t)(M / 64), (size_t)CEIL_DIV(n_tiles, 4) * 4 };
|
||||
backend_ctx->enqueue_ndrange_kernel(kernel, 3, c_global, c_local, dst);
|
||||
return true;
|
||||
}
|
||||
|
||||
static void ggml_cl_mul_mat_q4_0_f32_adreno_ila(ggml_backend_t backend, const ggml_tensor * src0,
|
||||
const ggml_tensor * src1, ggml_tensor * dst) {
|
||||
GGML_ASSERT(src0);
|
||||
@@ -26119,11 +25934,6 @@ static void ggml_cl_mul_mat_q4_0_f32_adreno(ggml_backend_t backend, const ggml_t
|
||||
|
||||
const bool use_bin = use_q4_0_bin_kernels(backend_ctx, src0);
|
||||
|
||||
if (use_bin && ggml_cl_mul_mat_coopmat(backend_ctx, backend_ctx->kernel_gemm_noshuffle_q4_0_f32_coopmat,
|
||||
{ extra0_q4_0->q, extra0_q4_0->d }, src1, dst, K, M, N)) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (use_bin) {
|
||||
if (use_q40_mc3) {
|
||||
static bool warned = false;
|
||||
@@ -26688,11 +26498,6 @@ static void ggml_cl_mul_mat_q5_0_f32_adreno(ggml_backend_t backend, const ggml_t
|
||||
int N = ne1;
|
||||
int K = ne00;
|
||||
|
||||
if (ggml_cl_mul_mat_coopmat(backend_ctx, backend_ctx->kernel_gemm_noshuffle_q5_0_f32_coopmat,
|
||||
{ extra0_q5_0->qs, extra0_q5_0->qh, extra0_q5_0->d }, src1, dst, K, M, N)) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (ne1 == 1) {
|
||||
cl_mem qs_img = nullptr;
|
||||
cl_mem b_sub_buf = nullptr;
|
||||
@@ -27535,11 +27340,6 @@ static void ggml_cl_mul_mat_q8_0_f32_adreno(ggml_backend_t backend, const ggml_t
|
||||
int N = ne1;
|
||||
int K = ne00;
|
||||
|
||||
if (ggml_cl_mul_mat_coopmat(backend_ctx, backend_ctx->kernel_gemm_noshuffle_q8_0_f32_coopmat,
|
||||
{ extra0_q8_0->q, extra0_q8_0->d }, src1, dst, K, M, N)) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Multi-column verify GEMV for ne1 2..4, reuses the ne1 == 1 image setup.
|
||||
// Default on for X2E, GGML_OPENCL_Q8_0_MC=0/1 forces it.
|
||||
static const char * q8_mc_env = getenv("GGML_OPENCL_Q8_0_MC");
|
||||
@@ -28521,11 +28321,6 @@ static void ggml_cl_mul_mat_q4_k_f32_adreno(ggml_backend_t backend, const ggml_t
|
||||
|
||||
const bool use_bin = use_q4_k_bin_kernels(backend_ctx, src0);
|
||||
|
||||
if (use_bin && ggml_cl_mul_mat_coopmat(backend_ctx, backend_ctx->kernel_gemm_noshuffle_q4_k_f32_coopmat,
|
||||
{ extra0_q4_k->q, extra0_q4_k->s, extra0_q4_k->d, extra0_q4_k->dm }, src1, dst, K, M, N)) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (use_bin) {
|
||||
if (use_mc3) {
|
||||
static bool warned = false;
|
||||
@@ -29436,11 +29231,6 @@ static void ggml_cl_mul_mat_q6_K_f32_adreno(ggml_backend_t backend, const ggml_t
|
||||
|
||||
const bool use_bin = use_q6_k_bin_kernels(backend_ctx, src0);
|
||||
|
||||
if (use_bin && ggml_cl_mul_mat_coopmat(backend_ctx, backend_ctx->kernel_gemm_noshuffle_q6_k_f32_coopmat,
|
||||
{ extra0_q6_K->ql, extra0_q6_K->qh, extra0_q6_K->s, extra0_q6_K->d }, src1, dst, ne00, ne01, ne1)) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (use_bin) {
|
||||
if (use_q6k_mc3 || use_q6k_tiled_mc) {
|
||||
static bool warned = false;
|
||||
@@ -30181,11 +29971,6 @@ static void ggml_cl_mul_mat_q5_K_f32_adreno(ggml_backend_t backend, const ggml_t
|
||||
|
||||
const bool use_bin = use_q5_k_bin_kernels(backend_ctx, src0);
|
||||
|
||||
if (use_bin && ggml_cl_mul_mat_coopmat(backend_ctx, backend_ctx->kernel_gemm_noshuffle_q5_k_f32_coopmat,
|
||||
{ extra0_q5_k->q, extra0_q5_k->qh, extra0_q5_k->s, extra0_q5_k->d, extra0_q5_k->dm }, src1, dst, K, M, N)) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (use_bin) {
|
||||
if (use_q5k_mc3) {
|
||||
static bool warned = false;
|
||||
|
||||
@@ -1,145 +0,0 @@
|
||||
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
|
||||
#pragma OPENCL EXTENSION cl_qcom_kernel_cooperative_matrix : enable
|
||||
|
||||
// Q4_0 x F32 GEMM with cl_qcom_kernel_cooperative_matrix: one 64x64 f16 C tile per wave,
|
||||
// A dequantized in registers, B packed by kernel_pack_b_f16_coopmat.
|
||||
|
||||
typedef half mat_a_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 16, CLK_COOPERATIVE_MATRIX_A_QCOM)));
|
||||
typedef half mat_b_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 16, 64, CLK_COOPERATIVE_MATRIX_B_QCOM)));
|
||||
typedef half mat_c_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 64, CLK_COOPERATIVE_MATRIX_ACCUMULATOR_QCOM)));
|
||||
|
||||
typedef half half1024 __attribute__((ext_vector_type(1024)));
|
||||
typedef half half4096 __attribute__((ext_vector_type(4096)));
|
||||
|
||||
typedef union {
|
||||
half1024 v;
|
||||
half16 h;
|
||||
} a_row_t;
|
||||
|
||||
typedef union {
|
||||
half4096 v;
|
||||
struct {
|
||||
half16 h0;
|
||||
half16 h1;
|
||||
half16 h2;
|
||||
half16 h3;
|
||||
} s;
|
||||
} c_row_t;
|
||||
|
||||
inline half8 dequant8(uint v, half d) {
|
||||
uint4 t;
|
||||
t.s0 = ( v & 0x000F) | ((v << 12) & 0x000F0000);
|
||||
t.s1 = ((v >> 8) & 0x000F) | ((v << 4) & 0x000F0000);
|
||||
t.s2 = ((v >> 16) & 0x000F) | ((v >> 4) & 0x000F0000);
|
||||
t.s3 = ((v >> 24) & 0x000F) | ((v >> 12) & 0x000F0000);
|
||||
t |= (uint4)(0x64006400);
|
||||
|
||||
return (as_half8(t) - (half8)(1032.0h)) * d;
|
||||
}
|
||||
|
||||
inline void store_c16(global float * p, half16 h, int j0, int nv, int M) {
|
||||
float16 f = convert_float16(h);
|
||||
if (j0 + 0 < nv) p[(ulong)(j0 + 0) * M] = f.s0;
|
||||
if (j0 + 1 < nv) p[(ulong)(j0 + 1) * M] = f.s1;
|
||||
if (j0 + 2 < nv) p[(ulong)(j0 + 2) * M] = f.s2;
|
||||
if (j0 + 3 < nv) p[(ulong)(j0 + 3) * M] = f.s3;
|
||||
if (j0 + 4 < nv) p[(ulong)(j0 + 4) * M] = f.s4;
|
||||
if (j0 + 5 < nv) p[(ulong)(j0 + 5) * M] = f.s5;
|
||||
if (j0 + 6 < nv) p[(ulong)(j0 + 6) * M] = f.s6;
|
||||
if (j0 + 7 < nv) p[(ulong)(j0 + 7) * M] = f.s7;
|
||||
if (j0 + 8 < nv) p[(ulong)(j0 + 8) * M] = f.s8;
|
||||
if (j0 + 9 < nv) p[(ulong)(j0 + 9) * M] = f.s9;
|
||||
if (j0 + 10 < nv) p[(ulong)(j0 + 10) * M] = f.sa;
|
||||
if (j0 + 11 < nv) p[(ulong)(j0 + 11) * M] = f.sb;
|
||||
if (j0 + 12 < nv) p[(ulong)(j0 + 12) * M] = f.sc;
|
||||
if (j0 + 13 < nv) p[(ulong)(j0 + 13) * M] = f.sd;
|
||||
if (j0 + 14 < nv) p[(ulong)(j0 + 14) * M] = f.se;
|
||||
if (j0 + 15 < nv) p[(ulong)(j0 + 15) * M] = f.sf;
|
||||
}
|
||||
|
||||
#define LOAD_Q(qv, dd, o) { \
|
||||
qv.s0 = qp[(4 * (o) + 0) * M]; \
|
||||
qv.s1 = qp[(4 * (o) + 1) * M]; \
|
||||
qv.s2 = qp[(4 * (o) + 2) * M]; \
|
||||
qv.s3 = qp[(4 * (o) + 3) * M]; \
|
||||
dd = dp[(o) * M]; }
|
||||
|
||||
#define MMA_K32(qv, dd, b) { \
|
||||
a_row_t ua0; \
|
||||
a_row_t ua1; \
|
||||
ua0.h = (half16)(dequant8(qv.s0, dd), dequant8(qv.s1, dd)); \
|
||||
ua1.h = (half16)(dequant8(qv.s2, dd), dequant8(qv.s3, dd)); \
|
||||
mat_a_t a0 = __builtin_astype(ua0.v, mat_a_t); \
|
||||
mat_a_t a1 = __builtin_astype(ua1.v, mat_a_t); \
|
||||
mat_b_t b0 = qcom_coop_mat_load((b), CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
|
||||
mat_b_t b1 = qcom_coop_mat_load((b) + 1024, CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
|
||||
acc = qcom_coop_mat_mulAdd(a0, b0, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); \
|
||||
acc = qcom_coop_mat_mulAdd(a1, b1, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); }
|
||||
|
||||
kernel void kernel_gemm_noshuffle_q4_0_f32_coopmat(
|
||||
global const uint * src0_q, // A: q4_0 quants, [K/8][M] u32
|
||||
global const half * src0_d, // A: q4_0 scales, [K/32][M] f16
|
||||
global const half * src1, // B: packed f16 tiles
|
||||
global char * dst, // C: [N][M] f32
|
||||
ulong offsetd,
|
||||
int K,
|
||||
int M,
|
||||
int N
|
||||
) {
|
||||
const int lane = get_local_id(0);
|
||||
const int row = get_global_id(1) * 64;
|
||||
const int tn = get_global_id(2);
|
||||
const int n_tiles = (N + 63) / 64;
|
||||
|
||||
if (tn >= n_tiles) {
|
||||
return;
|
||||
}
|
||||
|
||||
global const uint * qp = src0_q + row + lane;
|
||||
global const half * dp = src0_d + row + lane;
|
||||
global const half * bp = src1 + (ulong)tn * (K / 16) * 1024;
|
||||
|
||||
mat_c_t acc = qcom_coop_mat_init(0.0);
|
||||
|
||||
{
|
||||
const int odd = (K / 32) & 1;
|
||||
const int ob = odd ? 0 : 1;
|
||||
uint4 qa;
|
||||
uint4 qb;
|
||||
half da;
|
||||
half db;
|
||||
LOAD_Q(qa, da, 0)
|
||||
LOAD_Q(qb, db, ob)
|
||||
da = odd ? 0.0h : da;
|
||||
MMA_K32(qa, da, bp)
|
||||
MMA_K32(qb, db, bp + ob * 2048)
|
||||
qp += (ob + 1) * 4 * M;
|
||||
dp += (ob + 1) * M;
|
||||
bp += (ob + 1) * 2048;
|
||||
}
|
||||
|
||||
#pragma unroll 1
|
||||
for (int k0 = (K / 32) & 1 ? 32 : 64; k0 < K; k0 += 64) {
|
||||
uint4 qa;
|
||||
uint4 qb;
|
||||
half da;
|
||||
half db;
|
||||
LOAD_Q(qa, da, 0)
|
||||
LOAD_Q(qb, db, 1)
|
||||
MMA_K32(qa, da, bp)
|
||||
MMA_K32(qb, db, bp + 2048)
|
||||
qp += 8 * M;
|
||||
dp += 2 * M;
|
||||
bp += 4096;
|
||||
}
|
||||
|
||||
c_row_t uc;
|
||||
uc.v = __builtin_astype(acc, half4096);
|
||||
|
||||
global float * p = (global float *)(dst + offsetd) + (ulong)tn * 64 * M + row + lane;
|
||||
const int nv = N - tn * 64;
|
||||
store_c16(p, uc.s.h0, 0, nv, M);
|
||||
store_c16(p, uc.s.h1, 16, nv, M);
|
||||
store_c16(p, uc.s.h2, 32, nv, M);
|
||||
store_c16(p, uc.s.h3, 48, nv, M);
|
||||
}
|
||||
@@ -1,148 +0,0 @@
|
||||
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
|
||||
#pragma OPENCL EXTENSION cl_qcom_kernel_cooperative_matrix : enable
|
||||
|
||||
// Q4_K x F32 GEMM with cl_qcom_kernel_cooperative_matrix: one 64x64 f16 C tile per wave,
|
||||
// A dequantized in registers, B packed by kernel_pack_b_f16_coopmat.
|
||||
|
||||
#define K_SCALE_SIZE 12
|
||||
|
||||
typedef half mat_a_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 16, CLK_COOPERATIVE_MATRIX_A_QCOM)));
|
||||
typedef half mat_b_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 16, 64, CLK_COOPERATIVE_MATRIX_B_QCOM)));
|
||||
typedef half mat_c_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 64, CLK_COOPERATIVE_MATRIX_ACCUMULATOR_QCOM)));
|
||||
|
||||
typedef half half1024 __attribute__((ext_vector_type(1024)));
|
||||
typedef half half4096 __attribute__((ext_vector_type(4096)));
|
||||
|
||||
typedef union {
|
||||
half1024 v;
|
||||
half16 h;
|
||||
} a_row_t;
|
||||
|
||||
typedef union {
|
||||
half4096 v;
|
||||
struct {
|
||||
half16 h0;
|
||||
half16 h1;
|
||||
half16 h2;
|
||||
half16 h3;
|
||||
} s;
|
||||
} c_row_t;
|
||||
|
||||
inline half8 dequant8(uint v, half scale, half nminv) {
|
||||
uint4 t;
|
||||
t.s0 = ( v & 0x000F) | ((v << 12) & 0x000F0000);
|
||||
t.s1 = ((v >> 8) & 0x000F) | ((v << 4) & 0x000F0000);
|
||||
t.s2 = ((v >> 16) & 0x000F) | ((v >> 4) & 0x000F0000);
|
||||
t.s3 = ((v >> 24) & 0x000F) | ((v >> 12) & 0x000F0000);
|
||||
t |= (uint4)(0x64006400);
|
||||
|
||||
return (as_half8(t) - (half8)(1024.0h)) * scale + nminv;
|
||||
}
|
||||
|
||||
inline void store_c16(global float * p, half16 h, int j0, int nv, int M) {
|
||||
float16 f = convert_float16(h);
|
||||
if (j0 + 0 < nv) p[(ulong)(j0 + 0) * M] = f.s0;
|
||||
if (j0 + 1 < nv) p[(ulong)(j0 + 1) * M] = f.s1;
|
||||
if (j0 + 2 < nv) p[(ulong)(j0 + 2) * M] = f.s2;
|
||||
if (j0 + 3 < nv) p[(ulong)(j0 + 3) * M] = f.s3;
|
||||
if (j0 + 4 < nv) p[(ulong)(j0 + 4) * M] = f.s4;
|
||||
if (j0 + 5 < nv) p[(ulong)(j0 + 5) * M] = f.s5;
|
||||
if (j0 + 6 < nv) p[(ulong)(j0 + 6) * M] = f.s6;
|
||||
if (j0 + 7 < nv) p[(ulong)(j0 + 7) * M] = f.s7;
|
||||
if (j0 + 8 < nv) p[(ulong)(j0 + 8) * M] = f.s8;
|
||||
if (j0 + 9 < nv) p[(ulong)(j0 + 9) * M] = f.s9;
|
||||
if (j0 + 10 < nv) p[(ulong)(j0 + 10) * M] = f.sa;
|
||||
if (j0 + 11 < nv) p[(ulong)(j0 + 11) * M] = f.sb;
|
||||
if (j0 + 12 < nv) p[(ulong)(j0 + 12) * M] = f.sc;
|
||||
if (j0 + 13 < nv) p[(ulong)(j0 + 13) * M] = f.sd;
|
||||
if (j0 + 14 < nv) p[(ulong)(j0 + 14) * M] = f.se;
|
||||
if (j0 + 15 < nv) p[(ulong)(j0 + 15) * M] = f.sf;
|
||||
}
|
||||
|
||||
#define MMA_K32(q0, q1, q2, q3, sc, nm, b) { \
|
||||
a_row_t ua0; \
|
||||
a_row_t ua1; \
|
||||
ua0.h = (half16)(dequant8(q0, sc, nm), dequant8(q1, sc, nm)); \
|
||||
ua1.h = (half16)(dequant8(q2, sc, nm), dequant8(q3, sc, nm)); \
|
||||
mat_a_t a0 = __builtin_astype(ua0.v, mat_a_t); \
|
||||
mat_a_t a1 = __builtin_astype(ua1.v, mat_a_t); \
|
||||
mat_b_t b0 = qcom_coop_mat_load((b), CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
|
||||
mat_b_t b1 = qcom_coop_mat_load((b) + 1024, CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
|
||||
acc = qcom_coop_mat_mulAdd(a0, b0, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); \
|
||||
acc = qcom_coop_mat_mulAdd(a1, b1, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); }
|
||||
|
||||
#define STEP64(sa, na, sb_, nb_) { \
|
||||
uint8 qv; \
|
||||
qv.s0 = qp[0]; \
|
||||
qv.s1 = qp[M]; \
|
||||
qv.s2 = qp[2 * M]; \
|
||||
qv.s3 = qp[3 * M]; \
|
||||
qv.s4 = qp[4 * M]; \
|
||||
qv.s5 = qp[5 * M]; \
|
||||
qv.s6 = qp[6 * M]; \
|
||||
qv.s7 = qp[7 * M]; \
|
||||
qp += 8 * M; \
|
||||
MMA_K32(qv.s0, qv.s1, qv.s2, qv.s3, sa, na, bp) \
|
||||
MMA_K32(qv.s4, qv.s5, qv.s6, qv.s7, sb_, nb_, bp + 2048) \
|
||||
bp += 4096; }
|
||||
|
||||
kernel void kernel_gemm_noshuffle_q4_k_f32_coopmat(
|
||||
global const uint * src0_q, // A: q4_K quants, [K/8][M] u32
|
||||
global const uchar * src0_s, // A: 6-bit scale/min codes, [K/256*12][M] u8
|
||||
global const half * src0_d, // A: per-superblock scale, [K/256][M] f16
|
||||
global const half * src0_dm, // A: per-superblock min, [K/256][M] f16
|
||||
global const half * src1, // B: packed f16 tiles
|
||||
global char * dst, // C: [N][M] f32
|
||||
ulong offsetd,
|
||||
int K,
|
||||
int M,
|
||||
int N
|
||||
) {
|
||||
const int lane = get_local_id(0);
|
||||
const int row = get_global_id(1) * 64;
|
||||
const int tn = get_global_id(2);
|
||||
const int n_tiles = (N + 63) / 64;
|
||||
|
||||
if (tn >= n_tiles) {
|
||||
return;
|
||||
}
|
||||
|
||||
const int lrow = row + lane;
|
||||
|
||||
global const uint * qp = src0_q + lrow;
|
||||
global const half * bp = src1 + (ulong)tn * (K / 16) * 1024;
|
||||
|
||||
mat_c_t acc = qcom_coop_mat_init(0.0);
|
||||
|
||||
#pragma unroll 1
|
||||
for (int k0 = 0; k0 < K; k0 += 128) {
|
||||
const int sb = k0 >> 8;
|
||||
const half dd = src0_d [(ulong)sb * M + lrow];
|
||||
const half dmm = src0_dm[(ulong)sb * M + lrow];
|
||||
|
||||
global const uchar * sp = src0_s + (ulong)(sb * K_SCALE_SIZE) * M + lrow;
|
||||
const uchar4 c0 = (uchar4)(sp[0], sp[M], sp[2 * M], sp[3 * M]);
|
||||
const uchar4 c1 = (uchar4)(sp[4 * M], sp[5 * M], sp[6 * M], sp[7 * M]);
|
||||
const uchar4 c2 = (uchar4)(sp[8 * M], sp[9 * M], sp[10 * M], sp[11 * M]);
|
||||
const uchar4 sl = c0 & (uchar)0x3F;
|
||||
const uchar4 ml = c1 & (uchar)0x3F;
|
||||
const uchar4 sh = (c2 & (uchar)0x0F) | ((c0 >> (uchar)2) & (uchar)0x30);
|
||||
const uchar4 mh = (c2 >> (uchar)4) | ((c1 >> (uchar)2) & (uchar)0x30);
|
||||
const int hi = (k0 >> 7) & 1;
|
||||
const half4 S = dd * convert_half4(hi ? sh : sl);
|
||||
const half4 Nm = -(dmm * convert_half4(hi ? mh : ml));
|
||||
|
||||
STEP64(S.s0, Nm.s0, S.s1, Nm.s1)
|
||||
STEP64(S.s2, Nm.s2, S.s3, Nm.s3)
|
||||
}
|
||||
|
||||
c_row_t uc;
|
||||
uc.v = __builtin_astype(acc, half4096);
|
||||
|
||||
global float * p = (global float *)(dst + offsetd) + (ulong)tn * 64 * M + row + lane;
|
||||
const int nv = N - tn * 64;
|
||||
store_c16(p, uc.s.h0, 0, nv, M);
|
||||
store_c16(p, uc.s.h1, 16, nv, M);
|
||||
store_c16(p, uc.s.h2, 32, nv, M);
|
||||
store_c16(p, uc.s.h3, 48, nv, M);
|
||||
}
|
||||
@@ -1,157 +0,0 @@
|
||||
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
|
||||
#pragma OPENCL EXTENSION cl_qcom_kernel_cooperative_matrix : enable
|
||||
|
||||
// Q5_0 x F32 GEMM with cl_qcom_kernel_cooperative_matrix: one 64x64 f16 C tile per wave,
|
||||
// A dequantized in registers, B packed by kernel_pack_b_f16_coopmat.
|
||||
|
||||
typedef half mat_a_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 16, CLK_COOPERATIVE_MATRIX_A_QCOM)));
|
||||
typedef half mat_b_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 16, 64, CLK_COOPERATIVE_MATRIX_B_QCOM)));
|
||||
typedef half mat_c_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 64, CLK_COOPERATIVE_MATRIX_ACCUMULATOR_QCOM)));
|
||||
|
||||
typedef half half1024 __attribute__((ext_vector_type(1024)));
|
||||
typedef half half4096 __attribute__((ext_vector_type(4096)));
|
||||
|
||||
typedef union {
|
||||
half1024 v;
|
||||
half16 h;
|
||||
} a_row_t;
|
||||
|
||||
typedef union {
|
||||
half4096 v;
|
||||
struct {
|
||||
half16 h0;
|
||||
half16 h1;
|
||||
half16 h2;
|
||||
half16 h3;
|
||||
} s;
|
||||
} c_row_t;
|
||||
|
||||
inline half8 dequant8(uint v, uint h, half d) {
|
||||
uint4 t;
|
||||
t.s0 = ( v & 0x000F) | ((v << 12) & 0x000F0000) | ((h << 4) & 0x0010) | ((h << 19) & 0x00100000);
|
||||
t.s1 = ((v >> 8) & 0x000F) | ((v << 4) & 0x000F0000) | ((h << 2) & 0x0010) | ((h << 17) & 0x00100000);
|
||||
t.s2 = ((v >> 16) & 0x000F) | ((v >> 4) & 0x000F0000) | ( h & 0x0010) | ((h << 15) & 0x00100000);
|
||||
t.s3 = ((v >> 24) & 0x000F) | ((v >> 12) & 0x000F0000) | ((h >> 2) & 0x0010) | ((h << 13) & 0x00100000);
|
||||
t |= (uint4)(0x64006400);
|
||||
|
||||
return (as_half8(t) - (half8)(1040.0h)) * d;
|
||||
}
|
||||
|
||||
inline void store_c16(global float * p, half16 h, int j0, int nv, int M) {
|
||||
float16 f = convert_float16(h);
|
||||
if (j0 + 0 < nv) p[(ulong)(j0 + 0) * M] = f.s0;
|
||||
if (j0 + 1 < nv) p[(ulong)(j0 + 1) * M] = f.s1;
|
||||
if (j0 + 2 < nv) p[(ulong)(j0 + 2) * M] = f.s2;
|
||||
if (j0 + 3 < nv) p[(ulong)(j0 + 3) * M] = f.s3;
|
||||
if (j0 + 4 < nv) p[(ulong)(j0 + 4) * M] = f.s4;
|
||||
if (j0 + 5 < nv) p[(ulong)(j0 + 5) * M] = f.s5;
|
||||
if (j0 + 6 < nv) p[(ulong)(j0 + 6) * M] = f.s6;
|
||||
if (j0 + 7 < nv) p[(ulong)(j0 + 7) * M] = f.s7;
|
||||
if (j0 + 8 < nv) p[(ulong)(j0 + 8) * M] = f.s8;
|
||||
if (j0 + 9 < nv) p[(ulong)(j0 + 9) * M] = f.s9;
|
||||
if (j0 + 10 < nv) p[(ulong)(j0 + 10) * M] = f.sa;
|
||||
if (j0 + 11 < nv) p[(ulong)(j0 + 11) * M] = f.sb;
|
||||
if (j0 + 12 < nv) p[(ulong)(j0 + 12) * M] = f.sc;
|
||||
if (j0 + 13 < nv) p[(ulong)(j0 + 13) * M] = f.sd;
|
||||
if (j0 + 14 < nv) p[(ulong)(j0 + 14) * M] = f.se;
|
||||
if (j0 + 15 < nv) p[(ulong)(j0 + 15) * M] = f.sf;
|
||||
}
|
||||
|
||||
#define LOAD_Q(qv, hv, dd, o) { \
|
||||
qv.s0 = (uint)qp[(8 * (o) + 0) * M] | ((uint)qp[(8 * (o) + 1) * M] << 16); \
|
||||
qv.s1 = (uint)qp[(8 * (o) + 2) * M] | ((uint)qp[(8 * (o) + 3) * M] << 16); \
|
||||
qv.s2 = (uint)qp[(8 * (o) + 4) * M] | ((uint)qp[(8 * (o) + 5) * M] << 16); \
|
||||
qv.s3 = (uint)qp[(8 * (o) + 6) * M] | ((uint)qp[(8 * (o) + 7) * M] << 16); \
|
||||
hv.s0 = hp[(4 * (o) + 0) * M]; \
|
||||
hv.s1 = hp[(4 * (o) + 1) * M]; \
|
||||
hv.s2 = hp[(4 * (o) + 2) * M]; \
|
||||
hv.s3 = hp[(4 * (o) + 3) * M]; \
|
||||
dd = dp[(o) * M]; }
|
||||
|
||||
#define MMA_K32(qv, hv, dd, b) { \
|
||||
a_row_t ua0; \
|
||||
a_row_t ua1; \
|
||||
ua0.h = (half16)(dequant8(qv.s0, hv.s0, dd), dequant8(qv.s1, hv.s1, dd)); \
|
||||
ua1.h = (half16)(dequant8(qv.s2, hv.s2, dd), dequant8(qv.s3, hv.s3, dd)); \
|
||||
mat_a_t a0 = __builtin_astype(ua0.v, mat_a_t); \
|
||||
mat_a_t a1 = __builtin_astype(ua1.v, mat_a_t); \
|
||||
mat_b_t b0 = qcom_coop_mat_load((b), CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
|
||||
mat_b_t b1 = qcom_coop_mat_load((b) + 1024, CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
|
||||
acc = qcom_coop_mat_mulAdd(a0, b0, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); \
|
||||
acc = qcom_coop_mat_mulAdd(a1, b1, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); }
|
||||
|
||||
kernel void kernel_gemm_noshuffle_q5_0_f32_coopmat(
|
||||
global const ushort * src0_q, // A: q5_0 quants, [K/4][M] u16
|
||||
global const uchar * src0_qh, // A: q5_0 high bits, [K/8][M] u8
|
||||
global const half * src0_d, // A: q5_0 scales, [K/32][M] f16
|
||||
global const half * src1, // B: packed f16 tiles
|
||||
global char * dst, // C: [N][M] f32
|
||||
ulong offsetd,
|
||||
int K,
|
||||
int M,
|
||||
int N
|
||||
) {
|
||||
const int lane = get_local_id(0);
|
||||
const int row = get_global_id(1) * 64;
|
||||
const int tn = get_global_id(2);
|
||||
const int n_tiles = (N + 63) / 64;
|
||||
|
||||
if (tn >= n_tiles) {
|
||||
return;
|
||||
}
|
||||
|
||||
global const ushort * qp = src0_q + row + lane;
|
||||
global const uchar * hp = src0_qh + row + lane;
|
||||
global const half * dp = src0_d + row + lane;
|
||||
global const half * bp = src1 + (ulong)tn * (K / 16) * 1024;
|
||||
|
||||
mat_c_t acc = qcom_coop_mat_init(0.0);
|
||||
|
||||
{
|
||||
const int odd = (K / 32) & 1;
|
||||
const int ob = odd ? 0 : 1;
|
||||
uint4 qa;
|
||||
uint4 qb;
|
||||
uint4 ha;
|
||||
uint4 hb;
|
||||
half da;
|
||||
half db;
|
||||
LOAD_Q(qa, ha, da, 0)
|
||||
LOAD_Q(qb, hb, db, ob)
|
||||
da = odd ? 0.0h : da;
|
||||
MMA_K32(qa, ha, da, bp)
|
||||
MMA_K32(qb, hb, db, bp + ob * 2048)
|
||||
qp += (ob + 1) * 8 * M;
|
||||
hp += (ob + 1) * 4 * M;
|
||||
dp += (ob + 1) * M;
|
||||
bp += (ob + 1) * 2048;
|
||||
}
|
||||
|
||||
#pragma unroll 1
|
||||
for (int k0 = (K / 32) & 1 ? 32 : 64; k0 < K; k0 += 64) {
|
||||
uint4 qa;
|
||||
uint4 qb;
|
||||
uint4 ha;
|
||||
uint4 hb;
|
||||
half da;
|
||||
half db;
|
||||
LOAD_Q(qa, ha, da, 0)
|
||||
LOAD_Q(qb, hb, db, 1)
|
||||
MMA_K32(qa, ha, da, bp)
|
||||
MMA_K32(qb, hb, db, bp + 2048)
|
||||
qp += 16 * M;
|
||||
hp += 8 * M;
|
||||
dp += 2 * M;
|
||||
bp += 4096;
|
||||
}
|
||||
|
||||
c_row_t uc;
|
||||
uc.v = __builtin_astype(acc, half4096);
|
||||
|
||||
global float * p = (global float *)(dst + offsetd) + (ulong)tn * 64 * M + row + lane;
|
||||
const int nv = N - tn * 64;
|
||||
store_c16(p, uc.s.h0, 0, nv, M);
|
||||
store_c16(p, uc.s.h1, 16, nv, M);
|
||||
store_c16(p, uc.s.h2, 32, nv, M);
|
||||
store_c16(p, uc.s.h3, 48, nv, M);
|
||||
}
|
||||
@@ -1,164 +0,0 @@
|
||||
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
|
||||
#pragma OPENCL EXTENSION cl_qcom_kernel_cooperative_matrix : enable
|
||||
|
||||
// Q5_K x F32 GEMM with cl_qcom_kernel_cooperative_matrix: one 64x64 f16 C tile per wave,
|
||||
// A dequantized in registers, B packed by kernel_pack_b_f16_coopmat.
|
||||
|
||||
#define K_SCALE_SIZE 12
|
||||
|
||||
typedef half mat_a_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 16, CLK_COOPERATIVE_MATRIX_A_QCOM)));
|
||||
typedef half mat_b_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 16, 64, CLK_COOPERATIVE_MATRIX_B_QCOM)));
|
||||
typedef half mat_c_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 64, CLK_COOPERATIVE_MATRIX_ACCUMULATOR_QCOM)));
|
||||
|
||||
typedef half half1024 __attribute__((ext_vector_type(1024)));
|
||||
typedef half half4096 __attribute__((ext_vector_type(4096)));
|
||||
|
||||
typedef union {
|
||||
half1024 v;
|
||||
half16 h;
|
||||
} a_row_t;
|
||||
|
||||
typedef union {
|
||||
half4096 v;
|
||||
struct {
|
||||
half16 h0;
|
||||
half16 h1;
|
||||
half16 h2;
|
||||
half16 h3;
|
||||
} s;
|
||||
} c_row_t;
|
||||
|
||||
inline half8 dequant8(uint v, uint h, half scale, half nminv) {
|
||||
uint4 t;
|
||||
t.s0 = ( v & 0x000F) | ((v << 12) & 0x000F0000);
|
||||
t.s1 = ((v >> 8) & 0x000F) | ((v << 4) & 0x000F0000);
|
||||
t.s2 = ((v >> 16) & 0x000F) | ((v >> 4) & 0x000F0000);
|
||||
t.s3 = ((v >> 24) & 0x000F) | ((v >> 12) & 0x000F0000);
|
||||
t.s0 |= ((h & 0x01u) << 4) | ((h & 0x02u) << 19);
|
||||
t.s1 |= ((h & 0x04u) << 2) | ((h & 0x08u) << 17);
|
||||
t.s2 |= ( h & 0x10u ) | ((h & 0x20u) << 15);
|
||||
t.s3 |= ((h & 0x40u) >> 2) | ((h & 0x80u) << 13);
|
||||
t |= (uint4)(0x64006400);
|
||||
|
||||
return (as_half8(t) - (half8)(1024.0h)) * scale + nminv;
|
||||
}
|
||||
|
||||
inline void store_c16(global float * p, half16 h, int j0, int nv, int M) {
|
||||
float16 f = convert_float16(h);
|
||||
if (j0 + 0 < nv) p[(ulong)(j0 + 0) * M] = f.s0;
|
||||
if (j0 + 1 < nv) p[(ulong)(j0 + 1) * M] = f.s1;
|
||||
if (j0 + 2 < nv) p[(ulong)(j0 + 2) * M] = f.s2;
|
||||
if (j0 + 3 < nv) p[(ulong)(j0 + 3) * M] = f.s3;
|
||||
if (j0 + 4 < nv) p[(ulong)(j0 + 4) * M] = f.s4;
|
||||
if (j0 + 5 < nv) p[(ulong)(j0 + 5) * M] = f.s5;
|
||||
if (j0 + 6 < nv) p[(ulong)(j0 + 6) * M] = f.s6;
|
||||
if (j0 + 7 < nv) p[(ulong)(j0 + 7) * M] = f.s7;
|
||||
if (j0 + 8 < nv) p[(ulong)(j0 + 8) * M] = f.s8;
|
||||
if (j0 + 9 < nv) p[(ulong)(j0 + 9) * M] = f.s9;
|
||||
if (j0 + 10 < nv) p[(ulong)(j0 + 10) * M] = f.sa;
|
||||
if (j0 + 11 < nv) p[(ulong)(j0 + 11) * M] = f.sb;
|
||||
if (j0 + 12 < nv) p[(ulong)(j0 + 12) * M] = f.sc;
|
||||
if (j0 + 13 < nv) p[(ulong)(j0 + 13) * M] = f.sd;
|
||||
if (j0 + 14 < nv) p[(ulong)(j0 + 14) * M] = f.se;
|
||||
if (j0 + 15 < nv) p[(ulong)(j0 + 15) * M] = f.sf;
|
||||
}
|
||||
|
||||
#define MMA_K32(q0, q1, q2, q3, h0_, h1_, h2_, h3_, sc, nm, b) { \
|
||||
a_row_t ua0; \
|
||||
a_row_t ua1; \
|
||||
ua0.h = (half16)(dequant8(q0, h0_, sc, nm), dequant8(q1, h1_, sc, nm)); \
|
||||
ua1.h = (half16)(dequant8(q2, h2_, sc, nm), dequant8(q3, h3_, sc, nm)); \
|
||||
mat_a_t a0 = __builtin_astype(ua0.v, mat_a_t); \
|
||||
mat_a_t a1 = __builtin_astype(ua1.v, mat_a_t); \
|
||||
mat_b_t b0 = qcom_coop_mat_load((b), CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
|
||||
mat_b_t b1 = qcom_coop_mat_load((b) + 1024, CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
|
||||
acc = qcom_coop_mat_mulAdd(a0, b0, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); \
|
||||
acc = qcom_coop_mat_mulAdd(a1, b1, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); }
|
||||
|
||||
#define STEP64(sa, na, sb_, nb_) { \
|
||||
uint8 qv; \
|
||||
qv.s0 = qp[0]; \
|
||||
qv.s1 = qp[M]; \
|
||||
qv.s2 = qp[2 * M]; \
|
||||
qv.s3 = qp[3 * M]; \
|
||||
qv.s4 = qp[4 * M]; \
|
||||
qv.s5 = qp[5 * M]; \
|
||||
qv.s6 = qp[6 * M]; \
|
||||
qv.s7 = qp[7 * M]; \
|
||||
qp += 8 * M; \
|
||||
uint8 hv; \
|
||||
hv.s0 = qhp[0]; \
|
||||
hv.s1 = qhp[M]; \
|
||||
hv.s2 = qhp[2 * M]; \
|
||||
hv.s3 = qhp[3 * M]; \
|
||||
hv.s4 = qhp[4 * M]; \
|
||||
hv.s5 = qhp[5 * M]; \
|
||||
hv.s6 = qhp[6 * M]; \
|
||||
hv.s7 = qhp[7 * M]; \
|
||||
qhp += 8 * M; \
|
||||
MMA_K32(qv.s0, qv.s1, qv.s2, qv.s3, hv.s0, hv.s1, hv.s2, hv.s3, sa, na, bp) \
|
||||
MMA_K32(qv.s4, qv.s5, qv.s6, qv.s7, hv.s4, hv.s5, hv.s6, hv.s7, sb_, nb_, bp + 2048) \
|
||||
bp += 4096; }
|
||||
|
||||
kernel void kernel_gemm_noshuffle_q5_k_f32_coopmat(
|
||||
global const uint * src0_q, // A: q5_K low nibbles, [K/8][M] u32
|
||||
global const uchar * src0_qh, // A: q5_K high bits, [K/8][M] u8
|
||||
global const uchar * src0_s, // A: 6-bit scale/min codes, [K/256*12][M] u8
|
||||
global const half * src0_d, // A: per-superblock scale, [K/256][M] f16
|
||||
global const half * src0_dm, // A: per-superblock min, [K/256][M] f16
|
||||
global const half * src1, // B: packed f16 tiles
|
||||
global char * dst, // C: [N][M] f32
|
||||
ulong offsetd,
|
||||
int K,
|
||||
int M,
|
||||
int N
|
||||
) {
|
||||
const int lane = get_local_id(0);
|
||||
const int row = get_global_id(1) * 64;
|
||||
const int tn = get_global_id(2);
|
||||
const int n_tiles = (N + 63) / 64;
|
||||
|
||||
if (tn >= n_tiles) {
|
||||
return;
|
||||
}
|
||||
|
||||
const int lrow = row + lane;
|
||||
|
||||
global const uint * qp = src0_q + lrow;
|
||||
global const uchar * qhp = src0_qh + lrow;
|
||||
global const half * bp = src1 + (ulong)tn * (K / 16) * 1024;
|
||||
|
||||
mat_c_t acc = qcom_coop_mat_init(0.0);
|
||||
|
||||
#pragma unroll 1
|
||||
for (int k0 = 0; k0 < K; k0 += 128) {
|
||||
const int sb = k0 >> 8;
|
||||
const half dd = src0_d [(ulong)sb * M + lrow];
|
||||
const half dmm = src0_dm[(ulong)sb * M + lrow];
|
||||
|
||||
global const uchar * sp = src0_s + (ulong)(sb * K_SCALE_SIZE) * M + lrow;
|
||||
const uchar4 c0 = (uchar4)(sp[0], sp[M], sp[2 * M], sp[3 * M]);
|
||||
const uchar4 c1 = (uchar4)(sp[4 * M], sp[5 * M], sp[6 * M], sp[7 * M]);
|
||||
const uchar4 c2 = (uchar4)(sp[8 * M], sp[9 * M], sp[10 * M], sp[11 * M]);
|
||||
const uchar4 sl = c0 & (uchar)0x3F;
|
||||
const uchar4 ml = c1 & (uchar)0x3F;
|
||||
const uchar4 sh = (c2 & (uchar)0x0F) | ((c0 >> (uchar)2) & (uchar)0x30);
|
||||
const uchar4 mh = (c2 >> (uchar)4) | ((c1 >> (uchar)2) & (uchar)0x30);
|
||||
const int hi = (k0 >> 7) & 1;
|
||||
const half4 S = dd * convert_half4(hi ? sh : sl);
|
||||
const half4 Nm = -(dmm * convert_half4(hi ? mh : ml));
|
||||
|
||||
STEP64(S.s0, Nm.s0, S.s1, Nm.s1)
|
||||
STEP64(S.s2, Nm.s2, S.s3, Nm.s3)
|
||||
}
|
||||
|
||||
c_row_t uc;
|
||||
uc.v = __builtin_astype(acc, half4096);
|
||||
|
||||
global float * p = (global float *)(dst + offsetd) + (ulong)tn * 64 * M + row + lane;
|
||||
const int nv = N - tn * 64;
|
||||
store_c16(p, uc.s.h0, 0, nv, M);
|
||||
store_c16(p, uc.s.h1, 16, nv, M);
|
||||
store_c16(p, uc.s.h2, 32, nv, M);
|
||||
store_c16(p, uc.s.h3, 48, nv, M);
|
||||
}
|
||||
@@ -1,145 +0,0 @@
|
||||
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
|
||||
#pragma OPENCL EXTENSION cl_qcom_kernel_cooperative_matrix : enable
|
||||
|
||||
// Q6_K x F32 GEMM with cl_qcom_kernel_cooperative_matrix: one 64x64 f16 C tile per wave,
|
||||
// A dequantized in registers, B packed by kernel_pack_b_f16_coopmat.
|
||||
|
||||
typedef half mat_a_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 16, CLK_COOPERATIVE_MATRIX_A_QCOM)));
|
||||
typedef half mat_b_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 16, 64, CLK_COOPERATIVE_MATRIX_B_QCOM)));
|
||||
typedef half mat_c_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 64, CLK_COOPERATIVE_MATRIX_ACCUMULATOR_QCOM)));
|
||||
|
||||
typedef half half1024 __attribute__((ext_vector_type(1024)));
|
||||
typedef half half4096 __attribute__((ext_vector_type(4096)));
|
||||
|
||||
typedef union {
|
||||
half1024 v;
|
||||
half16 h;
|
||||
} a_row_t;
|
||||
|
||||
typedef union {
|
||||
half4096 v;
|
||||
struct {
|
||||
half16 h0;
|
||||
half16 h1;
|
||||
half16 h2;
|
||||
half16 h3;
|
||||
} s;
|
||||
} c_row_t;
|
||||
|
||||
inline half8 dequant8(uint v, uint hh, half scale) {
|
||||
uint4 t;
|
||||
t.s0 = ( v & 0x000F) | ((v << 12) & 0x000F0000);
|
||||
t.s1 = ((v >> 8) & 0x000F) | ((v << 4) & 0x000F0000);
|
||||
t.s2 = ((v >> 16) & 0x000F) | ((v >> 4) & 0x000F0000);
|
||||
t.s3 = ((v >> 24) & 0x000F) | ((v >> 12) & 0x000F0000);
|
||||
t.s0 |= ((hh & 0x0003u) << 4) | ((hh & 0x000Cu) << 18);
|
||||
t.s1 |= ( hh & 0x0030u ) | ((hh & 0x00C0u) << 14);
|
||||
t.s2 |= ((hh & 0x0300u) >> 4) | ((hh & 0x0C00u) << 10);
|
||||
t.s3 |= ((hh & 0x3000u) >> 8) | ((hh & 0xC000u) << 6);
|
||||
t |= (uint4)(0x64006400);
|
||||
|
||||
return (as_half8(t) - (half8)(1056.0h)) * scale;
|
||||
}
|
||||
|
||||
inline void store_c16(global float * p, half16 h, int j0, int nv, int M) {
|
||||
float16 f = convert_float16(h);
|
||||
if (j0 + 0 < nv) p[(ulong)(j0 + 0) * M] = f.s0;
|
||||
if (j0 + 1 < nv) p[(ulong)(j0 + 1) * M] = f.s1;
|
||||
if (j0 + 2 < nv) p[(ulong)(j0 + 2) * M] = f.s2;
|
||||
if (j0 + 3 < nv) p[(ulong)(j0 + 3) * M] = f.s3;
|
||||
if (j0 + 4 < nv) p[(ulong)(j0 + 4) * M] = f.s4;
|
||||
if (j0 + 5 < nv) p[(ulong)(j0 + 5) * M] = f.s5;
|
||||
if (j0 + 6 < nv) p[(ulong)(j0 + 6) * M] = f.s6;
|
||||
if (j0 + 7 < nv) p[(ulong)(j0 + 7) * M] = f.s7;
|
||||
if (j0 + 8 < nv) p[(ulong)(j0 + 8) * M] = f.s8;
|
||||
if (j0 + 9 < nv) p[(ulong)(j0 + 9) * M] = f.s9;
|
||||
if (j0 + 10 < nv) p[(ulong)(j0 + 10) * M] = f.sa;
|
||||
if (j0 + 11 < nv) p[(ulong)(j0 + 11) * M] = f.sb;
|
||||
if (j0 + 12 < nv) p[(ulong)(j0 + 12) * M] = f.sc;
|
||||
if (j0 + 13 < nv) p[(ulong)(j0 + 13) * M] = f.sd;
|
||||
if (j0 + 14 < nv) p[(ulong)(j0 + 14) * M] = f.se;
|
||||
if (j0 + 15 < nv) p[(ulong)(j0 + 15) * M] = f.sf;
|
||||
}
|
||||
|
||||
#define MMA_K32(q0, q1, q2, q3, hA, hB, sc0, sc1, b) { \
|
||||
a_row_t ua0; \
|
||||
a_row_t ua1; \
|
||||
ua0.h = (half16)(dequant8(q0, (hA), sc0), dequant8(q1, (hA) >> 16, sc0)); \
|
||||
ua1.h = (half16)(dequant8(q2, (hB), sc1), dequant8(q3, (hB) >> 16, sc1)); \
|
||||
mat_a_t a0 = __builtin_astype(ua0.v, mat_a_t); \
|
||||
mat_a_t a1 = __builtin_astype(ua1.v, mat_a_t); \
|
||||
mat_b_t b0 = qcom_coop_mat_load((b), CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
|
||||
mat_b_t b1 = qcom_coop_mat_load((b) + 1024, CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
|
||||
acc = qcom_coop_mat_mulAdd(a0, b0, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); \
|
||||
acc = qcom_coop_mat_mulAdd(a1, b1, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); }
|
||||
|
||||
kernel void kernel_gemm_noshuffle_q6_k_f32_coopmat(
|
||||
global const uint * src0_ql, // A: q6_K low nibbles, [K/8][M] u32
|
||||
global const uint * src0_qh, // A: q6_K high 2-bit, [K/16][M] u32
|
||||
global const uint * src0_s, // A: int8 sub-scales, [M][K/16] (4 per u32)
|
||||
global const half * src0_d, // A: per-superblock scale, [K/256][M] f16
|
||||
global const half * src1, // B: packed f16 tiles
|
||||
global char * dst, // C: [N][M] f32
|
||||
ulong offsetd,
|
||||
int K,
|
||||
int M,
|
||||
int N
|
||||
) {
|
||||
const int lane = get_local_id(0);
|
||||
const int row = get_global_id(1) * 64;
|
||||
const int tn = get_global_id(2);
|
||||
const int n_tiles = (N + 63) / 64;
|
||||
|
||||
if (tn >= n_tiles) {
|
||||
return;
|
||||
}
|
||||
|
||||
const int lrow = row + lane;
|
||||
|
||||
global const uint * qp = src0_ql + lrow;
|
||||
global const uint * qhp = src0_qh + lrow;
|
||||
global const half * bp = src1 + (ulong)tn * (K / 16) * 1024;
|
||||
global const uint * sp = src0_s + (ulong)lrow * (K / 64);
|
||||
|
||||
mat_c_t acc = qcom_coop_mat_init(0.0);
|
||||
|
||||
#pragma unroll 1
|
||||
for (int k0 = 0; k0 < K; k0 += 64) {
|
||||
const half dd = src0_d[(ulong)(k0 >> 8) * M + lrow];
|
||||
|
||||
const half4 g = convert_half4(as_char4(sp[0])) * dd;
|
||||
sp += 1;
|
||||
|
||||
uint8 qv;
|
||||
qv.s0 = qp[0];
|
||||
qv.s1 = qp[M];
|
||||
qv.s2 = qp[2 * M];
|
||||
qv.s3 = qp[3 * M];
|
||||
qv.s4 = qp[4 * M];
|
||||
qv.s5 = qp[5 * M];
|
||||
qv.s6 = qp[6 * M];
|
||||
qv.s7 = qp[7 * M];
|
||||
qp += 8 * M;
|
||||
|
||||
uint4 hv;
|
||||
hv.s0 = qhp[0];
|
||||
hv.s1 = qhp[M];
|
||||
hv.s2 = qhp[2 * M];
|
||||
hv.s3 = qhp[3 * M];
|
||||
qhp += 4 * M;
|
||||
|
||||
MMA_K32(qv.s0, qv.s1, qv.s2, qv.s3, hv.s0, hv.s1, g.s0, g.s1, bp)
|
||||
MMA_K32(qv.s4, qv.s5, qv.s6, qv.s7, hv.s2, hv.s3, g.s2, g.s3, bp + 2048)
|
||||
bp += 4096;
|
||||
}
|
||||
|
||||
c_row_t uc;
|
||||
uc.v = __builtin_astype(acc, half4096);
|
||||
|
||||
global float * p = (global float *)(dst + offsetd) + (ulong)tn * 64 * M + row + lane;
|
||||
const int nv = N - tn * 64;
|
||||
store_c16(p, uc.s.h0, 0, nv, M);
|
||||
store_c16(p, uc.s.h1, 16, nv, M);
|
||||
store_c16(p, uc.s.h2, 32, nv, M);
|
||||
store_c16(p, uc.s.h3, 48, nv, M);
|
||||
}
|
||||
@@ -1,157 +0,0 @@
|
||||
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
|
||||
#pragma OPENCL EXTENSION cl_qcom_kernel_cooperative_matrix : enable
|
||||
|
||||
// Q8_0 x F32 GEMM with cl_qcom_kernel_cooperative_matrix: one 64x64 f16 C tile per wave,
|
||||
// A dequantized in registers, B packed by kernel_pack_b_f16_coopmat.
|
||||
|
||||
typedef half mat_a_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 16, CLK_COOPERATIVE_MATRIX_A_QCOM)));
|
||||
typedef half mat_b_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 16, 64, CLK_COOPERATIVE_MATRIX_B_QCOM)));
|
||||
typedef half mat_c_t __attribute__((qcom_coop_mat(CLK_COOPERATIVE_MATRIX_SCOPE_SUBGROUP_QCOM, 64, 64, CLK_COOPERATIVE_MATRIX_ACCUMULATOR_QCOM)));
|
||||
|
||||
typedef half half1024 __attribute__((ext_vector_type(1024)));
|
||||
typedef half half4096 __attribute__((ext_vector_type(4096)));
|
||||
|
||||
typedef union {
|
||||
half1024 v;
|
||||
half16 h;
|
||||
} a_row_t;
|
||||
|
||||
typedef union {
|
||||
half4096 v;
|
||||
struct {
|
||||
half16 h0;
|
||||
half16 h1;
|
||||
half16 h2;
|
||||
half16 h3;
|
||||
} s;
|
||||
} c_row_t;
|
||||
|
||||
inline half4 dequant4(uint v, half d) {
|
||||
return convert_half4(as_char4(v)) * d;
|
||||
}
|
||||
|
||||
inline void store_c16(global float * p, half16 h, int j0, int nv, int M) {
|
||||
float16 f = convert_float16(h);
|
||||
if (j0 + 0 < nv) p[(ulong)(j0 + 0) * M] = f.s0;
|
||||
if (j0 + 1 < nv) p[(ulong)(j0 + 1) * M] = f.s1;
|
||||
if (j0 + 2 < nv) p[(ulong)(j0 + 2) * M] = f.s2;
|
||||
if (j0 + 3 < nv) p[(ulong)(j0 + 3) * M] = f.s3;
|
||||
if (j0 + 4 < nv) p[(ulong)(j0 + 4) * M] = f.s4;
|
||||
if (j0 + 5 < nv) p[(ulong)(j0 + 5) * M] = f.s5;
|
||||
if (j0 + 6 < nv) p[(ulong)(j0 + 6) * M] = f.s6;
|
||||
if (j0 + 7 < nv) p[(ulong)(j0 + 7) * M] = f.s7;
|
||||
if (j0 + 8 < nv) p[(ulong)(j0 + 8) * M] = f.s8;
|
||||
if (j0 + 9 < nv) p[(ulong)(j0 + 9) * M] = f.s9;
|
||||
if (j0 + 10 < nv) p[(ulong)(j0 + 10) * M] = f.sa;
|
||||
if (j0 + 11 < nv) p[(ulong)(j0 + 11) * M] = f.sb;
|
||||
if (j0 + 12 < nv) p[(ulong)(j0 + 12) * M] = f.sc;
|
||||
if (j0 + 13 < nv) p[(ulong)(j0 + 13) * M] = f.sd;
|
||||
if (j0 + 14 < nv) p[(ulong)(j0 + 14) * M] = f.se;
|
||||
if (j0 + 15 < nv) p[(ulong)(j0 + 15) * M] = f.sf;
|
||||
}
|
||||
|
||||
#define LOAD_Q(qv, dd) { \
|
||||
qv.s0 = qp[0]; \
|
||||
qv.s1 = qp[M]; \
|
||||
qv.s2 = qp[2 * M]; \
|
||||
qv.s3 = qp[3 * M]; \
|
||||
qv.s4 = qp[4 * M]; \
|
||||
qv.s5 = qp[5 * M]; \
|
||||
qv.s6 = qp[6 * M]; \
|
||||
qv.s7 = qp[7 * M]; \
|
||||
dd = dp[0]; \
|
||||
qp += 8 * M; \
|
||||
dp += M; }
|
||||
|
||||
#define MMA_K32(qv, dd, b) { \
|
||||
a_row_t ua0; \
|
||||
a_row_t ua1; \
|
||||
ua0.h = (half16)(dequant4(qv.s0, dd), dequant4(qv.s1, dd), dequant4(qv.s2, dd), dequant4(qv.s3, dd)); \
|
||||
ua1.h = (half16)(dequant4(qv.s4, dd), dequant4(qv.s5, dd), dequant4(qv.s6, dd), dequant4(qv.s7, dd)); \
|
||||
mat_a_t a0 = __builtin_astype(ua0.v, mat_a_t); \
|
||||
mat_a_t a1 = __builtin_astype(ua1.v, mat_a_t); \
|
||||
mat_b_t b0 = qcom_coop_mat_load((b), CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
|
||||
mat_b_t b1 = qcom_coop_mat_load((b) + 1024, CLK_COOPERATIVE_MATRIX_LAYOUT_COLUMN_MAJOR_QCOM, 16); \
|
||||
acc = qcom_coop_mat_mulAdd(a0, b0, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); \
|
||||
acc = qcom_coop_mat_mulAdd(a1, b1, acc, CLK_COOPERATIVE_MATRIX_OPERAND_NONE_QCOM); }
|
||||
|
||||
kernel void kernel_gemm_noshuffle_q8_0_f32_coopmat(
|
||||
global const uint * src0_q, // A: q8_0 quants, [K/4][M] u32
|
||||
global const half * src0_d, // A: q8_0 scales, [K/32][M] f16
|
||||
global const half * src1, // B: packed f16 tiles
|
||||
global char * dst, // C: [N][M] f32
|
||||
ulong offsetd,
|
||||
int K,
|
||||
int M,
|
||||
int N
|
||||
) {
|
||||
const int lane = get_local_id(0);
|
||||
const int row = get_global_id(1) * 64;
|
||||
const int tn = get_global_id(2);
|
||||
const int n_tiles = (N + 63) / 64;
|
||||
|
||||
if (tn >= n_tiles) {
|
||||
return;
|
||||
}
|
||||
|
||||
global const uint * qp = src0_q + row + lane;
|
||||
global const half * dp = src0_d + row + lane;
|
||||
global const half * bp = src1 + (ulong)tn * (K / 16) * 1024;
|
||||
|
||||
mat_c_t acc = qcom_coop_mat_init(0.0);
|
||||
|
||||
{
|
||||
const int odd = (K / 32) & 1;
|
||||
const int kb = odd ? 0 : 32;
|
||||
uint8 qa;
|
||||
uint8 qb;
|
||||
half da;
|
||||
half db;
|
||||
qa.s0 = qp[0];
|
||||
qa.s1 = qp[M];
|
||||
qa.s2 = qp[2 * M];
|
||||
qa.s3 = qp[3 * M];
|
||||
qa.s4 = qp[4 * M];
|
||||
qa.s5 = qp[5 * M];
|
||||
qa.s6 = qp[6 * M];
|
||||
qa.s7 = qp[7 * M];
|
||||
da = odd ? 0.0h : dp[0];
|
||||
qb.s0 = qp[(kb / 4 + 0) * M];
|
||||
qb.s1 = qp[(kb / 4 + 1) * M];
|
||||
qb.s2 = qp[(kb / 4 + 2) * M];
|
||||
qb.s3 = qp[(kb / 4 + 3) * M];
|
||||
qb.s4 = qp[(kb / 4 + 4) * M];
|
||||
qb.s5 = qp[(kb / 4 + 5) * M];
|
||||
qb.s6 = qp[(kb / 4 + 6) * M];
|
||||
qb.s7 = qp[(kb / 4 + 7) * M];
|
||||
db = dp[(kb / 32) * M];
|
||||
MMA_K32(qa, da, bp)
|
||||
MMA_K32(qb, db, bp + (kb / 16) * 1024)
|
||||
qp += (kb / 4 + 8) * M;
|
||||
dp += (kb / 32 + 1) * M;
|
||||
bp += (kb / 16 + 2) * 1024;
|
||||
}
|
||||
|
||||
#pragma unroll 1
|
||||
for (int k0 = (K / 32) & 1 ? 32 : 64; k0 < K; k0 += 64) {
|
||||
uint8 qa;
|
||||
uint8 qb;
|
||||
half da;
|
||||
half db;
|
||||
LOAD_Q(qa, da)
|
||||
LOAD_Q(qb, db)
|
||||
MMA_K32(qa, da, bp)
|
||||
MMA_K32(qb, db, bp + 2048)
|
||||
bp += 4096;
|
||||
}
|
||||
|
||||
c_row_t uc;
|
||||
uc.v = __builtin_astype(acc, half4096);
|
||||
|
||||
global float * p = (global float *)(dst + offsetd) + (ulong)tn * 64 * M + row + lane;
|
||||
const int nv = N - tn * 64;
|
||||
store_c16(p, uc.s.h0, 0, nv, M);
|
||||
store_c16(p, uc.s.h1, 16, nv, M);
|
||||
store_c16(p, uc.s.h2, 32, nv, M);
|
||||
store_c16(p, uc.s.h3, 48, nv, M);
|
||||
}
|
||||
@@ -1,48 +0,0 @@
|
||||
#pragma OPENCL EXTENSION cl_khr_fp16 : enable
|
||||
|
||||
// Pack B f32 [N][K] into f16 64x16 tiles (column-major, ld 16) at (tn * (K/16) + ks) * 1024.
|
||||
// Columns >= N are zero. A workgroup (64, 4) packs 64 columns x 64 k.
|
||||
#define PACK_LDS (64 + 8)
|
||||
|
||||
kernel void kernel_pack_b_f16_coopmat(
|
||||
global const char * src1,
|
||||
ulong offset1,
|
||||
global half * dst,
|
||||
int K,
|
||||
int N
|
||||
) {
|
||||
local half tile[64 * PACK_LDS];
|
||||
|
||||
const int l = get_local_id(1) * 64 + get_local_id(0);
|
||||
const int tn = get_group_id(0);
|
||||
const int kt = get_group_id(1);
|
||||
const int k0 = kt * 64;
|
||||
|
||||
global const float * src = (global const float *)(src1 + offset1);
|
||||
|
||||
for (int j = 0; j < 4; j++) {
|
||||
const int f = j * 256 + l;
|
||||
const int r = f >> 4;
|
||||
const int c = (f & 15) * 4;
|
||||
const int n = tn * 64 + r;
|
||||
float4 v = (float4)(0.0f);
|
||||
if (n < N && k0 + c < K) {
|
||||
v = vload4(0, src + (ulong)n * K + k0 + c);
|
||||
}
|
||||
vstore_half4(v, 0, tile + r * PACK_LDS + c);
|
||||
}
|
||||
|
||||
barrier(CLK_LOCAL_MEM_FENCE);
|
||||
|
||||
global half * out = dst + ((ulong)tn * (K / 16) + kt * 4) * 1024;
|
||||
for (int j = 0; j < 2; j++) {
|
||||
const int q = j * 256 + l;
|
||||
const int ks = q >> 7;
|
||||
const int n = (q >> 1) & 63;
|
||||
const int h = q & 1;
|
||||
if (kt * 4 + ks < K / 16) {
|
||||
half8 v = vload8(0, tile + n * PACK_LDS + ks * 16 + h * 8);
|
||||
vstore8(v, 0, out + q * 8);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -809,25 +809,9 @@ void ggml_vk_buffer_memset(vk_buffer& dst, size_t offset, uint32_t c, size_t siz
|
||||
return;
|
||||
}
|
||||
|
||||
// vkCmdFillBuffer requires a 4-byte aligned offset and size, so write the unaligned ends from the matching bytes of c
|
||||
const size_t end = offset + size;
|
||||
const size_t begin_aligned = std::min(GGML_PAD(offset, 4), end);
|
||||
const size_t end_aligned = std::max(end - end % 4, begin_aligned);
|
||||
|
||||
const uint8_t * c_bytes = (const uint8_t *) &c;
|
||||
if (begin_aligned > offset) {
|
||||
ggml_vk_buffer_write(dst, offset, c_bytes + offset % 4, begin_aligned - offset);
|
||||
}
|
||||
if (end > end_aligned) {
|
||||
ggml_vk_buffer_write(dst, end_aligned, c_bytes, end - end_aligned);
|
||||
}
|
||||
if (end_aligned == begin_aligned) {
|
||||
return;
|
||||
}
|
||||
|
||||
std::lock_guard<std::recursive_mutex> guard(dst->device->mutex);
|
||||
vk_context subctx = ggml_vk_begin_sync_ctx(dst->device, dst->device->transfer_queue->cmd_pool);
|
||||
subctx->s->buffer->buf.fillBuffer(dst->buffer, begin_aligned, end_aligned - begin_aligned, c);
|
||||
subctx->s->buffer->buf.fillBuffer(dst->buffer, offset, size, c);
|
||||
ggml_vk_submit_sync(dst->device, subctx);
|
||||
}
|
||||
|
||||
|
||||
@@ -14435,9 +14435,7 @@ bool ggml_backend_vk_buffer_cpy_tensor_n(ggml_backend_buffer_t buffer, const ggm
|
||||
void ggml_backend_vk_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) {
|
||||
ggml_backend_vk_buffer_context * ctx = (ggml_backend_vk_buffer_context *)buffer->context;
|
||||
|
||||
// set all bytes of a u32 to the same 1 byte value
|
||||
uint32_t val32 = (uint32_t)value * 0x01010101;
|
||||
ggml_vk_buffer_memset(ctx->dev_buffer, 0, val32, buffer->size);
|
||||
ggml_vk_buffer_memset(ctx->dev_buffer, 0, value, buffer->size);
|
||||
}
|
||||
|
||||
const char * ggml_backend_vk_buffer_type_name(ggml_backend_buffer_type_t buft) {
|
||||
|
||||
@@ -4066,7 +4066,7 @@ static void ggml_backend_webgpu_buffer_memset_tensor(ggml_backend_buffer_t buffe
|
||||
|
||||
size_t total_offset = ggml_webgpu_tensor_offset(tensor) + offset;
|
||||
|
||||
// set all bytes of a u32 to the same 1 byte value
|
||||
// This is a trick to set all bytes of a u32 to the same 1 byte value.
|
||||
uint32_t val32 = (uint32_t) value * 0x01010101;
|
||||
ggml_backend_webgpu_buffer_memset(buf_ctx->global_ctx, buf_ctx->buffer, val32, total_offset, size);
|
||||
WEBGPU_CPU_PROFILE_TOTAL_END(memset_tensor, buf_ctx->global_ctx);
|
||||
@@ -4406,10 +4406,7 @@ static void ggml_backend_webgpu_buffer_clear(ggml_backend_buffer_t buffer, uint8
|
||||
WEBGPU_LOG_DEBUG("ggml_backend_webgpu_buffer_clear(" << buffer << ", " << (uint32_t) value << ")");
|
||||
WEBGPU_CPU_PROFILE_TOTAL_START(clear);
|
||||
ggml_backend_webgpu_buffer_context * buf_ctx = (ggml_backend_webgpu_buffer_context *) buffer->context;
|
||||
|
||||
// set all bytes of a u32 to the same 1 byte value
|
||||
uint32_t val32 = (uint32_t) value * 0x01010101;
|
||||
ggml_backend_webgpu_buffer_memset(buf_ctx->global_ctx, buf_ctx->buffer, val32, 0, buffer->size);
|
||||
ggml_backend_webgpu_buffer_memset(buf_ctx->global_ctx, buf_ctx->buffer, value, 0, buffer->size);
|
||||
WEBGPU_CPU_PROFILE_TOTAL_END(clear, buf_ctx->global_ctx);
|
||||
}
|
||||
|
||||
@@ -5266,8 +5263,7 @@ static bool ggml_backend_webgpu_device_supports_op(ggml_backend_dev_t dev, const
|
||||
supports_op = op->type == GGML_TYPE_F32 && src0->type == GGML_TYPE_F32 && src1->type == GGML_TYPE_F32 &&
|
||||
src2->type == GGML_TYPE_F32 && op->src[3]->type == GGML_TYPE_F32 &&
|
||||
op->src[4]->type == GGML_TYPE_F32 && op->src[5]->type == GGML_TYPE_F32 &&
|
||||
s_v <= ctx->webgpu_global_ctx->capabilities.limits.maxComputeInvocationsPerWorkgroup &&
|
||||
ggml_are_same_stride(src0, src1);
|
||||
s_v <= ctx->webgpu_global_ctx->capabilities.limits.maxComputeInvocationsPerWorkgroup;
|
||||
}
|
||||
break;
|
||||
case GGML_OP_CLAMP:
|
||||
|
||||
+2
-26
@@ -1116,11 +1116,9 @@ static const char * GGML_OP_NAME[GGML_OP_COUNT] = {
|
||||
"GLU",
|
||||
|
||||
"SLEEP",
|
||||
|
||||
"SIGMOID_BACK",
|
||||
};
|
||||
|
||||
static_assert(GGML_OP_COUNT == 101, "GGML_OP_COUNT != 101");
|
||||
static_assert(GGML_OP_COUNT == 100, "GGML_OP_COUNT != 100");
|
||||
|
||||
static const char * GGML_OP_SYMBOL[GGML_OP_COUNT] = {
|
||||
"none",
|
||||
@@ -1233,11 +1231,9 @@ static const char * GGML_OP_SYMBOL[GGML_OP_COUNT] = {
|
||||
"glu(x)",
|
||||
|
||||
"sleep(x)",
|
||||
|
||||
"sigmoid_back(x)",
|
||||
};
|
||||
|
||||
static_assert(GGML_OP_COUNT == 101, "GGML_OP_COUNT != 101");
|
||||
static_assert(GGML_OP_COUNT == 100, "GGML_OP_COUNT != 100");
|
||||
|
||||
static_assert(GGML_OP_POOL_COUNT == 2, "GGML_OP_POOL_COUNT != 2");
|
||||
|
||||
@@ -2870,21 +2866,6 @@ struct ggml_tensor * ggml_silu_back(
|
||||
return result;
|
||||
}
|
||||
|
||||
// ggml_sigmoid_back
|
||||
|
||||
struct ggml_tensor * ggml_sigmoid_back(
|
||||
struct ggml_context * ctx,
|
||||
struct ggml_tensor * a,
|
||||
struct ggml_tensor * b) {
|
||||
struct ggml_tensor * result = ggml_dup_tensor(ctx, a);
|
||||
|
||||
result->op = GGML_OP_SIGMOID_BACK;
|
||||
result->src[0] = a;
|
||||
result->src[1] = b;
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
// ggml hardswish
|
||||
|
||||
struct ggml_tensor * ggml_hardswish(
|
||||
@@ -7195,11 +7176,6 @@ static void ggml_compute_backward(
|
||||
ggml_add_or_set(ctx, cgraph, isrc0, ggml_silu_back(ctx, grad, src0));
|
||||
}
|
||||
} break;
|
||||
case GGML_UNARY_OP_SIGMOID: {
|
||||
if (src0_needs_grads) {
|
||||
ggml_add_or_set(ctx, cgraph, isrc0, ggml_sigmoid_back(ctx, grad, src0));
|
||||
}
|
||||
} break;
|
||||
case GGML_UNARY_OP_EXP: {
|
||||
if (src0_needs_grads) {
|
||||
ggml_add_or_set(ctx, cgraph, isrc0, ggml_mul(ctx, tensor, grad));
|
||||
|
||||
+22
-9
@@ -322,6 +322,24 @@ struct gguf_reader {
|
||||
return true;
|
||||
}
|
||||
|
||||
bool read(enum ggml_type & dst) const {
|
||||
int32_t tmp = -1;
|
||||
if (!read(tmp)) {
|
||||
return false;
|
||||
}
|
||||
dst = ggml_type(tmp);
|
||||
return true;
|
||||
}
|
||||
|
||||
bool read(enum gguf_type & dst) const {
|
||||
int32_t tmp = -1;
|
||||
if (!read(tmp)) {
|
||||
return false;
|
||||
}
|
||||
dst = gguf_type(tmp);
|
||||
return true;
|
||||
}
|
||||
|
||||
bool read(std::string & dst) const {
|
||||
uint64_t size = 0;
|
||||
if (!read(size)) {
|
||||
@@ -537,7 +555,7 @@ static struct gguf_context * gguf_init_from_reader(const struct gguf_reader & gr
|
||||
|
||||
for (int64_t i = 0; ok && i < n_kv; ++i) {
|
||||
std::string key;
|
||||
int32_t type = -1;
|
||||
gguf_type type = gguf_type(-1);
|
||||
bool is_array = false;
|
||||
uint64_t n = 1;
|
||||
|
||||
@@ -706,20 +724,15 @@ static struct gguf_context * gguf_init_from_reader(const struct gguf_reader & gr
|
||||
|
||||
// tensor type
|
||||
{
|
||||
int32_t type = -1;
|
||||
ok = ok && gr.read(type);
|
||||
if (!ok) {
|
||||
break;
|
||||
}
|
||||
ok = ok && gr.read(info.t.type);
|
||||
|
||||
// check that tensor type is within defined range
|
||||
if (type < 0 || type >= GGML_TYPE_COUNT) {
|
||||
if (info.t.type < 0 || info.t.type >= GGML_TYPE_COUNT) {
|
||||
GGML_LOG_ERROR("%s: tensor '%s' has invalid ggml type %d. should be in [0, %d)\n",
|
||||
__func__, info.t.name, type, GGML_TYPE_COUNT);
|
||||
__func__, info.t.name, info.t.type, GGML_TYPE_COUNT);
|
||||
ok = false;
|
||||
break;
|
||||
}
|
||||
info.t.type = ggml_type(type);
|
||||
const size_t type_size = ggml_type_size(info.t.type);
|
||||
const int64_t blck_size = ggml_blck_size(info.t.type);
|
||||
|
||||
|
||||
+2
-4
@@ -73,13 +73,11 @@ Tokens allow grammars to match specific tokenizer tokens rather than character s
|
||||
|
||||
Tokens can be specified in two ways:
|
||||
|
||||
1. **Token ID**: Use angle brackets with the token ID in square brackets: `<[token-id]>`. For example, `<[1000]>` matches the token with ID 1000. Multiple IDs and inclusive ranges can be combined: `<[1000,1005-1010]>` matches token 1000 or any token from 1005 to 1010.
|
||||
1. **Token ID**: Use angle brackets with the token ID in square brackets: `<[token-id]>`. For example, `<[1000]>` matches the token with ID 1000.
|
||||
|
||||
2. **Token string**: Use angle brackets with the token text directly: `<token>`. For example, `<think>` will match the token whose text is exactly `<think>`. This only works if the string tokenizes to exactly one token in the vocabulary, otherwise the grammar will fail to parse.
|
||||
|
||||
You can negate token matches using the `!` prefix: `!<[1000]>` or `!<think>` matches any token *except* the specified one, and `!<[1000,1005-1010]>` matches any token outside the set.
|
||||
|
||||
Tokens are matched whole, so a token rule can only match at the start of a token, never after a character rule has consumed part of its text. Any token named by a token rule is also excluded from character rules: once `<think>` appears as a token in a grammar, `.*` in that grammar will not match the `<think>` token.
|
||||
You can negate token matches using the `!` prefix: `!<[1000]>` or `!<think>` matches any token *except* the specified one.
|
||||
|
||||
```
|
||||
# Match a thinking block: <think>...</think>
|
||||
|
||||
@@ -149,10 +149,7 @@ now_ms() { date +%s%3N; }
|
||||
|
||||
# ---------------------------------------------------------------- scratch + cleanup
|
||||
S=$(mktemp -d /tmp/fork-validate.XXXXXX)
|
||||
# the arena lives on tmpfs, like production's /dev/shm/llama-vram: on overlayfs (the pod's rootfs,
|
||||
# so /tmp, since 2026-11-22) stat reports the overlay device while /proc/locks prints the upper
|
||||
# filesystem's, so lock_id never matched and every token check failed
|
||||
ARENA=/dev/shm/fork-validate.${S##*.}
|
||||
ARENA=$S/arena
|
||||
mkdir -p "$ARENA"
|
||||
cd "$S" || exit 2 # anything written to a relative path lands in the scratch dir
|
||||
PIDS=() # "pid:starttime" of every background process we start (starttime guards PID reuse)
|
||||
@@ -186,7 +183,6 @@ finish() {
|
||||
local rc=$?
|
||||
trap - EXIT INT TERM HUP
|
||||
cleanup
|
||||
rm -rf "$ARENA" "$ARENA-moec"
|
||||
echo
|
||||
echo "==================== fork-validate summary ===================="
|
||||
echo "candidate: $BUILD ($(git -C "$BUILD/.." rev-parse --short HEAD 2>/dev/null || echo '?'))"
|
||||
@@ -826,7 +822,7 @@ run_moecache() {
|
||||
[ -f "$MOECACHE_MODEL" ] || { fail moecache "model missing: $MOECACHE_MODEL"; return; }
|
||||
local free_mb=$(( ( $(cat "$VRAM_TOTAL") - $(cat "$VRAM_USED") ) / 1048576 ))
|
||||
log "moecache: $free_mb MiB VRAM free; server on :$PC with --moe-cache-mib $MOECACHE_MIB"
|
||||
local arena=$ARENA-moec; mkdir -p "$arena"
|
||||
local arena=$S/arena-moec; mkdir -p "$arena"
|
||||
# -lv 4: the cache size and hit/miss stats are only logged at that verbosity
|
||||
# shellcheck disable=SC2086
|
||||
start_server M "$BUILD" "$PC" LLAMA_SLEEP_VRAM_ONLY=1 "LLAMA_VRAM_ARENA=$arena" LLAMA_SLEEP_EVICT_KV=1 -- \
|
||||
|
||||
@@ -69,7 +69,6 @@ These recur often enough in review comments on past add-model PRs that they're w
|
||||
- Before writing a dedicated tool-call/output parser, check whether the existing autoparser already handles the template (`test-chat-auto-parser <jinja>` shows what it detects).
|
||||
- Marking a custom EOS/closing-tag token as `eot` at conversion time isn't always sufficient - in long/agentic generations a model can emit the closing sequence as literal text instead of the token, so generation never stops on EOG and raw text leaks past the parser. Verify this case, not just the token path.
|
||||
- If reusing or aliasing an existing pre-tokenizer for convenience, justify and test that choice explicitly - silent reuse is an easy source of subtle tokenizer bugs.
|
||||
- The GGUF only stores the pre-tokenizer name (`tokenizer.pre`), the behavior flags are hard-coded per name in `src/llama-vocab.cpp`. When adding a new name, set each flag from the HF `tokenizer.json`, not from the case you copied. Examples: `model.ignore_merges` -> `ignore_merges`, the BOS/EOS of the post-processor -> `special_bos_id` / `add_bos`, prefix space and space cleanup -> `add_space_prefix` / `clean_spaces`. The `chkhsh` check in the converter only identifies the pre-tokenizer on a short text, it may not catch a wrong flag.
|
||||
- Watch for excessive graph splits caused by building per-layer view/index tensors inside the layer loop - hoist tensors that don't vary per layer out of the loop (relevant if you hit `GGML_SCHED_MAX_SPLIT_INPUTS`).
|
||||
- A custom KQ mask fed into flash attention must match FA's expected dtype - cast it to F16 before passing it to `build_attn_mha` when FA is enabled.
|
||||
- When padding a custom KV-cache size to an alignment (e.g. `GGML_PAD(..., 256)`), apply the padding after all other size adjustments, not before - otherwise later logic can un-align it again.
|
||||
@@ -84,13 +83,12 @@ These recur often enough in review comments on past add-model PRs that they're w
|
||||
Reference: `examples/model-conversion/README.md`.
|
||||
|
||||
1. Convert to GGUF, then inspect/run both the original and converted tensors.
|
||||
2. If the model adds a new pre-tokenizer, compare the token ids of `llama-tokenize -f <file> --ids --no-escape` with the HF tokenizer on whole files of prose, code and multilingual text (e.g. wikitext-2 and a few source files).
|
||||
3. Run logits verification (original vs converted). If this model is a new version of an already-supported family, verify the *previous* version still passes logits verification first - numerical differences may be pre-existing, not caused by the new work. The tools to perform full logits validation are available in `examples/model-conversion`.
|
||||
4. Quantize (including QAT variants if relevant) and re-verify.
|
||||
5. Run perplexity evaluation (simple and full).
|
||||
6. Sanity-check across `tools/cli`, `tools/completion`, `tools/imatrix`, `tools/quantize`, and `tools/server`.
|
||||
7. CPU backend first; other backends (CUDA, Metal, ...) can be separate follow-up PRs per `CONTRIBUTING.md`.
|
||||
8. Re-review every changed file against the coding/naming guidelines in `AGENTS.md` (and `CONTRIBUTING.md`'s "Coding guidelines"/"Naming guidelines" sections) - this is a separate pass from functional testing and is just as important: no forced line-wrapping, no unicode punctuation, minimal/non-redundant comments, `snake_case` naming (`kebab-case` for file names), matching indentation/brace style, etc.
|
||||
2. Run logits verification (original vs converted). If this model is a new version of an already-supported family, verify the *previous* version still passes logits verification first - numerical differences may be pre-existing, not caused by the new work. The tools to perform full logits validation are available in `examples/model-conversion`.
|
||||
3. Quantize (including QAT variants if relevant) and re-verify.
|
||||
4. Run perplexity evaluation (simple and full).
|
||||
5. Sanity-check across `tools/cli`, `tools/completion`, `tools/imatrix`, `tools/quantize`, and `tools/server`.
|
||||
6. CPU backend first; other backends (CUDA, Metal, ...) can be separate follow-up PRs per `CONTRIBUTING.md`.
|
||||
7. Re-review every changed file against the coding/naming guidelines in `AGENTS.md` (and `CONTRIBUTING.md`'s "Coding guidelines"/"Naming guidelines" sections) - this is a separate pass from functional testing and is just as important: no forced line-wrapping, no unicode punctuation, minimal/non-redundant comments, `snake_case` naming (`kebab-case` for file names), matching indentation/brace style, etc.
|
||||
|
||||
## Before opening a PR
|
||||
|
||||
|
||||
@@ -3230,19 +3230,6 @@ bool llama_context::sched_copy_experts(ggml_backend_t backend, const ggml_tensor
|
||||
return false;
|
||||
}
|
||||
|
||||
// Later loop passes can select different experts from the same device copy.
|
||||
// TODO: copy only the experts needed by each pass (see #32807 and #32809).
|
||||
// [fork F13] with the per-pass host-weight views every pass has its own copy, so the partial
|
||||
// copy is correct again; LLAMA_LOOP_FULL_COPY=0 skips this guard (A/B knob, only correct with
|
||||
// the views on, LLAMA_LOOP_HOST_VIEWS unset or 1)
|
||||
static const bool loop_full_copy = [] {
|
||||
const char * env = getenv("LLAMA_LOOP_FULL_COPY");
|
||||
return env == nullptr || strcmp(env, "0") != 0;
|
||||
}();
|
||||
if (loop_full_copy && lctx->model.arch == LLM_ARCH_INKLING2_LOOP) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const ggml_tensor * ids = node->src[2];
|
||||
if (ggml_nelements(ids) == 0) {
|
||||
return true;
|
||||
|
||||
@@ -112,9 +112,6 @@ using llama_memory_breakdown = std::map<ggml_backend_buffer_type_t, llama_memory
|
||||
LLAMA_API int32_t llama_model_n_expert (const struct llama_model * model);
|
||||
LLAMA_API int32_t llama_model_n_devices(const struct llama_model * model);
|
||||
|
||||
// number of logical passes that execute each layer's weights (0 for weight-sharing alias slots)
|
||||
LLAMA_API std::vector<uint32_t> llama_model_layer_visits(const struct llama_model * model);
|
||||
|
||||
LLAMA_API ggml_backend_dev_t llama_model_get_device(const struct llama_model * model, int i);
|
||||
|
||||
LLAMA_API llama_memory_breakdown llama_get_memory_breakdown(const struct llama_context * ctx);
|
||||
|
||||
+51
-165
@@ -7,7 +7,6 @@
|
||||
#include <cmath>
|
||||
#include <algorithm>
|
||||
#include <cstdint>
|
||||
#include <limits>
|
||||
#include <set>
|
||||
#include <numeric>
|
||||
#include <stdexcept>
|
||||
@@ -34,9 +33,9 @@ static std::pair<uint32_t, const char *> decode_utf8(const char * src) {
|
||||
return std::make_pair(value, pos);
|
||||
}
|
||||
|
||||
std::pair<std::vector<uint32_t>, llama_partial_utf8> llama_grammar_decode_utf8(
|
||||
const std::string & src,
|
||||
llama_partial_utf8 partial_start) {
|
||||
static std::pair<std::vector<uint32_t>, llama_partial_utf8> decode_utf8(
|
||||
const std::string & src,
|
||||
llama_partial_utf8 partial_start) {
|
||||
static const int lookup[] = { 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 2, 2, 3, 4 };
|
||||
const char * pos = src.c_str();
|
||||
std::vector<uint32_t> code_points;
|
||||
@@ -186,45 +185,23 @@ static std::pair<uint32_t, const char *> parse_char(const char * src) {
|
||||
throw std::runtime_error("unexpected end of input");
|
||||
}
|
||||
|
||||
static std::pair<uint32_t, const char *> parse_token_id(const char * src) {
|
||||
const char * int_end = parse_int(src);
|
||||
unsigned long id = std::stoul(std::string(src, int_end - src));
|
||||
if (id > std::numeric_limits<uint32_t>::max()) {
|
||||
throw std::runtime_error(std::string("parsed token id is too big at ") + src);
|
||||
}
|
||||
return std::make_pair(static_cast<uint32_t>(id), int_end);
|
||||
}
|
||||
|
||||
static const char * parse_token(const llama_vocab * vocab, const char * src, llama_gretype type, llama_grammar_rule & rule) {
|
||||
static std::pair<uint32_t, const char *> parse_token(const llama_vocab * vocab, const char * src) {
|
||||
const char * pos = src;
|
||||
if (*pos != '<') {
|
||||
throw std::runtime_error(std::string("expecting '<' at ") + pos);
|
||||
}
|
||||
pos++;
|
||||
|
||||
// Parse <[id,id-id,...]>
|
||||
// Parse <[id]>
|
||||
if (*pos == '[') {
|
||||
pos++;
|
||||
const size_t start = rule.size();
|
||||
while (true) {
|
||||
auto id_pair = parse_token_id(pos);
|
||||
uint32_t token_id = id_pair.first;
|
||||
pos = id_pair.second;
|
||||
rule.push_back({rule.size() == start ? type : LLAMA_GRETYPE_TOKEN_ALT, token_id});
|
||||
if (*pos == '-') {
|
||||
pos++;
|
||||
auto end_pair = parse_token_id(pos);
|
||||
if (end_pair.first < token_id) {
|
||||
throw std::runtime_error(std::string("invalid token range at ") + pos);
|
||||
}
|
||||
pos = end_pair.second;
|
||||
rule.push_back({LLAMA_GRETYPE_TOKEN_RNG_UPPER, end_pair.first});
|
||||
}
|
||||
if (*pos != ',') {
|
||||
break;
|
||||
}
|
||||
pos++;
|
||||
const char * int_end = parse_int(pos);
|
||||
unsigned long id = std::stoul(std::string(pos, int_end - pos));
|
||||
if (id > std::numeric_limits<uint32_t>::max()) {
|
||||
throw std::runtime_error(std::string("parsed token id is too big at ") + pos);
|
||||
}
|
||||
uint32_t token_id = static_cast<uint32_t>(id);
|
||||
pos = int_end;
|
||||
if (*pos != ']') {
|
||||
throw std::runtime_error(std::string("expecting ']' at ") + pos);
|
||||
}
|
||||
@@ -233,7 +210,7 @@ static const char * parse_token(const llama_vocab * vocab, const char * src, lla
|
||||
throw std::runtime_error(std::string("expecting '>' at ") + pos);
|
||||
}
|
||||
pos++;
|
||||
return pos;
|
||||
return std::make_pair(token_id, pos);
|
||||
}
|
||||
|
||||
if (vocab == nullptr) {
|
||||
@@ -255,8 +232,7 @@ static const char * parse_token(const llama_vocab * vocab, const char * src, lla
|
||||
// must tokenize to exactly 1 token
|
||||
throw std::runtime_error("invalid token '" + std::string(src, pos - src) + "'");
|
||||
}
|
||||
rule.push_back({type, static_cast<uint32_t>(tokens[0])});
|
||||
return pos;
|
||||
return std::make_pair(tokens[0], pos);
|
||||
}
|
||||
|
||||
static void print_grammar_char(FILE * file, uint32_t c) {
|
||||
@@ -268,16 +244,6 @@ static void print_grammar_char(FILE * file, uint32_t c) {
|
||||
}
|
||||
}
|
||||
|
||||
static bool is_token_element(llama_grammar_element elem) {
|
||||
switch (elem.type) {
|
||||
case LLAMA_GRETYPE_TOKEN: return true;
|
||||
case LLAMA_GRETYPE_TOKEN_NOT: return true;
|
||||
case LLAMA_GRETYPE_TOKEN_ALT: return true;
|
||||
case LLAMA_GRETYPE_TOKEN_RNG_UPPER: return true;
|
||||
default: return false;
|
||||
}
|
||||
}
|
||||
|
||||
static bool is_char_element(llama_grammar_element elem) {
|
||||
switch (elem.type) {
|
||||
case LLAMA_GRETYPE_CHAR: return true;
|
||||
@@ -292,18 +258,16 @@ static bool is_char_element(llama_grammar_element elem) {
|
||||
static void print_rule_binary(FILE * file, const llama_grammar_rule & rule) {
|
||||
for (auto elem : rule) {
|
||||
switch (elem.type) {
|
||||
case LLAMA_GRETYPE_END: fprintf(file, "END"); break;
|
||||
case LLAMA_GRETYPE_ALT: fprintf(file, "ALT"); break;
|
||||
case LLAMA_GRETYPE_RULE_REF: fprintf(file, "RULE_REF"); break;
|
||||
case LLAMA_GRETYPE_CHAR: fprintf(file, "CHAR"); break;
|
||||
case LLAMA_GRETYPE_CHAR_NOT: fprintf(file, "CHAR_NOT"); break;
|
||||
case LLAMA_GRETYPE_CHAR_RNG_UPPER: fprintf(file, "CHAR_RNG_UPPER"); break;
|
||||
case LLAMA_GRETYPE_CHAR_ALT: fprintf(file, "CHAR_ALT"); break;
|
||||
case LLAMA_GRETYPE_CHAR_ANY: fprintf(file, "CHAR_ANY"); break;
|
||||
case LLAMA_GRETYPE_TOKEN: fprintf(file, "TOKEN"); break;
|
||||
case LLAMA_GRETYPE_TOKEN_NOT: fprintf(file, "TOKEN_NOT"); break;
|
||||
case LLAMA_GRETYPE_TOKEN_RNG_UPPER: fprintf(file, "TOKEN_RNG_UPPER"); break;
|
||||
case LLAMA_GRETYPE_TOKEN_ALT: fprintf(file, "TOKEN_ALT"); break;
|
||||
case LLAMA_GRETYPE_END: fprintf(file, "END"); break;
|
||||
case LLAMA_GRETYPE_ALT: fprintf(file, "ALT"); break;
|
||||
case LLAMA_GRETYPE_RULE_REF: fprintf(file, "RULE_REF"); break;
|
||||
case LLAMA_GRETYPE_CHAR: fprintf(file, "CHAR"); break;
|
||||
case LLAMA_GRETYPE_CHAR_NOT: fprintf(file, "CHAR_NOT"); break;
|
||||
case LLAMA_GRETYPE_CHAR_RNG_UPPER: fprintf(file, "CHAR_RNG_UPPER"); break;
|
||||
case LLAMA_GRETYPE_CHAR_ALT: fprintf(file, "CHAR_ALT"); break;
|
||||
case LLAMA_GRETYPE_CHAR_ANY: fprintf(file, "CHAR_ANY"); break;
|
||||
case LLAMA_GRETYPE_TOKEN: fprintf(file, "TOKEN"); break;
|
||||
case LLAMA_GRETYPE_TOKEN_NOT: fprintf(file, "TOKEN_NOT"); break;
|
||||
}
|
||||
switch (elem.type) {
|
||||
case LLAMA_GRETYPE_END:
|
||||
@@ -331,10 +295,6 @@ static void print_rule_binary(FILE * file, const llama_grammar_rule & rule) {
|
||||
fprintf(file, "%u", elem.value);
|
||||
fprintf(file, "]> ");
|
||||
break;
|
||||
case LLAMA_GRETYPE_TOKEN_RNG_UPPER:
|
||||
case LLAMA_GRETYPE_TOKEN_ALT:
|
||||
fprintf(file, "(%u) ", elem.value);
|
||||
break;
|
||||
}
|
||||
}
|
||||
fprintf(file, "\n");
|
||||
@@ -394,32 +354,14 @@ static void print_rule(
|
||||
case LLAMA_GRETYPE_TOKEN:
|
||||
fprintf(file, "<[");
|
||||
fprintf(file, "%u", elem.value);
|
||||
fprintf(file, "]> ");
|
||||
break;
|
||||
case LLAMA_GRETYPE_TOKEN_NOT:
|
||||
fprintf(file, "!");
|
||||
fprintf(file, "<[");
|
||||
fprintf(file, "%u", elem.value);
|
||||
fprintf(file, "]> ");
|
||||
break;
|
||||
case LLAMA_GRETYPE_TOKEN_RNG_UPPER:
|
||||
if (i == 0 || !is_token_element(rule[i - 1])) {
|
||||
throw std::runtime_error(
|
||||
"LLAMA_GRETYPE_TOKEN_RNG_UPPER without preceding token: " +
|
||||
std::to_string(rule_id) + "," + std::to_string(i));
|
||||
}
|
||||
fprintf(file, "-%u", elem.value);
|
||||
break;
|
||||
case LLAMA_GRETYPE_TOKEN_ALT:
|
||||
if (i == 0 || !is_token_element(rule[i - 1])) {
|
||||
throw std::runtime_error(
|
||||
"LLAMA_GRETYPE_TOKEN_ALT without preceding token: " +
|
||||
std::to_string(rule_id) + "," + std::to_string(i));
|
||||
}
|
||||
fprintf(file, ",%u", elem.value);
|
||||
break;
|
||||
}
|
||||
if (is_token_element(elem) &&
|
||||
rule[i + 1].type != LLAMA_GRETYPE_TOKEN_ALT && rule[i + 1].type != LLAMA_GRETYPE_TOKEN_RNG_UPPER) {
|
||||
fprintf(file, "]> ");
|
||||
}
|
||||
if (is_char_element(elem)) {
|
||||
switch (rule[i + 1].type) {
|
||||
@@ -645,9 +587,11 @@ const char * llama_grammar_parser::parse_sequence(
|
||||
type = LLAMA_GRETYPE_TOKEN_NOT;
|
||||
pos++;
|
||||
}
|
||||
auto token_pair = parse_token(vocab, pos);
|
||||
const char * token_end = token_pair.second;
|
||||
last_sym_start = rule.size();
|
||||
n_prev_rules = 1;
|
||||
const char * token_end = parse_token(vocab, pos, type, rule);
|
||||
rule.push_back({type, token_pair.first});
|
||||
pos = parse_space(token_end, is_nested);
|
||||
} else if (is_word_char(*pos)) { // rule reference
|
||||
const char * name_end = parse_name(pos);
|
||||
@@ -910,58 +854,17 @@ static bool llama_grammar_match_partial_char(
|
||||
|
||||
// returns true iff token matches the rule at pos (regular or inverse)
|
||||
// asserts that pos is pointing to a token element
|
||||
static std::pair<bool, const llama_grammar_element *> llama_grammar_match_token(
|
||||
static bool llama_grammar_match_token(
|
||||
const llama_grammar_element * pos,
|
||||
const llama_token token) {
|
||||
bool is_positive_token = pos->type == LLAMA_GRETYPE_TOKEN;
|
||||
|
||||
GGML_ASSERT(is_positive_token || pos->type == LLAMA_GRETYPE_TOKEN_NOT);
|
||||
|
||||
const uint32_t id = static_cast<uint32_t>(token);
|
||||
|
||||
bool found = false;
|
||||
do {
|
||||
if (pos[1].type == LLAMA_GRETYPE_TOKEN_RNG_UPPER) {
|
||||
// inclusive range, e.g. <[1-5]>
|
||||
found = found || (pos->value <= id && id <= pos[1].value);
|
||||
pos += 2;
|
||||
} else {
|
||||
found = found || pos->value == id;
|
||||
pos += 1;
|
||||
}
|
||||
} while (pos->type == LLAMA_GRETYPE_TOKEN_ALT);
|
||||
|
||||
return std::make_pair(found == is_positive_token, pos);
|
||||
}
|
||||
|
||||
// every token id a token rule names, merged into sorted inclusive ranges
|
||||
static std::vector<std::pair<uint32_t, uint32_t>> llama_grammar_opaque_tokens(const llama_grammar_rules & rules) {
|
||||
std::vector<std::pair<uint32_t, uint32_t>> ranges;
|
||||
for (const auto & rule : rules) {
|
||||
for (size_t i = 0; i < rule.size(); i++) {
|
||||
const auto type = rule[i].type;
|
||||
if (type == LLAMA_GRETYPE_TOKEN || type == LLAMA_GRETYPE_TOKEN_NOT || type == LLAMA_GRETYPE_TOKEN_ALT) {
|
||||
const bool is_range = i + 1 < rule.size() && rule[i + 1].type == LLAMA_GRETYPE_TOKEN_RNG_UPPER;
|
||||
ranges.emplace_back(rule[i].value, is_range ? rule[i + 1].value : rule[i].value);
|
||||
}
|
||||
}
|
||||
GGML_ASSERT(pos->type == LLAMA_GRETYPE_TOKEN || pos->type == LLAMA_GRETYPE_TOKEN_NOT);
|
||||
if (pos->type == LLAMA_GRETYPE_TOKEN) {
|
||||
return pos->value == static_cast<uint32_t>(token);
|
||||
}
|
||||
std::sort(ranges.begin(), ranges.end());
|
||||
std::vector<std::pair<uint32_t, uint32_t>> merged;
|
||||
for (const auto & r : ranges) {
|
||||
if (!merged.empty() && r.first <= (uint64_t) merged.back().second + 1) {
|
||||
merged.back().second = std::max(merged.back().second, r.second);
|
||||
} else {
|
||||
merged.push_back(r);
|
||||
}
|
||||
if (pos->type == LLAMA_GRETYPE_TOKEN_NOT) {
|
||||
return pos->value != static_cast<uint32_t>(token);
|
||||
}
|
||||
return merged;
|
||||
}
|
||||
|
||||
bool llama_grammar_is_opaque(const llama_grammar & grammar, llama_token token) {
|
||||
const auto id = static_cast<uint32_t>(token);
|
||||
auto it = std::upper_bound(grammar.opaque_tokens.begin(), grammar.opaque_tokens.end(), std::make_pair(id, UINT32_MAX));
|
||||
return it != grammar.opaque_tokens.begin() && std::prev(it)->second >= id;
|
||||
return false;
|
||||
}
|
||||
|
||||
// transforms a grammar pushdown stack into N possible stacks, all ending
|
||||
@@ -1043,7 +946,8 @@ static void llama_grammar_advance_stack(
|
||||
break;
|
||||
default:
|
||||
// end of alternate (LLAMA_GRETYPE_END, LLAMA_GRETYPE_ALT) or middle of char range
|
||||
// (LLAMA_GRETYPE_CHAR_ALT, LLAMA_GRETYPE_CHAR_RNG_UPPER) or token set (LLAMA_GRETYPE_TOKEN_ALT, LLAMA_GRETYPE_TOKEN_RNG_UPPER); stack should never be left on those
|
||||
// (LLAMA_GRETYPE_CHAR_ALT, LLAMA_GRETYPE_CHAR_RNG_UPPER); stack should never be left on
|
||||
// those
|
||||
GGML_ABORT("fatal error");
|
||||
}
|
||||
}
|
||||
@@ -1334,13 +1238,13 @@ static bool llama_grammar_reject_step(
|
||||
if (stack_pos->type == LLAMA_GRETYPE_TOKEN || stack_pos->type == LLAMA_GRETYPE_TOKEN_NOT) {
|
||||
for (const uint32_t i : cur) {
|
||||
const auto & tok = candidates[i];
|
||||
if (depth > 0) {
|
||||
// char rules already consumed part of this token, so it cannot match by id.
|
||||
// accept it only if they consumed all of it and did not end in a partial sequence.
|
||||
if (tok.code_points[depth] != 0 || tok.partial_utf8.n_remain != 0) {
|
||||
if (tok.code_points[depth] == 0) {
|
||||
// reached the end of a token consumed by char rules, reject iff it ended
|
||||
// in a partial response
|
||||
if (tok.partial_utf8.n_remain != 0) {
|
||||
rejects.push_back(i);
|
||||
}
|
||||
} else if (tok.partial_utf8.n_remain < 0 || !llama_grammar_match_token(stack_pos, tok.id).first) {
|
||||
} else if (!llama_grammar_match_token(stack_pos, tok.id)) {
|
||||
rejects.push_back(i);
|
||||
}
|
||||
}
|
||||
@@ -1351,10 +1255,7 @@ static bool llama_grammar_reject_step(
|
||||
|
||||
for (const uint32_t i : cur) {
|
||||
const auto & tok = candidates[i];
|
||||
if (tok.opaque) {
|
||||
// a token named by a token rule only matches token rules
|
||||
rejects.push_back(i);
|
||||
} else if (tok.code_points[depth] == 0) {
|
||||
if (tok.code_points[depth] == 0) {
|
||||
// reached end of full codepoints in token, reject iff it ended in a partial sequence
|
||||
// that cannot satisfy this position in grammar
|
||||
if (tok.partial_utf8.n_remain != 0 &&
|
||||
@@ -1524,8 +1425,6 @@ struct llama_grammar * llama_grammar_init_impl(
|
||||
}
|
||||
} while (true);
|
||||
|
||||
auto opaque_tokens = llama_grammar_opaque_tokens(vec_rules);
|
||||
|
||||
// Important: vec_rules has to be moved here, not copied, because stacks contains
|
||||
// pointers to elements of vec_rules. If vec_rules were copied into llama_grammar
|
||||
// then the pointers would be invalidated when the local vec_rules goes out of scope.
|
||||
@@ -1540,7 +1439,6 @@ struct llama_grammar * llama_grammar_init_impl(
|
||||
/* .trigger_buffer_positions = */ {},
|
||||
/* .trigger_tokens = */ {},
|
||||
/* .trigger_patterns = */ {},
|
||||
/* .opaque_tokens = */ std::move(opaque_tokens),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1642,8 +1540,6 @@ struct llama_grammar * llama_grammar_init_impl(
|
||||
trigger.regex = std::regex(trigger.pattern);
|
||||
}
|
||||
|
||||
auto opaque_tokens = llama_grammar_opaque_tokens(vec_rules);
|
||||
|
||||
// Important: vec_rules has to be moved here, not copied, because stacks contains
|
||||
// pointers to elements of vec_rules. If vec_rules were copied into llama_grammar
|
||||
// then the pointers would be invalidated when the local vec_rules goes out of scope.
|
||||
@@ -1658,7 +1554,6 @@ struct llama_grammar * llama_grammar_init_impl(
|
||||
/* .trigger_buffer_positions = */ {},
|
||||
std::move(vec_trigger_tokens),
|
||||
std::move(vec_trigger_patterns),
|
||||
std::move(opaque_tokens),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1706,9 +1601,8 @@ void llama_grammar_apply_impl(const struct llama_grammar & grammar, llama_token_
|
||||
} else if (piece.empty() || piece[0] == 0) {
|
||||
cur_p->data[i].logit = -INFINITY;
|
||||
} else {
|
||||
candidates_decoded.push_back(llama_grammar_decode_utf8(piece, grammar.partial_utf8));
|
||||
candidates_grammar.push_back({ i, candidates_decoded.back().first.data(), candidates_decoded.back().second, id,
|
||||
llama_grammar_is_opaque(grammar, id) });
|
||||
candidates_decoded.push_back(decode_utf8(piece, grammar.partial_utf8));
|
||||
candidates_grammar.push_back({ i, candidates_decoded.back().first.data(), candidates_decoded.back().second, id });
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1779,7 +1673,7 @@ void llama_grammar_accept_impl(struct llama_grammar & grammar, llama_token token
|
||||
|
||||
void llama_grammar_accept_str(struct llama_grammar & grammar, const std::string & piece) {
|
||||
// Note terminating 0 in decoded string
|
||||
const auto decoded = llama_grammar_decode_utf8(piece, grammar.partial_utf8);
|
||||
const auto decoded = decode_utf8(piece, grammar.partial_utf8);
|
||||
const auto & code_points = decoded.first;
|
||||
|
||||
for (auto it = code_points.begin(), end = code_points.end() - 1; it != end; ++it) {
|
||||
@@ -1794,16 +1688,9 @@ void llama_grammar_accept_str(struct llama_grammar & grammar, const std::string
|
||||
|
||||
void llama_grammar_accept_token(struct llama_grammar & grammar, llama_token token, const std::string & piece) {
|
||||
// Note terminating 0 in decoded string
|
||||
const auto decoded = llama_grammar_decode_utf8(piece, grammar.partial_utf8);
|
||||
const auto decoded = decode_utf8(piece, grammar.partial_utf8);
|
||||
const auto & code_points = decoded.first;
|
||||
|
||||
// the mask rejects a piece that does not continue a partial character, so accepting one has to fail too
|
||||
if (decoded.second.n_remain < 0) {
|
||||
throw std::runtime_error("Invalid UTF-8 continuation in piece: " + piece + " (" + std::to_string(token) + ")");
|
||||
}
|
||||
|
||||
const bool opaque = llama_grammar_is_opaque(grammar, token);
|
||||
|
||||
llama_grammar_stacks stacks_new;
|
||||
stacks_new.reserve(grammar.stacks.size());
|
||||
|
||||
@@ -1815,15 +1702,14 @@ void llama_grammar_accept_token(struct llama_grammar & grammar, llama_token toke
|
||||
const llama_grammar_element * pos = stack.back();
|
||||
|
||||
if (pos->type == LLAMA_GRETYPE_TOKEN || pos->type == LLAMA_GRETYPE_TOKEN_NOT) {
|
||||
auto match = llama_grammar_match_token(pos, token);
|
||||
if (match.first) {
|
||||
if (llama_grammar_match_token(pos, token)) {
|
||||
llama_grammar_stack new_stack(stack.begin(), stack.end() - 1);
|
||||
if (!llama_grammar_is_end_of_sequence(match.second)) {
|
||||
new_stack.push_back(match.second);
|
||||
if (!llama_grammar_is_end_of_sequence(pos + 1)) {
|
||||
new_stack.push_back(pos + 1);
|
||||
}
|
||||
llama_grammar_advance_stack(*grammar.rules, new_stack, stacks_new);
|
||||
}
|
||||
} else if (!opaque) {
|
||||
} else {
|
||||
llama_grammar_stacks current_stacks = {stack};
|
||||
|
||||
for (auto it = code_points.begin(), end = code_points.end() - 1; it != end; ++it) {
|
||||
|
||||
+2
-18
@@ -38,17 +38,11 @@ enum llama_gretype {
|
||||
// any character (.)
|
||||
LLAMA_GRETYPE_CHAR_ANY = 7,
|
||||
|
||||
// terminal element: token (<[token-id]>, <[token-id,token-id-token-id]>)
|
||||
// terminal element: token (<[token-id]>)
|
||||
LLAMA_GRETYPE_TOKEN = 8,
|
||||
|
||||
// inverse token (!<[token-id]>, !<[token-id,token-id-token-id]>)
|
||||
// inverse token (!<[token-id]>)
|
||||
LLAMA_GRETYPE_TOKEN_NOT = 9,
|
||||
|
||||
// modifies a preceding LLAMA_GRETYPE_TOKEN, LLAMA_GRETYPE_TOKEN_NOT, or LLAMA_GRETYPE_TOKEN_ALT to be an inclusive range (<[1-5]>)
|
||||
LLAMA_GRETYPE_TOKEN_RNG_UPPER = 10,
|
||||
|
||||
// modifies a preceding LLAMA_GRETYPE_TOKEN, LLAMA_GRETYPE_TOKEN_NOT, or LLAMA_GRETYPE_TOKEN_RNG_UPPER to add an alternate token to match (<[1,5]>, <[1-5,9]>)
|
||||
LLAMA_GRETYPE_TOKEN_ALT = 11,
|
||||
};
|
||||
|
||||
typedef struct llama_grammar_element {
|
||||
@@ -66,7 +60,6 @@ struct llama_grammar_candidate {
|
||||
const uint32_t * code_points;
|
||||
llama_partial_utf8 partial_utf8;
|
||||
llama_token id;
|
||||
bool opaque;
|
||||
};
|
||||
|
||||
using llama_grammar_rule = std::vector< llama_grammar_element>;
|
||||
@@ -91,10 +84,6 @@ std::vector<llama_grammar_candidate> llama_grammar_reject_candidates_for_stack(
|
||||
const llama_grammar_stack & stack,
|
||||
const llama_grammar_candidates & candidates);
|
||||
|
||||
std::pair<std::vector<uint32_t>, llama_partial_utf8> llama_grammar_decode_utf8(
|
||||
const std::string & src,
|
||||
llama_partial_utf8 partial_start);
|
||||
|
||||
struct llama_grammar_parser {
|
||||
const llama_vocab * vocab;
|
||||
std::map<std::string, uint32_t> symbol_ids;
|
||||
@@ -162,8 +151,6 @@ struct llama_grammar {
|
||||
trigger_patterns; // Regular expressions that trigger a lazy grammar. Must be a full match of the entire generated
|
||||
// string, and the grammar will be given the string from the first match group onwards.
|
||||
|
||||
// token ids named by token rules, as sorted inclusive ranges. char rules never match these tokens.
|
||||
std::vector<std::pair<uint32_t, uint32_t>> opaque_tokens;
|
||||
};
|
||||
|
||||
//
|
||||
@@ -208,6 +195,3 @@ void llama_grammar_accept_token(
|
||||
struct llama_grammar & grammar,
|
||||
llama_token token,
|
||||
const std::string & piece);
|
||||
|
||||
// true if a token rule names the token, char rules do not match it
|
||||
bool llama_grammar_is_opaque(const llama_grammar & grammar, llama_token token);
|
||||
@@ -38,7 +38,6 @@
|
||||
#include <sstream>
|
||||
#include <stdexcept>
|
||||
#include <string>
|
||||
#include <unordered_map>
|
||||
#include <vector>
|
||||
|
||||
static llama_model * llama_model_mapping(llm_arch arch, const llama_model_params & params) {
|
||||
@@ -4282,38 +4281,6 @@ int32_t llama_model_n_expert(const struct llama_model * model) {
|
||||
return model->hparams.n_expert;
|
||||
}
|
||||
|
||||
std::vector<uint32_t> llama_model_layer_visits(const struct llama_model * model) {
|
||||
const size_t n_layer = model->layers.size();
|
||||
std::vector<uint32_t> visits(n_layer, 0);
|
||||
std::unordered_map<const ggml_tensor *, size_t> owner;
|
||||
owner.reserve(n_layer);
|
||||
|
||||
for (size_t il = 0; il < n_layer; ++il) {
|
||||
const auto & l = model->layers[il];
|
||||
const ggml_tensor * key =
|
||||
l.ffn_up_exps ? l.ffn_up_exps :
|
||||
l.ffn_gate_up_exps ? l.ffn_gate_up_exps :
|
||||
l.ffn_gate_exps ? l.ffn_gate_exps :
|
||||
l.ffn_down_exps ? l.ffn_down_exps :
|
||||
l.attn_norm ? l.attn_norm :
|
||||
l.ffn_norm ? l.ffn_norm :
|
||||
l.wq ? l.wq :
|
||||
l.wqkv ? l.wqkv :
|
||||
l.ffn_up ? l.ffn_up :
|
||||
l.ffn_gate ? l.ffn_gate :
|
||||
l.ffn_down ? l.ffn_down :
|
||||
nullptr;
|
||||
if (!key) {
|
||||
visits[il] = 1;
|
||||
continue;
|
||||
}
|
||||
auto [it, inserted] = owner.emplace(key, il);
|
||||
visits[it->second]++;
|
||||
}
|
||||
|
||||
return visits;
|
||||
}
|
||||
|
||||
int32_t llama_model_n_devices(const struct llama_model * model) {
|
||||
return (int32_t)model->devices.size();
|
||||
}
|
||||
|
||||
@@ -2673,7 +2673,6 @@ void llama_vocab::impl::load(llama_model_loader & ml, const LLM_KV & kv) {
|
||||
pre_type = LLAMA_VOCAB_PRE_TYPE_GPT4O;
|
||||
special_bos_id = LLAMA_TOKEN_NULL;
|
||||
clean_spaces = false;
|
||||
ignore_merges = true;
|
||||
} else if (
|
||||
tokenizer_pre == "granite-embed-multi-97m") {
|
||||
pre_type = LLAMA_VOCAB_PRE_TYPE_GRANITE_EMB_MULTI;
|
||||
|
||||
+16
-32
@@ -173,35 +173,22 @@ static clef_spans clef_get_spans(const llama_ubatch & ubatch) {
|
||||
return res;
|
||||
}
|
||||
|
||||
// the head always has the max number of spans, so its graph depends only on n_tokens
|
||||
// each span has 1 token or more and each question has an option, so there is always a spare question for the padding options
|
||||
static size_t clef_n_questions(int64_t n_tokens) { return n_tokens / 2 + 1; }
|
||||
static size_t clef_n_options (int64_t n_tokens) { return std::max<int64_t>(n_tokens - 1, 1); }
|
||||
|
||||
// pooling matrices and indices computed from the spans
|
||||
class llama_model_clef::input_decision : public llm_graph_input_i {
|
||||
public:
|
||||
input_decision(const llama_ubatch & ubatch) :
|
||||
n_tokens(ubatch.n_tokens), n_questions(clef_n_questions(n_tokens)), n_options(clef_n_options(n_tokens)) {}
|
||||
input_decision(const llama_ubatch & ubatch) : n_tokens(ubatch.n_tokens) {
|
||||
const auto spans = clef_get_spans(ubatch);
|
||||
n_questions = spans.questions.size();
|
||||
n_options = spans.options.size();
|
||||
}
|
||||
|
||||
void set_input(const llama_ubatch * ubatch) override {
|
||||
// a batch of embeddings has no token ids, and no usable spans
|
||||
const std::vector<llama_token> no_tokens(ubatch->token ? 0 : n_tokens, 0);
|
||||
ggml_backend_tensor_set(tokens, ubatch->token ? ubatch->token : no_tokens.data(), 0, n_tokens * sizeof(llama_token));
|
||||
|
||||
auto spans = clef_get_spans(*ubatch);
|
||||
const int32_t n_q_used = spans.questions.size();
|
||||
GGML_ASSERT((size_t) n_q_used < n_questions && spans.options.size() <= n_options);
|
||||
|
||||
// the padding options belong to the first padding question, the questions do not read the padding questions
|
||||
spans.questions.resize(n_questions, { 0, 0, 0 });
|
||||
spans.options.resize(n_options, { n_q_used, 0, 0 });
|
||||
|
||||
std::vector<float> q_mask(n_questions * n_questions, 0.0f);
|
||||
for (size_t i = 0; i < n_questions; i++) {
|
||||
std::fill(q_mask.begin() + i * n_questions + n_q_used, q_mask.begin() + (i + 1) * n_questions, -INFINITY);
|
||||
}
|
||||
ggml_backend_tensor_set(question_mask, q_mask.data(), 0, ggml_nbytes(question_mask));
|
||||
const auto spans = clef_get_spans(*ubatch);
|
||||
GGML_ASSERT(spans.questions.size() == n_questions && spans.options.size() == n_options);
|
||||
|
||||
// the scores are NaN if the batch has a decision order that cannot be used
|
||||
const float status_data = spans.valid || ubatch->decision_order == nullptr ? 0.0f : NAN;
|
||||
@@ -228,8 +215,6 @@ public:
|
||||
owner[i] = o.question;
|
||||
mask[o.question * n_options + i] = 0.0f;
|
||||
}
|
||||
// a padding question reads all options, a row of -inf gives NaN
|
||||
std::fill(mask.begin() + n_q_used * n_options, mask.end(), 0.0f);
|
||||
|
||||
ggml_backend_tensor_set(pool_q, pool_q_data.data(), 0, ggml_nbytes(pool_q));
|
||||
ggml_backend_tensor_set(pool_o, pool_o_data.data(), 0, ggml_nbytes(pool_o));
|
||||
@@ -240,7 +225,8 @@ public:
|
||||
|
||||
bool can_reuse(const llm_graph_params & params) override {
|
||||
// the values are computed again in set_input(), only the shapes must match
|
||||
return params.ubatch.n_tokens == (uint32_t) n_tokens;
|
||||
const auto spans = clef_get_spans(params.ubatch);
|
||||
return spans.questions.size() == n_questions && spans.options.size() == n_options;
|
||||
}
|
||||
|
||||
ggml_tensor * tokens = nullptr; // I32 [n_tokens]
|
||||
@@ -249,12 +235,11 @@ public:
|
||||
ggml_tensor * question_type = nullptr; // I32 [n_questions]
|
||||
ggml_tensor * option_question = nullptr; // I32 [n_options]
|
||||
ggml_tensor * option_mask = nullptr; // F32 [n_options, n_questions], 0 if the option belongs to the question, else -inf
|
||||
ggml_tensor * question_mask = nullptr; // F32 [n_questions, n_questions], -inf for the padding questions
|
||||
ggml_tensor * status = nullptr; // F32 [1], added to the scores: 0, or NaN on invalid input
|
||||
|
||||
const int64_t n_tokens;
|
||||
const size_t n_questions;
|
||||
const size_t n_options;
|
||||
size_t n_questions;
|
||||
size_t n_options;
|
||||
};
|
||||
|
||||
// the backbone is copied from llama_model_qwen35::graph, without the memory module
|
||||
@@ -494,8 +479,8 @@ ggml_tensor * llama_model_clef::graph::build_head_norm(ggml_tensor * cur, const
|
||||
return build_norm(cur, n.w, n.b, LLM_NORM, -1);
|
||||
}
|
||||
|
||||
// q: [n_embd_h, n_q], kv: [n_embd_h, n_kv], kq_mask: [n_kv, n_q] or null
|
||||
ggml_tensor * llama_model_clef::graph::build_head_attn(ggml_tensor * q, ggml_tensor * kv, const attn & a, ggml_tensor * kq_mask) {
|
||||
// q: [n_embd_h, n_q], kv: [n_embd_h, n_kv], no mask
|
||||
ggml_tensor * llama_model_clef::graph::build_head_attn(ggml_tensor * q, ggml_tensor * kv, const attn & a) {
|
||||
const int64_t n_embd_h = q->ne[0];
|
||||
const int64_t n_head_h = model.n_head_decision;
|
||||
const int64_t d_head = n_embd_h / n_head_h;
|
||||
@@ -512,7 +497,7 @@ ggml_tensor * llama_model_clef::graph::build_head_attn(ggml_tensor * q, ggml_ten
|
||||
Vcur = ggml_cont(ctx0, Vcur);
|
||||
|
||||
ggml_tensor * kq = ggml_mul_mat(ctx0, Kcur, Qcur); // [n_kv, n_q, n_head]
|
||||
kq = ggml_soft_max_ext(ctx0, kq, kq_mask, 1.0f / sqrtf(float(d_head)), 0.0f);
|
||||
kq = ggml_soft_max_ext(ctx0, kq, nullptr, 1.0f / sqrtf(float(d_head)), 0.0f);
|
||||
|
||||
ggml_tensor * cur = ggml_mul_mat(ctx0, Vcur, kq); // [d_head, n_q, n_head]
|
||||
cur = ggml_cont_2d(ctx0, ggml_permute(ctx0, cur, 0, 2, 1, 3), n_embd_h, n_q);
|
||||
@@ -538,9 +523,8 @@ ggml_tensor * llama_model_clef::graph::build_head(ggml_tensor * hidden, input_de
|
||||
inp->question_type = ggml_new_tensor_1d(ctx0, GGML_TYPE_I32, n_questions);
|
||||
inp->option_question = ggml_new_tensor_1d(ctx0, GGML_TYPE_I32, n_options);
|
||||
inp->option_mask = ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, n_options, n_questions);
|
||||
inp->question_mask = ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, n_questions, n_questions);
|
||||
inp->status = ggml_new_tensor_1d(ctx0, GGML_TYPE_F32, 1);
|
||||
for (ggml_tensor * t : { inp->tokens, inp->pool_q, inp->pool_o, inp->question_type, inp->option_question, inp->option_mask, inp->question_mask, inp->status }) {
|
||||
for (ggml_tensor * t : { inp->tokens, inp->pool_q, inp->pool_o, inp->question_type, inp->option_question, inp->option_mask, inp->status }) {
|
||||
ggml_set_input(t);
|
||||
}
|
||||
|
||||
@@ -600,7 +584,7 @@ ggml_tensor * llama_model_clef::graph::build_head(ggml_tensor * hidden, input_de
|
||||
const auto & layer = model.head_layers[il];
|
||||
|
||||
ggml_tensor * cur = build_head_norm(fields, layer.self_norm);
|
||||
fields = ggml_add(ctx0, fields, build_head_attn(cur, cur, layer.self_attn, inp->question_mask));
|
||||
fields = ggml_add(ctx0, fields, build_head_attn(cur, cur, layer.self_attn));
|
||||
|
||||
cur = build_head_norm(fields, layer.cross_norm);
|
||||
fields = ggml_add(ctx0, fields, build_head_attn(cur, memory, layer.cross_attn));
|
||||
|
||||
+1
-1
@@ -2930,7 +2930,7 @@ struct llama_model_clef : public llama_model_qwen35 {
|
||||
input_decision * inp);
|
||||
|
||||
ggml_tensor * build_head_norm(ggml_tensor * cur, const norm & n);
|
||||
ggml_tensor * build_head_attn(ggml_tensor * q, ggml_tensor * kv, const attn & a, ggml_tensor * kq_mask = nullptr);
|
||||
ggml_tensor * build_head_attn(ggml_tensor * q, ggml_tensor * kv, const attn & a);
|
||||
ggml_tensor * build_head_ffn (ggml_tensor * cur, const head_layer & layer);
|
||||
|
||||
const llama_model_clef & model;
|
||||
|
||||
@@ -99,19 +99,13 @@ static decision2_rows decision2_get_rows(const llama_ubatch & ubatch) {
|
||||
return res;
|
||||
}
|
||||
|
||||
// no decision order (e.g. reserve): use the max number of options
|
||||
static size_t decision2_n_options(const llama_ubatch & ubatch, uint32_t n_outputs) {
|
||||
return ubatch.decision_order ? decision2_get_rows(ubatch).options.size() : std::max<uint32_t>(n_outputs, 1);
|
||||
}
|
||||
|
||||
class llama_model_vllm_decision2::input_decision : public llm_graph_input_i {
|
||||
public:
|
||||
input_decision(const llama_ubatch & ubatch, uint32_t n_outputs) : n_options(decision2_n_options(ubatch, n_outputs)) {}
|
||||
input_decision(const llama_ubatch & ubatch) : n_options(decision2_get_rows(ubatch).options.size()) {}
|
||||
|
||||
void set_input(const llama_ubatch * ubatch) override {
|
||||
auto rows = decision2_get_rows(*ubatch);
|
||||
GGML_ASSERT(rows.options.size() <= n_options);
|
||||
rows.options.resize(n_options, 0);
|
||||
const auto rows = decision2_get_rows(*ubatch);
|
||||
GGML_ASSERT(rows.options.size() == n_options);
|
||||
|
||||
// the scores are NaN if the batch has a decision order that cannot be used
|
||||
const float status_data = rows.valid || ubatch->decision_order == nullptr ? 0.0f : NAN;
|
||||
@@ -123,7 +117,7 @@ public:
|
||||
|
||||
bool can_reuse(const llm_graph_params & params) override {
|
||||
// the values are computed again in set_input(), only the shapes must match
|
||||
return decision2_n_options(params.ubatch, params.n_outputs) == n_options;
|
||||
return decision2_get_rows(params.ubatch).options.size() == n_options;
|
||||
}
|
||||
|
||||
ggml_tensor * options = nullptr; // I32 [n_options]
|
||||
@@ -135,7 +129,7 @@ public:
|
||||
|
||||
llama_model_vllm_decision2::graph::graph(const llm_graph_params & params) :
|
||||
llama_model_qwen3::graph(params, no_build()), model(static_cast<const llama_model_vllm_decision2 &>(*params.model)) {
|
||||
auto inp_decision_ptr = std::make_unique<input_decision>(ubatch, n_outputs);
|
||||
auto inp_decision_ptr = std::make_unique<input_decision>(ubatch);
|
||||
auto * inp_decision = inp_decision_ptr.get();
|
||||
res->add_input(std::move(inp_decision_ptr));
|
||||
|
||||
|
||||
@@ -367,49 +367,6 @@ void test_basic(testing & t) {
|
||||
});
|
||||
|
||||
|
||||
t.test("through", [](testing & t) {
|
||||
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
|
||||
return p.through(p.tag("body", p.until_one_of({ "</a>", "</b>" })), { { p.literal("</a>"), p.literal("x") }, { p.literal("</b>") } }, true);
|
||||
});
|
||||
|
||||
t.test("consumes the delimiter and its rest", [&](testing & t) {
|
||||
auto ctx = common_peg_parse_context("hello</a>x");
|
||||
auto result = parser.parse(ctx);
|
||||
t.assert_equal("success", true, result.success());
|
||||
t.assert_equal("end", (size_t) 10, result.end);
|
||||
t.assert_equal("body", std::string("hello"), std::string(ctx.ast.get(result.nodes[0]).text));
|
||||
});
|
||||
|
||||
t.test("picks the branch of the delimiter found", [&](testing & t) {
|
||||
auto ctx = common_peg_parse_context("hello</b>x");
|
||||
auto result = parser.parse(ctx);
|
||||
t.assert_equal("success", true, result.success());
|
||||
t.assert_equal("end", (size_t) 9, result.end);
|
||||
});
|
||||
|
||||
t.test("waits for the rest of a partial delimiter", [&](testing & t) {
|
||||
auto ctx = common_peg_parse_context("hello</", COMMON_PEG_PARSE_FLAG_LENIENT);
|
||||
auto result = parser.parse(ctx);
|
||||
t.assert_equal("need_more_input", true, result.need_more_input());
|
||||
});
|
||||
|
||||
t.test("optional may end before a delimiter", [&](testing & t) {
|
||||
auto ctx = common_peg_parse_context("hello");
|
||||
auto result = parser.parse(ctx);
|
||||
t.assert_equal("success", true, result.success());
|
||||
t.assert_equal("end", (size_t) 5, result.end);
|
||||
});
|
||||
|
||||
t.test("fails when content stops off a delimiter", [&](testing & t) {
|
||||
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
|
||||
return p.through(p.literal("he"), "</a>");
|
||||
});
|
||||
auto ctx = common_peg_parse_context("hello</a>");
|
||||
auto result = parser.parse(ctx);
|
||||
t.assert_equal("fail", true, result.fail());
|
||||
});
|
||||
});
|
||||
|
||||
t.test("recursive rules", [](testing &t) {
|
||||
// Test simple number
|
||||
t.test("simple_number", [](testing &t) {
|
||||
|
||||
@@ -129,14 +129,14 @@ void test_gbnf_generation(testing &t) {
|
||||
});
|
||||
|
||||
assert_gbnf_equal(t, R"""(
|
||||
root ::= until-1
|
||||
root ::= until-0
|
||||
space ::= | " " | "\n"{1,2} [ \t]{0,20}
|
||||
until-1 ::= | [<] until-1-01 | [^<] until-1
|
||||
until-1-01 ::= | [<] until-1-01 | [/] until-1-02 | [^/<] until-1
|
||||
until-1-02 ::= | [<] until-1-01 | [t] until-1-03 | [^<t] until-1
|
||||
until-1-03 ::= | [<] until-1-01 | [a] until-1-04 | [^<a] until-1
|
||||
until-1-04 ::= | [<] until-1-01 | [g] until-1-05 | [^<g] until-1
|
||||
until-1-05 ::= | [<] until-1-01 | [^<>] until-1
|
||||
until-0 ::= | [<] until-0-01 | [^<] until-0
|
||||
until-0-01 ::= | [<] until-0-01 | [/] until-0-02 | [^/<] until-0
|
||||
until-0-02 ::= | [<] until-0-01 | [t] until-0-03 | [^<t] until-0
|
||||
until-0-03 ::= | [<] until-0-01 | [a] until-0-04 | [^<a] until-0
|
||||
until-0-04 ::= | [<] until-0-01 | [g] until-0-05 | [^<g] until-0
|
||||
until-0-05 ::= | [<] until-0-01 | [^<>] until-0
|
||||
)""", gbnf);
|
||||
});
|
||||
|
||||
@@ -150,22 +150,22 @@ void test_gbnf_generation(testing &t) {
|
||||
});
|
||||
|
||||
assert_gbnf_equal(t, R"""(
|
||||
root ::= until-1
|
||||
root ::= until-0
|
||||
space ::= | " " | "\n"{1,2} [ \t]{0,20}
|
||||
until-1 ::= | [\n] until-1-01 | [^\n] until-1
|
||||
until-1-01 ::= | [\n] until-1-01 | [<] until-1-02 | [^\n<] until-1
|
||||
until-1-02 ::= | [\n] until-1-01 | [/] until-1-03 | [^\n/] until-1
|
||||
until-1-03 ::= | [\n] until-1-01 | [p] until-1-04 | [^\np] until-1
|
||||
until-1-04 ::= | [\n] until-1-01 | [a] until-1-05 | [^\na] until-1
|
||||
until-1-05 ::= | [\n] until-1-01 | [r] until-1-06 | [^\nr] until-1
|
||||
until-1-06 ::= | [\n] until-1-01 | [a] until-1-07 | [^\na] until-1
|
||||
until-1-07 ::= | [\n] until-1-01 | [m] until-1-08 | [^\nm] until-1
|
||||
until-1-08 ::= | [\n] until-1-01 | [e] until-1-09 | [^\ne] until-1
|
||||
until-1-09 ::= | [\n] until-1-01 | [t] until-1-10 | [^\nt] until-1
|
||||
until-1-10 ::= | [\n] until-1-01 | [e] until-1-11 | [^\ne] until-1
|
||||
until-1-11 ::= | [\n] until-1-01 | [r] until-1-12 | [^\nr] until-1
|
||||
until-1-12 ::= | [\n] until-1-01 | [>] until-1-13 | [^\n>] until-1
|
||||
until-1-13 ::= | [^\n] until-1
|
||||
until-0 ::= | [\n] until-0-01 | [^\n] until-0
|
||||
until-0-01 ::= | [\n] until-0-01 | [<] until-0-02 | [^\n<] until-0
|
||||
until-0-02 ::= | [\n] until-0-01 | [/] until-0-03 | [^\n/] until-0
|
||||
until-0-03 ::= | [\n] until-0-01 | [p] until-0-04 | [^\np] until-0
|
||||
until-0-04 ::= | [\n] until-0-01 | [a] until-0-05 | [^\na] until-0
|
||||
until-0-05 ::= | [\n] until-0-01 | [r] until-0-06 | [^\nr] until-0
|
||||
until-0-06 ::= | [\n] until-0-01 | [a] until-0-07 | [^\na] until-0
|
||||
until-0-07 ::= | [\n] until-0-01 | [m] until-0-08 | [^\nm] until-0
|
||||
until-0-08 ::= | [\n] until-0-01 | [e] until-0-09 | [^\ne] until-0
|
||||
until-0-09 ::= | [\n] until-0-01 | [t] until-0-10 | [^\nt] until-0
|
||||
until-0-10 ::= | [\n] until-0-01 | [e] until-0-11 | [^\ne] until-0
|
||||
until-0-11 ::= | [\n] until-0-01 | [r] until-0-12 | [^\nr] until-0
|
||||
until-0-12 ::= | [\n] until-0-01 | [>] until-0-13 | [^\n>] until-0
|
||||
until-0-13 ::= | [^\n] until-0
|
||||
)""", gbnf);
|
||||
});
|
||||
|
||||
@@ -181,15 +181,15 @@ void test_gbnf_generation(testing &t) {
|
||||
});
|
||||
|
||||
assert_gbnf_equal(t, R"""(
|
||||
root ::= until-1
|
||||
root ::= until-0
|
||||
space ::= | " " | "\n"{1,2} [ \t]{0,20}
|
||||
until-1 ::= | [<] until-1-01 | [^<] until-1
|
||||
until-1-01 ::= | [<] until-1-01 | [\uFF5C] until-1-02 | [^<\uFF5C] until-1
|
||||
until-1-02 ::= | [<] until-1-01 | [D] until-1-03 | [^<D] until-1
|
||||
until-1-03 ::= | [<] until-1-01 | [S] until-1-04 | [^<S] until-1
|
||||
until-1-04 ::= | [<] until-1-01 | [M] until-1-05 | [^<M] until-1
|
||||
until-1-05 ::= | [<] until-1-01 | [L] until-1-06 | [^<L] until-1
|
||||
until-1-06 ::= | [<] until-1-01 | [^<\uFF5C] until-1
|
||||
until-0 ::= | [<] until-0-01 | [^<] until-0
|
||||
until-0-01 ::= | [<] until-0-01 | [\uFF5C] until-0-02 | [^<\uFF5C] until-0
|
||||
until-0-02 ::= | [<] until-0-01 | [D] until-0-03 | [^<D] until-0
|
||||
until-0-03 ::= | [<] until-0-01 | [S] until-0-04 | [^<S] until-0
|
||||
until-0-04 ::= | [<] until-0-01 | [M] until-0-05 | [^<M] until-0
|
||||
until-0-05 ::= | [<] until-0-01 | [L] until-0-06 | [^<L] until-0
|
||||
until-0-06 ::= | [<] until-0-01 | [^<\uFF5C] until-0
|
||||
)""", gbnf);
|
||||
});
|
||||
|
||||
@@ -203,18 +203,18 @@ void test_gbnf_generation(testing &t) {
|
||||
});
|
||||
|
||||
assert_gbnf_equal(t, R"""(
|
||||
root ::= until-3
|
||||
root ::= until-0
|
||||
space ::= | " " | "\n"{1,2} [ \t]{0,20}
|
||||
until-3 ::= | [a] until-3-01 | [c] until-3-03 | [e] until-3-05 | [^ace] until-3
|
||||
until-3-01 ::= | [a] until-3-01 | [c] until-3-03 | [e] until-3-05 | [^abce] until-3
|
||||
until-3-03 ::= | [a] until-3-01 | [c] until-3-03 | [e] until-3-05 | [^acde] until-3
|
||||
until-3-05 ::= | [a] until-3-01 | [c] until-3-03 | [e] until-3-05 | [^acef] until-3
|
||||
until-0 ::= | [a] until-0-01 | [c] until-0-03 | [e] until-0-05 | [^ace] until-0
|
||||
until-0-01 ::= | [a] until-0-01 | [c] until-0-03 | [e] until-0-05 | [^abce] until-0
|
||||
until-0-03 ::= | [a] until-0-01 | [c] until-0-03 | [e] until-0-05 | [^acde] until-0
|
||||
until-0-05 ::= | [a] until-0-01 | [c] until-0-03 | [e] until-0-05 | [^acef] until-0
|
||||
)""", gbnf);
|
||||
});
|
||||
|
||||
t.test("through grammar", [](testing &t) {
|
||||
t.test("ac grammar", [](testing &t) {
|
||||
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
|
||||
return p.through(p.until("</tag>"), "</tag>");
|
||||
return p.ac(p.until("</tag>") + p.literal("</tag>"), "</tag>");
|
||||
});
|
||||
|
||||
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
|
||||
@@ -222,20 +222,20 @@ void test_gbnf_generation(testing &t) {
|
||||
});
|
||||
|
||||
assert_gbnf_equal(t, R"""(
|
||||
root ::= until-3
|
||||
ac-3 ::= [<] ac-3-01 | [^<] ac-3
|
||||
ac-3-01 ::= [<] ac-3-01 | [/] ac-3-02 | [^/<] ac-3
|
||||
ac-3-02 ::= [<] ac-3-01 | [t] ac-3-03 | [^<t] ac-3
|
||||
ac-3-03 ::= [<] ac-3-01 | [a] ac-3-04 | [^<a] ac-3
|
||||
ac-3-04 ::= [<] ac-3-01 | [g] ac-3-05 | [^<g] ac-3
|
||||
ac-3-05 ::= [>] | [<] ac-3-01 | [^<>] ac-3
|
||||
root ::= ac-3
|
||||
space ::= | " " | "\n"{1,2} [ \t]{0,20}
|
||||
until-3 ::= [<] until-3-01 | [^<] until-3
|
||||
until-3-01 ::= [<] until-3-01 | [/] until-3-02 | [^/<] until-3
|
||||
until-3-02 ::= [<] until-3-01 | [t] until-3-03 | [^<t] until-3
|
||||
until-3-03 ::= [<] until-3-01 | [a] until-3-04 | [^<a] until-3
|
||||
until-3-04 ::= [<] until-3-01 | [g] until-3-05 | [^<g] until-3
|
||||
until-3-05 ::= [>] | [<] until-3-01 | [^<>] until-3
|
||||
)""", gbnf);
|
||||
});
|
||||
|
||||
t.test("through grammar terminates at first delimiter", [](testing &t) {
|
||||
t.test("ac grammar terminates at first delimiter", [](testing &t) {
|
||||
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
|
||||
return p.through(p.until("\n</parameter>\n"), "\n</parameter>\n");
|
||||
return p.ac(p.until("\n</parameter>\n") + p.literal("\n</parameter>\n"), "\n</parameter>\n");
|
||||
});
|
||||
|
||||
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
|
||||
@@ -243,28 +243,28 @@ void test_gbnf_generation(testing &t) {
|
||||
});
|
||||
|
||||
assert_gbnf_equal(t, R"""(
|
||||
root ::= until-3
|
||||
ac-3 ::= [\n] ac-3-01 | [^\n] ac-3
|
||||
ac-3-01 ::= [\n] ac-3-01 | [<] ac-3-02 | [^\n<] ac-3
|
||||
ac-3-02 ::= [\n] ac-3-01 | [/] ac-3-03 | [^\n/] ac-3
|
||||
ac-3-03 ::= [\n] ac-3-01 | [p] ac-3-04 | [^\np] ac-3
|
||||
ac-3-04 ::= [\n] ac-3-01 | [a] ac-3-05 | [^\na] ac-3
|
||||
ac-3-05 ::= [\n] ac-3-01 | [r] ac-3-06 | [^\nr] ac-3
|
||||
ac-3-06 ::= [\n] ac-3-01 | [a] ac-3-07 | [^\na] ac-3
|
||||
ac-3-07 ::= [\n] ac-3-01 | [m] ac-3-08 | [^\nm] ac-3
|
||||
ac-3-08 ::= [\n] ac-3-01 | [e] ac-3-09 | [^\ne] ac-3
|
||||
ac-3-09 ::= [\n] ac-3-01 | [t] ac-3-10 | [^\nt] ac-3
|
||||
ac-3-10 ::= [\n] ac-3-01 | [e] ac-3-11 | [^\ne] ac-3
|
||||
ac-3-11 ::= [\n] ac-3-01 | [r] ac-3-12 | [^\nr] ac-3
|
||||
ac-3-12 ::= [\n] ac-3-01 | [>] ac-3-13 | [^\n>] ac-3
|
||||
ac-3-13 ::= [\n] | [^\n] ac-3
|
||||
root ::= ac-3
|
||||
space ::= | " " | "\n"{1,2} [ \t]{0,20}
|
||||
until-3 ::= [\n] until-3-01 | [^\n] until-3
|
||||
until-3-01 ::= [\n] until-3-01 | [<] until-3-02 | [^\n<] until-3
|
||||
until-3-02 ::= [\n] until-3-01 | [/] until-3-03 | [^\n/] until-3
|
||||
until-3-03 ::= [\n] until-3-01 | [p] until-3-04 | [^\np] until-3
|
||||
until-3-04 ::= [\n] until-3-01 | [a] until-3-05 | [^\na] until-3
|
||||
until-3-05 ::= [\n] until-3-01 | [r] until-3-06 | [^\nr] until-3
|
||||
until-3-06 ::= [\n] until-3-01 | [a] until-3-07 | [^\na] until-3
|
||||
until-3-07 ::= [\n] until-3-01 | [m] until-3-08 | [^\nm] until-3
|
||||
until-3-08 ::= [\n] until-3-01 | [e] until-3-09 | [^\ne] until-3
|
||||
until-3-09 ::= [\n] until-3-01 | [t] until-3-10 | [^\nt] until-3
|
||||
until-3-10 ::= [\n] until-3-01 | [e] until-3-11 | [^\ne] until-3
|
||||
until-3-11 ::= [\n] until-3-01 | [r] until-3-12 | [^\nr] until-3
|
||||
until-3-12 ::= [\n] until-3-01 | [>] until-3-13 | [^\n>] until-3
|
||||
until-3-13 ::= [\n] | [^\n] until-3
|
||||
)""", gbnf);
|
||||
});
|
||||
|
||||
t.test("through grammar multiple delimiters", [](testing &t) {
|
||||
t.test("ac grammar multiple delimiters", [](testing &t) {
|
||||
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
|
||||
return p.through(p.eps(), { { p.literal("ab") }, { p.literal("cd") }, { p.literal("ef") } });
|
||||
return p.ac(p.eps(), std::vector<std::string>{"ab", "cd", "ef"});
|
||||
});
|
||||
|
||||
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
|
||||
@@ -272,238 +272,12 @@ void test_gbnf_generation(testing &t) {
|
||||
});
|
||||
|
||||
assert_gbnf_equal(t, R"""(
|
||||
root ::= until-4
|
||||
ac-1 ::= [a] ac-1-01 | [c] ac-1-03 | [e] ac-1-05 | [^ace] ac-1
|
||||
ac-1-01 ::= [b] | [a] ac-1-01 | [c] ac-1-03 | [e] ac-1-05 | [^abce] ac-1
|
||||
ac-1-03 ::= [d] | [a] ac-1-01 | [c] ac-1-03 | [e] ac-1-05 | [^acde] ac-1
|
||||
ac-1-05 ::= [f] | [a] ac-1-01 | [c] ac-1-03 | [e] ac-1-05 | [^acef] ac-1
|
||||
root ::= ac-1
|
||||
space ::= | " " | "\n"{1,2} [ \t]{0,20}
|
||||
until-4 ::= [a] until-4-01 | [c] until-4-03 | [e] until-4-05 | [^ace] until-4
|
||||
until-4-01 ::= [b] | [a] until-4-01 | [c] until-4-03 | [e] until-4-05 | [^abce] until-4
|
||||
until-4-03 ::= [d] | [a] until-4-01 | [c] until-4-03 | [e] until-4-05 | [^acde] until-4
|
||||
until-4-05 ::= [f] | [a] until-4-01 | [c] until-4-03 | [e] until-4-05 | [^acef] until-4
|
||||
)""", gbnf);
|
||||
});
|
||||
|
||||
t.test("until grammar with parser delimiters", [](testing &t) {
|
||||
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
|
||||
return p.until((p.literal("a") + p.literal("b")) | p.literal("cd"));
|
||||
});
|
||||
|
||||
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
|
||||
parser.build_grammar(builder);
|
||||
});
|
||||
|
||||
assert_gbnf_equal(t, R"""(
|
||||
root ::= until-5
|
||||
space ::= | " " | "\n"{1,2} [ \t]{0,20}
|
||||
until-5 ::= | [a] until-5-01 | [c] until-5-03 | [^ac] until-5
|
||||
until-5-01 ::= | [a] until-5-01 | [c] until-5-03 | [^abc] until-5
|
||||
until-5-03 ::= | [a] until-5-01 | [c] until-5-03 | [^acd] until-5
|
||||
)""", gbnf);
|
||||
});
|
||||
|
||||
t.test("delimiters see through tags and atomics", [](testing &t) {
|
||||
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
|
||||
return p.until(p.tag("open", p.literal("a") + p.atomic(p.literal("b"))));
|
||||
});
|
||||
|
||||
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
|
||||
parser.build_grammar(builder);
|
||||
});
|
||||
|
||||
assert_gbnf_equal(t, R"""(
|
||||
root ::= until-5
|
||||
space ::= | " " | "\n"{1,2} [ \t]{0,20}
|
||||
until-5 ::= | [a] until-5-01 | [^a] until-5
|
||||
until-5-01 ::= | [a] until-5-01 | [^ab] until-5
|
||||
)""", gbnf);
|
||||
});
|
||||
|
||||
t.test("delimiters expand a choice in a sequence", [](testing &t) {
|
||||
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
|
||||
return p.until(p.literal("a") + (p.literal("b") | p.literal("d")));
|
||||
});
|
||||
|
||||
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
|
||||
parser.build_grammar(builder);
|
||||
});
|
||||
|
||||
assert_gbnf_equal(t, R"""(
|
||||
root ::= until-5
|
||||
space ::= | " " | "\n"{1,2} [ \t]{0,20}
|
||||
until-5 ::= | [a] until-5-01 | [^a] until-5
|
||||
until-5-01 ::= | [a] until-5-01 | [^abd] until-5
|
||||
)""", gbnf);
|
||||
});
|
||||
|
||||
t.test("delimiters reject anything but sequences, choices, literals, and tokens", [](testing &t) {
|
||||
bool threw = false;
|
||||
try {
|
||||
build_peg_parser([](common_peg_parser_builder & p) {
|
||||
return p.until(p.literal("a") + p.space());
|
||||
});
|
||||
} catch (const std::invalid_argument &) {
|
||||
threw = true;
|
||||
}
|
||||
t.assert_equal("space_rejected", true, threw);
|
||||
});
|
||||
|
||||
t.test("through branches reject a shared delimiter", [](testing &t) {
|
||||
bool threw = false;
|
||||
try {
|
||||
build_peg_parser([](common_peg_parser_builder & p) {
|
||||
return p.through(p.eps(), { { p.literal("ab"), p.literal("x") }, { p.literal("ab"), p.literal("y") } });
|
||||
});
|
||||
} catch (const std::invalid_argument &) {
|
||||
threw = true;
|
||||
}
|
||||
t.assert_equal("duplicate_rejected", true, threw);
|
||||
});
|
||||
|
||||
t.test("through grammar branches continue with their rest", [](testing &t) {
|
||||
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
|
||||
return p.through(p.eps(), { { p.literal("ab"), p.literal("x") }, { p.literal("cd") } });
|
||||
});
|
||||
|
||||
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
|
||||
parser.build_grammar(builder);
|
||||
});
|
||||
|
||||
assert_gbnf_equal(t, R"""(
|
||||
root ::= until-4
|
||||
space ::= | " " | "\n"{1,2} [ \t]{0,20}
|
||||
until-4 ::= [a] until-4-01 | [c] until-4-03 | [^ac] until-4
|
||||
until-4-01 ::= [b] until-4-rest-0 | [a] until-4-01 | [c] until-4-03 | [^abc] until-4
|
||||
until-4-03 ::= [d] | [a] until-4-01 | [c] until-4-03 | [^acd] until-4
|
||||
until-4-rest-0 ::= "x"
|
||||
)""", gbnf);
|
||||
});
|
||||
|
||||
t.test("through grammar optional may end before a delimiter", [](testing &t) {
|
||||
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
|
||||
return p.through(p.eps(), { { p.literal("ab"), p.literal("x") } }, true);
|
||||
});
|
||||
|
||||
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
|
||||
parser.build_grammar(builder);
|
||||
});
|
||||
|
||||
assert_gbnf_equal(t, R"""(
|
||||
root ::= until-3
|
||||
space ::= | " " | "\n"{1,2} [ \t]{0,20}
|
||||
until-3 ::= | [a] until-3-01 | [^a] until-3
|
||||
until-3-01 ::= | [b] until-3-rest-0 | [a] until-3-01 | [^ab] until-3
|
||||
until-3-rest-0 ::= "x"
|
||||
)""", gbnf);
|
||||
});
|
||||
|
||||
t.test("trigger rules emit their start and rest when not lazy", [](testing &t) {
|
||||
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
|
||||
auto ab = p.literal("ab");
|
||||
auto cd = p.literal("cd");
|
||||
auto scan = p.until(ab | cd);
|
||||
return scan + (p.trigger_rule("one", ab, p.literal("x")) | p.trigger_rule("two", cd, p.literal("y")));
|
||||
});
|
||||
|
||||
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
|
||||
parser.build_grammar(builder, false);
|
||||
});
|
||||
|
||||
assert_gbnf_equal(t, R"""(
|
||||
one ::= "ab" "x"
|
||||
root ::= until-3 (one | two)
|
||||
space ::= | " " | "\n"{1,2} [ \t]{0,20}
|
||||
two ::= "cd" "y"
|
||||
until-3 ::= | [a] until-3-01 | [c] until-3-03 | [^ac] until-3
|
||||
until-3-01 ::= | [a] until-3-01 | [c] until-3-03 | [^abc] until-3
|
||||
until-3-03 ::= | [a] until-3-01 | [c] until-3-03 | [^acd] until-3
|
||||
)""", gbnf);
|
||||
});
|
||||
|
||||
t.test("trigger rules make the lazy grammar scan for their starts", [](testing &t) {
|
||||
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
|
||||
auto ab = p.literal("ab");
|
||||
auto cd = p.literal("cd");
|
||||
auto scan = p.until(ab | cd);
|
||||
return scan + (p.trigger_rule("one", ab, p.literal("x")) | p.trigger_rule("two", cd, p.literal("y")));
|
||||
});
|
||||
|
||||
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
|
||||
parser.build_grammar(builder, true);
|
||||
});
|
||||
|
||||
assert_gbnf_equal(t, R"""(
|
||||
one-rest ::= "x"
|
||||
root ::= trigger
|
||||
space ::= | " " | "\n"{1,2} [ \t]{0,20}
|
||||
trigger ::= | [a] trigger-01 | [c] trigger-03 | [^ac] trigger
|
||||
trigger-01 ::= | [b] one-rest | [a] trigger-01 | [c] trigger-03 | [^abc] trigger
|
||||
trigger-03 ::= | [d] two-rest | [a] trigger-01 | [c] trigger-03 | [^acd] trigger
|
||||
two-rest ::= "y"
|
||||
)""", gbnf);
|
||||
});
|
||||
|
||||
t.test("scanning grammar lets other named tokens through", [](testing &t) {
|
||||
common_peg_tokens tokens;
|
||||
tokens.ids.emplace("<a>", 1);
|
||||
tokens.ids.emplace("<b>", 2);
|
||||
tokens.tokens.emplace(1, common_peg_token{ "<a>", LLAMA_TOKEN_ATTR_CONTROL });
|
||||
tokens.tokens.emplace(2, common_peg_token{ "<b>", LLAMA_TOKEN_ATTR_CONTROL });
|
||||
|
||||
common_peg_parser_builder p(tokens);
|
||||
auto start = p.token("<a>") + p.literal("x");
|
||||
p.set_root(p.until(start) + p.trigger_rule("tool", start, p.token("<b>")));
|
||||
auto parser = p.build();
|
||||
|
||||
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
|
||||
parser.build_grammar(builder, true);
|
||||
});
|
||||
|
||||
assert_gbnf_equal(t, R"""(
|
||||
root ::= trigger
|
||||
space ::= | " " | "\n"{1,2} [ \t]{0,20}
|
||||
tool-rest ::= <[2]>
|
||||
trigger ::= | <[1]> trigger-01 | (. | <[2]>) trigger
|
||||
trigger-01 ::= | [x] tool-rest | <[1]> trigger-01 | ([^x] | <[2]>) trigger
|
||||
)""", gbnf);
|
||||
});
|
||||
|
||||
t.test("token sets", [](testing &t) {
|
||||
common_peg_tokens tokens;
|
||||
tokens.ids.emplace("<a>", 1);
|
||||
tokens.ids.emplace("<b>", 2);
|
||||
tokens.tokens.emplace(1, common_peg_token{ "<a>", LLAMA_TOKEN_ATTR_CONTROL });
|
||||
tokens.tokens.emplace(2, common_peg_token{ "<b>", LLAMA_TOKEN_ATTR_CONTROL });
|
||||
|
||||
common_peg_parser_builder p(tokens);
|
||||
p.set_root(p.token({ "<a>", "<b>" }) + p.token({ "<a>", "<b>" }, true));
|
||||
auto parser = p.build();
|
||||
|
||||
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
|
||||
parser.build_grammar(builder);
|
||||
});
|
||||
|
||||
assert_gbnf_equal(t, R"""(
|
||||
root ::= (<[1]> | <[2]>) !<[1-2]>
|
||||
space ::= | " " | "\n"{1,2} [ \t]{0,20}
|
||||
)""", gbnf);
|
||||
});
|
||||
|
||||
t.test("through grammar suffix delimiter offers both rests", [](testing &t) {
|
||||
auto parser = build_peg_parser([](common_peg_parser_builder & p) {
|
||||
return p.through(p.eps(), { { p.literal("xab"), p.literal("1") }, { p.literal("ab"), p.literal("2") } });
|
||||
});
|
||||
|
||||
auto gbnf = build_grammar([&](const common_grammar_builder & builder) {
|
||||
parser.build_grammar(builder);
|
||||
});
|
||||
|
||||
assert_gbnf_equal(t, R"""(
|
||||
root ::= until-5
|
||||
space ::= | " " | "\n"{1,2} [ \t]{0,20}
|
||||
until-5 ::= [x] until-5-01 | [a] until-5-04 | [^ax] until-5
|
||||
until-5-01 ::= [x] until-5-01 | [a] until-5-02 | [^ax] until-5
|
||||
until-5-02 ::= [b] (until-5-rest-0 | until-5-rest-1) | [x] until-5-01 | [a] until-5-04 | [^abx] until-5
|
||||
until-5-04 ::= [b] until-5-rest-1 | [x] until-5-01 | [a] until-5-04 | [^abx] until-5
|
||||
until-5-rest-0 ::= "1"
|
||||
until-5-rest-1 ::= "2"
|
||||
)""", gbnf);
|
||||
});
|
||||
|
||||
|
||||
@@ -2176,7 +2176,7 @@ struct test_unary : public test_case {
|
||||
ggml_tensor * build_graph(ggml_context * ctx) override {
|
||||
const bool grad_supported = op == GGML_UNARY_OP_ABS || op == GGML_UNARY_OP_SGN || op == GGML_UNARY_OP_NEG ||
|
||||
op == GGML_UNARY_OP_STEP || op == GGML_UNARY_OP_RELU || op == GGML_UNARY_OP_SILU ||
|
||||
op == GGML_UNARY_OP_EXPM1 || op == GGML_UNARY_OP_SOFTPLUS || op == GGML_UNARY_OP_SIGMOID;
|
||||
op == GGML_UNARY_OP_EXPM1 || op == GGML_UNARY_OP_SOFTPLUS;
|
||||
|
||||
ggml_tensor * a;
|
||||
if (v & 1) {
|
||||
@@ -2218,12 +2218,6 @@ struct test_unary : public test_case {
|
||||
max = 10.f;
|
||||
}
|
||||
|
||||
// saturated sigmoid inputs give zero analytic and numeric gradients, which the grad check cannot compare
|
||||
if (mode == MODE_GRAD && op == GGML_UNARY_OP_SIGMOID) {
|
||||
min = -2.f;
|
||||
max = 2.f;
|
||||
}
|
||||
|
||||
for (ggml_tensor * t = ggml_get_first_tensor(ctx); t != NULL; t = ggml_get_next_tensor(ctx, t)) {
|
||||
// test extended range of values to check for NaNs in GELU
|
||||
init_tensor_uniform(t, min, max);
|
||||
@@ -2231,16 +2225,7 @@ struct test_unary : public test_case {
|
||||
}
|
||||
|
||||
float grad_eps() override {
|
||||
return op == GGML_UNARY_OP_SIGMOID ? 0.25f : 15.0f;
|
||||
}
|
||||
|
||||
bool grad_precise() override {
|
||||
return op == GGML_UNARY_OP_SIGMOID;
|
||||
}
|
||||
|
||||
// f32 rounding of a large summed loss exceeds the tolerance, so only the small sigmoid case is grad checked
|
||||
int64_t grad_nmax() override {
|
||||
return op == GGML_UNARY_OP_SIGMOID ? 64 : test_case::grad_nmax();
|
||||
return 15.0f;
|
||||
}
|
||||
|
||||
std::vector<float> grad_expect() override {
|
||||
@@ -3913,37 +3898,6 @@ struct test_silu_back : public test_case {
|
||||
}
|
||||
};
|
||||
|
||||
// GGML_OP_SIGMOID_BACK
|
||||
struct test_sigmoid_back : public test_case {
|
||||
const ggml_type type;
|
||||
const std::array<int64_t, 4> ne;
|
||||
|
||||
std::string vars() override {
|
||||
return VARS_TO_STR2(type, ne);
|
||||
}
|
||||
|
||||
test_sigmoid_back(ggml_type type = GGML_TYPE_F32,
|
||||
std::array<int64_t, 4> ne = {64, 5, 4, 3})
|
||||
: type(type), ne(ne) {}
|
||||
|
||||
ggml_tensor * build_graph(ggml_context * ctx) override {
|
||||
ggml_tensor * a = ggml_new_tensor(ctx, type, 4, ne.data());
|
||||
ggml_set_name(a, "a");
|
||||
|
||||
ggml_tensor * grad = ggml_new_tensor(ctx, type, 4, ne.data());
|
||||
ggml_set_name(grad, "grad");
|
||||
|
||||
ggml_tensor * out = ggml_sigmoid_back(ctx, a, grad);
|
||||
ggml_set_name(out, "out");
|
||||
|
||||
return out;
|
||||
}
|
||||
|
||||
bool grad_precise() override {
|
||||
return true;
|
||||
}
|
||||
};
|
||||
|
||||
// GGML_OP_NORM
|
||||
struct test_norm : public test_case {
|
||||
const ggml_type type;
|
||||
@@ -10827,7 +10781,6 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
|
||||
}
|
||||
}
|
||||
}
|
||||
test_cases.emplace_back(new test_unary(GGML_UNARY_OP_SIGMOID, GGML_TYPE_F32, { 16, 4, 1, 1 }));
|
||||
|
||||
// fused relu + sqr (squared ReLU)
|
||||
for (ggml_type type : {GGML_TYPE_F16, GGML_TYPE_F32}) {
|
||||
@@ -11808,7 +11761,6 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {
|
||||
test_cases.emplace_back(new test_scale(GGML_TYPE_F16, {10, 10, 10, 10}, 2.0f, 1.0f, false, 1)); // view test
|
||||
test_cases.emplace_back(new test_softcap(GGML_TYPE_F32, {10, 10, 10, 10}, 50.0f));
|
||||
test_cases.emplace_back(new test_silu_back());
|
||||
test_cases.emplace_back(new test_sigmoid_back());
|
||||
|
||||
for (float eps : { 0.0f, 1e-6f, 1e-4f, 1e-1f, 10.f }) {
|
||||
for (uint32_t n : { 64, 1025 }) {
|
||||
|
||||
@@ -170,25 +170,6 @@ static bool test_tensor_n(ggml_backend_buffer_type_t buft) {
|
||||
}
|
||||
}
|
||||
|
||||
{
|
||||
uint8_t expected[24], out[24] = {};
|
||||
memset(expected, 0x11, sizeof(expected));
|
||||
ggml_backend_tensor_set(t_big, expected, 0, sizeof(expected));
|
||||
|
||||
// Verify memset of unaligned ranges, such as single F16 elements.
|
||||
const size_t ranges[][2] = { { 2, 2 }, { 1, 1 }, { 5, 2 }, { 9, 13 }, { 22, 2 } };
|
||||
for (const auto & range : ranges) {
|
||||
const uint8_t value = (uint8_t) (0xA0 + range[0]);
|
||||
memset(expected + range[0], value, range[1]);
|
||||
ggml_backend_tensor_memset(t_big, value, range[0], range[1]);
|
||||
}
|
||||
|
||||
ggml_backend_tensor_get(t_big, out, 0, sizeof(out));
|
||||
if (memcmp(out, expected, sizeof(out)) != 0) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
{
|
||||
std::vector<ggml_tensor *> batch;
|
||||
ggml_backend_buffer_ptr scatter_buf = alloc_tensors(ctx.get(), buft, { 64 }, batch);
|
||||
@@ -223,19 +204,6 @@ static bool test_tensor_n(ggml_backend_buffer_type_t buft) {
|
||||
}
|
||||
}
|
||||
|
||||
{
|
||||
uint8_t expected[24], out[24] = {};
|
||||
memset(expected, 0x5A, sizeof(expected));
|
||||
|
||||
// Verify that clear fills every byte with a nonzero value.
|
||||
ggml_backend_buffer_clear(buf.get(), 0x5A);
|
||||
|
||||
ggml_backend_tensor_get(t_big, out, 0, sizeof(out));
|
||||
if (memcmp(out, expected, sizeof(out)) != 0) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
|
||||
+50
-251
@@ -305,66 +305,6 @@ static std::string get_expected_description(const llama_grammar_rules & rules, c
|
||||
return result;
|
||||
}
|
||||
|
||||
// Every text token shares one ID, the parsers and grammars only tell special tokens apart
|
||||
static constexpr llama_token FAKE_TEXT_TOKEN = 0;
|
||||
|
||||
struct fake_token {
|
||||
llama_token id;
|
||||
std::string piece;
|
||||
};
|
||||
|
||||
// Splits text into the special tokens it spells, longest first, and one text token per UTF-8 character.
|
||||
static std::vector<fake_token> fake_tokenize(const common_peg_tokens & specials, const std::string & text) {
|
||||
std::vector<fake_token> result;
|
||||
size_t pos = 0;
|
||||
while (pos < text.size()) {
|
||||
llama_token best = LLAMA_TOKEN_NULL;
|
||||
size_t best_len = 0;
|
||||
for (const auto & [piece, id] : specials.ids) {
|
||||
if (piece.size() > best_len && text.compare(pos, piece.size(), piece) == 0) {
|
||||
best = id;
|
||||
best_len = piece.size();
|
||||
}
|
||||
}
|
||||
if (best != LLAMA_TOKEN_NULL) {
|
||||
result.push_back({ best, text.substr(pos, best_len) });
|
||||
pos += best_len;
|
||||
continue;
|
||||
}
|
||||
size_t len = 1;
|
||||
while (pos + len < text.size() && (static_cast<unsigned char>(text[pos + len]) & 0xC0) == 0x80) {
|
||||
len++;
|
||||
}
|
||||
result.push_back({ FAKE_TEXT_TOKEN, text.substr(pos, len) });
|
||||
pos += len;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
// Returns the tokens from byte offset pos. A token split by pos keeps the rest of its piece as text.
|
||||
static std::vector<fake_token> fake_tokens_from(const std::vector<fake_token> & tokens, size_t pos) {
|
||||
std::vector<fake_token> result;
|
||||
size_t start = 0;
|
||||
for (const auto & token : tokens) {
|
||||
const size_t end = start + token.piece.size();
|
||||
if (start >= pos) {
|
||||
result.push_back(token);
|
||||
} else if (end > pos) {
|
||||
result.push_back({ FAKE_TEXT_TOKEN, token.piece.substr(pos - start) });
|
||||
}
|
||||
start = end;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
static common_chat_input fake_input(const std::vector<fake_token> & tokens) {
|
||||
common_chat_input input;
|
||||
for (const auto & token : tokens) {
|
||||
input.append(token.piece, token.id);
|
||||
}
|
||||
return input;
|
||||
}
|
||||
|
||||
// Result of a detailed grammar match attempt
|
||||
struct grammar_match_result {
|
||||
bool success = false; // Did the string fully match the grammar?
|
||||
@@ -378,49 +318,56 @@ struct grammar_match_result {
|
||||
bool incomplete = false; // True if matched all input but grammar expects more
|
||||
};
|
||||
|
||||
// Detailed version of match_string that returns failure information, accepting the input token by token
|
||||
static grammar_match_result match_string_detailed(const std::vector<fake_token> & tokens, llama_grammar * grammar) {
|
||||
std::string input;
|
||||
for (const auto & token : tokens) {
|
||||
input += token.piece;
|
||||
}
|
||||
|
||||
// Detailed version of match_string that returns failure information
|
||||
static grammar_match_result match_string_detailed(const std::string & input, llama_grammar * grammar) {
|
||||
grammar_match_result result;
|
||||
result.total_bytes = input.size();
|
||||
result.total_codepoints = unicode_cpts_from_utf8(input).size();
|
||||
result.total_bytes = input.size();
|
||||
|
||||
const auto cpts = unicode_cpts_from_utf8(input);
|
||||
result.total_codepoints = cpts.size();
|
||||
|
||||
auto & stacks_cur = llama_grammar_get_stacks(grammar);
|
||||
const auto & rules = llama_grammar_get_rules(grammar);
|
||||
|
||||
size_t byte_pos = 0;
|
||||
size_t cpt_pos = 0;
|
||||
|
||||
for (const auto & token : tokens) {
|
||||
for (size_t i = 0; i < cpts.size(); i++) {
|
||||
const auto & cpt = cpts[i];
|
||||
|
||||
// Get expected before accepting (for error reporting)
|
||||
std::string expected_before = get_expected_description(rules, stacks_cur);
|
||||
|
||||
try {
|
||||
llama_grammar_accept_token(*grammar, token.id, token.piece);
|
||||
} catch (const std::runtime_error & /*e*/) {
|
||||
llama_grammar_accept(grammar, cpt);
|
||||
|
||||
// Calculate byte position for this codepoint
|
||||
size_t cpt_bytes = 0;
|
||||
if (cpt < 0x80) {
|
||||
cpt_bytes = 1;
|
||||
} else if (cpt < 0x800) {
|
||||
cpt_bytes = 2;
|
||||
} else if (cpt < 0x10000) {
|
||||
cpt_bytes = 3;
|
||||
} else {
|
||||
cpt_bytes = 4;
|
||||
}
|
||||
|
||||
if (stacks_cur.empty()) {
|
||||
// Grammar failed to match at this point
|
||||
result.matched_bytes = byte_pos;
|
||||
result.matched_codepoints = cpt_pos;
|
||||
result.matched_codepoints = i;
|
||||
result.matched_prefix = input.substr(0, byte_pos);
|
||||
result.failing_char = token.id == FAKE_TEXT_TOKEN
|
||||
? format_codepoint(unicode_cpts_from_utf8(token.piece).at(0))
|
||||
: token.piece + " (token " + std::to_string(token.id) + ")";
|
||||
result.failing_char = format_codepoint(cpt);
|
||||
result.expected_description = expected_before;
|
||||
result.incomplete = false;
|
||||
return result;
|
||||
}
|
||||
|
||||
byte_pos += token.piece.size();
|
||||
cpt_pos += unicode_cpts_from_utf8(token.piece).size();
|
||||
byte_pos += cpt_bytes;
|
||||
}
|
||||
|
||||
// All input matched - check if grammar is complete
|
||||
result.matched_bytes = input.size();
|
||||
result.matched_codepoints = result.total_codepoints;
|
||||
result.matched_codepoints = cpts.size();
|
||||
result.matched_prefix = input;
|
||||
|
||||
if (std::any_of(stacks_cur.begin(), stacks_cur.end(), [](const auto & stack) { return stack.empty(); })) {
|
||||
@@ -1171,7 +1118,6 @@ struct peg_test_case {
|
||||
common_chat_msg expect;
|
||||
bool is_partial = false;
|
||||
bool expect_reconstruction = false;
|
||||
bool expect_grammar = false;
|
||||
std::vector<std::string> expect_rules;
|
||||
};
|
||||
|
||||
@@ -1182,7 +1128,6 @@ static std::string g_template_filter;
|
||||
static bool g_force_reconstruction_test = false;
|
||||
|
||||
static void test_peg_parser(common_chat_templates * tmpls,
|
||||
const common_peg_tokens & specials,
|
||||
const std::function<void(peg_test_case &)> & init,
|
||||
bool detailed_debug) {
|
||||
// UTF-8-safe truncation helper (same as in test_parser_with_streaming)
|
||||
@@ -1227,15 +1172,8 @@ static void test_peg_parser(common_chat_templates * tmpls,
|
||||
tc.expect.role = "assistant";
|
||||
}
|
||||
|
||||
const bool tokenized = !specials.ids.empty();
|
||||
|
||||
common_chat_session_params session_params;
|
||||
session_params.debug = detailed_debug;
|
||||
if (tokenized) {
|
||||
session_params.tokenize = [&](const std::string & text) {
|
||||
return fake_input(fake_tokenize(specials, text));
|
||||
};
|
||||
}
|
||||
common_chat_session session(tmpls, nullptr, tc.params, session_params);
|
||||
const auto & parser = session.parser();
|
||||
common_params_sampling sampling;
|
||||
@@ -1253,35 +1191,18 @@ static void test_peg_parser(common_chat_templates * tmpls,
|
||||
}
|
||||
}
|
||||
|
||||
if (tc.expect_grammar && session.grammar().empty()) {
|
||||
throw std::runtime_error("expected a grammar constraining the tool calls, but none was produced");
|
||||
}
|
||||
|
||||
common_chat_msg msg_accum;
|
||||
common_chat_msg msg_prev;
|
||||
msg_accum.role = msg_prev.role = "assistant";
|
||||
|
||||
const auto input_tokens = fake_tokenize(specials, tc.input);
|
||||
|
||||
// A token arrives whole, so a special token is never split across chunks
|
||||
std::vector<common_chat_input> chunks;
|
||||
if (tokenized) {
|
||||
for (const auto & token : input_tokens) {
|
||||
chunks.push_back(fake_input({ token }));
|
||||
}
|
||||
} else {
|
||||
size_t fed = 0;
|
||||
for (size_t i = 1; i <= tc.input.size(); ++i) {
|
||||
// Use UTF-8 safe truncation to avoid corrupting multi-byte characters
|
||||
size_t safe_len = utf8_truncate_safe_len(std::string_view(tc.input).substr(0, i));
|
||||
chunks.emplace_back(tc.input.substr(fed, safe_len - fed));
|
||||
fed = safe_len;
|
||||
}
|
||||
}
|
||||
|
||||
for (size_t i = 0; i < chunks.size(); ++i) {
|
||||
auto is_partial = i + 1 < chunks.size() || tc.is_partial;
|
||||
const common_chat_msg & msg_current = is_partial ? session.feed(chunks[i]) : session.finish(chunks[i]);
|
||||
size_t fed = 0;
|
||||
for (size_t i = 1; i <= tc.input.size(); ++i) {
|
||||
auto is_partial = i < tc.input.size() || tc.is_partial;
|
||||
// Use UTF-8 safe truncation to avoid corrupting multi-byte characters
|
||||
size_t safe_len = utf8_truncate_safe_len(std::string_view(tc.input).substr(0, i));
|
||||
common_chat_input chunk(tc.input.substr(fed, safe_len - fed));
|
||||
fed = safe_len;
|
||||
const common_chat_msg & msg_current = is_partial ? session.feed(chunk) : session.finish(chunk);
|
||||
|
||||
for (const auto & diff : common_chat_msg_diff::compute_diffs(msg_prev, msg_current)) {
|
||||
if (!diff.reasoning_content_delta.empty()) {
|
||||
@@ -1296,15 +1217,10 @@ static void test_peg_parser(common_chat_templates * tmpls,
|
||||
while (msg_accum.tool_calls.size() <= diff.tool_call_index) {
|
||||
msg_accum.tool_calls.push_back({ "", "", "" });
|
||||
}
|
||||
// The name and id are set once the delta carries them.
|
||||
// A name that is already set must not change, clients concatenate name deltas.
|
||||
// Always update name and id from diff (may change during incremental parsing), but only if the delta
|
||||
// actually contains them
|
||||
if (!diff.tool_call_delta.name.empty()) {
|
||||
auto & name = msg_accum.tool_calls[diff.tool_call_index].name;
|
||||
if (!name.empty() && name != diff.tool_call_delta.name) {
|
||||
throw std::runtime_error("Tool call " + std::to_string(diff.tool_call_index) + " name changed from " +
|
||||
name + " to " + diff.tool_call_delta.name + " during incremental parsing");
|
||||
}
|
||||
name = diff.tool_call_delta.name;
|
||||
msg_accum.tool_calls[diff.tool_call_index].name = diff.tool_call_delta.name;
|
||||
}
|
||||
if (!diff.tool_call_delta.id.empty()) {
|
||||
msg_accum.tool_calls[diff.tool_call_index].id = diff.tool_call_delta.id;
|
||||
@@ -1492,10 +1408,9 @@ static void test_peg_parser(common_chat_templates * tmpls,
|
||||
// sampler was already active before model output began — constrain from the
|
||||
// start of the model output (i.e. tc.input).
|
||||
std::string constrained = full_input;
|
||||
size_t constrain_from = 0;
|
||||
bool grammar_triggered = false;
|
||||
if (earliest_trigger_pos != std::string::npos) {
|
||||
constrain_from = std::max(earliest_trigger_pos, gen_prompt.size());
|
||||
auto constrain_from = std::max(earliest_trigger_pos, gen_prompt.size());
|
||||
constrained = full_input.substr(constrain_from);
|
||||
grammar_triggered = true;
|
||||
} else if (!sampling.grammar_lazy) {
|
||||
@@ -1505,9 +1420,7 @@ static void test_peg_parser(common_chat_templates * tmpls,
|
||||
|
||||
// Test the constrained portion against the grammar
|
||||
if (grammar_triggered && !tc.is_partial) {
|
||||
auto full_tokens = fake_tokenize(specials, gen_prompt);
|
||||
full_tokens.insert(full_tokens.end(), input_tokens.begin(), input_tokens.end());
|
||||
auto result = match_string_detailed(fake_tokens_from(full_tokens, constrain_from), grammar.get());
|
||||
auto result = match_string_detailed(constrained, grammar.get());
|
||||
if (!result.success) {
|
||||
std::string error_msg;
|
||||
if (result.incomplete) {
|
||||
@@ -1596,7 +1509,6 @@ class peg_tester {
|
||||
common_chat_templates_ptr tmpls_;
|
||||
std::string template_path_;
|
||||
bool detailed_debug_;
|
||||
common_peg_tokens specials_;
|
||||
friend class peg_test_builder;
|
||||
|
||||
public:
|
||||
@@ -1612,18 +1524,6 @@ class peg_tester {
|
||||
|
||||
const std::string & template_path() const { return template_path_; }
|
||||
|
||||
// Run the tests tokenized with these pieces as special tokens, as if the vocab defined them
|
||||
peg_tester & specials(const std::vector<std::string> & pieces) {
|
||||
llama_token id = FAKE_TEXT_TOKEN + 1;
|
||||
for (const auto & piece : pieces) {
|
||||
specials_.ids.emplace(piece, id);
|
||||
specials_.tokens.emplace(id, common_peg_token{ piece, LLAMA_TOKEN_ATTR_CONTROL });
|
||||
id++;
|
||||
}
|
||||
common_chat_templates_set_specials(tmpls_.get(), specials_);
|
||||
return *this;
|
||||
}
|
||||
|
||||
peg_test_builder test(const std::string & input);
|
||||
};
|
||||
|
||||
@@ -1714,11 +1614,6 @@ class peg_test_builder {
|
||||
return *this;
|
||||
}
|
||||
|
||||
peg_test_builder & expect_grammar() {
|
||||
tc_.expect_grammar = true;
|
||||
return *this;
|
||||
}
|
||||
|
||||
peg_test_builder & tool_choice(common_chat_tool_choice choice) {
|
||||
tc_.params.tool_choice = choice;
|
||||
return *this;
|
||||
@@ -1744,10 +1639,8 @@ class peg_test_builder {
|
||||
return;
|
||||
}
|
||||
}
|
||||
LOG_INF("\n\x1b[38;5;126m[%s%s]\x1b[0m\n%s\n\n", tester_.template_path().c_str(),
|
||||
tester_.specials_.ids.empty() ? "" : ", tokenized", tc_.input.c_str());
|
||||
test_peg_parser(tester_.tmpls_.get(), tester_.specials_, [this](peg_test_case & t) { t = tc_; },
|
||||
tester_.detailed_debug_);
|
||||
LOG_INF("\n\x1b[38;5;126m[%s]\x1b[0m\n%s\n\n", tester_.template_path().c_str(), tc_.input.c_str());
|
||||
test_peg_parser(tester_.tmpls_.get(), [this](peg_test_case & t) { t = tc_; }, tester_.detailed_debug_);
|
||||
}
|
||||
};
|
||||
|
||||
@@ -2366,17 +2259,9 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
|
||||
"const": "42"
|
||||
})";
|
||||
|
||||
// Special tokens of the templates that use the Qwen3-Coder format.
|
||||
// Check that the vocab of a model defines them before reuse. A model can share the syntax without the tokens.
|
||||
const std::vector<std::string> qwen3_coder_specials = {
|
||||
"<|im_start|>", "<|im_end|>", "<think>", "</think>",
|
||||
"<tool_call>", "</tool_call>", "<tool_response>", "</tool_response>",
|
||||
};
|
||||
|
||||
{
|
||||
// Qwen3.5 (basically same as Nemotron, but keeping separate tests just in case)
|
||||
auto tst = peg_tester("models/templates/Qwen3.5-4B.jinja", detailed_debug);
|
||||
tst.specials(qwen3_coder_specials);
|
||||
|
||||
tst.test("I'm\nthinking\n</think>\n\nHello, world!\nWhat's up?")
|
||||
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
|
||||
@@ -2948,7 +2833,6 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
|
||||
{
|
||||
// NVIDIA Nemotron-3 Nano
|
||||
auto tst = peg_tester("models/templates/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.jinja", detailed_debug);
|
||||
tst.specials(qwen3_coder_specials);
|
||||
|
||||
tst.test("Hello, world!\nWhat's up?").
|
||||
enable_thinking(false).
|
||||
@@ -3244,8 +3128,6 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
|
||||
{
|
||||
// Google Gemma 4 (tool calling with Gemma4 dict format)
|
||||
auto tst = peg_tester("models/templates/google-gemma-4-31B-it.jinja");
|
||||
tst.specials({ "<|turn>", "<turn|>", "<|channel>", "<channel|>", "<|tool_call>", "<tool_call|>",
|
||||
"<|tool_response>", "<tool_response|>", "<|tool>", "<tool|>", "<|\"|>", "<|think|>" });
|
||||
|
||||
tst.test("Hello, world!").expect(simple_assist_msg("Hello, world!")).run();
|
||||
|
||||
@@ -3874,7 +3756,6 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
|
||||
{
|
||||
// Qwen3-Coder (tool calling with XML-style format)
|
||||
auto tst = peg_tester("models/templates/Qwen3-Coder.jinja", detailed_debug);
|
||||
tst.specials(qwen3_coder_specials);
|
||||
|
||||
tst.test("Hello, world!\nWhat's up?").expect(message_assist).expect_reconstruction().run();
|
||||
|
||||
@@ -3903,20 +3784,6 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
|
||||
.expect(message_assist_call)
|
||||
.run();
|
||||
|
||||
// Required tool choice constrains from the first token instead of gating on a trigger
|
||||
tst.test(
|
||||
"<tool_call>\n"
|
||||
"<function=special_function>\n"
|
||||
"<parameter=arg1>\n"
|
||||
"1\n"
|
||||
"</parameter>\n"
|
||||
"</function>\n"
|
||||
"</tool_call>")
|
||||
.tools({ special_function_tool })
|
||||
.tool_choice(COMMON_CHAT_TOOL_CHOICE_REQUIRED)
|
||||
.expect(message_assist_call)
|
||||
.run();
|
||||
|
||||
tst.test(
|
||||
"Let me call it.\n"
|
||||
"<function=special_function>\n"
|
||||
@@ -6018,7 +5885,6 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
|
||||
|
||||
{
|
||||
auto tst = peg_tester("models/templates/LFM3.jinja", detailed_debug);
|
||||
tst.specials(qwen3_coder_specials);
|
||||
|
||||
tst.test("Hello, world!\nWhat's up?")
|
||||
.enable_thinking(false)
|
||||
@@ -7242,19 +7108,6 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
|
||||
auto tst = peg_tester("models/templates/meta-llama-Llama-3.2-3B-Instruct.jinja", detailed_debug);
|
||||
tst.test("Hello, world!\nWhat's up?").tools({ special_function_tool }).expect(message_assist).expect_reconstruction().expect_rules({ "tool-0" }).run();
|
||||
|
||||
// bare JSON calls have no start marker, auto mode must still produce a lazy grammar
|
||||
tst.test("{\"name\": \"special_function\", \"parameters\": {\"arg1\": 1}}")
|
||||
.tools({ special_function_tool })
|
||||
.expect(message_assist_call)
|
||||
.expect_grammar()
|
||||
.run();
|
||||
// same without spaces
|
||||
tst.test("{\"name\":\"special_function\",\"parameters\":{\"arg1\":1}}")
|
||||
.tools({ special_function_tool })
|
||||
.expect(message_assist_call)
|
||||
.expect_grammar()
|
||||
.run();
|
||||
|
||||
// Continuation tests
|
||||
tst.test("world!\nWhat's up?")
|
||||
.messages({ message_user, message_assist_prefill_content })
|
||||
@@ -7281,7 +7134,6 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
|
||||
// Muse Glimmer format tests
|
||||
{
|
||||
auto tst = peg_tester("models/templates/muse-glimmer.jinja", detailed_debug);
|
||||
tst.specials({ "<|start|>", "<|message|>", "<|eom|>" });
|
||||
|
||||
const std::string call_markup =
|
||||
"<atem:function_calls>\n"
|
||||
@@ -7291,7 +7143,7 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
|
||||
"</atem:function_calls>";
|
||||
|
||||
// A plain answer is unaffected
|
||||
tst.test(" to=user<|message|>Hello, world!\nWhat's up?")
|
||||
tst.test(" to=user<|message|>Hello, world!\nWhat's up?<|eot|>")
|
||||
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
|
||||
.expect(message_assist)
|
||||
.run();
|
||||
@@ -7302,34 +7154,7 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
|
||||
.tools({ special_function_tool })
|
||||
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
|
||||
.expect(message_assist_call)
|
||||
.expect_rules({ "tool-0", "tool-calls" })
|
||||
.run();
|
||||
|
||||
// Parallel calls, one message per call
|
||||
tst.test(" to=special_function<|message|>" + call_markup + "<|eom|>"
|
||||
"<|start|>assistant to=special_function_with_opt<|message|>"
|
||||
"<atem:function_calls>\n"
|
||||
"<atem:invoke name=\"special_function_with_opt\">\n"
|
||||
"<atem:parameter name=\"arg1\">1</atem:parameter>\n"
|
||||
"<atem:parameter name=\"arg2\">2</atem:parameter>\n"
|
||||
"</atem:invoke>\n"
|
||||
"</atem:function_calls>")
|
||||
.tools({ special_function_tool, special_function_tool_with_optional_param })
|
||||
.parallel_tool_calls(true)
|
||||
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
|
||||
.expect_tool_calls({
|
||||
{ "special_function", R"({"arg1": 1})", {} },
|
||||
{ "special_function_with_opt", R"({"arg1": 1, "arg2": 2})", {} },
|
||||
})
|
||||
.run();
|
||||
|
||||
// Required tool choice after a reasoning message
|
||||
tst.test(" to=self<|message|>I'm\nthinking<|eom|>"
|
||||
"<|start|>assistant to=special_function<|message|>" + call_markup)
|
||||
.tools({ special_function_tool })
|
||||
.tool_choice(COMMON_CHAT_TOOL_CHOICE_REQUIRED)
|
||||
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
|
||||
.expect(message_assist_call_thoughts)
|
||||
.expect_rules({ "tool-0" })
|
||||
.run();
|
||||
|
||||
// "Inform then act": the model answers the user and calls a tool in ONE generation,
|
||||
@@ -7345,7 +7170,7 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
|
||||
|
||||
// Markup quoted in an answer has no preceding <|eom|>, so it stays content instead of
|
||||
// becoming an invocation the user never asked for
|
||||
tst.test(" to=user<|message|>You invoke it like this:\n" + call_markup)
|
||||
tst.test(" to=user<|message|>You invoke it like this:\n" + call_markup + "<|eot|>")
|
||||
.tools({ special_function_tool })
|
||||
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
|
||||
.expect_content("You invoke it like this:\n" + call_markup)
|
||||
@@ -7369,7 +7194,7 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
|
||||
|
||||
// Tool markup inside the analysis channel is reasoning, not a call
|
||||
tst.test(" to=self<|message|>I could use " + call_markup + " here<|eom|>"
|
||||
"<|start|>assistant to=user<|message|>Hello!")
|
||||
"<|start|>assistant to=user<|message|>Hello!<|eot|>")
|
||||
.tools({ special_function_tool })
|
||||
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
|
||||
.expect_reasoning("I could use " + call_markup + " here")
|
||||
@@ -7440,7 +7265,6 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
|
||||
// GPT-OSS format tests
|
||||
{
|
||||
auto tst = peg_tester("models/templates/openai-gpt-oss-120b.jinja", detailed_debug);
|
||||
tst.specials({ "<|start|>", "<|end|>", "<|message|>", "<|channel|>", "<|constrain|>", "<|return|>", "<|call|>" });
|
||||
|
||||
// Basic content only - final channel
|
||||
tst.test("<|channel|>final<|message|>Hello, world!\nWhat's up?").expect(message_assist).run();
|
||||
@@ -7480,30 +7304,7 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
|
||||
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
|
||||
.tools({ special_function_tool })
|
||||
.expect(message_assist_call)
|
||||
.expect_rules({ "tool-0-in-role", "tool-0-in-channel" })
|
||||
.run();
|
||||
|
||||
// A tool whose name prefixes another's must not be reported while the header is still open
|
||||
tst.test(" to=functions.special_function_with_opt<|channel|>analysis<|message|>{\"arg1\": 1}")
|
||||
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
|
||||
.tools({ special_function_tool, special_function_tool_with_optional_param })
|
||||
.expect(simple_assist_msg("", "", "special_function_with_opt", "{\"arg1\": 1}"))
|
||||
.run();
|
||||
|
||||
tst.test("<|channel|>commentary to=functions.special_function_with_opt<|message|>{\"arg1\": 1}")
|
||||
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
|
||||
.tools({ special_function_tool, special_function_tool_with_optional_param })
|
||||
.expect(simple_assist_msg("", "", "special_function_with_opt", "{\"arg1\": 1}"))
|
||||
.run();
|
||||
|
||||
// Required tool choice constrains from the first token instead of gating on a trigger
|
||||
tst.test(
|
||||
"<|channel|>analysis<|message|>I'm\nthinking<|end|>"
|
||||
"<|start|>assistant to=functions.special_function<|channel|>analysis<|message|>{\"arg1\": 1}")
|
||||
.reasoning_format(COMMON_REASONING_FORMAT_AUTO)
|
||||
.tools({ special_function_tool })
|
||||
.tool_choice(COMMON_CHAT_TOOL_CHOICE_REQUIRED)
|
||||
.expect(message_assist_call_thoughts)
|
||||
.expect_rules({ "tool-0" })
|
||||
.run();
|
||||
|
||||
// Tool call with recipient in channel header: "<|channel|>analysis to=functions.NAME<|message|>JSON"
|
||||
@@ -7720,7 +7521,6 @@ static void test_template_output_peg_parsers(bool detailed_debug) {
|
||||
|
||||
{
|
||||
auto tst = peg_tester("models/templates/StepFun3.5-Flash.jinja", detailed_debug);
|
||||
tst.specials(qwen3_coder_specials);
|
||||
|
||||
tst.test("I was thinking\n</think>\nNow I'm not.").
|
||||
enable_thinking(true).
|
||||
@@ -8920,7 +8720,7 @@ static void test_reasoning_effort_caps() {
|
||||
assert_equals(json(expected).dump(), common_chat_templates_get_caps(tmpls.get()).at("reasoning_effort_levels").dump());
|
||||
};
|
||||
|
||||
const std::vector<std::string> all_levels = { "none", "minimal", "low", "medium", "high", "xhigh", "max" };
|
||||
const std::vector<std::string> all_levels = { "minimal", "low", "medium", "high", "xhigh", "max" };
|
||||
|
||||
assert_effort_levels("models/templates/deepseek-ai-DeepSeek-V4.jinja", { "max" });
|
||||
assert_effort_levels("models/templates/deepseek-ai-DeepSeek-V4-Flash-0731.jinja", { "high", "max" });
|
||||
@@ -8930,8 +8730,7 @@ static void test_reasoning_effort_caps() {
|
||||
assert_effort_levels("models/templates/upstage-Solar-Open-100B.jinja", { "low" });
|
||||
assert_effort_levels("models/templates/muse-glimmer.jinja", all_levels);
|
||||
assert_effort_levels("models/templates/openai-gpt-oss-120b.jinja", all_levels);
|
||||
assert_effort_levels("models/templates/Cohere2MoE.jinja", { "none" });
|
||||
assert_effort_levels("models/templates/thinkingmachines-Inkling-2-Small.jinja", { "none", "minimal", "low", "medium", "high", "max" });
|
||||
assert_effort_levels("models/templates/Cohere2MoE.jinja", {});
|
||||
assert_effort_levels("models/templates/Qwen-Qwen3-0.6B.jinja", {});
|
||||
|
||||
auto assert_effort_rejected = [](const std::string & path, const std::vector<std::string> & expected) {
|
||||
|
||||
@@ -41,21 +41,15 @@ struct token_and_piece {
|
||||
std::string piece;
|
||||
};
|
||||
|
||||
// token() encodes a token and its piece as a 0xff marker, the ID in big-endian order, the piece length, and the piece.
|
||||
static std::string token(llama_token id, const std::string & piece) {
|
||||
// token() encodes a 32-bit ID as 5 bytes: a 0xff marker followed by the ID in big-endian order.
|
||||
static std::string token(llama_token id) {
|
||||
return std::string{
|
||||
static_cast<char>(0xff),
|
||||
static_cast<char>((id >> 24) & 0xff),
|
||||
static_cast<char>((id >> 16) & 0xff),
|
||||
static_cast<char>((id >> 8) & 0xff),
|
||||
static_cast<char>(id & 0xff),
|
||||
static_cast<char>(piece.size())
|
||||
} + piece;
|
||||
}
|
||||
|
||||
// token() without a piece encodes a token whose piece is its <[id]> spelling.
|
||||
static std::string token(llama_token id) {
|
||||
return token(id, "<[" + std::to_string(id) + "]>");
|
||||
static_cast<char>(id & 0xff)
|
||||
};
|
||||
}
|
||||
|
||||
// parse_tokens() parses the token encodes above and UTF-8 text.
|
||||
@@ -66,7 +60,7 @@ static std::vector<token_and_piece> parse_tokens(const std::string & input) {
|
||||
while (offset < input.size()) {
|
||||
try {
|
||||
if (static_cast<unsigned char>(input[offset]) == 0xff) {
|
||||
if (offset + 6 > input.size()) {
|
||||
if (offset + 5 > input.size()) {
|
||||
throw std::runtime_error("not enough bytes for token id");
|
||||
}
|
||||
uint32_t val =
|
||||
@@ -74,12 +68,9 @@ static std::vector<token_and_piece> parse_tokens(const std::string & input) {
|
||||
(static_cast<unsigned char>(input[offset + 2]) << 16) |
|
||||
(static_cast<unsigned char>(input[offset + 3]) << 8) |
|
||||
(static_cast<unsigned char>(input[offset + 4]));
|
||||
size_t len = static_cast<unsigned char>(input[offset + 5]);
|
||||
if (offset + 6 + len > input.size()) {
|
||||
throw std::runtime_error("not enough bytes for token piece");
|
||||
}
|
||||
result.push_back({static_cast<llama_token>(val), input.substr(offset + 6, len)});
|
||||
offset += 6 + len;
|
||||
auto piece = "<[" + std::to_string(val) + "]>";
|
||||
result.push_back({static_cast<llama_token>(val), piece});
|
||||
offset += 5;
|
||||
} else {
|
||||
uint32_t cpt = unicode_cpt_from_utf8(input, offset);
|
||||
result.push_back({0, unicode_cpt_to_utf8(cpt)});
|
||||
@@ -93,42 +84,23 @@ static std::vector<token_and_piece> parse_tokens(const std::string & input) {
|
||||
return result;
|
||||
}
|
||||
|
||||
// Returns true if the sampler mask allows the token. The mask rejects a token only when every stack rejects it.
|
||||
static bool mask_allows(llama_grammar * grammar, const token_and_piece & in) {
|
||||
const auto decoded = llama_grammar_decode_utf8(in.piece, grammar->partial_utf8);
|
||||
|
||||
llama_grammar_candidates rejects = { { 0, decoded.first.data(), decoded.second, in.token, llama_grammar_is_opaque(*grammar, in.token) } };
|
||||
for (const auto & stack : llama_grammar_get_stacks(grammar)) {
|
||||
rejects = llama_grammar_reject_candidates_for_stack(llama_grammar_get_rules(grammar), stack, rejects);
|
||||
}
|
||||
return rejects.empty();
|
||||
}
|
||||
|
||||
static bool match_string(const std::string & input, llama_grammar * grammar) {
|
||||
const auto parsed = parse_tokens(input);
|
||||
|
||||
auto & stacks_cur = llama_grammar_get_stacks(grammar);
|
||||
|
||||
for (const auto & in : parsed) {
|
||||
const bool allowed = mask_allows(grammar, in);
|
||||
|
||||
// the mask decides what can be sampled, so it has to agree with what accepting the token does
|
||||
try {
|
||||
llama_grammar_accept_token(*grammar, in.token, in.piece);
|
||||
} catch (const std::runtime_error & /*e*/) {
|
||||
if (allowed) {
|
||||
fprintf(stderr, "❌ (token %d \"%s\" is allowed by the mask but not accepted)\n", in.token, in.piece.c_str());
|
||||
}
|
||||
assert(!allowed);
|
||||
|
||||
// no stacks means that the grammar failed to match at this point
|
||||
// normally this shouldn't get hit because of llama_grammar_apply
|
||||
return false;
|
||||
}
|
||||
|
||||
if (!allowed) {
|
||||
fprintf(stderr, "❌ (token %d \"%s\" is masked but accepted)\n", in.token, in.piece.c_str());
|
||||
if (stacks_cur.empty()) {
|
||||
// no stacks means that the grammar failed to match at this point
|
||||
return false;
|
||||
}
|
||||
assert(allowed);
|
||||
}
|
||||
|
||||
for (const auto & stack : stacks_cur) {
|
||||
@@ -149,7 +121,6 @@ static void test(const std::string & test_desc, const std::string & grammar_str,
|
||||
|
||||
// Save the original grammar stacks so that we can reset after every new string we want to test
|
||||
const llama_grammar_stacks stacks_org = llama_grammar_get_stacks(grammar); // copy
|
||||
const llama_partial_utf8 partial_org = grammar->partial_utf8;
|
||||
|
||||
llama_grammar_stacks & stacks_cur = llama_grammar_get_stacks(grammar);
|
||||
|
||||
@@ -189,7 +160,6 @@ static void test(const std::string & test_desc, const std::string & grammar_str,
|
||||
|
||||
// Reset the grammar stacks
|
||||
stacks_cur = stacks_org;
|
||||
grammar->partial_utf8 = partial_org;
|
||||
}
|
||||
|
||||
fprintf(stderr, " 🟠 Invalid strings:\n");
|
||||
@@ -210,7 +180,6 @@ static void test(const std::string & test_desc, const std::string & grammar_str,
|
||||
|
||||
// Reset the grammar stacks
|
||||
stacks_cur = stacks_org;
|
||||
grammar->partial_utf8 = partial_org;
|
||||
}
|
||||
|
||||
// Clean up allocated memory
|
||||
@@ -553,137 +522,6 @@ static void test_simple_grammar() {
|
||||
);
|
||||
}
|
||||
|
||||
static void test_token_rule_utf8() {
|
||||
// A byte-level token can hold part of a multibyte character, a token rule still takes it as a whole token
|
||||
test_grammar(
|
||||
"byte-split characters under token rules",
|
||||
R"""(
|
||||
root ::= <[10]> content <[11]>
|
||||
content ::= (!<[11]>)*)""",
|
||||
// Passing strings
|
||||
{
|
||||
token(10) + token(20, "\xF0") + token(21, "\x9F\xA6\x99") + token(11),
|
||||
token(10) + token(22, "\xE2") + token(23, "\x98\x80") + token(11),
|
||||
token(10) + token(24, "a\xF0") + token(21, "\x9F\xA6\x99") + token(11),
|
||||
},
|
||||
// Failing strings
|
||||
{
|
||||
token(10) + token(20, "\xF0"),
|
||||
token(10) + token(20, "\xF0") + token(11),
|
||||
}
|
||||
);
|
||||
|
||||
// A token rule can only start where a token starts, not after char rules consumed part of it
|
||||
test_grammar(
|
||||
"token rule after a char rule",
|
||||
R"""(
|
||||
root ::= "a" <[11]>)""",
|
||||
// Passing strings
|
||||
{
|
||||
"a" + token(11),
|
||||
token(12, "a") + token(11),
|
||||
},
|
||||
// Failing strings
|
||||
{
|
||||
token(11, "ab"),
|
||||
token(12, "ab") + token(11),
|
||||
}
|
||||
);
|
||||
|
||||
// A token partly matched by a char rule stays partly matched for every alternative, whichever is checked first
|
||||
test_grammar(
|
||||
"token rule alternative after a char rule",
|
||||
R"""(
|
||||
root ::= "a" (!<[7]> | "bx"))""",
|
||||
// Passing strings
|
||||
{
|
||||
"abx",
|
||||
token(12, "abx"),
|
||||
"a" + token(5),
|
||||
},
|
||||
// Failing strings
|
||||
{
|
||||
token(5, "aby"),
|
||||
token(5, "ab"),
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
static void test_token_sets() {
|
||||
test_grammar(
|
||||
"token set with alternatives and ranges",
|
||||
R"""(
|
||||
root ::= <[1,5-9,12]>+)""",
|
||||
// Passing strings
|
||||
{
|
||||
token(1),
|
||||
token(5) + token(9),
|
||||
token(7) + token(12) + token(1),
|
||||
},
|
||||
// Failing strings
|
||||
{
|
||||
token(2),
|
||||
token(4),
|
||||
token(10),
|
||||
token(13),
|
||||
token(5) + token(11),
|
||||
}
|
||||
);
|
||||
|
||||
test_grammar(
|
||||
"negated token set",
|
||||
R"""(
|
||||
root ::= <[10]> (!<[11,20-29]>)* <[11]>)""",
|
||||
// Passing strings
|
||||
{
|
||||
token(10) + token(11),
|
||||
token(10) + "text" + token(12) + token(19) + token(30) + token(11),
|
||||
},
|
||||
// Failing strings
|
||||
{
|
||||
token(10) + token(20) + token(11),
|
||||
token(10) + token(29) + token(11),
|
||||
token(10) + token(11) + token(11),
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
static void test_opaque_tokens() {
|
||||
// Char rules do not match a token that a token rule names, other tokens still match as text
|
||||
test_grammar(
|
||||
"tokens named by token rules are opaque to char rules",
|
||||
R"""(
|
||||
root ::= [^x]* <[11]> "done")""",
|
||||
// Passing strings
|
||||
{
|
||||
"abc" + token(11) + "done",
|
||||
token(11) + "done",
|
||||
token(12) + token(11) + "done",
|
||||
},
|
||||
// Failing strings
|
||||
{
|
||||
"abc" + token(11) + "more" + token(11) + "done",
|
||||
token(11) + token(11) + "done",
|
||||
}
|
||||
);
|
||||
|
||||
test_grammar(
|
||||
"tokens named by a negated set are opaque to char rules",
|
||||
R"""(
|
||||
root ::= [^x]* !<[11,20-29]>)""",
|
||||
// Passing strings
|
||||
{
|
||||
"abc" + token(30),
|
||||
token(12) + token(30),
|
||||
},
|
||||
// Failing strings
|
||||
{
|
||||
token(25) + token(30),
|
||||
token(11) + token(30),
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
static void test_complex_grammar() {
|
||||
// Test case for a more complex grammar, with both failure strings and success strings
|
||||
test_grammar(
|
||||
@@ -1701,9 +1539,6 @@ int main() {
|
||||
fprintf(stdout, "Running grammar integration tests...\n");
|
||||
test_simple_grammar();
|
||||
test_complex_grammar();
|
||||
test_token_rule_utf8();
|
||||
test_token_sets();
|
||||
test_opaque_tokens();
|
||||
test_special_chars();
|
||||
test_quantifiers();
|
||||
test_failure_missing_root();
|
||||
|
||||
@@ -179,14 +179,6 @@ int main()
|
||||
verify_failure(deep_grammar.c_str());
|
||||
}
|
||||
|
||||
verify_failure(R"""(
|
||||
root ::= <[9-5]>
|
||||
)""");
|
||||
|
||||
verify_failure(R"""(
|
||||
root ::= <[4294967296]>
|
||||
)""");
|
||||
|
||||
verify_parsing(R"""(
|
||||
root ::= "a"{0,5000}
|
||||
)""", {
|
||||
@@ -602,21 +594,5 @@ int main()
|
||||
{LLAMA_GRETYPE_END, 0},
|
||||
});
|
||||
|
||||
verify_parsing(R"""(
|
||||
root ::= <[1,5-9,12]> !<[11,20-29]>
|
||||
)""", {
|
||||
{"root", 0}
|
||||
}, {
|
||||
// root (index 0)
|
||||
{LLAMA_GRETYPE_TOKEN, 1},
|
||||
{LLAMA_GRETYPE_TOKEN_ALT, 5},
|
||||
{LLAMA_GRETYPE_TOKEN_RNG_UPPER, 9},
|
||||
{LLAMA_GRETYPE_TOKEN_ALT, 12},
|
||||
{LLAMA_GRETYPE_TOKEN_NOT, 11},
|
||||
{LLAMA_GRETYPE_TOKEN_ALT, 20},
|
||||
{LLAMA_GRETYPE_TOKEN_RNG_UPPER, 29},
|
||||
{LLAMA_GRETYPE_END, 0},
|
||||
});
|
||||
|
||||
return 0;
|
||||
}
|
||||
@@ -202,7 +202,7 @@ int main()
|
||||
uint32_t *cp = new uint32_t[2]; // dynamically allocate memory for code_point
|
||||
cp[0] = 37 + i;
|
||||
cp[1] = 0;
|
||||
next_candidates[i] = {i, cp, {}, 0, false};
|
||||
next_candidates[i] = {i, cp, {}, 0};
|
||||
}
|
||||
|
||||
std::vector<std::vector<std::pair<uint32_t, uint16_t>>> expected_reject = {
|
||||
|
||||
@@ -2435,8 +2435,6 @@ Response:
|
||||
|
||||
### GET `/models/sse`: Real-time events
|
||||
|
||||
This endpoint is also available in single-model mode (`-m`, `-hf`, ...), where it is served before the model is loaded, to follow the download and loading progress. As `GET /models` is unavailable until the model is loaded, each new client first receives the last event sent. If the download or the loading fails, a `status_change` event with status `unloaded` is sent before the server exits.
|
||||
|
||||
Example events:
|
||||
|
||||
```js
|
||||
|
||||
@@ -1,4 +1,3 @@
|
||||
#include "arg.h"
|
||||
#include "common.h"
|
||||
#include "download.h"
|
||||
#include "log.h"
|
||||
@@ -1642,18 +1641,13 @@ json oaicompat_chat_params_parse(
|
||||
// Parse the OAI "reasoning_effort" field; "none" disables reasoning.
|
||||
if (body.contains("reasoning_effort")) {
|
||||
auto reasoning_effort = json_value(body, "reasoning_effort", std::string(""));
|
||||
const auto levels = caps.at("reasoning_effort_levels").get<std::vector<std::string>>();
|
||||
if (reasoning_effort == "none") {
|
||||
inputs.enable_thinking = false;
|
||||
// Some models use reasoning_effort = none and some throw an exception, so check the model's list of effort levels first
|
||||
if (std::find(levels.begin(), levels.end(), reasoning_effort) != levels.end()) {
|
||||
inputs.chat_template_kwargs["reasoning_effort"] = json(reasoning_effort).dump();
|
||||
} else {
|
||||
inputs.chat_template_kwargs.erase("reasoning_effort");
|
||||
}
|
||||
inputs.chat_template_kwargs.erase("reasoning_effort");
|
||||
} else if (!reasoning_effort.empty()) {
|
||||
const auto rejected = caps.at("reasoning_effort_rejected").get<std::vector<std::string>>();
|
||||
if (std::find(rejected.begin(), rejected.end(), reasoning_effort) != rejected.end()) {
|
||||
const auto levels = caps.at("reasoning_effort_levels").get<std::vector<std::string>>();
|
||||
throw std::invalid_argument("reasoning_effort \"" + reasoning_effort + "\" is not supported by the chat template, supported levels: " + string_join(levels, ", "));
|
||||
}
|
||||
inputs.chat_template_kwargs["reasoning_effort"] = json(reasoning_effort).dump();
|
||||
@@ -2387,22 +2381,6 @@ server_role::~server_role() {
|
||||
}
|
||||
}
|
||||
|
||||
std::string server_get_model_name(const common_params & params) {
|
||||
if (!params.model_alias.empty()) {
|
||||
// backward compat: use first alias as model name
|
||||
return *params.model_alias.begin();
|
||||
}
|
||||
if (!params.model.get_name().empty()) {
|
||||
return params.model.get_name();
|
||||
}
|
||||
if (!params.model.url.empty()) {
|
||||
// -mu before the download: the path the model will be stored at
|
||||
return common_get_default_local_path(params.model.url);
|
||||
}
|
||||
// fallback: derive model name from file name
|
||||
return std::filesystem::path(params.model.path).filename().string();
|
||||
}
|
||||
|
||||
bool server_role::is_managed_by_router() {
|
||||
return std::getenv(SERVER_ENV_ROUTER_PORT) != nullptr;
|
||||
}
|
||||
|
||||
@@ -837,9 +837,6 @@ static inline server_state server_state_from_str(const std::string & str) {
|
||||
GGML_ASSERT(false && "invalid server_state string");
|
||||
}
|
||||
|
||||
// the model name reported by the server: the first alias, else the model's repo or path
|
||||
std::string server_get_model_name(const common_params & params);
|
||||
|
||||
// the role of this process and, when it is a child of a router, its side of the protocol
|
||||
struct server_role {
|
||||
// serializes the notify_to_router writes
|
||||
|
||||
@@ -2170,7 +2170,16 @@ private:
|
||||
SRV_TRC("%s", "context checkpoints disabled\n");
|
||||
}
|
||||
|
||||
model_name = server_get_model_name(params_base);
|
||||
if (!params_base.model_alias.empty()) {
|
||||
// backward compat: use first alias as model name
|
||||
model_name = *params_base.model_alias.begin();
|
||||
} else if (!params_base.model.get_name().empty()) {
|
||||
model_name = params_base.model.get_name();
|
||||
} else {
|
||||
// fallback: derive model name from file name
|
||||
auto model_path = std::filesystem::path(params_base.model.path);
|
||||
model_name = model_path.filename().string();
|
||||
}
|
||||
|
||||
model_aliases = params_base.model_alias;
|
||||
model_tags = params_base.model_tags;
|
||||
@@ -4813,9 +4822,7 @@ private:
|
||||
|
||||
if (!slot.can_split()) {
|
||||
// cannot fit the prompt in the current batch - will try next iter
|
||||
// with non-causal attention the whole batch must fit in one ubatch, see llama_decode
|
||||
const int32_t n_batch_max = llama_get_causal_attn(ctx_tgt) ? n_batch : std::min(n_batch, n_ubatch);
|
||||
if (batch.size() + slot.task->n_tokens() > n_batch_max) {
|
||||
if (batch.size() + slot.task->n_tokens() > n_batch) {
|
||||
return;
|
||||
}
|
||||
}
|
||||
@@ -4948,23 +4955,6 @@ private:
|
||||
const bool merge_user_starts = userstart_merge && !has_mtmd &&
|
||||
slot.task->n_tokens() - slot.prompt.n_tokens() <= n_ubatch;
|
||||
|
||||
// [fork F14] merged or not, keep the break at the first user message, the end of
|
||||
// the system prompt: the checkpoint there is what a new chat with the same system
|
||||
// prompt restores. The batch only reaches it when the prompt is processed from
|
||||
// before it (a fresh prompt; checkpoints past the restore point are erased), so
|
||||
// follow-up turns are not affected. Below 8 tokens there is no system prompt
|
||||
// worth a checkpoint (only a BOS before the first user message).
|
||||
int32_t first_user_pos = -1;
|
||||
for (const auto & s : spans.spans) {
|
||||
if (s.role == COMMON_CHAT_ROLE_USER) {
|
||||
first_user_pos = (int32_t) s.pos;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (first_user_pos < 8) {
|
||||
first_user_pos = -1;
|
||||
}
|
||||
|
||||
// add prompt tokens for processing in the current batch
|
||||
while (slot.prompt.n_tokens() < slot.task->n_tokens() && batch.size() < n_batch) {
|
||||
// get next token to process
|
||||
@@ -5008,15 +4998,11 @@ private:
|
||||
slot.prompt.tokens.push_back(cur_tok);
|
||||
|
||||
// break at the last user message, or at user messages at least min step past the last checkpoint
|
||||
if (do_checkpoint && spans.is_user_start(slot.prompt.n_tokens())) {
|
||||
if (do_checkpoint && !merge_user_starts && spans.is_user_start(slot.prompt.n_tokens())) {
|
||||
const auto pos = slot.prompt.n_tokens();
|
||||
const auto & checkpoints = slot.prompt.checkpoints;
|
||||
|
||||
if (merge_user_starts) {
|
||||
if (pos == first_user_pos) {
|
||||
break;
|
||||
}
|
||||
} else if (pos == last_user_pos || checkpoints.empty() || pos > checkpoints.back()->n_tokens + params_base.checkpoint_min_step) {
|
||||
if (pos == last_user_pos || checkpoints.empty() || pos > checkpoints.back()->n_tokens + params_base.checkpoint_min_step) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -396,9 +396,6 @@ bool server_http_context::init_listener(const common_params & params) {
|
||||
if (frontend_paths.count(req.path)) {
|
||||
return true; // frontend asset, allow it to load and show "loading"
|
||||
}
|
||||
if (req.path == "/models/sse") {
|
||||
return true; // reports the download and loading progress
|
||||
}
|
||||
// no endpoints are allowed to be accessed when the server is not ready
|
||||
// this is to prevent any data races or inconsistent states
|
||||
res.status = 503;
|
||||
|
||||
@@ -734,7 +734,16 @@ bool server_models::remove_sidecar(const std::string & name, const std::string &
|
||||
}
|
||||
|
||||
void server_models::notify_sse(const std::string & event, const std::string & model_id, const json & data) {
|
||||
sse.notify(event, model_id, data);
|
||||
std::unique_ptr<server_task_result_router> result = std::make_unique<server_task_result_router>();
|
||||
result->data = {
|
||||
{"model", model_id},
|
||||
{"event", event},
|
||||
};
|
||||
if (!data.is_null()) {
|
||||
result->data["data"] = data;
|
||||
}
|
||||
SRV_DBG("notifying SSE clients about event '%s' for model '%s': %s\n", event.c_str(), model_id.c_str(), safe_json_to_str(result->data).c_str());
|
||||
sse.broadcast(std::move(result));
|
||||
}
|
||||
|
||||
void server_models::load_models() {
|
||||
@@ -1922,7 +1931,7 @@ void server_models::handle_child_state(const std::string & name, const std::stri
|
||||
}
|
||||
|
||||
//
|
||||
// server_models_sse
|
||||
// server_models_routes
|
||||
//
|
||||
|
||||
// RAII wrapper similar to server_response_reader, but doesn't use server_queue
|
||||
@@ -1961,61 +1970,6 @@ struct server_models_sse_client {
|
||||
}
|
||||
};
|
||||
|
||||
void server_models_sse::notify(const std::string & event, const std::string & model_id, const json & data) {
|
||||
std::unique_ptr<server_task_result_router> result = std::make_unique<server_task_result_router>();
|
||||
result->data = {
|
||||
{"model", model_id},
|
||||
{"event", event},
|
||||
};
|
||||
if (!data.is_null()) {
|
||||
result->data["data"] = data;
|
||||
}
|
||||
SRV_DBG("notifying SSE clients about event '%s' for model '%s': %s\n", event.c_str(), model_id.c_str(), safe_json_to_str(result->data).c_str());
|
||||
// under the lock, so that a client connecting meanwhile gets this event either replayed or queued, not both
|
||||
std::lock_guard<std::mutex> lk(mutex);
|
||||
if (replay_last) {
|
||||
last_event = result->data;
|
||||
}
|
||||
queue.broadcast(std::move(result));
|
||||
}
|
||||
|
||||
server_http_context::handler_t server_models_sse::make_handler(std::function<bool()> should_stop) {
|
||||
return [this, should_stop = std::move(should_stop)](const server_http_req & req) {
|
||||
auto res = std::make_unique<server_http_res>();
|
||||
res->status = 200;
|
||||
res->content_type = "text/event-stream";
|
||||
std::shared_ptr<server_models_sse_client> sse_client;
|
||||
std::string pending; // the replayed event, sent first
|
||||
{
|
||||
std::lock_guard<std::mutex> lk(mutex);
|
||||
sse_client = std::make_shared<server_models_sse_client>(queue);
|
||||
if (replay_last && !last_event.is_null()) {
|
||||
pending = "data: " + safe_json_to_str(last_event) + "\n\n";
|
||||
}
|
||||
}
|
||||
res->next = [should_stop, sse_client, pending = std::move(pending), &req](std::string & output) mutable -> bool {
|
||||
if (!pending.empty()) {
|
||||
output = std::move(pending);
|
||||
pending.clear();
|
||||
return true;
|
||||
}
|
||||
auto result = sse_client->next([&]() {
|
||||
return should_stop() || req.should_stop();
|
||||
});
|
||||
if (result == nullptr) {
|
||||
return false; // client disconnected or should_stop
|
||||
}
|
||||
output = "data: " + safe_json_to_str(result->to_json()) + "\n\n";
|
||||
return true; // listen for the next event
|
||||
};
|
||||
return res;
|
||||
};
|
||||
}
|
||||
|
||||
//
|
||||
// server_models_routes
|
||||
//
|
||||
|
||||
static void res_ok(std::unique_ptr<server_http_res> & res, const json & response_data) {
|
||||
res->status = 200;
|
||||
res->data = safe_json_to_str(response_data);
|
||||
@@ -2358,9 +2312,23 @@ void server_models_routes::init_routes() {
|
||||
return res;
|
||||
};
|
||||
|
||||
this->get_router_models_sse = models.sse.make_handler([this]() {
|
||||
return stopping.load(std::memory_order_relaxed);
|
||||
});
|
||||
this->get_router_models_sse = [this](const server_http_req & req) {
|
||||
auto res = std::make_unique<server_http_res>();
|
||||
res->status = 200;
|
||||
res->content_type = "text/event-stream";
|
||||
auto sse_client = std::make_shared<server_models_sse_client>(models.sse);
|
||||
res->next = [this, sse_client, &req](std::string & output) -> bool {
|
||||
auto result = sse_client->next([&]() {
|
||||
return stopping.load(std::memory_order_relaxed) || req.should_stop();
|
||||
});
|
||||
if (result == nullptr) {
|
||||
return false; // client disconnected or should_stop
|
||||
}
|
||||
output = "data: " + safe_json_to_str(result->to_json()) + "\n\n";
|
||||
return true; // listen for the next event
|
||||
};
|
||||
return res;
|
||||
};
|
||||
|
||||
this->post_router_models = [this](const server_http_req & req) {
|
||||
auto res = std::make_unique<server_http_res>();
|
||||
|
||||
@@ -157,25 +157,6 @@ private:
|
||||
void delete_partials(const std::vector<common_download_partial> & parts);
|
||||
};
|
||||
|
||||
// broadcasts model events to the clients of GET /models/sse, used by both router and single-model mode
|
||||
struct server_models_sse {
|
||||
// send the last event to each new client, for single-model mode where GET /models is unavailable while loading
|
||||
const bool replay_last;
|
||||
|
||||
server_models_sse(bool replay_last = false) : replay_last(replay_last) {}
|
||||
|
||||
// thread-safe
|
||||
void notify(const std::string & event, const std::string & model_id, const json & data = nullptr);
|
||||
|
||||
// handler for GET /models/sse, a client is disconnected once should_stop() returns true
|
||||
server_http_context::handler_t make_handler(std::function<bool()> should_stop);
|
||||
|
||||
private:
|
||||
std::mutex mutex;
|
||||
json last_event;
|
||||
server_response queue;
|
||||
};
|
||||
|
||||
struct server_models_routes;
|
||||
struct server_monitor; // defined in server-models.cpp
|
||||
|
||||
@@ -308,7 +289,7 @@ public:
|
||||
server_models(const common_params & params, int argc, char ** argv);
|
||||
~server_models();
|
||||
|
||||
server_models_sse sse; // for real-time updates via SSE endpoint
|
||||
server_response sse; // for real-time updates via SSE endpoint
|
||||
|
||||
// (re-)load the list of models from various sources and prepare the metadata mapping
|
||||
// - if this is called the first time, simply populate the metadata
|
||||
|
||||
@@ -1121,7 +1121,6 @@ struct server_tool_grep_search : server_tool {
|
||||
{"literal", {{"type", "boolean"}, {"description", "Treat pattern as a literal string instead of a regular expression (default: false)"}}},
|
||||
{"ignore_case", {{"type", "boolean"}, {"description", "Case-insensitive search (default: false)"}}},
|
||||
{"context_lines", {{"type", "integer"}, {"description", "Number of lines of context to show before and after each match (default: 0)"}}},
|
||||
{"offset", {{"type", "integer"}, {"description", "Number of matching lines to skip (default: 0)"}}},
|
||||
}},
|
||||
{"required", json::array({"path", "pattern"})},
|
||||
}},
|
||||
@@ -1138,7 +1137,6 @@ struct server_tool_grep_search : server_tool {
|
||||
bool literal = json_value(params, "literal", false);
|
||||
bool ignore_case = json_value(params, "ignore_case", false);
|
||||
int ctx_lines = std::max(0, json_value(params, "context_lines", 0));
|
||||
uint64_t offset = std::max<int64_t>(0, json_value(params, "offset", int64_t{0}));
|
||||
|
||||
std::string pattern_src = pat_str;
|
||||
if (literal) {
|
||||
@@ -1166,7 +1164,6 @@ struct server_tool_grep_search : server_tool {
|
||||
// collect (absolute_path, display_path) pairs to search
|
||||
std::vector<std::pair<std::string, std::string>> files;
|
||||
|
||||
bool listing_truncated = false;
|
||||
const std::string abs_path = io->resolve(path);
|
||||
if (io->is_regular_file(abs_path)) {
|
||||
files.emplace_back(abs_path, path);
|
||||
@@ -1175,7 +1172,6 @@ struct server_tool_grep_search : server_tool {
|
||||
if (!listing.err.empty()) {
|
||||
return {{"error", listing.err + ": " + path}};
|
||||
}
|
||||
listing_truncated = listing.truncated;
|
||||
for (const auto & entry : listing.entries) {
|
||||
if (!path_glob_match(include, entry.rel)) continue;
|
||||
if (!exclude.empty() && path_glob_match(exclude, entry.rel)) continue;
|
||||
@@ -1184,18 +1180,14 @@ struct server_tool_grep_search : server_tool {
|
||||
} else {
|
||||
return {{"error", "path does not exist: " + path}};
|
||||
}
|
||||
std::sort(files.begin(), files.end(), [](const auto & a, const auto & b) {
|
||||
return a.second < b.second;
|
||||
});
|
||||
|
||||
std::ostringstream output_text;
|
||||
uint64_t skipped = 0;
|
||||
size_t total = 0;
|
||||
bool has_more = false;
|
||||
bool limit_reached = false;
|
||||
bool show_num = show_lineno || ctx_lines > 0;
|
||||
|
||||
for (const auto & file_entry : files) {
|
||||
if (has_more) break;
|
||||
if (limit_reached) break;
|
||||
const std::string & fpath = file_entry.first;
|
||||
const std::string & display_path = file_entry.second;
|
||||
|
||||
@@ -1209,15 +1201,11 @@ struct server_tool_grep_search : server_tool {
|
||||
}
|
||||
|
||||
for (size_t i = 0; i < lines.size(); i++) {
|
||||
if (!std::regex_search(lines[i], pattern)) continue;
|
||||
if (skipped < offset) {
|
||||
skipped++;
|
||||
continue;
|
||||
}
|
||||
if (total >= SERVER_TOOL_GREP_SEARCH_MAX_RESULTS) {
|
||||
has_more = true;
|
||||
limit_reached = true;
|
||||
break;
|
||||
}
|
||||
if (!std::regex_search(lines[i], pattern)) continue;
|
||||
|
||||
long ctx_start = ctx_lines > 0 ? std::max<long>(0, (long) i - ctx_lines) : (long) i;
|
||||
long ctx_end = ctx_lines > 0 ? std::min<long>((long) lines.size() - 1, (long) i + ctx_lines) : (long) i;
|
||||
@@ -1238,11 +1226,10 @@ struct server_tool_grep_search : server_tool {
|
||||
}
|
||||
|
||||
output_text << "\n---\nTotal matches: " << total << "\n";
|
||||
if (has_more) {
|
||||
output_text << "Continue with offset=" << offset + total << " and the same search arguments.\n";
|
||||
}
|
||||
if (listing_truncated) {
|
||||
output_text << "[results truncated: time budget or unreadable directory]\n";
|
||||
if (limit_reached) {
|
||||
output_text << string_format(
|
||||
"[%zu matches limit reached. Narrow the path/pattern/include to see more.]\n",
|
||||
SERVER_TOOL_GREP_SEARCH_MAX_RESULTS);
|
||||
}
|
||||
|
||||
return {{"plain_text_response", output_text.str()}};
|
||||
|
||||
+65
-125
@@ -160,7 +160,6 @@ static server_role_type server_role_detect(const common_params & params, server_
|
||||
}
|
||||
// a server given no model is a router
|
||||
const bool has_model = !params.model.path.empty()
|
||||
|| !params.model.url.empty()
|
||||
|| !params.model.hf_repo.empty()
|
||||
|| !params.model.docker_repo.empty();
|
||||
return has_model ? SERVER_ROLE_INSTANCE : SERVER_ROLE_ROUTER;
|
||||
@@ -415,45 +414,66 @@ static bool server_on_listening(server_http_context & ctx_http, const common_par
|
||||
return true;
|
||||
}
|
||||
|
||||
// throttled download progress, shared by the download worker and single-model mode
|
||||
struct server_download_progress : public common_download_callback {
|
||||
// called under the lock, the files are downloaded in parallel threads
|
||||
std::function<void(const common_download_progress &)> on_progress;
|
||||
std::function<bool()> should_stop;
|
||||
// throttle each file on its own, a shared timer lets a small, fast file (e.g. a draft) starve the updates of the others
|
||||
std::mutex mutex;
|
||||
std::unordered_map<std::string, int64_t> last_progress_time;
|
||||
//
|
||||
// download worker: a child of the router that pulls a model into the cache and reports the progress
|
||||
//
|
||||
|
||||
void notify(const common_download_progress & p) {
|
||||
std::lock_guard<std::mutex> lk(mutex);
|
||||
on_progress(p);
|
||||
struct server_download_state : public common_download_callback {
|
||||
server_role * role;
|
||||
std::function<bool()> should_stop;
|
||||
// files download in parallel threads: throttle each file on its own, a shared
|
||||
// timer lets a small, fast file (e.g. a draft) starve the updates of the others
|
||||
std::mutex progress_mutex;
|
||||
std::unordered_map<std::string, int64_t> last_progress_time;
|
||||
bool is_ok = false;
|
||||
|
||||
server_download_state(server_role * r) : role(r) {}
|
||||
|
||||
bool run(common_params & params) {
|
||||
try {
|
||||
common_models_handler handler = common_models_handler_init(params, LLAMA_EXAMPLE_SERVER);
|
||||
common_models_handler_apply(handler, params, this);
|
||||
is_ok = true;
|
||||
} catch (const std::exception & e) {
|
||||
auto model_name = params.model.get_name();
|
||||
SRV_ERR("download failed for model name=%s: %s\n", model_name.c_str(), e.what());
|
||||
is_ok = false;
|
||||
}
|
||||
return is_ok;
|
||||
}
|
||||
void on_progress(const common_download_progress & p) {
|
||||
json data = {
|
||||
{"url", p.url},
|
||||
{"downloaded", p.downloaded},
|
||||
{"total", p.total},
|
||||
};
|
||||
role->notify_to_router(server_state_to_str(SERVER_STATE_DOWNLOADING), data);
|
||||
}
|
||||
void on_start(const common_download_progress & p) override {
|
||||
notify(p);
|
||||
on_progress(p);
|
||||
}
|
||||
void on_update(const common_download_progress & p) override {
|
||||
int64_t now = ggml_time_ms();
|
||||
// the first update of a file always passes, it carries the file's total size
|
||||
std::lock_guard<std::mutex> lk(mutex);
|
||||
auto it = last_progress_time.find(p.url);
|
||||
if (it != last_progress_time.end() && now - it->second < 100) {
|
||||
return;
|
||||
// throttle progress updates to avoid flooding logs; the first update of a file
|
||||
// always passes, it carries the file's total size
|
||||
{
|
||||
std::lock_guard<std::mutex> lk(progress_mutex);
|
||||
auto it = last_progress_time.find(p.url);
|
||||
if (it != last_progress_time.end() && now - it->second < 100) {
|
||||
return;
|
||||
}
|
||||
last_progress_time[p.url] = now;
|
||||
}
|
||||
last_progress_time[p.url] = now;
|
||||
on_progress(p);
|
||||
}
|
||||
void on_done(const common_download_progress & p, bool) override {
|
||||
notify(p);
|
||||
on_progress(p);
|
||||
}
|
||||
bool is_cancelled() const override {
|
||||
return should_stop ? should_stop() : false;
|
||||
}
|
||||
};
|
||||
|
||||
//
|
||||
// download worker: a child of the router that pulls a model into the cache and reports the progress
|
||||
//
|
||||
|
||||
static int server_run_download(common_params & params, server_role & role) {
|
||||
auto cancelled = std::make_shared<std::atomic<bool>>(false);
|
||||
|
||||
@@ -462,27 +482,12 @@ static int server_run_download(common_params & params, server_role & role) {
|
||||
cancelled->store(true, std::memory_order_relaxed);
|
||||
});
|
||||
|
||||
server_download_progress dl;
|
||||
dl.on_progress = [&role](const common_download_progress & p) {
|
||||
role.notify_to_router(server_state_to_str(SERVER_STATE_DOWNLOADING), {
|
||||
{"url", p.url},
|
||||
{"downloaded", p.downloaded},
|
||||
{"total", p.total},
|
||||
});
|
||||
};
|
||||
server_download_state dl(&role);
|
||||
dl.should_stop = [cancelled]() {
|
||||
return cancelled->load(std::memory_order_relaxed);
|
||||
};
|
||||
|
||||
bool ok = true;
|
||||
try {
|
||||
common_models_handler handler = common_models_handler_init(params, LLAMA_EXAMPLE_SERVER);
|
||||
common_models_handler_apply(handler, params, &dl);
|
||||
} catch (const std::exception & e) {
|
||||
auto model_name = params.model.get_name();
|
||||
SRV_ERR("download failed for model name=%s: %s\n", model_name.c_str(), e.what());
|
||||
ok = false;
|
||||
}
|
||||
bool ok = dl.run(params);
|
||||
|
||||
role.notify_to_router(server_state_to_str(SERVER_STATE_DOWNLOADING), {
|
||||
{"result", ok ? "download_finished" : "download_failed"},
|
||||
@@ -684,21 +689,15 @@ static int server_run_router(common_params & params, int argc, char ** argv, con
|
||||
//
|
||||
|
||||
static int server_run_instance(common_params & params, server_role & role, common_models_handler & models_handler, const server_run_opts & opts) {
|
||||
// a router child reports its state to the router, a single-model server to its own /models/sse clients
|
||||
const bool managed = role.is_managed_by_router();
|
||||
const std::string model_name = server_get_model_name(params);
|
||||
server_models_sse sse(/* replay_last */ true);
|
||||
std::atomic<bool> sse_stopping = false;
|
||||
|
||||
auto notify = [&](const std::string & event, const json & data = nullptr) {
|
||||
if (!managed) {
|
||||
sse.notify(event, model_name, data);
|
||||
// -hf / docker: download if needed and resolve the local path. a child resolves what the router downloaded
|
||||
if (opts.resolve_model) {
|
||||
try {
|
||||
common_models_handler_apply(models_handler, params);
|
||||
} catch (const std::exception & e) {
|
||||
SRV_ERR("failed to download model: %s\n", e.what());
|
||||
return 1;
|
||||
}
|
||||
};
|
||||
auto notify_status = [&](server_model_status status, json data = json::object()) {
|
||||
data["status"] = server_model_status_to_string(status);
|
||||
notify("status_change", data);
|
||||
};
|
||||
}
|
||||
|
||||
// note: this is guaranteed to out-live ctx_http and tools
|
||||
server_mcp mcp_mgr;
|
||||
@@ -719,82 +718,37 @@ static int server_run_instance(common_params & params, server_role & role, commo
|
||||
server_stream_make_get_handler(),
|
||||
server_stream_make_lookup_handler(),
|
||||
server_stream_make_delete_handler());
|
||||
if (!managed) {
|
||||
ctx_http.get("/models/sse", ex_wrapper(sse.make_handler([&sse_stopping]() {
|
||||
return sse_stopping.load(std::memory_order_relaxed);
|
||||
})));
|
||||
}
|
||||
|
||||
if (!server_setup_features(ctx_http, params, mcp_mgr, tools, SERVER_ROLE_INSTANCE)) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
// setup clean up function, to be called before exit
|
||||
auto clean_up = [&ctx_http, &ctx_server, &mcp_mgr, &sse_stopping]() {
|
||||
auto clean_up = [&ctx_http, &ctx_server, &mcp_mgr]() {
|
||||
SRV_INF("%s: cleaning up before exit...\n", __func__);
|
||||
// stop the session GC first, it finalizes live sessions and wakes pending readers
|
||||
server_stream_session_manager_stop();
|
||||
sse_stopping.store(true); // disconnect the SSE clients
|
||||
ctx_http.stop();
|
||||
ctx_server.terminate();
|
||||
mcp_mgr.shutdown();
|
||||
llama_backend_free();
|
||||
};
|
||||
|
||||
// start the HTTP server before downloading and loading the model to be able to serve /health and /models/sse requests
|
||||
// start the HTTP server before loading the model to be able to serve /health requests
|
||||
if (!ctx_http.start()) {
|
||||
clean_up();
|
||||
SRV_ERR("%s", "exiting due to HTTP server error\n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
// -hf / docker: download if needed and resolve the local path. a child resolves what the router downloaded
|
||||
if (opts.resolve_model) {
|
||||
json progress = json::object(); // url -> {done, total}, the same shape as the router sends
|
||||
server_download_progress dl;
|
||||
dl.on_progress = [&](const common_download_progress & p) {
|
||||
if (p.total == 0) {
|
||||
return; // not started yet, or found in the cache
|
||||
}
|
||||
progress[p.url] = {
|
||||
{"done", p.downloaded},
|
||||
{"total", p.total},
|
||||
};
|
||||
notify("download_progress", {{"progress", progress}});
|
||||
};
|
||||
try {
|
||||
common_models_handler_apply(models_handler, params, managed ? nullptr : &dl);
|
||||
if (!progress.empty()) {
|
||||
notify("download_finished");
|
||||
}
|
||||
} catch (const std::exception & e) {
|
||||
SRV_ERR("failed to download model: %s\n", e.what());
|
||||
notify("download_failed");
|
||||
notify_status(SERVER_MODEL_STATUS_UNLOADED, {{"exit_code", 1}});
|
||||
clean_up();
|
||||
ctx_http.join();
|
||||
return 1;
|
||||
}
|
||||
// setup communication child --> router if necessary
|
||||
if (role.is_managed_by_router()) {
|
||||
ctx_server.set_state_callback([&](server_state state, json payload) {
|
||||
role.notify_to_router(server_state_to_str(state), payload);
|
||||
});
|
||||
}
|
||||
|
||||
// setup communication child --> router, or to the SSE clients
|
||||
ctx_server.set_state_callback([&](server_state state, json payload) {
|
||||
if (managed) {
|
||||
role.notify_to_router(server_state_to_str(state), payload);
|
||||
return;
|
||||
}
|
||||
switch (state) {
|
||||
case SERVER_STATE_LOADING: notify_status(SERVER_MODEL_STATUS_LOADING, {{"progress", payload}}); break;
|
||||
case SERVER_STATE_READY: notify_status(SERVER_MODEL_STATUS_LOADED); break; // wakeup from sleeping
|
||||
case SERVER_STATE_SLEEPING: notify_status(SERVER_MODEL_STATUS_SLEEPING); break;
|
||||
default: break;
|
||||
}
|
||||
});
|
||||
|
||||
notify_status(SERVER_MODEL_STATUS_LOADING);
|
||||
|
||||
if (!ctx_server.load_model(params)) {
|
||||
notify_status(SERVER_MODEL_STATUS_UNLOADED, {{"exit_code", 1}});
|
||||
clean_up();
|
||||
ctx_http.join();
|
||||
SRV_ERR("%s", "exiting due to model loading error\n");
|
||||
@@ -806,22 +760,7 @@ static int server_run_instance(common_params & params, server_role & role, commo
|
||||
|
||||
SRV_INF("%s", "model loaded\n");
|
||||
|
||||
json info = routes.get_model_info();
|
||||
json memory;
|
||||
if (const auto * ll_ctx = ctx_server.get_llama_context()) {
|
||||
memory = server_model_memory(llama_n_ctx(ll_ctx), common_memory_breakdown_rows(ll_ctx)).to_json();
|
||||
}
|
||||
|
||||
{
|
||||
json data = {{"info", info}};
|
||||
if (!memory.is_null()) {
|
||||
data["memory"] = memory;
|
||||
}
|
||||
notify_status(SERVER_MODEL_STATUS_LOADED, data);
|
||||
}
|
||||
|
||||
shutdown_handler = [&](int) {
|
||||
sse_stopping.store(true);
|
||||
mcp_mgr.shutdown();
|
||||
// this will unblock start_loop()
|
||||
ctx_server.terminate();
|
||||
@@ -847,11 +786,12 @@ static int server_run_instance(common_params & params, server_role & role, commo
|
||||
|
||||
// optionally, notify router server that this instance is ready
|
||||
std::thread monitor_thread;
|
||||
if (managed) {
|
||||
if (role.is_managed_by_router()) {
|
||||
monitor_thread = role.setup(shutdown_handler);
|
||||
json info = routes.get_model_info();
|
||||
// the router tracks device memory from these reports
|
||||
if (!memory.is_null()) {
|
||||
info["memory"] = memory;
|
||||
if (const auto * ll_ctx = ctx_server.get_llama_context()) {
|
||||
info["memory"] = server_model_memory(llama_n_ctx(ll_ctx), common_memory_breakdown_rows(ll_ctx)).to_json();
|
||||
}
|
||||
role.notify_to_router(server_state_to_str(SERVER_STATE_READY), info);
|
||||
}
|
||||
|
||||
@@ -143,19 +143,3 @@ def test_server_model_aliases_and_tags():
|
||||
assert set(model["tags"]) == {"chat", "fim", "small"}
|
||||
# id is derived from first alias (alphabetical order from std::set)
|
||||
assert model["id"] == "code"
|
||||
|
||||
|
||||
def test_server_models_sse_single_model():
|
||||
global server
|
||||
server.start()
|
||||
# a client connecting after the load gets the last status replayed
|
||||
with requests.get(server.make_url("/models/sse"), stream=True, timeout=10) as res:
|
||||
assert res.status_code == 200
|
||||
assert res.headers["Content-Type"].startswith("text/event-stream")
|
||||
line = next(x for x in res.iter_lines() if x)
|
||||
event = json.loads(line.decode("utf-8").removeprefix("data: "))
|
||||
model_id = server.make_request("GET", "/models").body["data"][0]["id"]
|
||||
assert event["model"] == model_id
|
||||
assert event["event"] == "status_change"
|
||||
assert event["data"]["status"] == "loaded"
|
||||
assert event["data"]["info"]["id"] == model_id
|
||||
@@ -375,30 +375,6 @@ def test_systemone_d1_parallel():
|
||||
assert_answers_close(a[qid], b[qid], 1e-6)
|
||||
|
||||
|
||||
def test_systemone_noncausal_parallel():
|
||||
global server
|
||||
server = ServerPreset.tinydecision("vllm-d3")
|
||||
# n_batch > n_ubatch: each prompt fits in one ubatch, but several of them do not
|
||||
server.n_batch = 2048
|
||||
server.n_ubatch = 512
|
||||
server.n_slots = 4
|
||||
server.start()
|
||||
state = " ".join(f"The order {i} arrived damaged and the customer wants a replacement." for i in range(14))
|
||||
questions = {"angry": TEST_QUESTIONS["angry"]}
|
||||
|
||||
def decide():
|
||||
res = server.make_request("POST", "/v1/systemone", data={"state": state, "questions": questions})
|
||||
assert res.status_code == 200
|
||||
assert res.body["usage"]["input_tokens"] > 512 // 2
|
||||
return res.body["answers"]
|
||||
|
||||
# a non-causal batch is evaluated in one ubatch, the concurrent prompts are spread over several batches
|
||||
single = decide()
|
||||
parallel = parallel_function_calls([(decide, ()) for _ in range(4)])
|
||||
for answers in parallel:
|
||||
assert_answers_close(single["angry"], answers["angry"], 1e-6)
|
||||
|
||||
|
||||
def test_systemone_d1_special_tokens():
|
||||
global server
|
||||
server = ServerPreset.tinydecision("lfm3-d1-omni")
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
@@ -54,85 +53,6 @@ def test_tools_builtin_grep_search():
|
||||
assert "Total matches: 1" in text
|
||||
|
||||
|
||||
def _grep_matches(text: str) -> list[tuple[str, int, str]]:
|
||||
matches = []
|
||||
for line in text.splitlines():
|
||||
match = re.match(r"^(.*):(\d+):(match-\d+)$", line)
|
||||
if match:
|
||||
path, line_number, identity = match.groups()
|
||||
matches.append((path, int(line_number), identity))
|
||||
return matches
|
||||
|
||||
|
||||
def test_tools_builtin_grep_search_pagination(tmp_path):
|
||||
global server
|
||||
server.start()
|
||||
|
||||
path = tmp_path / "matches.txt"
|
||||
expected = [f"match-{i:03d}" for i in range(231)]
|
||||
path.write_text("\n".join(expected) + "\n")
|
||||
params = {"path": str(path), "pattern": "match-", "return_line_numbers": True}
|
||||
|
||||
pages = []
|
||||
next_offset = 0
|
||||
for page_index, expected_size in enumerate([100, 100, 31]):
|
||||
page_params = params if not pages else {**params, "offset": next_offset}
|
||||
text = call_tool("grep_search", page_params)["plain_text_response"]
|
||||
matches = _grep_matches(text)
|
||||
pages.append(matches)
|
||||
assert len(matches) == expected_size
|
||||
assert f"\n---\nTotal matches: {expected_size}\n" in text
|
||||
|
||||
continuation = re.search(r"Continue with offset=(\d+) and the same search arguments\.", text)
|
||||
if page_index < 2:
|
||||
assert continuation is not None
|
||||
next_offset = int(continuation.group(1))
|
||||
assert next_offset == (page_index + 1) * 100
|
||||
else:
|
||||
assert continuation is None
|
||||
|
||||
assert [identity for page in pages for _, _, identity in page] == expected
|
||||
|
||||
|
||||
def test_tools_builtin_grep_search_exact_page_has_no_continuation(tmp_path):
|
||||
global server
|
||||
server.start()
|
||||
|
||||
path = tmp_path / "matches.txt"
|
||||
path.write_text("".join(f"match-{i:03d}\n" for i in range(100)) + "unrelated\n")
|
||||
res = call_tool("grep_search", {"path": str(path), "pattern": "match-", "return_line_numbers": True})
|
||||
text = res["plain_text_response"]
|
||||
|
||||
assert len(_grep_matches(text)) == 100
|
||||
assert "Total matches: 100" in text
|
||||
assert "Continue with offset=" not in text
|
||||
assert "matches limit reached" not in text
|
||||
|
||||
|
||||
def test_tools_builtin_grep_search_page_order_with_context(tmp_path):
|
||||
global server
|
||||
server.start()
|
||||
|
||||
(tmp_path / "z.txt").write_text("".join(f"match-{i:03d}\n" for i in range(50)))
|
||||
(tmp_path / "a.txt").write_text("".join(f"match-{i:03d}\n" for i in range(50, 120)))
|
||||
params = {"path": str(tmp_path), "pattern": "match-", "context_lines": 1}
|
||||
|
||||
first = call_tool("grep_search", {**params, "offset": 0})["plain_text_response"]
|
||||
second = call_tool("grep_search", {**params, "offset": 100})["plain_text_response"]
|
||||
past_end = [call_tool("grep_search", {**params, "offset": offset})["plain_text_response"] for offset in (121, 1 << 32)]
|
||||
|
||||
expected = [("a.txt", line, f"match-{line + 49:03d}") for line in range(1, 71)]
|
||||
expected += [("z.txt", line, f"match-{line - 1:03d}") for line in range(1, 51)]
|
||||
assert _grep_matches(first) == expected[:100]
|
||||
assert _grep_matches(second) == expected[100:]
|
||||
assert "Continue with offset=100 and the same search arguments." in first
|
||||
assert "Continue with offset=" not in second
|
||||
for text in past_end:
|
||||
assert _grep_matches(text) == []
|
||||
assert "Total matches: 0" in text
|
||||
assert "Continue with offset=" not in text
|
||||
|
||||
|
||||
def test_tools_builtin_read_file(tmp_path):
|
||||
global server
|
||||
server.start()
|
||||
|
||||
Reference in new issue
Block a user