Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

server: add ctx-per-slot (--kv-unified-per-slot) (#24124)

Browse Source 
* Add ctx-per-slot argument for unifid KV cache

* Swap out ctx fractions for ctx pool slots

* Formatting cleanup

* Remove ctx-pool-slots, make ctx-per-slot an int

* refactor it

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
This commit is contained in:
Bartowski
2026-08-27 22:39:14 +02:00
committed by GitHub
co-authored by Xuan Son Nguyen
parent 32176338a6
commit 18443257a3
5 changed files with 59 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
common/arg.cpp
+8
View File
@@ -1643,6 +1643,14 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
}
}
).set_env("LLAMA_ARG_CTX_SIZE"));
add_opt(common_arg(
{ "--kv-unified-per-slot" }, "N",
"context limit per parallel slot (default: unset, behavior unchanged).\n"
"when set without -c/--ctx-size, the shared KV pool is sized to n_parallel*N",
[](common_params & params, int value) {
params.kv_unified_per_slot = value;
}
).set_env("LLAMA_ARG_KV_UNIFIED_PER_SLOT").set_examples({ LLAMA_EXAMPLE_SERVER }));
add_opt(common_arg(
{"-n", "--predict", "--n-predict"}, "N",
string_format(