Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

server: add ctx-per-slot (--kv-unified-per-slot) (#24124)

Browse Source 
* Add ctx-per-slot argument for unifid KV cache

* Swap out ctx fractions for ctx pool slots

* Formatting cleanup

* Remove ctx-pool-slots, make ctx-per-slot an int

* refactor it

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
This commit is contained in:
Bartowski
2026-08-27 22:39:14 +02:00
committed by GitHub
co-authored by Xuan Son Nguyen
parent 32176338a6
commit 18443257a3
5 changed files with 59 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
common/common.h
+1
View File
@@ -627,6 +627,7 @@ struct common_params {
bool cache_prompt = true; // whether to enable prompt caching
bool cache_idle_slots = true; // save and clear idle slots upon starting a new task
int32_t n_ctx_checkpoints = 32; // max number of context checkpoints per slot
int32_t kv_unified_per_slot = 0; // max context per parallel slot; 0 = unset
int32_t checkpoint_min_step = 8192; // minimum spacing between context checkpoints
int32_t cache_ram_mib = 8192; // -1 = no limit, 0 - disable, 1 = 1 MiB, etc.