server: fix the off-by-two in the SWA reuse threshold (#30580)
The first evaluated token attends to the positions above pos - n_swa, so a SWA state starting at pos - n_swa + 1 holds the whole window, but the reuse threshold asked for two more cells. A saved SWA state keeps only the cells inside its window, so a state read back from a slot file or from the RAM prompt cache was always rejected and the slot rolled back to a checkpoint. The SWA threshold now matches the window, while a state without SWA keeps the exact bound. The test harness gains override_kv, used to give the SWA test model a window smaller than the test context. [no release]
This commit is contained in:
1 parent
a6b91d8c5e
commit
61ad80bf93
3 files changed
+95
-2
No files matched your search
@@ -3784,8 +3784,12 @@ private:
|
||||
// ref: https://github.com/ggml-org/llama.cpp/pull/24110
|
||||
const bool has_new_tokens = (n_past < slot.task->n_tokens());
|
||||
|
||||
// the largest pos_min required for a checkpoint to be useful
|
||||
const auto pos_min_thold = std::max(0, pos_next - n_swa - (has_new_tokens ? 0 : 1));
|
||||
// the first evaluated token attends to the positions > pos_eval - n_swa, so a SWA state
|
||||
// starting at pos_eval - n_swa + 1 is enough, while a state without SWA must end before pos_eval
|
||||
const llama_pos pos_eval = pos_next - (has_new_tokens ? 0 : 1);
|
||||
|
||||
// a state or a checkpoint is useful when its pos_min is below this threshold
|
||||
const auto pos_min_thold = std::max(0, n_swa > 0 ? pos_eval - n_swa + 2 : pos_eval);
|
||||
|
||||
if (n_past > 0 && n_past <= slot.prompt.n_tokens()) {
|
||||
const auto pos_min = llama_memory_seq_pos_min(llama_get_memory(ctx_tgt), slot.id);
|
||||
|
||||
Reference in new issue
Block a user