kv-cache : keep rotation constants on the GPU (#31073)

* kv-cache : keep rotation constants on the GPU

Assisted-by: Codex

* tests : verify resident KV rotation buffers

Assisted-by: Codex

* kv-cache : document remaining multi-device rotation work

Assisted-by: Codex

* kv-cache : support per-layer rotation buffers

Assisted-by: Codex

* kv-cache : fix hybrid SWA rotation inputs and mixed placement check

Assisted-by: Codex

* models : use per-layer rotation in mistral-large4 and dflash

* tests: drop the rotation buffer tests

---------

Co-authored-by: Pascal <admin@serveurperso.com>
This commit is contained in:
Anton IandPascal authored and GitHub committed 2026-10-30 10:15:06 +02:00
1 parent ab3768cdc2
commit fb5997274e
15 files changed
+179 -246

No files matched your search

+2 -2
View File
@@ -243,9 +243,9 @@ llama_model_deepseek32::graph::graph(const llama_model & model, const llm_graph_
cb(indexer_k, "indexer_k", il);
// perform Hadamard transform on indexer q and k
indexer_q = ggml_mul_mat(ctx0, inp_attn_dsa->self_k_rot_lid, indexer_q);
indexer_q = ggml_mul_mat(ctx0, inp_attn_dsa->self_k_rot_lid[il], indexer_q);
cb(indexer_q, "indexer_q", il);
indexer_k = ggml_mul_mat(ctx0, inp_attn_dsa->self_k_rot_lid, indexer_k);
indexer_k = ggml_mul_mat(ctx0, inp_attn_dsa->self_k_rot_lid[il], indexer_k);
cb(indexer_k, "indexer_k", il);
// store indexer keys to KV cache