kv-cache : keep rotation constants on the GPU (#31073)
* kv-cache : keep rotation constants on the GPU Assisted-by: Codex * tests : verify resident KV rotation buffers Assisted-by: Codex * kv-cache : document remaining multi-device rotation work Assisted-by: Codex * kv-cache : support per-layer rotation buffers Assisted-by: Codex * kv-cache : fix hybrid SWA rotation inputs and mixed placement check Assisted-by: Codex * models : use per-layer rotation in mistral-large4 and dflash * tests: drop the rotation buffer tests --------- Co-authored-by: Pascal <admin@serveurperso.com>
This commit is contained in:
1 parent
ab3768cdc2
commit
fb5997274e
15 files changed
+179
-246
No files matched your search
@@ -632,7 +632,7 @@ ggml_tensor * llama_model_deepseek41::graph::build_attention41(
|
||||
GGML_ASSERT(top_k && "DeepSeek-V4.1: top-k source layer produced no top-k");
|
||||
|
||||
// a quantized K cache stores the window Hadamard-rotated (as build_raw_attention)
|
||||
ggml_tensor * k_rot = inp_attn->self_k_rot;
|
||||
ggml_tensor * k_rot = inp_attn->self_k_rot[il];
|
||||
if (k_rot) {
|
||||
q = llama_mul_mat_hadamard(ctx0, q, k_rot);
|
||||
kv = llama_mul_mat_hadamard(ctx0, kv, k_rot);
|
||||
|
||||
Reference in new issue
Block a user