kv-cache : keep rotation constants on the GPU (#31073)
* kv-cache : keep rotation constants on the GPU Assisted-by: Codex * tests : verify resident KV rotation buffers Assisted-by: Codex * kv-cache : document remaining multi-device rotation work Assisted-by: Codex * kv-cache : support per-layer rotation buffers Assisted-by: Codex * kv-cache : fix hybrid SWA rotation inputs and mixed placement check Assisted-by: Codex * models : use per-layer rotation in mistral-large4 and dflash * tests: drop the rotation buffer tests --------- Co-authored-by: Pascal <admin@serveurperso.com>
This commit is contained in:
1 parent
ab3768cdc2
commit
fb5997274e
15 files changed
+179
-246
No files matched your search
@@ -230,9 +230,9 @@ llama_model_dots3note::graph::graph(const llama_model & model, const llm_graph_p
|
||||
cb(indexer_k, "indexer_k", il);
|
||||
|
||||
// perform Hadamard transform on indexer q and k
|
||||
indexer_q = ggml_mul_mat(ctx0, inp_attn->get_dsa()->self_k_rot_lid, indexer_q);
|
||||
indexer_q = ggml_mul_mat(ctx0, inp_attn->get_dsa()->self_k_rot_lid[il], indexer_q);
|
||||
cb(indexer_q, "indexer_q", il);
|
||||
indexer_k = ggml_mul_mat(ctx0, inp_attn->get_dsa()->self_k_rot_lid, indexer_k);
|
||||
indexer_k = ggml_mul_mat(ctx0, inp_attn->get_dsa()->self_k_rot_lid[il], indexer_k);
|
||||
cb(indexer_k, "indexer_k", il);
|
||||
|
||||
// store indexer keys to KV cache
|
||||
|
||||
Reference in new issue
Block a user