Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

common : fractional -ncmoe for tensor-granularity expert placement

Browse Source 
Extends --n-cpu-moe to accept a fractional layer count. The integer part
offloads whole layers as before; the fractional part offloads a subset of the
boundary layer expert tensors (gate, then up), keeping down_proj resident.
This realizes ATSInfer tensor-granularity static placement, giving sub-layer
control over expert VRAM residency. Placement-only, lossless.

Assisted-by: Claude
This commit is contained in:
Lumpiasty
2026-07-26 21:58:43 +02:00
parent 34ee143caa
commit a9c986086b
2 changed files with 29 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
common/common.h
+22
View File
@@ -1083,6 +1083,28 @@ inline llama_model_tensor_buft_override llm_ffn_exps_cpu_override() {
return { LLM_FFN_EXPS_REGEX, ggml_backend_cpu_buffer_type() };
}
// ATSInfer-style tensor-granularity static placement of MoE expert weights.
// Offloads the expert weights of the first floor(n) layers to the CPU, plus a
// subset of the boundary layer's three expert tensors for the fractional part.
// Expert tensors are dropped from the GPU in ascending performance-density
// order (gate, then up), keeping the higher-value down_proj resident longest.
inline std::vector<std::string> llm_ffn_exps_cpu_block_regexes(double n_cpu_moe) {
std::vector<std::string> regexes;
const int n_full = n_cpu_moe > 0 ? (int) n_cpu_moe : 0;
for (int i = 0; i < n_full; ++i) {
regexes.push_back(llm_ffn_exps_block_regex(i));
}
const int k = (int) ((n_cpu_moe - n_full) * 3.0 + 0.5);
if (k >= 3) {
regexes.push_back(llm_ffn_exps_block_regex(n_full));
} else if (k == 2) {
regexes.push_back(string_format("blk\\.%d\\.ffn_(gate|up)_(ch|)exps", n_full));
} else if (k == 1) {
regexes.push_back(string_format("blk\\.%d\\.ffn_gate_(ch|)exps", n_full));
}
return regexes;
}
//
// training utils
//