llama : report compute graph reallocations during inference (#30329)
Add an optional ggml_backend_sched_realloc_callback that user code registers with ggml_backend_sched_set_realloc_callback to be notified when the buffers reserved for a graph are not sufficient and have to be reserved again. The callback reports whether the backend assignments and the graph size changed since the previous scheduling. llama_context registers the callback when creating the scheduler (wrapped into the new llama_context::sched_new helper) and treats the reallocations that follow a change of the backend assignments as expected, counting the rest by graph size - unchanged or changed from the previously scheduled graph. Both counters are reported on one line in the perf stats. Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
This commit is contained in:
1 parent
2f160ba59a
commit
972b5fcc19
7 files changed
+92
-14
No files matched your search
@@ -693,6 +693,10 @@ struct server_slot {
|
||||
" graphs reused = %10d\n",
|
||||
llama_perf_context(ctx_tgt).n_reused);
|
||||
|
||||
SLT_INF(*this,
|
||||
" graph reallocs = %10d / %5d (same size) / (changed size)\n",
|
||||
llama_perf_context(ctx_tgt).n_reallocs_same, llama_perf_context(ctx_tgt).n_reallocs_diff);
|
||||
|
||||
const int32_t n_draft_total = stats.n_draft_tokens;
|
||||
const int32_t n_draft_accepted = stats.n_draft_accepted;
|
||||
const int32_t n_draft_verif_steps = stats.n_draft_verif_steps;
|
||||
|
||||
Reference in new issue
Block a user