llama : report compute graph reallocations during inference (#30329)

Add an optional ggml_backend_sched_realloc_callback that user code registers with
ggml_backend_sched_set_realloc_callback to be notified when the buffers reserved for a
graph are not sufficient and have to be reserved again. The callback reports whether the
backend assignments and the graph size changed since the previous scheduling.

llama_context registers the callback when creating the scheduler (wrapped into the new
llama_context::sched_new helper) and treats the reallocations that follow a change of the
backend assignments as expected, counting the rest by graph size - unchanged or changed
from the previously scheduled graph. Both counters are reported on one line in the perf
stats.

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
This commit is contained in:
Georgi Gerganov authored and GitHub committed 2026-10-12 11:53:58 +03:00
1 parent 2f160ba59a
commit 972b5fcc19
7 files changed
+92 -14

No files matched your search

+4
View File
@@ -693,6 +693,10 @@ struct server_slot {
" graphs reused = %10d\n",
llama_perf_context(ctx_tgt).n_reused);
SLT_INF(*this,
" graph reallocs = %10d / %5d (same size) / (changed size)\n",
llama_perf_context(ctx_tgt).n_reallocs_same, llama_perf_context(ctx_tgt).n_reallocs_diff);
const int32_t n_draft_total = stats.n_draft_tokens;
const int32_t n_draft_accepted = stats.n_draft_accepted;
const int32_t n_draft_verif_steps = stats.n_draft_verif_steps;