common : add --spec-draft-n-ubatch (#30459)

* common : add --spec-draft-n-ubatch

Allow configuring the physical batch size of the draft context. The
drafters feed it at most the drafted tokens per sequence and chunk larger
batches by its own n_ubatch, so it does not need the ubatch of the target.
A smaller value shrinks the draft compute buffer and leaves more VRAM for
the target KV cache (the draft is fitted together with the target).

The default 0 keeps the current behavior of following --n-ubatch.

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next

* cont : remove comment

[no ci]
This commit is contained in:
Georgi Gerganov authored and GitHub committed 2026-10-14 13:58:05 +03:00
1 parent 318a471a62
commit 3fb47677ea
9 files changed
+63 -3

No files matched your search

+10
View File
@@ -4173,6 +4173,16 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
params.speculative.draft.n_min = value;
}
).set_spec().set_examples({LLAMA_EXAMPLE_SPECULATIVE, LLAMA_EXAMPLE_LOOKUP, LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_CLI}).set_env("LLAMA_ARG_SPEC_DRAFT_N_MIN"));
add_opt(common_arg(
{"--spec-draft-n-ubatch"}, "N",
string_format("physical maximum batch size of the draft context, use a smaller value to spare VRAM (default: %d - same as --n-ubatch)", params.speculative.draft.n_ubatch),
[](common_params & params, int value) {
if (value < 0) {
throw std::invalid_argument("invalid value");
}
params.speculative.draft.n_ubatch = value;
}
).set_spec().set_examples({LLAMA_EXAMPLE_SPECULATIVE, LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_CLI}).set_env("LLAMA_ARG_SPEC_DRAFT_N_UBATCH"));
add_opt(common_arg(
{"--spec-synth-len"}, "L",
"target mean synthetic acceptance length, including the target token (benchmarking only)",