common : add --spec-draft-n-ubatch (#30459)
* common : add --spec-draft-n-ubatch Allow configuring the physical batch size of the draft context. The drafters feed it at most the drafted tokens per sequence and chunk larger batches by its own n_ubatch, so it does not need the ubatch of the target. A smaller value shrinks the draft compute buffer and leaves more VRAM for the target KV cache (the draft is fitted together with the target). The default 0 keeps the current behavior of following --n-ubatch. Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next * cont : remove comment [no ci]
This commit is contained in:
1 parent
318a471a62
commit
3fb47677ea
9 files changed
+63
-3
No files matched your search
@@ -4173,6 +4173,16 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
|
||||
params.speculative.draft.n_min = value;
|
||||
}
|
||||
).set_spec().set_examples({LLAMA_EXAMPLE_SPECULATIVE, LLAMA_EXAMPLE_LOOKUP, LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_CLI}).set_env("LLAMA_ARG_SPEC_DRAFT_N_MIN"));
|
||||
add_opt(common_arg(
|
||||
{"--spec-draft-n-ubatch"}, "N",
|
||||
string_format("physical maximum batch size of the draft context, use a smaller value to spare VRAM (default: %d - same as --n-ubatch)", params.speculative.draft.n_ubatch),
|
||||
[](common_params & params, int value) {
|
||||
if (value < 0) {
|
||||
throw std::invalid_argument("invalid value");
|
||||
}
|
||||
params.speculative.draft.n_ubatch = value;
|
||||
}
|
||||
).set_spec().set_examples({LLAMA_EXAMPLE_SPECULATIVE, LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_CLI}).set_env("LLAMA_ARG_SPEC_DRAFT_N_UBATCH"));
|
||||
add_opt(common_arg(
|
||||
{"--spec-synth-len"}, "L",
|
||||
"target mean synthetic acceptance length, including the target token (benchmarking only)",
|
||||
|
||||
Reference in new issue
Block a user