Files
llama.cpp/common/speculative.h
T
Ruixiang Wang 32bdc3a89e spec + model: refactor DFlash based speculative decoding methods (#32016)
* dflash : move the DSpark markov head and backbone graphs to dspark.cpp

Move-only: build_dspark_markov_head, graph_dsv4, graph_dsv41 and
graph_mistral_large4 move verbatim. build_inp_target_features and
build_dspark_markov_head become static members of llama_model_dflash so
both files can use them.

* dflash : detect the head and the backbone once, split the loaders per file

load_arch_hparams sets llama_model_dflash::variant (DFlash, DFlash2,
DSpark, LiliCorr) and ::backbone (base, DSV4, DSV4.1, Mistral Large 4).
The head, backbone and base layer loaders become member helpers in
dflash.cpp, dflash2.cpp, lilicorr.cpp and dspark.cpp. build_arch_graph
dispatches on the backbone.

The DFlash2 and LiliCorr loaders create the DFlash2 conv tensors through
a helper in dflash2.cpp, right after their head tensors instead of in
the base layer loop; the other tensors keep their creation order. The
DFlash2 loader drops its copy of the conv group size check, which master
never reached.

* dflash : share the injection input, token embeddings and lm head lookups

build_inject_inp (target features + fc + encoder norm), get_tok_embd and
get_lm_head replace the copies in the base, DSV4, DSV4.1 and Mistral
Large 4 graphs. build_inp_target_features is file-static again.

* dflash : split the DFlash decoder into a graph per head

graph_base creates the shared inputs and provides build_inject,
build_inp_tokens, build_layers and build_logits. graph_dflash
(dflash.cpp), graph_dflash2 (dflash2.cpp), graph_dspark (dspark.cpp) and
graph_lilicorr (lilicorr.cpp) assemble them and add their head; the
selector, scorer and markov head builders move to the file of their
graph, and the DFlash2 local conv run by build_layers to dflash2.cpp.
build_arch_graph dispatches on the backbone, then on the head.

The op order and the tensor names are unchanged.

* speculative : split the DFlash driver into a class per head

common_speculative_impl_draft_dflash stays the DFlash implementation and
becomes the base of draft_dspark, draft_dflash2 and draft_lilicorr. Its
protected constructor only reads the draft metadata; init() then sets up
the contexts with the options of the draft (draft length, backend
sampling, device features, masked h_nextn). Each derived class overrides
draft() with the path of its head from the former draft(). DFlash2 also
overrides begin() to reseed its generator per request, and LiliCorr
overrides process() to store the context features of each injected
chunk, which its noise block carries as its state.

The factory keeps the current routing: draft-dspark picks DSpark,
draft-dflash picks DFlash2 or LiliCorr from the draft metadata and
DFlash otherwise. draft-dspark with a DFlash2 or LiliCorr draft now runs
the DSpark reader instead of a mix of both. sample_from_anchor is logged
by DSpark only.

* speculative : add the draft-dflash2 and draft-lilicorr types

draft-dflash2 and draft-lilicorr name the DFlash2 and LiliCorr drafts,
which draft-dflash keeps running too: it picks them from the draft
metadata. draft-dflash2 and draft-lilicorr must match the draft model,
draft-dspark rejects a DFlash2 or LiliCorr draft, a model that is not a
DFlash draft is rejected, and a run uses at most one DFlash family type;
these errors leave speculative decoding uninitialized.

The type detected from the draft GGUF is now the concrete one, and
draft-dflash2 and draft-lilicorr download the dflash sidecar.

* docs : describe the DFlash2 and LiliCorr drafts

Add the DFlash2 and LiliCorr sections and the new types, and explain
that draft-dflash also runs DFlash2 and LiliCorr drafts.

* speculative : drop the unused DFlash is_mrope

Nothing reads it since the migration to common_batch (#29385), which
picks the number of M-RoPE position rows from the rope type of the draft.

* doc: update spec dec

* address review comments
2026-11-14 09:49:27 +01:00

126 lines
5.2 KiB
C++

#pragma once
#include "llama.h"
#include "common.h"
struct common_speculative;
// comma separated list the provided types
std::string common_speculative_type_name_str(const std::vector<enum common_speculative_type> & types);
// comma separated list of all types
const char * common_speculative_all_types_str();
// parse user provided types
std::vector<enum common_speculative_type> common_speculative_types_from_names(const std::vector<std::string> & names);
// infer the spec types from the GGUF metadata of a draft model; empty if unknown
std::vector<enum common_speculative_type> common_speculative_types_from_gguf(const std::string & path);
// convert string to type
enum common_speculative_type common_speculative_type_from_name(const std::string & name);
// convert type to string
std::string common_speculative_type_to_str(enum common_speculative_type type);
// return the max number of draft tokens based on the speculative parameters
int32_t common_speculative_n_max(const common_params_speculative * spec);
// return the max number of draft tokens from the initialized implementations
int32_t common_speculative_n_max(const common_speculative * spec);
// validate and resolve the unconditional synthetic acceptance rates
std::vector<double> common_speculative_synth_rates_resolve(const common_params_speculative * spec, int32_t n_max);
// return the conditional synthetic acceptance probabilities
const std::vector<double> & common_speculative_get_synth_probs(const common_speculative * spec);
common_params common_base_params_to_speculative(const common_params & params);
struct common_speculative_output_limits {
int32_t total;
int32_t per_seq;
};
// return the output limits needed for speculative decoding
common_speculative_output_limits common_speculative_get_output_limits(
int32_t n_batch, int32_t n_parallel, int32_t n_draft);
// return true if the target and draft models have compatible vocabs
bool common_speculative_are_compatible(const llama_model * model_tgt, const llama_model * model_dft);
common_speculative * common_speculative_init(common_params_speculative & params, uint32_t n_seq);
void common_speculative_free(common_speculative * spec);
struct common_speculative_draft_params {
// this flag is used to chain the drafts through all the available implementations
// after the first successful draft from an implementation, we set it
// to false to prevent further drafts for that sequence
// at the end of the draft() call, all drafting flags will be reset to false
bool drafting = false;
// overrides individual configurations (-1 disabled)
// can be used to constraint the max draft based on the remaining context size
int32_t n_max = -1;
llama_pos pos0;
llama_token id_last;
// TODO: remove in the future by keeping track of the prompt from the _begin() call and the consecutive accept calls
const llama_tokens * prompt;
// the generated draft from the last _draft() call
llama_tokens * result;
// candidate distribution per drafted token; set it to make draft-simple, draft-mtp and draft-dflash2 sample
std::vector<std::vector<llama_token_data>> * result_q = nullptr;
// the target's temp and seed, read only when the drafter samples probabilistically
float temp = 1.0f;
uint32_t seed = LLAMA_DEFAULT_SEED;
};
common_speculative_draft_params & common_speculative_get_draft_params(common_speculative * spec, llama_seq_id seq_id);
// optionally call once at the beginning of a new generation
void common_speculative_begin(common_speculative * spec, llama_seq_id seq_id, const llama_tokens & prompt);
// process the batch and update the internal state of the speculative context
bool common_speculative_process(common_speculative * spec, const common_batch & batch);
// generate drafts for the sequences specified with `common_speculative_get_draft_params`
void common_speculative_draft(common_speculative * spec);
// informs the speculative context that n_accepted tokens were accepted by the target model
void common_speculative_accept(common_speculative * spec, llama_seq_id, uint16_t n_accepted);
// (optional) get/set internal state
bool common_speculative_get_state(common_speculative * spec, llama_seq_id seq_id, std::vector<uint8_t> & data);
void common_speculative_set_state(common_speculative * spec, llama_seq_id seq_id, const std::vector<uint8_t> & data);
// print statistics about the speculative decoding
void common_speculative_print_stats(const common_speculative * spec);
struct common_speculative_deleter {
void operator()(common_speculative * s) { common_speculative_free(s); }
};
typedef std::unique_ptr<common_speculative, common_speculative_deleter> common_speculative_ptr;
struct common_speculative_init_result {
common_speculative_init_result(common_params & params, llama_model * model_tgt, llama_context * ctx_tgt);
~common_speculative_init_result();
llama_model * model();
llama_context * context();
private:
struct impl;
std::unique_ptr<impl> pimpl;
};
using common_speculative_init_result_ptr = std::unique_ptr<common_speculative_init_result>;
common_speculative_init_result_ptr common_speculative_init_from_params(common_params & params, llama_model * model_tgt, llama_context * ctx_tgt);