model: support speculators-format checkpoints for DSpark (#26275)
* dspark: support speculators-format checkpoints (SpecForge exports) Speculators-format DSpark drafts (e.g. SpecForge exports for the Gemma-4-26B-A4B target) differ from the dense DeepSpec checkpoints in three ways: - the config nests the backbone hparams under transformer_layer_config and gives the extract layers as aux_hidden_state_layer_ids - the block is the DFlash 1+N fill-in layout: the anchor slot is a bonus token, not a prediction slot. Written as dflash.bonus_anchor; such drafts build the block and read the mask positions exactly like DFlash (n_max drafts from a 1+n_max block), only the Markov/confidence sampling comes from DSpark - the draft output vocab may be reduced (draft_vocab_size < vocab_size) with a d2t remap table. The converter expands lm_head/markov_w2 back to the full vocab and synthesizes an lm_head bias of -1e9 on the rows the draft cannot produce, so the runtime needs no d2t remapping. Such drafts ship their own (now optional) token_embd/output tensors instead of sharing the target's Verified against gemma4-26b-a4b-dspark: greedy outputs are byte-identical with and without the draft; acceptance 0.46, mean draft len 3.7 (n_max 6). Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable 5 * dspark: fold the speculators draft class into DSparkModel One class now covers every DSpark variant. What used to pick the class is a single flag, because the arch name turns out to be the only thing that separates the two families: SpecForge also exports a flat schema that carries no speculators_* fields yet still uses the 1+N bonus-anchor block, so keying on those fields would silently mis-read its drafts. Also rename i0 to i_first_pred in the draft read loop and the Markov head, and give the head a real bonus_anchor bool instead of testing i0 > 0. Converting the Qwen3-8B DeepSpec draft and both gemma-4 speculators drafts produces byte-identical GGUFs. The one behaviour change is that the markov_head_type check now also covers the DeepSpec checkpoints, which previously skipped it. Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Opus 5 * dspark: address review comments - rename bonus_anchor to sample_from_anchor (GGUF key and code), matching the checkpoint config field; absent key still means anchor-first - rework the reduced draft vocab to match EAGLE3: d2t is written as I64 absolute target ids and the logits are scattered at runtime, instead of expanding lm_head/markov_w2 and synthesizing an output bias at conversion - move the t2d skip to modify_tensors, like EAGLE3 - drop _is_specforge: the arch name only picks the sample_from_anchor default, embed/lm_head sharing is decided by the draft vocab size - deduplicate the tok_embd create_tensor left behind by the rebase Verified with the RedHat gemma-4-31b speculator draft: greedy output is byte-identical with and without the draft; acceptance 0.26 (n_max 7). Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable 5 * dspark: fold the sample_from_anchor read into the block_size block * dspark: fix flake8 continuation indent * clean up * dspark: key the sample_from_anchor default off the export format Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable * dspark: drop t2d in filter_tensors Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable * dspark: map model.lm_head instead of bypassing the dflash prefix Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable --------- Co-authored-by: desovo7 <942845546@qq.com> Co-authored-by: ruixiang63 <wangruixiang07@outlook.com>
This commit is contained in:
co-authored by
desovo7
ruixiang63
parent
7077abbe14
commit
9cd719af21
+15
-8
@@ -926,6 +926,9 @@ struct common_speculative_impl_draft_dflash : public common_speculative_impl {
|
||||
// draft-dspark: the draft carries a Markov head and uses an anchor-first block layout
|
||||
const bool is_dspark;
|
||||
|
||||
// dspark speculators
|
||||
bool sample_from_anchor = true;
|
||||
|
||||
const int32_t * target_layer_ids = nullptr; // model_dft's extract layer indices
|
||||
uint32_t target_layer_ids_n = 0;
|
||||
|
||||
@@ -960,16 +963,20 @@ struct common_speculative_impl_draft_dflash : public common_speculative_impl {
|
||||
if (llama_model_meta_val_str(model_dft, "dflash.block_size", buf, sizeof(buf)) >= 0) {
|
||||
block_size = std::atoi(buf);
|
||||
}
|
||||
if (llama_model_meta_val_str(model_dft, "dflash.sample_from_anchor", buf, sizeof(buf)) >= 0) {
|
||||
sample_from_anchor = std::strcmp(buf, "true") == 0;
|
||||
}
|
||||
}
|
||||
mask_token_id = llama_vocab_mask(llama_model_get_vocab(model_dft));
|
||||
|
||||
LOG_INF("%s: adding speculative implementation '%s'\n", __func__, common_speculative_type_to_str(type).c_str());
|
||||
LOG_INF("%s: - n_max=%d, n_min=%d, p_min=%.2f\n", __func__, this->params.n_max, this->params.n_min, this->params.p_min);
|
||||
LOG_INF("%s: - block_size=%d, mask_token_id=%d, n_extract=%u\n", __func__, block_size, mask_token_id, target_layer_ids_n);
|
||||
LOG_INF("%s: - block_size=%d, mask_token_id=%d, n_extract=%u, sample_from_anchor=%s\n", __func__,
|
||||
block_size, mask_token_id, target_layer_ids_n, sample_from_anchor ? "true" : "false");
|
||||
|
||||
// DFlash input is [id_last, <mask> * (block_size-1)]: in-place denoising yields at most
|
||||
// block_size-1 draft tokens, DSpark yield a full block_size draft tokens
|
||||
const int32_t n_draft_max = is_dspark ? block_size : block_size - 1;
|
||||
// block_size-1 draft tokens, anchor-first DSpark yields a full block_size draft tokens
|
||||
const int32_t n_draft_max = is_dspark && sample_from_anchor ? block_size : block_size - 1;
|
||||
if (this->params.n_max > n_draft_max || this->params.n_min > n_draft_max) {
|
||||
LOG_WRN("%s: requested draft size (n_max=%d, n_min=%d) exceeds the trained block size %d -- clamping to %d\n",
|
||||
__func__, this->params.n_max, this->params.n_min, block_size, n_draft_max);
|
||||
@@ -1175,7 +1182,7 @@ struct common_speculative_impl_draft_dflash : public common_speculative_impl {
|
||||
|
||||
const int32_t n_draft = params.n_max;
|
||||
|
||||
const int32_t n_block_tokens = n_draft + (is_dspark ? 0 : 1);
|
||||
const int32_t n_block_tokens = n_draft + (is_dspark && sample_from_anchor ? 0 : 1);
|
||||
i_block_beg[seq_id] = batch.n_tokens;
|
||||
n_block [seq_id] = n_block_tokens;
|
||||
for (int32_t i = 0; i < n_block_tokens; ++i) {
|
||||
@@ -1208,11 +1215,11 @@ struct common_speculative_impl_draft_dflash : public common_speculative_impl {
|
||||
auto & result = *dp.result;
|
||||
|
||||
if (is_dspark) {
|
||||
// DSpark predicts the next token from position 0 and optionally truncates
|
||||
// at the first position below the confidence threshold.
|
||||
// DSpark: read from the first draft slot, truncate below the confidence threshold
|
||||
const float * conf = params.p_min > 0.0f ? llama_get_embeddings_nextn(ctx_dft) : nullptr;
|
||||
|
||||
for (int32_t i = 0; i < n_block_tokens; ++i) {
|
||||
// bonus-anchor drafts read the mask positions only, like DFlash
|
||||
const int32_t i_draft_beg = sample_from_anchor ? 0 : 1;
|
||||
for (int32_t i = i_draft_beg; i < n_block_tokens; ++i) {
|
||||
const int32_t idx = beg + i;
|
||||
|
||||
if (conf && conf[(size_t) idx * n_embd_dec] < params.p_min) {
|
||||
|
||||
Reference in New Issue
Block a user