model: Add support for SenseNova U1 model (#28919)

* model : add SenseNova U1 understanding support

Add support for the dense SenseNova U1 understanding branch, including
the U1.5-8B-MoT model.

- Add HF-to-GGUF conversion for NEOChatModel, excluding MoT generation
  tensors and concatenating temporal and spatial Q/K normalization
  weights.
- Register the sensenova_u1 GGUF architecture and custom spatial RoPE
  metadata.
- Implement temporal/spatial Q/K normalization with 4-axis RoPE.
- Keep U1-specific text position handling in the model implementation.
- Add synthetic architecture coverage for sensenova_u1.

Only the dense understanding branch is supported; generation
branches are intentionally rejected during conversion.

* conversion : treat one-expert SenseNova U1 configs as dense

* mtmd : add SenseNova U1 vision support

Add conversion support for the SenseNova U1 vision encoder and implement
its CPU mtmd projector graph.

The graph applies patch embedding, GELU, adjacent-pair 2D RoPE, spatial
downsampling, and dense projection for dynamic-resolution image inputs.

* mtmd : fix SenseNova U1 vision token layout

* model : clean up SenseNova U1 RoPE metadata

Replace the hard-coded spatial RoPE key with shared GGUF and LLM_KV
definitions used by conversion, model loading, and architecture tests.

Clarify why SenseNova U1 retains its custom text-position input: U1 uses
[t, 0, 0, 0], while the generic M-RoPE input uses [t, t, t, 0].

* mtmd : simplify SenseNova U1 projector integration

Map the SenseNova U1 dense projection to the standard mm.0 weight and
bias tensors and reuse the shared projector fields in the loader and graph.

Read vision parameters from the checkpoint configuration and remove the
U1-specific tensor names and model fields while preserving its strided
projection, bias, and row-major token layout.

* mtmd : fix SenseNova U1 image positions and attention

* fix spacing

* mtmd : restore generic test input chunks

* mtmd : restore SenseNova U1 integration after upstream merge

* Update conversion/sensenova_u1.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* Update src/models/sensenova-u1.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

* model : remove SenseNova U1 BF16 activation casts

Use standard backend precision handling for QKV, FFN, and output
projections instead of casting inputs based on the output tensor type.

Assisted-by: Codex

* Update src/models/sensenova-u1.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
This commit is contained in:
Maphist0andSigbjørn Skjæret authored and GitHub committed 2026-10-21 21:54:16 +02:00
1 parent d454a204ef
commit 7775ffb7e4
18 files changed
+470 -2

No files matched your search

+23 -1
View File
@@ -225,6 +225,7 @@ enum mtmd_pos_type {
MTMD_POS_TYPE_NORMAL, // number of positions equals to number of tokens
MTMD_POS_TYPE_MROPE, // qwen-vl mrope style, each image takes max(t,h,w) position indexes
MTMD_POS_TYPE_HUNYUANVL, // HunyuanVL mrope + BOI/EOI/newline layout with XD-RoPE dim-3
MTMD_POS_TYPE_SENSENOVA_U1, // U1 uses zero-based H/W and one temporal position per image
MTMD_POS_TYPE_CANVAS, // MiniCPM-V 4.7: overview + slices in one chunk, sharing one 2D canvas (see mtmd_image_tokens::canvas_tile_grid)
MTMD_POS_TYPE_COUNT, // for validation
};
@@ -751,6 +752,12 @@ struct mtmd_context {
img_end = "]<]end of image[>[";
image_preproc = std::make_unique<mtmd_image_preprocessor_dyn_size>(ctx_v);
} break;
case PROJECTOR_TYPE_SENSENOVA_U1:
{
img_beg = "<img>";
img_end = "</img>";
image_preproc = std::make_unique<mtmd_image_preprocessor_dyn_size>(ctx_v);
} break;
case PROJECTOR_TYPE_MUSE_GLIMMER:
{
img_beg = "<|image_start|>";
@@ -1615,10 +1622,15 @@ struct mtmd_tokenizer {
image_tokens->ny = 1;
}
image_tokens->pos = ctx->pos_type;
// U1 uses M-RoPE, but unlike Qwen its image H/W coordinates are
// zero-based and the whole image occupies one temporal position.
if (ctx->proj_type_v() == PROJECTOR_TYPE_SENSENOVA_U1) {
image_tokens->pos = MTMD_POS_TYPE_SENSENOVA_U1;
}
// HunyuanVL wraps the image grid with BOI/EOI and adds one newline per row,
// and uses XD-RoPE (dim-3 = image index). Override the position type so that
// n_tokens() and mtmd_image_tokens_get_decoder_pos pick the HunyuanVL layout.
if (ctx->proj_type_v() == PROJECTOR_TYPE_HUNYUANVL) {
else if (ctx->proj_type_v() == PROJECTOR_TYPE_HUNYUANVL) {
image_tokens->pos = MTMD_POS_TYPE_HUNYUANVL;
image_tokens->image_idx = n_images_added;
GGML_ASSERT(n_tokens == (size_t)image_tokens->n_tokens());
@@ -2398,6 +2410,7 @@ bool mtmd_decode_use_non_causal(const mtmd_context * ctx, const mtmd_input_chunk
}
case PROJECTOR_TYPE_GEMMA4UV:
case PROJECTOR_TYPE_GEMMA3:
case PROJECTOR_TYPE_SENSENOVA_U1:
case PROJECTOR_TYPE_DEEPSEEK4V:
return true;
default:
@@ -2761,6 +2774,13 @@ mtmd_decoder_pos mtmd_image_tokens_get_decoder_pos(const mtmd_image_tokens * ima
pos.y = pos_0 + i;
pos.z = pos_0 + i;
} break;
case MTMD_POS_TYPE_SENSENOVA_U1:
{
pos.t = pos_0;
pos.x = i % image_tokens->nx;
pos.y = i / image_tokens->nx;
pos.z = 0;
} break;
case MTMD_POS_TYPE_HUNYUANVL:
{
// HunyuanVL layout: [BOI] [row0 tokens + newline] ... [row(ny-1) tokens + newline] [EOI]
@@ -2813,6 +2833,8 @@ llama_pos mtmd_image_tokens_get_n_pos(const mtmd_image_tokens * image_tokens) {
return std::max(image_tokens->nx, image_tokens->ny);
case MTMD_POS_TYPE_NORMAL:
return image_tokens->n_tokens();
case MTMD_POS_TYPE_SENSENOVA_U1:
return 1;
case MTMD_POS_TYPE_HUNYUANVL:
// HunyuanVL: the sequential (dim-0) position advances by the full token count
// (includes BOI/EOI and row newline tokens), not by max(nx, ny)