a1f4222608594c57202fdaa6a45dca1ef05abc62
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
a1f4222608 |
model : load ffn_gate_up_exps scales (#32657)
* model: load `ffn_gate_up_exps` scales The generic scale pass loads per-expert scales for ffn_gate_exps, ffn_up_exps and ffn_down_exps but not for the merged ffn_gate_up_exps. Load ~ffn_gate_up_exps_s there~ and expose it as ffn_up_exps_s ~when the layer has no separate up tensor,~ since build_moe_ffn applies up_exps_s to the merged gate_up output. Inkling 2 now passes the expert scales to build_moe_ffn like the other MoE models. * Update src/llama-model.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Update src/llama-model.h Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> |
||
|
|
8af68d486a |
llama: fixes for Inkling 2 attention mask (#32429)
* llama : give Inkling2 its own relative-position attention input The Inkling2 graph built its flash attention mask from the hybrid iswa input and passed it to build_attn through a new kq_mask argument. That argument let a caller pair its own dense mask with the kv_rows of llm_graph_input_attn_kv_iswa, whose mask follows the per-sequence cell lists whenever kv_rows is enabled. Add llm_graph_input_attn_rel_pos instead. It owns dense KQ masks and the query-to-cell distances, fills both from the same cells, and its build_attn overload adds the per-head bias to its own mask and never passes kv_rows. The iswa build_attn is back to its previous signature; its body moves to build_attn_iswa_impl, which both overloads call. Inkling2 now folds the per-query sink into the bias with and without flash attention, so build_attn_mha no longer needs the per-query sink path. Assisted-By: Claude Opus 5.5 <noreply@anthropic.com> * inkling2 : make the expert weight scale and final logit softcap optional Load both keys as optional and skip them when they are zero, as other models do, instead of rejecting zero at load. Assisted-By: Claude Opus 5.5 <noreply@anthropic.com> * llama : build kv_rows masks for the hybrid iswa memory llm_graph_input_mem_hybrid_iswa checked its masks with can_reuse_attn_mask, which expects kv_rows whenever they are enabled, but built them dense. With a unified cache and flash attention the check always failed, and hybrid iswa models (LFM2, LFM3, Inkling2) never reused a graph. Build the masks with build_attn_inp_attn_mask, like the plain iswa input. Assisted-By: Claude Opus 5.5 <noreply@anthropic.com> * llama : make the relative-position input a hybrid iswa input Address review: llm_graph_input_mem_hybrid_iswa_rel derives from llm_graph_input_mem_hybrid_iswa and owns the attention and recurrent inputs, so Inkling2 builds one memory input instead of two. Its attention input is built with dense masks, can_reuse checks them as dense, and its build_attn asserts that the mask has no kv_rows (TODO: a kv_rows variant). build_attn_iswa_impl is now a static helper. Assisted-By: Claude Opus 5.5 <noreply@anthropic.com> * llama : compose the hybrid iswa rel input, format inkling2 Address review: llm_graph_input_mem_hybrid_iswa_rel owns its attention and recurrent inputs like the other composite inputs instead of deriving from llm_graph_input_mem_hybrid_iswa. Format src/models/inkling2.cpp like the other model sources: grouped and aligned key and tensor loading, descriptive per-layer names, blank lines and section comments between the steps, and cb() names for the intermediate tensors. The graph is unchanged. Assisted-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5c008e0008 |
feat: add Inkling 2 support (text + vision) (#32217)
* convert : add Inkling2 text GGUF conversion Assisted-by: Codex * llama : implement Inkling2 text inference Assisted-by: Codex * tests : cover Inkling2 text support Assisted-by: Codex * conversion : dequantize packed Inkling2 experts in row chunks Decode per-layer NVFP4 and MXFP8 schemes, validate companions, and use stored scale shapes with a warning for the declared MXFP8 [32,32] grid. Reorder packed w13 rows and their scale rows together. Keep normal GGUF outtype selection and 3D expert tensors. Stream 128 rows per expert through LazyChunkedTensor. Real w13 (384x4608x4608) needs a 2.25 MiB F32 chunk instead of a 30.375 GiB expanded layer. Real-shaped sparse probes peak below 5.8 MiB of Python/NumPy chunk allocations, plus 5.2 MiB of callback metadata: budget about 16 MiB per projection including mapped pages, excluding runtime, other tensors and OS cache. Full-model chunk metadata is roughly 370 MiB. Packed synthetic GGUF weights match the independent pinned vLLM decode bit-for-bit. CPU prefill/decode logits remain within 3.1e-6 absolute error, with 100% argmax agreement. Float fixtures, original-name conversion and CPU architecture checks pass. Real checkpoint conversion remains untested. Assisted-by: Codex * convert : add Inkling2 vision mmproj support Assisted-by: Codex * mtmd : implement Inkling2 still-image encoder Assisted-by: Codex * inkling2 : scope image loading Assisted-by: Codex * inkling2 : enable flash attention with folded bias and sinks Assisted-by: Codex * cuda : route per-head flash attention masks through vec Assisted-by: Codex * mtmd : support Inkling2 temporal frame means Assisted-by: Codex * style : restore shared image comment Assisted-by: Codex * tests : enlarge Inkling2 context fixture Assisted-by: Codex * inkling2 : load per-layer expert feed-forward widths Assisted-by: Codex * tests : add Inkling2 Metal fusion baselines Assisted-by: Codex * inkling2 : group and align registry entries Assisted-by: Fable * convert : regenerate the Inkling2 tokenizer hash with the update script Assisted-by: Fable * conversion : revert the print_registered_models change Assisted-by: Fable * conversion : sort the Inkling2 model map entries Assisted-by: Fable * convert : map Inkling2 vision checkpoint names in the tensor table Assisted-by: Fable * convert : map Inkling2 text checkpoint names in the tensor table Assisted-by: Fable * tests : document the Inkling2 packed expert conversion test Assisted-by: Fable * tests : cover Inkling2 checkpoint and transformers tensor names Assisted-by: Fable * llama : place Inkling2 fields and list entries by file convention Assisted-by: Fable * gguf-py : align Inkling2 tensor mapping comments Assisted-by: Fable * mtmd : place Inkling2 entries by file convention Assisted-by: Fable * tests : place Inkling2 cases after the existing ones Assisted-by: Fable * cuda : index per-head flash attention masks in the tile and mma kernels Assisted-by: Claude Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q * inkling2 : project the relative bias before the gather, drop arch branches Assisted-by: Claude Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q * convert : require Inkling2 hparams, add inkling2 pre-tokenizer Assisted-by: Claude Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q * mtmd : cap Inkling2 image size, drop video frame averaging Assisted-by: Claude Claude-Session: https://claude.ai/code/session_01GbYSza1yxgmTz8wdr6uV8Q * inkling2 : fix CI build, Meta device assert and Windows conversion test Assisted-by: Claude * tests : use the default sliding window for the Inkling2 test model Assisted-by: Claude * simplify mtmd part * give _v arch for vision * remove alpha channel handling, it should be a follow up * explicit rule about audio + vision models * add audio support * rm audio * rm gguf mtmd rope base * address review comment * nits * do not normalize embd from mtmd --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> |