Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

mtmd: add mtmd_tokenize_from_parts() (#28250)

Browse Source 
* add mtmd_tokenize_from_parts

* use it in mtmd-cli

* move add_special to call level
This commit is contained in:
Xuan-Son Nguyen
2026-09-02 21:20:10 +02:00
committed by GitHub
parent 9cc33944f9
commit 7339054744
7 changed files with 154 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tools/mtmd/mtmd-internal.h
+4 -2
View File
@@ -10,10 +10,12 @@
#define MTMD_INTERNAL_HEADER
// bitmap is null for text parts
struct mtmd_input_part {
struct mtmd_internal_part {
std::string text;
const mtmd_bitmap * bitmap;
// only used for text parts
bool parse_special = false;
};
// [QWEN_VIDEO] merged parts are erased from `parts`, so one group always maps to one part
std::vector<std::vector<const mtmd_bitmap *>> mtmd_group_mergeable_bitmaps(std::vector<mtmd_input_part> & parts, int n_merge);
std::vector<std::vector<const mtmd_bitmap *>> mtmd_group_mergeable_bitmaps(std::vector<mtmd_internal_part> & parts, int n_merge);