llama : optimize vocabulary loading (#30679)

* llama : optimize vocabulary loading

* Update src/unicode.h

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* CR : added reserve in gguf_kv_to_str

* CR : trailing space fix

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
This commit is contained in:
Evgen MykhailenkoandGeorgi Gerganov authored and GitHub committed 2026-10-21 12:03:27 +03:00
1 parent eb182d85c0
commit 824eaecf48
7 files changed
+75 -39

No files matched your search

+1
View File
@@ -104,6 +104,7 @@ unicode_cpt_flags unicode_cpt_flags_from_utf8(const std::string & utf8);
std::string unicode_byte_to_utf8(uint8_t byte);
uint8_t unicode_utf8_to_byte(const std::string & utf8);
uint8_t unicode_cpt_to_byte (uint32_t cpt);
uint32_t unicode_tolower(uint32_t cpt);