llama : optimize vocabulary loading (#30679)
* llama : optimize vocabulary loading * Update src/unicode.h Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * CR : added reserve in gguf_kv_to_str * CR : trailing space fix --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
This commit is contained in:
1 parent
eb182d85c0
commit
824eaecf48
7 files changed
+75
-39
No files matched your search
@@ -104,6 +104,7 @@ unicode_cpt_flags unicode_cpt_flags_from_utf8(const std::string & utf8);
|
||||
|
||||
std::string unicode_byte_to_utf8(uint8_t byte);
|
||||
uint8_t unicode_utf8_to_byte(const std::string & utf8);
|
||||
uint8_t unicode_cpt_to_byte (uint32_t cpt);
|
||||
|
||||
uint32_t unicode_tolower(uint32_t cpt);
|
||||
|
||||
|
||||
Reference in new issue
Block a user