unicode : include '~' in collapsed symbol class (#26972)

The collapsed \p{S} class was missing '~', which split " ~" into
separate pre-tokens and prevented the Ġ~ BPE merge used by DeepSeek V4.
This caused re-tokenized prompts to diverge from sampled tokens and
broke KV cache reuse.

Assisted-by: Codex
This commit is contained in:
Thiago Padilha
2026-08-18 15:15:22 +02:00
committed by GitHub
parent 169e4a7ff2
commit afd439df1f
3 changed files with 27 additions and 1 deletions
+2
View File
@@ -116,6 +116,8 @@ function(llama_build_and_test source)
set_property(TEST ${TEST_TARGET} PROPERTY LABELS ${LLAMA_TEST_LABEL})
endfunction()
llama_build_and_test(test-unicode.cpp)
# build test-tokenizer-0 target once and add many tests
llama_build(test-tokenizer-0.cpp)