Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

vulkan: add TQ2_0 (ternary) support (#25850)

Browse Source 
* vulkan: TQ2_0 (ternary) support — dequant + dedicated mul_mat_vec + matmul via dequant_funcs

First Vulkan ternary type in ggml. Correctness: OM-125m TQ2_0 vs F16 top-12
logprobs identical to 4 decimals fully offloaded (float dequant path, no Q8_K
activation quant). Speed at 125m ~= F16 (overhead-bound at this scale); the
bandwidth win targets larger BitNet SKUs. MMQ/int-dot path intentionally not
wired yet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* tests: enable TQ2_0 in backend-ops type lists

Vulkan now implements TQ2_0 (dequant, mul_mat_vec, mul_mm, get_rows); backends
without support skip via not-supported as usual. TQ1_0 stays disabled.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Michael Trabalka <michael.trabalka@sqv.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
michaeltrabalka-tech
2026-08-12 08:07:23 +03:00
committed by GitHub
co-authored by Claude Fable 5 Michael Trabalka
parent 55f453b924
commit a7cd2f0e98
9 changed files with 257 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp
+2 -1
View File
@@ -72,6 +72,7 @@ const std::vector<std::string> type_names = {
"iq4_nl",
"mxfp4",
"nvfp4",
"tq2_0",
"bf16",
};
@@ -733,7 +734,7 @@ void process_shaders() {
for (const auto& tname : type_names) {
// mul mat vec
std::string data_a_key = "DATA_A_" + to_uppercase(tname);
std::string shader = (string_ends_with(tname, "_k") || string_starts_with(tname, "iq1_") || string_starts_with(tname, "iq2_") || string_starts_with(tname, "iq3_")) ? "mul_mat_vec_" + tname + ".comp" : "mul_mat_vec.comp";
std::string shader = (string_ends_with(tname, "_k") || string_starts_with(tname, "iq1_") || string_starts_with(tname, "iq2_") || string_starts_with(tname, "iq3_") || tname == "tq2_0") ? "mul_mat_vec_" + tname + ".comp" : "mul_mat_vec.comp";
string_to_spv("mul_mat_vec_" + tname + "_f32_f32", shader, merge_maps(base_dict, {{data_a_key, "1"}, {"B_TYPE", "float"}, {"B_TYPEV2", "vec2"}, {"B_TYPEV4", "vec4"}, {"D_TYPE", "float"}}));
string_to_spv("mul_mat_vec_" + tname + "_f16_f32", shader, merge_maps(base_dict, {{data_a_key, "1"}, {"B_TYPE", "float16_t"}, {"B_TYPEV2", "f16vec2"}, {"B_TYPEV4", "f16vec4"}, {"D_TYPE", "float"}}));