Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

vulkan: add TQ2_0 (ternary) support (#25850)

Browse Source 
* vulkan: TQ2_0 (ternary) support — dequant + dedicated mul_mat_vec + matmul via dequant_funcs

First Vulkan ternary type in ggml. Correctness: OM-125m TQ2_0 vs F16 top-12
logprobs identical to 4 decimals fully offloaded (float dequant path, no Q8_K
activation quant). Speed at 125m ~= F16 (overhead-bound at this scale); the
bandwidth win targets larger BitNet SKUs. MMQ/int-dot path intentionally not
wired yet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* tests: enable TQ2_0 in backend-ops type lists

Vulkan now implements TQ2_0 (dequant, mul_mat_vec, mul_mm, get_rows); backends
without support skip via not-supported as usual. TQ1_0 stays disabled.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Michael Trabalka <michael.trabalka@sqv.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
michaeltrabalka-tech
2026-08-12 08:07:23 +03:00
committed by GitHub
co-authored by Claude Fable 5 Michael Trabalka
parent 55f453b924
commit a7cd2f0e98
9 changed files with 257 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ggml/src/ggml-vulkan/vulkan-shaders/dequant_funcs_cm2.glsl
+41
View File
@@ -247,6 +247,44 @@ f16vec4 dequantFuncQ8_0_v(const in decodeBufQ8_0 bl, const in uint blockCoords[2
return f16vec4(vec4(qi) * vec4(float(d)));
}
layout(buffer_reference, std430, buffer_reference_align = 2) buffer decodeBufTQ2_0 {
block_tq2_0 block;
};
layout(buffer_reference, std430, buffer_reference_align = 2) buffer decodeBufTQ2_0_packed16 {
block_tq2_0_packed16 block;
};
float16_t dequantFuncTQ2_0(const in decodeBufTQ2_0 bl, const in uint blockCoords[2], const in uint coordInBlock[2])
{
decodeBufTQ2_0_packed16 bl16 = decodeBufTQ2_0_packed16(bl);
const uint idx = coordInBlock[1];
const uint qsshift = (idx & 0x60) >> 4; // 0,2,4,6
uint qs = uint32_t(bl16.block.qs[((idx & 0x80) >> 3) + ((idx & 0x1E) >> 1)]);
qs = (qs >> qsshift) & 0x0303;
qs = unpack8(qs)[idx & 1];
return bl.block.d * (float16_t(int(qs)) - float16_t(1.0));
}
f16vec4 dequantFuncTQ2_0_v(const in decodeBufTQ2_0 bl, const in uint blockCoords[2], const in uint coordInBlock[2])
{
const uint idx = coordInBlock[1];
const uint qsshift = (idx & 0x60) >> 4; // 0,2,4,6
const uint qsi = ((idx & 0x80) >> 2) + (idx & 0x1C); // byte index of 4-aligned group
const uint qsw = (uint(bl.block.qs[qsi]))
| (uint(bl.block.qs[qsi + 1]) << 8)
| (uint(bl.block.qs[qsi + 2]) << 16)
| (uint(bl.block.qs[qsi + 3]) << 24);
const u8vec4 q = unpack8((qsw >> qsshift) & 0x03030303);
return bl.block.d * (f16vec4(q) - f16vec4(1.0));
}
layout(buffer_reference, std430, buffer_reference_align = 4) buffer decodeBufQ2_K {
block_q2_K block;
};
@@ -1368,6 +1406,9 @@ f16vec4 dequantFuncNVFP4_v(const in decodeBufNVFP4 bl, const in uint blockCoords
#elif defined(DATA_A_Q8_0)
#define dequantFuncA dequantFuncQ8_0
#define dequantFuncA_v dequantFuncQ8_0_v
#elif defined(DATA_A_TQ2_0)
#define dequantFuncA dequantFuncTQ2_0
#define dequantFuncA_v dequantFuncTQ2_0_v
#elif defined(DATA_A_Q2_K)
#define dequantFuncA dequantFuncQ2_K
#define dequantFuncA_v dequantFuncQ2_K_v