hexagon: add support for SSM_SCAN op (#31921)
* hexagon: add SSM_SCAN Op support * hex-ssm-scan: fix multi-device spliting * hex-ssm-scan: remove redundant checks and make kernel use host config * hex-ssm-scan: remove direct ddr access * hex-ssm-scan: remove DMA ordering * hex-ops: update supported ops * hex-ssm-scan: rewrite the dma pipeline to match other ops (dbl.buf, etc) * hex-ssm-scan: remove all fallbacks, only dma/vtcm paths * hex-ssm-scan: add kernel_type to kparams and remove redundant recomputes * hex-ssm-scan: avoid stalls in the overall pipeline (dma flush between chunks) * hex-ssm-scan: vectorize softplus * hex-ssm-scan: optimize hvx loops * hex-ssm-scan: add HMX support * hex-ssm-scan: add support for GGML_HEXAGON_SSM_SELECT * hex-ssm-scan: cleanup kernel names * hex-ssm-scan: simplify mdev checks * hex-mdev: improve mdev row-split solver --------- Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
This commit is contained in:
1 parent
5193ef1d4f
commit
ded522f830
16 files changed
+9242
-6000
No files matched your search
+11
-11
@@ -28,9 +28,9 @@ Legend:
|
||||
| CONCAT | ❌ | ✅ | ✅ | 🟡 | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | 🟡 | ❌ | ❌ |
|
||||
| CONT | ❌ | 🟡 | ✅ | ✅ | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | 🟡 | ❌ | ❌ |
|
||||
| CONV_1D_DW | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ✅ |
|
||||
| CONV_2D | ❌ | ❌ | 🟡 | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | 🟡 | ✅ | ❌ | ❌ |
|
||||
| CONV_2D_DW | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| CONV_3D | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
|
||||
| CONV_2D | ❌ | ❌ | 🟡 | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | 🟡 | ✅ | ❌ | ❌ |
|
||||
| CONV_2D_DW | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| CONV_3D | ❌ | ❌ | ✅ | ✅ | ❌ | 🟡 | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
|
||||
| CONV_TRANSPOSE_1D | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
|
||||
| CONV_TRANSPOSE_2D | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
|
||||
| COS | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
@@ -42,8 +42,8 @@ Legend:
|
||||
| DIAG | ❌ | ❌ | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| DIV | ❌ | ✅ | ✅ | ✅ | ❌ | 🟡 | 🟡 | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| DSV4_HC_COMB | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
|
||||
| DSV4_HC_POST | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
|
||||
| DSV4_HC_PRE | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
|
||||
| DSV4_HC_POST | ❌ | ❌ | ✅ | ❌ | ❌ | 🟡 | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
|
||||
| DSV4_HC_PRE | ❌ | ❌ | ✅ | ❌ | ❌ | 🟡 | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
|
||||
| DUP | ❌ | ✅ | ✅ | ✅ | ❌ | 🟡 | 🟡 | 🟡 | ✅ | ✅ | ❌ | ❌ | ❌ |
|
||||
| ELU | ❌ | ✅ | ✅ | 🟡 | 🟡 | ❌ | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| EXP | ❌ | ✅ | ✅ | 🟡 | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
@@ -65,10 +65,10 @@ Legend:
|
||||
| HARDSIGMOID | ❌ | ✅ | ✅ | 🟡 | 🟡 | ❌ | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| HARDSWISH | ❌ | ✅ | ✅ | 🟡 | 🟡 | ❌ | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| IM2COL | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| IM2COL_3D | ❌ | ❌ | ✅ | ✅ | ❌ | 🟡 | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
|
||||
| IM2COL_3D | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
|
||||
| L2_NORM | ❌ | ✅ | ✅ | ✅ | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | 🟡 | ❌ | ❌ |
|
||||
| LEAKY_RELU | ❌ | ✅ | ✅ | ✅ | ❌ | 🟡 | 🟡 | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
|
||||
| LIGHTNING_INDEXER | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | 🟡 | ✅ | ❌ | ❌ | ❌ | ❌ |
|
||||
| LIGHTNING_INDEXER | ❌ | ❌ | ✅ | ❌ | ❌ | 🟡 | ❌ | 🟡 | ✅ | ❌ | ❌ | ❌ | ❌ |
|
||||
| LOG | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| MEAN | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | 🟡 | ✅ | ✅ | ❌ | ❌ |
|
||||
| MUL | ❌ | ✅ | ✅ | ✅ | 🟡 | 🟡 | 🟡 | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
@@ -103,7 +103,7 @@ Legend:
|
||||
| SCALE | ❌ | 🟡 | ✅ | ✅ | ✅ | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| SET | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | 🟡 | ✅ | ✅ | ❌ | ❌ |
|
||||
| SET_ROWS | ❌ | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | 🟡 | ❌ | ❌ |
|
||||
| SGN | ❌ | ✅ | ✅ | 🟡 | 🟡 | ❌ | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| SGN | ❌ | ✅ | ✅ | 🟡 | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| SIGMOID | ❌ | ✅ | ✅ | 🟡 | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| SILU | ❌ | ✅ | ✅ | 🟡 | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| SILU_BACK | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
|
||||
@@ -115,7 +115,7 @@ Legend:
|
||||
| SQR | ❌ | ✅ | ✅ | ✅ | 🟡 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| SQRT | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| SSM_CONV | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| SSM_SCAN | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | 🟡 | 🟡 | 🟡 | ✅ | ❌ | ❌ |
|
||||
| SSM_SCAN | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | 🟡 | 🟡 | 🟡 | ✅ | ❌ | ❌ |
|
||||
| STEP | ❌ | ✅ | ✅ | 🟡 | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| SUB | ❌ | ✅ | ✅ | ✅ | 🟡 | 🟡 | 🟡 | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| SUM | ❌ | 🟡 | ✅ | 🟡 | ❌ | 🟡 | 🟡 | ❌ | 🟡 | 🟡 | 🟡 | ❌ | ❌ |
|
||||
@@ -126,8 +126,8 @@ Legend:
|
||||
| TANH | ❌ | ✅ | ✅ | 🟡 | 🟡 | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| TIMESTEP_EMBEDDING | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
|
||||
| TOP_K | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | 🟡 | ✅ | ❌ | ❌ |
|
||||
| TOP_K_MASK | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
|
||||
| TOP_K_MASK | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ |
|
||||
| TRI | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| TRUNC | ❌ | ❌ | ✅ | 🟡 | 🟡 | ❌ | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| UPSCALE | ❌ | 🟡 | ✅ | ✅ | ❌ | ❌ | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| UPSCALE | ❌ | 🟡 | ✅ | ✅ | ❌ | 🟡 | ✅ | 🟡 | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| XIELU | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
+7280
-5968
File diff suppressed because it is too large.
Load diff
@@ -137,6 +137,7 @@ int opt_fa_select = 2; // 2 = HMX -> HVX -> CPU, 1 = HVX -> CPU, 0 = CPU
|
||||
int opt_fa_split = 1; // 1 = head-split + team-split when n_kv_heads < N_cores (default), 0 = off
|
||||
int opt_li_select = 2; // 2 = HMX -> HVX, 1 = HVX, 0 = CPU (unsupported)
|
||||
int opt_gdn_select = 2; // 2 = HMX -> HVX, 1 = HVX, 0 = CPU (unsupported)
|
||||
int opt_ssm_select = 2; // 2 = HMX -> HVX, 1 = HVX, 0 = CPU (unsupported)
|
||||
int opt_ar_select = 2; // 2 = fused ALLREDUCE+ADD (default), 1 = unfused ALLREDUCE, 0 = fallback to CPY+FENCE
|
||||
int opt_ar_scatter = 1; // 1 = reduce-scatter the fused ALLREDUCE+ADD (default), 0 = full reduction
|
||||
|
||||
@@ -3542,6 +3543,7 @@ static void ggml_hexagon_init(ggml_backend_reg * reg) {
|
||||
const char * str_fa_split = getenv("GGML_HEXAGON_FA_SPLIT");
|
||||
const char * str_li_select = getenv("GGML_HEXAGON_LI_SELECT");
|
||||
const char * str_gdn_select = getenv("GGML_HEXAGON_GDN_SELECT");
|
||||
const char * str_ssm_select = getenv("GGML_HEXAGON_SSM_SELECT");
|
||||
const char * str_ar_select = getenv("GGML_HEXAGON_AR_SELECT");
|
||||
const char * str_ar_scatter = getenv("GGML_HEXAGON_AR_SCATTER");
|
||||
const char * str_ndev = getenv("GGML_HEXAGON_NDEV");
|
||||
@@ -3609,6 +3611,7 @@ static void ggml_hexagon_init(ggml_backend_reg * reg) {
|
||||
opt_fa_split = str_fa_split ? atoi(str_fa_split) : opt_fa_split;
|
||||
opt_li_select = str_li_select ? atoi(str_li_select) : opt_li_select;
|
||||
opt_gdn_select = str_gdn_select ? atoi(str_gdn_select) : opt_gdn_select;
|
||||
opt_ssm_select = str_ssm_select ? atoi(str_ssm_select) : opt_ssm_select;
|
||||
opt_ar_select = str_ar_select ? atoi(str_ar_select) : opt_ar_select;
|
||||
opt_ar_scatter = str_ar_scatter ? atoi(str_ar_scatter) : opt_ar_scatter;
|
||||
|
||||
|
||||
@@ -25,6 +25,7 @@ extern int opt_fa_select;
|
||||
extern int opt_fa_split;
|
||||
extern int opt_li_select;
|
||||
extern int opt_gdn_select;
|
||||
extern int opt_ssm_select;
|
||||
extern int opt_ar_select;
|
||||
extern int opt_ar_scatter;
|
||||
|
||||
@@ -93,6 +94,23 @@ static inline bool htp_can_row_partition(const struct ggml_tensor * t) {
|
||||
return true;
|
||||
}
|
||||
|
||||
static inline uint32_t htp_mdev_rows_per_chunk(const struct ggml_tensor * t) {
|
||||
if ((t->data && ((uintptr_t) t->data & 127) != 0) || (t->view_offs & 127) != 0) return 0;
|
||||
const size_t elem_size = ggml_type_size(t->type);
|
||||
if (t->ne[0] != 1 && t->nb[0] != elem_size) return 0;
|
||||
if (ggml_is_permuted(t)) return 0;
|
||||
if (t->ne[2] > 1 && (t->nb[2] & 127) != 0) return 0;
|
||||
if (t->ne[3] > 1 && (t->nb[3] & 127) != 0) return 0;
|
||||
if (t->ne[1] > 1 && (t->nb[1] & 127) == 0) return 1;
|
||||
const size_t row_size = (size_t) t->ne[0] * elem_size;
|
||||
if (t->nb[1] == row_size &&
|
||||
(t->ne[2] <= 1 || t->nb[2] == t->nb[1] * (size_t) t->ne[1]) &&
|
||||
(t->ne[3] <= 1 || t->nb[3] == t->nb[2] * (size_t) t->ne[2])) {
|
||||
return (row_size > 0) ? (HEX_L2_LINE_SIZE / hex_gcd_u32((uint32_t) row_size, HEX_L2_LINE_SIZE)) : 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
struct ggml_hexagon_session;
|
||||
|
||||
const char * htp_session_name(const struct ggml_hexagon_session * sess);
|
||||
|
||||
@@ -46,6 +46,7 @@ htp_op_code htp_op_remap(const struct ggml_tensor * t) {
|
||||
case GGML_OP_LOG: return HTP_OP_UNARY_LOG;
|
||||
case GGML_OP_SOFT_MAX: return HTP_OP_SOFTMAX;
|
||||
case GGML_OP_SSM_CONV: return HTP_OP_SSM_CONV;
|
||||
case GGML_OP_SSM_SCAN: return HTP_OP_SSM_SCAN;
|
||||
case GGML_OP_GATED_DELTA_NET: return HTP_OP_GATED_DELTA_NET;
|
||||
case GGML_OP_ROPE: return HTP_OP_ROPE;
|
||||
case GGML_OP_REPEAT: return HTP_OP_REPEAT;
|
||||
@@ -684,11 +685,14 @@ static bool htp_kparams_solve_hmx_mm(
|
||||
int m_for_solver = ne11;
|
||||
int m_for_solver_padded = ne11_padded;
|
||||
if (!is_matmul_id && mdev_count > 1 && ((uint32_t) ne11 >= mdev_count)) {
|
||||
const bool dst_row_split = dst ? htp_can_row_partition(dst) : true;
|
||||
const bool act_row_split = htp_can_row_partition(act);
|
||||
if (dst_row_split && act_row_split) {
|
||||
m_for_solver = (ne11 + (int) mdev_count - 1) / (int) mdev_count;
|
||||
m_for_solver_padded = hex_round_up(std::max(m_for_solver, 32), 32);
|
||||
const uint32_t rows_per_chunk = dst ? htp_mdev_rows_per_chunk(dst) : 1;
|
||||
const uint32_t total_chunks = rows_per_chunk ? ((uint32_t) ne11 / rows_per_chunk) : 0;
|
||||
if (total_chunks >= mdev_count) {
|
||||
const uint32_t chunks_per_mdev = (total_chunks + mdev_count - 1) / mdev_count;
|
||||
const uint32_t m_dev0 = chunks_per_mdev * rows_per_chunk;
|
||||
const uint32_t m_last = (uint32_t) ne11 > (mdev_count - 1) * m_dev0 ? (uint32_t) ne11 - (mdev_count - 1) * m_dev0 : 0;
|
||||
m_for_solver = (int) std::max(m_dev0, m_last);
|
||||
m_for_solver_padded = hex_round_up(std::max(m_for_solver, 32), 32);
|
||||
}
|
||||
}
|
||||
const bool pipeline = is_matmul_id ? false : htp_mm_hmx_pipeline(m_for_solver);
|
||||
@@ -2243,6 +2247,152 @@ static bool htp_kparams_solve_ssm_conv(
|
||||
return L.total_bytes <= vtcm_size;
|
||||
}
|
||||
|
||||
static bool htp_kparams_solve_ssm_scan(
|
||||
const struct ggml_hexagon_session * sess,
|
||||
const struct ggml_tensor * op,
|
||||
struct htp_ssm_scan_kernel_params * kparams
|
||||
) {
|
||||
if (opt_ssm_select < 1) {
|
||||
return false;
|
||||
}
|
||||
|
||||
memset(kparams, 0, sizeof(*kparams));
|
||||
|
||||
const struct ggml_tensor * s = op->src[0];
|
||||
const struct ggml_tensor * x = op->src[1];
|
||||
const struct ggml_tensor * dt = op->src[2];
|
||||
const struct ggml_tensor * A = op->src[3];
|
||||
const struct ggml_tensor * B = op->src[4];
|
||||
const struct ggml_tensor * C = op->src[5];
|
||||
|
||||
if (s->type != GGML_TYPE_F32 || x->type != GGML_TYPE_F32 || dt->type != GGML_TYPE_F32 || A->type != GGML_TYPE_F32 ||
|
||||
B->type != GGML_TYPE_F32 || C->type != GGML_TYPE_F32) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const uint32_t d_state = (uint32_t) s->ne[0];
|
||||
const uint32_t n_a = (uint32_t) A->ne[0];
|
||||
const uint32_t head_dim = (uint32_t) x->ne[0];
|
||||
const uint32_t n_head = (uint32_t) x->ne[1];
|
||||
const uint32_t n_tokens = (uint32_t) x->ne[2];
|
||||
const uint32_t n_seqs = (uint32_t) x->ne[3];
|
||||
const uint32_t n_group = (uint32_t) B->ne[1];
|
||||
const uint32_t K = (uint32_t) ggml_get_op_params_i32(op, 0);
|
||||
|
||||
if (n_head % n_group != 0 || (n_a != 1 && n_a != d_state)) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const uint8_t fast_step4 = (n_a == 1 && (d_state & 31U) == 0) ? 1 : 0;
|
||||
const uint32_t sess_hvx = htp_session_n_hvx(sess);
|
||||
const size_t vtcm_size = htp_session_vtcm_size(sess);
|
||||
|
||||
const uint64_t y_token_stride = (uint64_t) n_head * head_dim * sizeof(float);
|
||||
const uint64_t y_seq_stride = (uint64_t) n_tokens * y_token_stride;
|
||||
const uint64_t y_bytes = (uint64_t) n_seqs * y_seq_stride;
|
||||
if (y_bytes > UINT32_MAX) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const bool can_use_hmx = (opt_ssm_select >= 2) &&
|
||||
(htp_session_n_hmx(sess) > 0) &&
|
||||
(n_tokens >= HTP_SSM_SCAN_HMX_TOKEN_TILE) &&
|
||||
(n_a == 1) &&
|
||||
(K == 1) &&
|
||||
(d_state >= 32U && d_state <= HTP_SSM_SCAN_HMX_MAX_DIM && (d_state & 31U) == 0) &&
|
||||
(head_dim >= 32U && head_dim <= HTP_SSM_SCAN_HMX_MAX_DIM && (head_dim & 31U) == 0);
|
||||
|
||||
struct htp_ssm_scan_vtcm_layout layout;
|
||||
bool hmx_selected = false;
|
||||
|
||||
if (can_use_hmx) {
|
||||
const uint64_t total_units = (uint64_t) n_seqs * n_head;
|
||||
if (total_units <= UINT32_MAX) {
|
||||
const uint32_t n_hvx = (std::min)(sess_hvx, (uint32_t) total_units);
|
||||
htp_ssm_scan_vtcm_layout_build(&layout, d_state, head_dim, HTP_SSM_SCAN_HMX_TOKEN_TILE, n_hvx, true);
|
||||
if (layout.total_bytes <= vtcm_size) {
|
||||
kparams->kernel_type = HTP_SSM_SCAN_KERNEL_HMX;
|
||||
kparams->fast_step4 = fast_step4;
|
||||
kparams->row_tile = (uint16_t) head_dim;
|
||||
kparams->token_tile = (uint16_t) HTP_SSM_SCAN_HMX_TOKEN_TILE;
|
||||
kparams->n_chunks = (n_tokens + HTP_SSM_SCAN_HMX_TOKEN_TILE - 1) / HTP_SSM_SCAN_HMX_TOKEN_TILE;
|
||||
kparams->d_state = d_state;
|
||||
kparams->n_a = n_a;
|
||||
kparams->head_dim = head_dim;
|
||||
kparams->n_tokens = n_tokens;
|
||||
kparams->n_seqs = n_seqs;
|
||||
kparams->K = K;
|
||||
kparams->heads_per_group = n_head / n_group;
|
||||
kparams->n_row_blocks = 1;
|
||||
kparams->group_units = kparams->heads_per_group;
|
||||
kparams->units_per_seq = n_head;
|
||||
kparams->total_units = (uint32_t) total_units;
|
||||
kparams->n_hvx = (uint16_t) n_hvx;
|
||||
kparams->units_per_hvx = (kparams->total_units + kparams->n_hvx - 1) / kparams->n_hvx;
|
||||
hmx_selected = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!hmx_selected) {
|
||||
const uint8_t kernel_type = (n_tokens == 1) ? HTP_SSM_SCAN_KERNEL_HVX_DECODE : HTP_SSM_SCAN_KERNEL_HVX_PREFILL;
|
||||
uint32_t row_tile = (std::min)(head_dim, (n_tokens > 1 && head_dim >= 64) ? 64U : HTP_SSM_SCAN_ROW_TILE_DEFAULT);
|
||||
uint32_t token_tile = (K > 1) ? 1U : (std::min)(n_tokens, HTP_SSM_SCAN_TOKEN_TILE_DEFAULT);
|
||||
|
||||
for (;;) {
|
||||
const uint32_t n_row_blocks = (head_dim + row_tile - 1) / row_tile;
|
||||
const uint64_t total_units = (uint64_t) n_seqs * n_head * n_row_blocks;
|
||||
if (total_units > UINT32_MAX) {
|
||||
return false;
|
||||
}
|
||||
|
||||
kparams->kernel_type = kernel_type;
|
||||
kparams->fast_step4 = fast_step4;
|
||||
kparams->row_tile = (uint16_t) row_tile;
|
||||
kparams->token_tile = (uint16_t) token_tile;
|
||||
kparams->n_chunks = (n_tokens + token_tile - 1) / token_tile;
|
||||
kparams->d_state = d_state;
|
||||
kparams->n_a = n_a;
|
||||
kparams->head_dim = head_dim;
|
||||
kparams->n_tokens = n_tokens;
|
||||
kparams->n_seqs = n_seqs;
|
||||
kparams->K = K;
|
||||
kparams->heads_per_group = n_head / n_group;
|
||||
kparams->n_row_blocks = n_row_blocks;
|
||||
kparams->group_units = kparams->heads_per_group * n_row_blocks;
|
||||
kparams->units_per_seq = n_head * n_row_blocks;
|
||||
kparams->total_units = (uint32_t) total_units;
|
||||
kparams->n_hvx = (uint16_t) (std::min)(sess_hvx, kparams->total_units);
|
||||
kparams->units_per_hvx = (kparams->total_units + kparams->n_hvx - 1) / kparams->n_hvx;
|
||||
|
||||
htp_ssm_scan_vtcm_layout_build(&layout, d_state, kparams->row_tile, kparams->token_tile, kparams->n_hvx, false);
|
||||
if (layout.total_bytes <= vtcm_size) {
|
||||
break;
|
||||
}
|
||||
if (row_tile > 1) {
|
||||
row_tile = (row_tile > 4) ? ((row_tile / 2) & ~3U) : 1U;
|
||||
} else if (token_tile > 1) {
|
||||
token_tile /= 2;
|
||||
} else {
|
||||
kparams->kernel_type = HTP_SSM_SCAN_KERNEL_UNSUPPORTED;
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
kparams->y_token_stride = (uint32_t) y_token_stride;
|
||||
kparams->y_seq_stride = (uint32_t) y_seq_stride;
|
||||
kparams->y_bytes = (uint32_t) y_bytes;
|
||||
kparams->vtcm_per_hvx = (uint32_t) layout.bytes_per_hvx;
|
||||
kparams->vtcm_size = (uint32_t) layout.total_bytes;
|
||||
kparams->div_units_per_seq = init_fastdiv_values(kparams->units_per_seq);
|
||||
kparams->div_group_units = init_fastdiv_values(kparams->group_units);
|
||||
kparams->div_n_row_blocks = init_fastdiv_values(kparams->n_row_blocks);
|
||||
kparams->div_n_hvx = init_fastdiv_values(kparams->n_hvx);
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
static bool htp_kparams_solve_sort(
|
||||
const struct ggml_hexagon_session * sess,
|
||||
const struct ggml_tensor * op,
|
||||
@@ -5447,6 +5597,9 @@ bool htp_kparams_solve(
|
||||
case HTP_OP_SSM_CONV:
|
||||
return htp_kparams_invoke<htp_kparams_solve_ssm_conv>(sess, op, kparams);
|
||||
|
||||
case HTP_OP_SSM_SCAN:
|
||||
return htp_kparams_invoke<htp_kparams_solve_ssm_scan>(sess, op, kparams);
|
||||
|
||||
case HTP_OP_SOFTMAX:
|
||||
return htp_kparams_invoke<htp_kparams_solve_softmax>(sess, op, kparams);
|
||||
|
||||
|
||||
@@ -29,6 +29,7 @@
|
||||
#include "htp/softmax-ops.h"
|
||||
#include "htp/solve-tri-ops.h"
|
||||
#include "htp/ssm-conv.h"
|
||||
#include "htp/ssm-scan-ops.h"
|
||||
#include "htp/sum-rows-ops.h"
|
||||
#include "htp/top-k-mask-ops.h"
|
||||
#include "htp/unary-ops.h"
|
||||
|
||||
@@ -19,6 +19,7 @@
|
||||
#include "htp/softmax-ops.h"
|
||||
#include "htp/solve-tri-ops.h"
|
||||
#include "htp/ssm-conv.h"
|
||||
#include "htp/ssm-scan-ops.h"
|
||||
#include "htp/sum-rows-ops.h"
|
||||
#include "htp/top-k-mask-ops.h"
|
||||
#include "htp/unary-ops.h"
|
||||
@@ -182,6 +183,13 @@ static std::string format_kernel_params(const htp_opnode & node) {
|
||||
const auto * kparams = node.kparams<htp_ssm_conv_kernel_params>();
|
||||
snprintf(str, max_size, "%s nhvx %u vtcm %u", kparams->n_t == 1 ? "decode" : "prefill",
|
||||
(unsigned int) kparams->n_hvx, (unsigned int) kparams->vtcm_size);
|
||||
} else if (node.opcode == HTP_OP_SSM_SCAN) {
|
||||
const auto * kparams = node.kparams<htp_ssm_scan_kernel_params>();
|
||||
const char * ktype_str = kparams->kernel_type == HTP_SSM_SCAN_KERNEL_HMX ? "hmx" :
|
||||
kparams->kernel_type == HTP_SSM_SCAN_KERNEL_HVX_DECODE ? "hvx-decode" :
|
||||
"hvx-prefill";
|
||||
snprintf(str, max_size, "%s nhvx %u vtcm %u", ktype_str,
|
||||
(unsigned int) kparams->n_hvx, (unsigned int) kparams->vtcm_size);
|
||||
} else if (node.opcode == HTP_OP_SOFTMAX) {
|
||||
const auto * kparams = node.kparams<htp_softmax_kernel_params>();
|
||||
snprintf(str, max_size, "k%d nhvx %u vtcm %u", (int) kparams->kernel_type,
|
||||
|
||||
@@ -36,6 +36,7 @@ add_library(${HTP_LIB} SHARED
|
||||
gather-rows.c
|
||||
repeat-ops.c
|
||||
ssm-conv.c
|
||||
ssm-scan-ops.c
|
||||
cumsum-ops.c
|
||||
top-k-mask-ops.c
|
||||
fill-ops.c
|
||||
|
||||
@@ -184,6 +184,7 @@ int op_repeat(struct htp_ops_context * octx);
|
||||
int op_argsort(struct htp_ops_context * octx);
|
||||
int op_top_k(struct htp_ops_context * octx);
|
||||
int op_ssm_conv(struct htp_ops_context * octx);
|
||||
int op_ssm_scan(struct htp_ops_context * octx);
|
||||
int op_cumsum(struct htp_ops_context * octx);
|
||||
int op_top_k_mask(struct htp_ops_context * octx);
|
||||
int op_fill(struct htp_ops_context * octx);
|
||||
|
||||
@@ -125,6 +125,7 @@ enum htp_op_code {
|
||||
HTP_OP_DSV4_HC_POST,
|
||||
HTP_OP_TOP_K_MASK,
|
||||
HTP_OP_LIGHTNING_INDEXER,
|
||||
HTP_OP_SSM_SCAN,
|
||||
|
||||
HTP_OP_INVALID
|
||||
};
|
||||
|
||||
@@ -1019,6 +1019,9 @@ static int execute_op(struct htp_ops_context * octx) {
|
||||
case HTP_OP_SSM_CONV:
|
||||
return op_ssm_conv(octx);
|
||||
|
||||
case HTP_OP_SSM_SCAN:
|
||||
return op_ssm_scan(octx);
|
||||
|
||||
case HTP_OP_CUMSUM:
|
||||
return op_cumsum(octx);
|
||||
|
||||
|
||||
@@ -2961,17 +2961,17 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_k
|
||||
int m_start = 0;
|
||||
int m_rows = m;
|
||||
if (octx->ctx->mdev.count > 1) {
|
||||
bool can_split = htp_tensor_can_row_partition(act, act_elem_size);
|
||||
if (can_split) {
|
||||
for (uint32_t p = 0; p < n_weights; ++p) {
|
||||
const struct htp_tensor * restrict dst = octx->dsts[p];
|
||||
if (!htp_tensor_can_row_partition(dst, sizeof(float))) {
|
||||
can_split = false;
|
||||
break;
|
||||
}
|
||||
uint32_t rows_per_chunk = 1;
|
||||
for (uint32_t p = 0; p < n_weights; ++p) {
|
||||
const struct htp_tensor * restrict dst = octx->dsts[p];
|
||||
uint32_t dst_rpc = 0;
|
||||
if (!htp_tensor_mdev_rows_per_chunk(dst, sizeof(float), dst->ne[0] * sizeof(float), &dst_rpc)) {
|
||||
rows_per_chunk = 0;
|
||||
break;
|
||||
}
|
||||
rows_per_chunk = MAX(rows_per_chunk, dst_rpc);
|
||||
}
|
||||
const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition((uint32_t) m, can_split ? 1 : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);
|
||||
const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition((uint32_t) m, rows_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);
|
||||
m_start = (int) range.start;
|
||||
m_rows = (int) range.count;
|
||||
}
|
||||
@@ -3646,12 +3646,12 @@ static int hmx_mm_op_matmul(struct htp_ops_context * octx, const struct htp_mm_k
|
||||
int m_start = 0;
|
||||
int m_rows = m_total;
|
||||
if (octx->ctx->mdev.count > 1) {
|
||||
bool can_split = htp_tensor_can_row_partition(dst, sizeof(float)) &&
|
||||
htp_tensor_can_row_partition(act, act_elem_size);
|
||||
if (src2 && src2->ne[1] > 1 && !htp_tensor_can_row_partition(src2, sizeof(float))) {
|
||||
can_split = false;
|
||||
uint32_t rows_per_chunk = 0;
|
||||
if ((dst->ne[2] <= 1 || (dst->nb[2] & (HTP_TENSOR_MDEV_LINE_SIZE - 1)) == 0) &&
|
||||
(dst->ne[3] <= 1 || (dst->nb[3] & (HTP_TENSOR_MDEV_LINE_SIZE - 1)) == 0)) {
|
||||
htp_tensor_mdev_rows_per_chunk(dst, sizeof(float), dst->ne[0] * sizeof(float), &rows_per_chunk);
|
||||
}
|
||||
const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition((uint32_t) m_total, can_split ? 1 : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);
|
||||
const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition((uint32_t) m_total, rows_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);
|
||||
m_start = (int) range.start;
|
||||
m_rows = (int) range.count;
|
||||
}
|
||||
@@ -3667,8 +3667,9 @@ static int hmx_mm_op_matmul(struct htp_ops_context * octx, const struct htp_mm_k
|
||||
size_t src2_nb3 = 0;
|
||||
if (src2) {
|
||||
src2_stride = (src2->ne[1] == 1) ? 0 : (uint32_t) (src2->nb[1] / sizeof(float));
|
||||
src2_addr = src2->data + m_start * src2_stride * sizeof(float);
|
||||
src2_bytes = (size_t) kparams->vtcm_bias_size;
|
||||
src2_addr = src2->data + (size_t) m_start * src2_stride * sizeof(float);
|
||||
const size_t slice_bytes = (src2->ne[1] == 1) ? src2->nb[1] : (size_t) m_rows * src2->nb[1];
|
||||
src2_bytes = MIN((size_t) kparams->vtcm_bias_size, slice_bytes);
|
||||
src2_nb2 = (src2->ne[2] == 1) ? 0 : src2->nb[2];
|
||||
src2_nb3 = (src2->ne[3] == 1) ? 0 : src2->nb[3];
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,165 @@
|
||||
#ifndef HTP_SSM_SCAN_OPS_H
|
||||
#define HTP_SSM_SCAN_OPS_H
|
||||
|
||||
#include <stdbool.h>
|
||||
#include <stddef.h>
|
||||
#include <stdint.h>
|
||||
|
||||
#include "hex-common.h"
|
||||
#include "hex-fastdiv.h"
|
||||
|
||||
#define HTP_SSM_SCAN_ROW_TILE_DEFAULT 32U
|
||||
#define HTP_SSM_SCAN_TOKEN_TILE_DEFAULT 32U
|
||||
#define HTP_SSM_SCAN_HMX_TOKEN_TILE 64U
|
||||
#define HTP_SSM_SCAN_HMX_MAX_DIM 128U
|
||||
|
||||
enum htp_ssm_scan_kernel_type {
|
||||
HTP_SSM_SCAN_KERNEL_UNSUPPORTED = 0,
|
||||
HTP_SSM_SCAN_KERNEL_HVX_DECODE = 1,
|
||||
HTP_SSM_SCAN_KERNEL_HVX_PREFILL = 2,
|
||||
HTP_SSM_SCAN_KERNEL_HMX = 3,
|
||||
};
|
||||
|
||||
struct htp_ssm_scan_kernel_params {
|
||||
uint8_t kernel_type;
|
||||
uint8_t fast_step4;
|
||||
uint16_t row_tile;
|
||||
uint16_t token_tile;
|
||||
uint16_t n_hvx;
|
||||
|
||||
uint32_t n_chunks;
|
||||
uint32_t d_state;
|
||||
uint32_t n_a;
|
||||
uint32_t head_dim;
|
||||
uint32_t n_tokens;
|
||||
uint32_t n_seqs;
|
||||
uint32_t K;
|
||||
|
||||
uint32_t heads_per_group;
|
||||
uint32_t n_row_blocks;
|
||||
uint32_t group_units;
|
||||
uint32_t units_per_seq;
|
||||
uint32_t total_units;
|
||||
uint32_t units_per_hvx;
|
||||
|
||||
uint32_t y_token_stride;
|
||||
uint32_t y_seq_stride;
|
||||
uint32_t y_bytes;
|
||||
|
||||
uint32_t vtcm_per_hvx;
|
||||
uint32_t vtcm_size;
|
||||
|
||||
struct fastdiv_values div_units_per_seq;
|
||||
struct fastdiv_values div_group_units;
|
||||
struct fastdiv_values div_n_row_blocks;
|
||||
struct fastdiv_values div_n_hvx;
|
||||
};
|
||||
|
||||
#if defined(__cplusplus)
|
||||
static_assert(sizeof(struct htp_ssm_scan_kernel_params) <= 128, "htp_ssm_scan_kernel_params is too large for kernel_params blob");
|
||||
#else
|
||||
_Static_assert(sizeof(struct htp_ssm_scan_kernel_params) <= 128, "htp_ssm_scan_kernel_params is too large for kernel_params blob");
|
||||
#endif
|
||||
|
||||
// Decode updates state out of place (off_state -> off_state_out) so writeback(N) and fetch(N+2)
|
||||
// never share a VTCM buffer. Prefill updates state in place across token chunks using off_state[0].
|
||||
struct htp_ssm_scan_vtcm_layout {
|
||||
size_t off_state[2];
|
||||
size_t off_state_out[2];
|
||||
size_t off_B[2];
|
||||
size_t off_C[2];
|
||||
size_t off_x[2];
|
||||
size_t off_dt[2];
|
||||
size_t off_A[2];
|
||||
size_t off_y[2];
|
||||
|
||||
size_t off_hmx_scales;
|
||||
size_t off_hmx_c_row;
|
||||
size_t off_hmx_b_t_col;
|
||||
size_t off_hmx_b_col;
|
||||
size_t off_hmx_s_t_col;
|
||||
size_t off_hmx_q_m;
|
||||
size_t off_hmx_y_tiles;
|
||||
size_t off_hmx_x_tilde_col;
|
||||
size_t off_hmx_x_hat_t_row;
|
||||
size_t off_hmx_L;
|
||||
size_t off_hmx_p;
|
||||
|
||||
size_t state_row_stride;
|
||||
size_t xy_row_stride;
|
||||
size_t bytes_per_hvx;
|
||||
size_t total_bytes;
|
||||
};
|
||||
|
||||
static inline void htp_ssm_scan_vtcm_layout_build(
|
||||
struct htp_ssm_scan_vtcm_layout * layout,
|
||||
uint32_t d_state,
|
||||
uint32_t row_tile,
|
||||
uint32_t token_tile,
|
||||
uint32_t n_hvx,
|
||||
bool use_hmx
|
||||
) {
|
||||
const size_t state_row_stride = hex_align_up((size_t) d_state * sizeof(float), 128);
|
||||
const size_t state_slot_bytes = (size_t) row_tile * state_row_stride;
|
||||
const size_t bc_slot_bytes = (size_t) token_tile * state_row_stride;
|
||||
const size_t xy_row_stride = hex_align_up((size_t) row_tile * sizeof(float), 128);
|
||||
const size_t xy_slot_bytes = (size_t) token_tile * xy_row_stride;
|
||||
const size_t scalar_slot_bytes = hex_align_up((size_t) token_tile * sizeof(float), 128);
|
||||
const size_t a_slot_bytes = state_row_stride;
|
||||
|
||||
size_t off = 0;
|
||||
for (uint32_t i = 0; i < 2; ++i) {
|
||||
layout->off_state[i] = off; off += state_slot_bytes;
|
||||
layout->off_state_out[i] = off; off += state_slot_bytes;
|
||||
layout->off_B[i] = off; off += bc_slot_bytes;
|
||||
layout->off_C[i] = off; off += bc_slot_bytes;
|
||||
layout->off_x[i] = off; off += xy_slot_bytes;
|
||||
layout->off_dt[i] = off; off += scalar_slot_bytes;
|
||||
layout->off_A[i] = off; off += a_slot_bytes;
|
||||
layout->off_y[i] = off; off += xy_slot_bytes;
|
||||
}
|
||||
|
||||
if (use_hmx) {
|
||||
const size_t n_ds_tiles = (size_t) (d_state / 32U);
|
||||
const size_t n_d_tiles = (size_t) (row_tile / 32U);
|
||||
const size_t tile_64x64_sz = 2U * 2U * 2048U;
|
||||
const size_t tile_64xds_sz = 2U * n_ds_tiles * 2048U;
|
||||
const size_t tile_64xd_sz = 2U * n_d_tiles * 2048U;
|
||||
const size_t tile_dxds_sz = n_d_tiles * n_ds_tiles * 2048U;
|
||||
const size_t l_f32_sz = 64U * 64U * sizeof(float);
|
||||
const size_t p_f32_sz = hex_align_up(4U * 64U * sizeof(float), 2048U);
|
||||
|
||||
off = hex_align_up(off, 2048U);
|
||||
layout->off_hmx_scales = off; off += 2048U;
|
||||
layout->off_hmx_c_row = off; off += tile_64xds_sz;
|
||||
layout->off_hmx_b_t_col = off; off += tile_64xds_sz;
|
||||
layout->off_hmx_b_col = off; off += tile_64xds_sz;
|
||||
layout->off_hmx_s_t_col = off; off += tile_dxds_sz;
|
||||
layout->off_hmx_q_m = off; off += tile_64x64_sz;
|
||||
layout->off_hmx_y_tiles = off; off += tile_64xd_sz;
|
||||
layout->off_hmx_x_tilde_col = off; off += tile_64xd_sz;
|
||||
layout->off_hmx_x_hat_t_row = off; off += tile_64xd_sz;
|
||||
layout->off_hmx_L = off; off += l_f32_sz;
|
||||
layout->off_hmx_p = off; off += p_f32_sz;
|
||||
off = hex_align_up(off, 2048U);
|
||||
} else {
|
||||
layout->off_hmx_scales = 0;
|
||||
layout->off_hmx_c_row = 0;
|
||||
layout->off_hmx_b_t_col = 0;
|
||||
layout->off_hmx_b_col = 0;
|
||||
layout->off_hmx_s_t_col = 0;
|
||||
layout->off_hmx_q_m = 0;
|
||||
layout->off_hmx_y_tiles = 0;
|
||||
layout->off_hmx_x_tilde_col = 0;
|
||||
layout->off_hmx_x_hat_t_row = 0;
|
||||
layout->off_hmx_L = 0;
|
||||
layout->off_hmx_p = 0;
|
||||
}
|
||||
|
||||
layout->state_row_stride = state_row_stride;
|
||||
layout->xy_row_stride = xy_row_stride;
|
||||
layout->bytes_per_hvx = off;
|
||||
layout->total_bytes = off * n_hvx;
|
||||
}
|
||||
|
||||
#endif // HTP_SSM_SCAN_OPS_H
|
||||
@@ -35,6 +35,7 @@ MANAGED_ENV_NAMES = (
|
||||
"GGML_HEXAGON_FA_HEAD_SPLIT",
|
||||
"GGML_HEXAGON_LI_SELECT",
|
||||
"GGML_HEXAGON_GDN_SELECT",
|
||||
"GGML_HEXAGON_SSM_SELECT",
|
||||
"GGML_HEXAGON_AR_SELECT",
|
||||
"GGML_HEXAGON_AR_SCATTER",
|
||||
"GGML_HEXAGON_ETM",
|
||||
@@ -222,6 +223,7 @@ def main():
|
||||
parser.add_argument("--hex-fa-head-split", help="Enable (1) or disable (0) head-parallel flash_attn partitioning (GGML_HEXAGON_FA_HEAD_SPLIT)")
|
||||
parser.add_argument("--hex-li-select", help="Select Lightning Indexer kernel (GGML_HEXAGON_LI_SELECT) 2:HMX,1:HVX,0:disable")
|
||||
parser.add_argument("--hex-gdn-select", help="Select Gated Delta Net kernel (GGML_HEXAGON_GDN_SELECT) 2:HMX,1:HVX,0:disable")
|
||||
parser.add_argument("--hex-ssm-select", help="Select SSM Scan kernel (GGML_HEXAGON_SSM_SELECT) 2:HMX,1:HVX,0:disable")
|
||||
parser.add_argument("--hex-ar-select", help="Select All-Reduce kernel (GGML_HEXAGON_AR_SELECT) 1:enable,0:disable")
|
||||
parser.add_argument("--hex-ar-scatter", help="Enable (1) or disable (0) reduce-scatter for fused ALLREDUCE+ADD (GGML_HEXAGON_AR_SCATTER)")
|
||||
parser.add_argument("--hex-etm", help="Enable Embedded Trace Macrocell hardware tracing / trace logging (GGML_HEXAGON_ETM)")
|
||||
@@ -377,6 +379,7 @@ def main():
|
||||
set_env("GGML_HEXAGON_FA_HEAD_SPLIT", args.hex_fa_head_split)
|
||||
set_env("GGML_HEXAGON_LI_SELECT", args.hex_li_select)
|
||||
set_env("GGML_HEXAGON_GDN_SELECT", args.hex_gdn_select)
|
||||
set_env("GGML_HEXAGON_SSM_SELECT", args.hex_ssm_select)
|
||||
set_env("GGML_HEXAGON_AR_SELECT", args.hex_ar_select)
|
||||
set_env("GGML_HEXAGON_AR_SCATTER", args.hex_ar_scatter)
|
||||
set_env("GGML_HEXAGON_ETM", args.hex_etm)
|
||||
|
||||
@@ -14024,6 +14024,12 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_perf() {
|
||||
test_cases.emplace_back(new test_ssm_scan(GGML_TYPE_F32, 128, 80, 128, 1, 512, 1)); // Nemotron-9B prefill
|
||||
test_cases.emplace_back(new test_ssm_scan(GGML_TYPE_F32, 128, 80, 128, 1, 1, 1)); // Nemotron-9B generate
|
||||
|
||||
test_cases.emplace_back(new test_ssm_scan(GGML_TYPE_F32, 128, 64, 16, 2, 512, 1)); // Mamba-2 prefill
|
||||
test_cases.emplace_back(new test_ssm_scan(GGML_TYPE_F32, 128, 64, 16, 2, 1, 1)); // Mamba-2 generate
|
||||
|
||||
test_cases.emplace_back(new test_ssm_scan(GGML_TYPE_F32, 128, 64, 64, 8, 512, 1)); // Nemotron-3.5-Lightning prefill
|
||||
test_cases.emplace_back(new test_ssm_scan(GGML_TYPE_F32, 128, 64, 64, 8, 1, 1)); // Nemotron-3.5-Lightning generate
|
||||
|
||||
// acc
|
||||
test_cases.emplace_back(new test_acc(GGML_TYPE_F32, {256, 17, 1, 1}, {256, 16, 1, 1}, -1));
|
||||
test_cases.emplace_back(new test_acc(GGML_TYPE_F32, {256, 17, 2, 3}, {256, 16, 2, 3}, -1));
|
||||
|
||||
Reference in new issue
Block a user