hexagon: tensor repacking rework and support for tiled and flat quantized and fp weights (#30635)
* hex-repack: Remove need_repack hack and streamline the rules -- breaks GET_ROWS tests for K-Quants, will fix in the next round Add support for GET_ROWS with tiled Q5_K Add support for raw k-quant tensors * hex-get-rows: vectorize raw k-quant ukernels * hex-binary: refactor dma-loop thread functions (no need for macros, etc) * hex-quant: move/cleanup flat and tiled dequant ukernels * hex-repack: add support for repacking/tiling F16/F32/BF16 2D weights * hex-get-rows: remove redundant cast * hex-repack: proper filtering for BF16 that were leaking through during the tests * hex-repack: streamling tiling rules (broken MM for now) * hex-repack: fix mmid after tiling updates * hex-repack: rename repacked to tiled in all flags/funcs/kernels/vars * hex-mm: fix valid_rows calc in various kernels * hex-mm: remove unused hvx_mm_nx_2d_flat * hex-mm: cleanup and optimize hvx-tiled vec_dot_f16 * hex-mm: another pass at hvx-tiled f16 matmul * hex-get-rows: optimize small src1 * hex-get-rows: one more pass at recovering get-rows perf regression * hex-repack: export GGML_HEXAGON_RT_REPACK via run.py * hex-mm-tiled: fix accuracy regression with hmx-tiled-f16 matmul * hex-mm: fold tiled_f16 checks into the general flow * hex-get-rows: remove dead checks for buffer_is_hexagon and tiling during supports_op * hex-quant: proper tail handling in new dequant ukernels * hex-get-rows: remove unused code and cleanup support checks * hex-mm: cleanup ne1 padding/tiling checks * hex-mm: cleanup fusion checks for tiled tensors * hex-repack: add a little can_repack helper to reduce duplication * hex-repack: never repack host buffers * hex-repack: factor out repack functions into separate .h and .cpp * hex-tensor: cleanup tensor ifaces for tiled/padded sizes * hex-repack: update k-quant tiling comments and cleanup #defines * hex-dcache: factor out l2/dcache dirty tracking and flushing * hex-tensor: update docs and remove #defs duplication * hex-tensor: remove unused/incorrect get_data interface * hex-get-rows: cleanup vtcm layout build * tests: disable huge MM tests * hex-dcache: add missing header
This commit is contained in:
1 parent
6afa9a3134
commit
b797b9ca81
24 files changed
+3497
-3396
No files matched your search
@@ -27,6 +27,7 @@ MANAGED_ENV_NAMES = (
|
||||
"GGML_HEXAGON_OPPOLL",
|
||||
"GGML_HEXAGON_OPFILTER",
|
||||
"GGML_HEXAGON_OPFUSION",
|
||||
"GGML_HEXAGON_RT_REPACK",
|
||||
"GGML_HEXAGON_VMEM",
|
||||
"GGML_HEXAGON_MBUF",
|
||||
"GGML_HEXAGON_MM_SELECT",
|
||||
@@ -167,6 +168,7 @@ def main():
|
||||
parser.add_argument("--hex-oppoll", default="1", help="Enable (1) or Disable (0) polling for NPU opbatch completion (GGML_HEXAGON_OPPOLL) (default: 1)")
|
||||
parser.add_argument("--hex-opfilter", help="Regex pattern to filter/select which operators are offloaded to NPU (GGML_HEXAGON_OPFILTER)")
|
||||
parser.add_argument("--hex-opfusion", help="NPU graph node fusion optimization level (0: disabled, 1: enabled) (GGML_HEXAGON_OPFUSION)")
|
||||
parser.add_argument("--hex-rt-repack", help="Runtime weight repacking mode (0: disabled, 1: quantized only, 2: always) (GGML_HEXAGON_RT_REPACK)")
|
||||
parser.add_argument("--hex-vmem", help="Maximum NPU VMEM size limit in MB to allocate (GGML_HEXAGON_VMEM)")
|
||||
parser.add_argument("--hex-mbuf", help="Host buffer size limits in MB (supports K/M/G suffix): <dyn>[,<static>[,<total>]] (default: 512,1024,0) (GGML_HEXAGON_MBUF)")
|
||||
parser.add_argument("--hex-mm-select", help="Select MUL_MAT and MUL_MAT_ID kernel (GGML_HEXAGON_MM_SELECT) 2:HMX,1:HVX,0:disable")
|
||||
@@ -311,6 +313,7 @@ def main():
|
||||
set_env("GGML_HEXAGON_OPPOLL", args.hex_oppoll)
|
||||
set_env("GGML_HEXAGON_OPFILTER", args.hex_opfilter)
|
||||
set_env("GGML_HEXAGON_OPFUSION", args.hex_opfusion)
|
||||
set_env("GGML_HEXAGON_RT_REPACK", args.hex_rt_repack)
|
||||
set_env("GGML_HEXAGON_VMEM", args.hex_vmem)
|
||||
set_env("GGML_HEXAGON_MBUF", args.hex_mbuf)
|
||||
set_env("GGML_HEXAGON_MM_SELECT", args.hex_mm_select)
|
||||
|
||||
Reference in new issue
Block a user