Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

ggml : process data in smaller chunks in CUDA ggml_top_k() and ggml_argsort() to reduce temporary buffers memory usage (#24776)

Browse Source 
* ggml : process data in smaller chunks in CUDA ggml_top_k() implementation to reduce temporary buffers memory usage

* ggml : allocate tmp_dst only only once before the loop

* chore : whitespaces

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* ggml : use chunked processing in both CUDA CUB top-k and argsort implementations

* chore : separate argsort_f32_i32_cuda_bitonic() call from return statement

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* chore : replace ternary operators with min/max

---------

Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
This commit is contained in:
fairydreaming
2026-07-09 20:07:12 +02:00
committed by GitHub
co-authored by Georgi Gerganov Johannes Gäßler Stanisław Szymczyk
parent 3de7dd4c8f
commit 074944998d
3 changed files with 47 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ggml/src/ggml-cuda/argsort.cuh
+1
View File
@@ -3,6 +3,7 @@
void ggml_cuda_op_argsort(ggml_backend_cuda_context & ctx, ggml_tensor * dst);
#ifdef GGML_CUDA_USE_CUB
int argsort_f32_i32_cuda_cub_chunk_nrows(const size_t nb01, const int64_t nrows);
void argsort_f32_i32_cuda_cub(ggml_cuda_pool & pool,
const float * x,
int * dst,