Lumpiasty/llama.cpp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

CUDA: Support CUDA Virtual Devices (#25228)

Browse Source 
* support cuda virtual devices

* disable NCCL path when virtual devices are used

* label virtual devices in description; add GPUx2 server CI jobs

* code refactor
This commit is contained in:
Anav Prasad
2026-07-16 13:37:35 +03:00
committed by GitHub
parent 3f08ef2c51
commit 79bba02a67
3 changed files with 173 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ggml/src/ggml-cuda/common.cuh
+5 -1
View File
@@ -1115,7 +1115,8 @@ struct ggml_cuda_type_traits<GGML_TYPE_IQ3_S> {
//////////////////////
struct ggml_cuda_device_info {
int device_count;
int device_count; // number of (possibly virtual) devices exposed to the rest of ggml
int physical_device_count; // number of physical CUDA devices actually present
struct cuda_device_info {
int cc; // compute capability
@@ -1128,6 +1129,9 @@ struct ggml_cuda_device_info {
size_t total_vram;
int warp_size; // Number of threads in a dispatch
bool supports_cooperative_launch; // whether cooperative launch is supported
int physical_device; // backing physical CUDA device for this (virtual) device
int physical_share_count; // number of (virtual) devices sharing this device's physical GPU
int virtual_index; // index of this (virtual) device among those sharing its physical GPU
};
cuda_device_info devices[GGML_CUDA_MAX_DEVICES] = {};