* Adds initial scheduler stress tests * Adds new sleep op to simplify scheduler testing * Adds testcase addressing https://github.com/ggml-org/llama.cpp/issues/23321 * Adds experimental vulkan GGML_OP_SLEEP * Improves structure and naming of previous tests. Adds new tests, including one which detects the nkvo-bug correctly, and validates the proposed hotfix. * Cleanup: Removes CUDA specific comments and nvtx-markers * Fix: forgot to add sleep specific test case * Fix oversights identified by CI * Drops default prints so that it behaves like other ctests. Adds -v flag to toggle this behavior * Fix: Early exit in CPU-only settings (cause for graviton CI fails) * Improves stress_test_linked_list by removing overlap with test_chain_all_backends and test_pair_user_inputs. The latter ones test either activations or user inputs in isolation and just once, whereas the stress test tests all combinations multiple rounds * Exclude ACCEL devices from new tests due to incompatibility * Fixes vulkan test failure in llvmpipe * Adds asserts, improves comments and some checks * Fixes nitpicks in white-space and variable naming * Adds comments that certain vk drivers could time OP_SLEEP incorrectly * Drops limitiation to CPU/GPU in test-sleep * Aligns OP_SLEEP guards in cuda with vk guards (input shape and contiguity) * Implements feedback for OP_SLEEP to work in place (no copy) * Apply non-code suggestions from JohannesGaesslers code review Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * Removes struct backend_supports; enables tests to return skip, instead of only pass/fail * Make clang-tidy happy * Removes sched_backend_caps, improves create_test_scheduler according to feedback (L149) * Removes "name" field from struct sched_check in favor of ggml_set_name/ggml_get_name * Drops unnecessary shader_clock.comp test and guards * Replaces sched_ prefixes with helper_ prefix * Moves test suite to new RAII types for better memory management * Adds sleep option to test_chain_all_backends * Adds sleep option to first-run arm of the y-shaped test * Fix switchcase order for legibility in stress_test_dag * Simplfies stress_test_linked_list via double for-loop * Adds pseudo-random input to stress_test_linked_list, and runs all transitions both with and without CPU input (next to activation copy as input) * Fix RPC Proto patch due to GGML_OPS bump * Adds ggml-cpp.h treatment to test-sleep.cpp * Adds TODO pointers for subsequent additional tests. * Apply suggestion to ggml.h Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * Apply suggestion to tests/test-backend-sched.cpp Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * Apply suggestion to tests/test-backend-sched.cpp Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * Fixes node compatibility check to not take buffer locations into account. This led to skipped tests due to incorrectly rejected ops due to wrong buffer locations in multi-GPU (which is one of the scheduler behaviors under test) * Apply batched suggestions from gg code review Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Completes callsite change of gg-suggestion * Switch test-sleep from std::chrono::steady_clock::now() to ggml_time_us to avoid mac clock drift and spurious test failures. This makes the test a bit nonsensical on CPU (same clock checks itself), but the test still has merit on the other backends where independent device clocks are used. --------- Co-authored-by: Johannes Gäßler <johannesg@5d6.de> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
41 lines
1.1 KiB
C
41 lines
1.1 KiB
C
#pragma once
|
|
|
|
#include "ggml-backend.h"
|
|
|
|
#ifdef __cplusplus
|
|
extern "C" {
|
|
#endif
|
|
|
|
#define RPC_PROTO_MAJOR_VERSION 10
|
|
#define RPC_PROTO_MINOR_VERSION 0
|
|
#define RPC_PROTO_PATCH_VERSION 1
|
|
|
|
#ifdef __cplusplus
|
|
static_assert(GGML_OP_COUNT == 100, "GGML_OP_COUNT has changed - update RPC_PROTO_PATCH_VERSION");
|
|
#endif
|
|
|
|
#define GGML_RPC_MAX_SERVERS 16
|
|
|
|
// backend API
|
|
GGML_BACKEND_API ggml_backend_t ggml_backend_rpc_init(const char * endpoint, uint32_t device);
|
|
|
|
GGML_BACKEND_API bool ggml_backend_is_rpc(ggml_backend_t backend);
|
|
|
|
GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_rpc_buffer_type(const char * endpoint, uint32_t device);
|
|
|
|
GGML_BACKEND_API void ggml_backend_rpc_get_device_memory(const char * endpoint, uint32_t device, size_t * free, size_t * total);
|
|
|
|
GGML_BACKEND_API void ggml_backend_rpc_start_server(
|
|
const char * endpoint,
|
|
const char * cache_dir,
|
|
size_t n_threads,
|
|
size_t n_devices,
|
|
ggml_backend_dev_t * devices);
|
|
|
|
GGML_BACKEND_API ggml_backend_reg_t ggml_backend_rpc_reg(void);
|
|
GGML_BACKEND_API ggml_backend_reg_t ggml_backend_rpc_add_server(const char * endpoint);
|
|
|
|
#ifdef __cplusplus
|
|
}
|
|
#endif
|