server: slot_sched: fix lora and recurrent (#32434)
* server: fix lora and recurrent * tests: run the hybrid SWA checkpoint test with one token per ubatch With and without the cache, the prompt was not split into the same ubatches, and the generated tokens are decoded one at a time while the uncached run processes them in a batch. On CUDA this moves the top logprobs of the tiny model by up to 0.03 and swaps near ties, so the test failed while the cache behaved as expected. With one token per ubatch both runs go through the same kernels and the output is compared exactly. --------- Co-authored-by: Pascal <admin@serveurperso.com>
This commit is contained in:
1 parent
3574df84b7
commit
52975342ff
8 files changed
+244
-24
No files matched your search
@@ -306,10 +306,9 @@ bool lora_should_clear_cache(
|
||||
const std::vector<common_adapter_lora_info> & current,
|
||||
const std::vector<common_adapter_lora_info> & next) {
|
||||
|
||||
// This should always be called after determining that the two sets are
|
||||
// _not_ equal. This assert is therefore some slightly wasted work and
|
||||
// should be safe to remove as long as this method is called correctly.
|
||||
GGML_ASSERT(!are_lora_equal(current, next));
|
||||
if (are_lora_equal(current, next)) {
|
||||
return false;
|
||||
}
|
||||
|
||||
return (
|
||||
!(lora_get_enabled_ids(current).empty() || lora_all_alora(current)) ||
|
||||
|
||||
Reference in new issue
Block a user