model : Granite switch decision 256 options (#30693)

* feat(convert): Attempt to come up with 256 single-token, semantic option markers

Each switch model needs to identify a collection of single-token option
markers that can be mapped to their respective logits for the output
probability distribution. The simple upper/lower/numbers set only yields 66
unique markers, but the model can support up to 255. The challenge is that
which combinations of characters result in single-token markers is model-
specific, so we do a discovery exercise at conversion time by looking
through the specific model's vocab for single-token sequences of 1+ plain
letter strings (eg `A`, `AA`, `BB`, etc).

Branch: GraniteSwitchDecisionFullOptions
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* docs: Note granite-switch with 256 option markers

Branch: GraniteSwitchDecisionFullOptions
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* style: Remove extra newline

Branch: GraniteSwitchDecisionFullOptions
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat(convert): Use the computed labels in the template so c++ side is source of truth

Branch: GraniteSwitchDecisionFullOptions
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* feat: Use LEV case for GRANITE_SWITCH w/ computed labels

Branch: GraniteSwitchDecisionFullOptions
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* docs: Consolidate g-s into 255 bucket

Branch: GraniteSwitchDecisionFullOptions
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
This commit is contained in:
Gabe Goodhart authored and GitHub committed 2026-10-17 09:14:32 +02:00
1 parent f74e2fa6a3
commit 1268edac57
3 files changed
+10 -6

No files matched your search

+5 -2
View File
@@ -87,7 +87,7 @@ void server_decision_context::init(const llama_model * model) {
bias[level] = std::strtof(val, nullptr);
}
if (model_type == COMMON_DECISION_TYPE_OPENJEV || model_type == COMMON_DECISION_TYPE_GRANITE_SWITCH) {
if (model_type == COMMON_DECISION_TYPE_OPENJEV) {
// one letter per option, each must be a single token
const std::string letters = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz";
for (const char c : letters) {
@@ -99,7 +99,7 @@ void server_decision_context::init(const llama_model * model) {
}
n_options_max = labels.size();
noul_true_first = true;
} else if (model_type == COMMON_DECISION_TYPE_LEV || model_type == COMMON_DECISION_TYPE_NIMBLE || model_type == COMMON_DECISION_TYPE_PPLX_DECIDER || model_type == COMMON_DECISION_TYPE_VLLM_D3) {
} else if (model_type == COMMON_DECISION_TYPE_LEV || model_type == COMMON_DECISION_TYPE_GRANITE_SWITCH || model_type == COMMON_DECISION_TYPE_NIMBLE || model_type == COMMON_DECISION_TYPE_PPLX_DECIDER || model_type == COMMON_DECISION_TYPE_VLLM_D3) {
// label codes are A..Z then AA..ZZ, only the ones that are a single token are used
std::vector<std::string> codes;
for (char a = 'A'; a <= 'Z'; a++) {
@@ -121,6 +121,9 @@ void server_decision_context::init(const llama_model * model) {
if (model_type == COMMON_DECISION_TYPE_VLLM_D3) {
canonical_legend = true;
}
if (model_type == COMMON_DECISION_TYPE_GRANITE_SWITCH) {
noul_true_first = true;
}
} else if (model_type == COMMON_DECISION_TYPE_KEV) {
// the hidden state of an option is read at the token that ends it
const auto toks = common_tokenize(vocab, "<|box_end|>", false, true);