Commit Graph
554 Commits
Author SHA1 Message Date
LumpiastyandClaude Opus 5.5 9537f885e0 scripts: fork-validate.sh - --moe-exact, greedy identity for the readback skip
F1 (the routing-ids readback skip in sched_copy_experts) had no automated
check. --moe-exact MODEL reads expert_count / expert_used_count from the GGUF
header, computes the skip threshold ceil(4 * n_expert / n_expert_used) tokens
per ubatch (128 on the 35B), and runs one greedy request with a prompt of
about 2x that in a single ubatch, every expert on the host, so the prefill
takes the skip and each decode step takes the readback. The candidate's 64
tokens must equal the reference's byte for byte; with the LLAMA_MOE_IDS_SKIP
knob in the candidate, also the candidate with the skip forced off (=0), which
proves the skip exact even when the reference carries it, and =log must show
skips, all at >= 4 * n_expert ids. Without the knob skip-engaged SKIPs with
the batch-size argument. It needs the GPU to itself (same guard as --perf,
now shared as gpu_exclusive); FV_TEST_SHARED_GPU=1 bypasses the guard for
script tests against stub binaries only.

Also unset inherited LLAMA_MOE* variables.

Not run on the GPU yet (needs a swapdown window). Dry-run on the pod
against stub servers and the real 35B GGUF header: threshold 128, all-PASS
when outputs match, FAIL on a differing reference, SKIP without the knob,
FAIL for a dense model and while llama-swap runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 12a2add2c5 scripts: fork-validate.sh - check that a cold server gives its VRAM back
Nothing checked that a release actually frees VRAM. --arbiter now takes a
baseline of mem_info_vram_used before any scratch server starts, then with A
warm and alone on the scratch arena makes it release through a fake waiter
on want.lock. v-vram-release requires the card-wide delta to drop by at
least 70% of what A added warm; v-vram-restore requires it back (and the
output equal to the reference) after the next request. Production's own
share is re-read at every point from the DRM fdinfo of every process the
script did not start (deduplicated by drm-client-id); if it moved by more
than 64 MiB the check SKIPs instead of blaming the candidate.

Run beside production on the pod (reb1024 vs itself): warm +546 MiB, cold
+4 MiB, restored +546 MiB, production steady at 5053 MiB; 16 PASS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 deb09fb305 scripts: fork-validate.sh - A/B ordinary handoff latency against --ref
--arbiter only reported the candidate's handoff latency, while the fork's
process rule (two production deploys lost) is to A/B the ordinary handoff,
not only the pathological cases. After the arbiter cases, a-handoff-latency
starts fresh A/B server pairs of each build on the scratch arena and runs
blocks ref, cand, cand, ref of 16 alternating handoffs each. It FAILs if the
candidate median exceeds the reference median by more than max(25%, 200 ms)
and reports both medians, maxima and per-block medians.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 3b2c7188fb scripts: fork-validate.sh - gate --perf on medians, not means
--perf averaged the two per-arm llama-bench means of each build, so one
catastrophic sample (CPU contention from the ~60 other pods on the node) moved
the result, against the fork's own rule of medians, never means. llama-bench
now writes -o jsonl (every repetition sample) next to the markdown table, and
the gates use the median of all samples of a build pooled over its two arms
(4 per prefill cell, 6 for decode); the table shows min-max and n.

--perf-report TSV re-aggregates a saved <scratch>/perf.tsv without a GPU.

Checked offline on the pod: the 2026-09 A/B samples in results.txt give
pp2048 reference 250.24 (old mean) vs 247.76 (new median), candidate 280.15
vs 276.93; a synthetic 2026-10-24-like set with one 180 t/s candidate sample
read -9.3% (FAIL) the old way and -0.2% (PASS) with medians.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 3141ad2a9e scripts: rx580-bench - decode at tg256 depth 4096, not tg32
The llama-bench sweep measured decode as tg32 at the default depth and the
README called it the continuity metric, while tg32 at n_ctx 32 is launch /
barrier bound on this card and is not a valid decode metric (it once read
-36% between builds at parity at tg256). lbsweep.sh now runs prefill
(-p $LB_P -n 0, depth 0) and decode (-p 0 -n 256 -d 4096, LB_N / LB_D) as
two invocations and labels depth cells (tg256@d4096). summarize.py marks
old depth-less tg rows as invalid-metric history. The README examples and
the reference values are updated to the 2026-10-24 production numbers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 a213514855 scripts: add fork-validate.sh, post-rebase validation of the fork
Checks the op kernels, the concat fast path, the VRAM arbiter incl. every
shutdown path that has caused a production incident, and prefill/decode
against a reference build. Prints PASS/FAIL per check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5.5 3d4e7bc065 scripts: rx580-bench - use -lm none, document the MoE cache option
Upstream #28334 removed --mmap/--no-mmap/--mlock and llama-bench's -mmp, so the
harness flags no longer parse: the server would refuse to start and llama-bench
would silently produce no rows. Switch both harnesses to -lm none, the same
load mode, and note --moe-cache-mib as a server-only option worth measuring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-21 19:39:51 +01:00
LumpiastyandClaude Opus 5 c61db474d8 scripts: add the RX 580 benchmark harness
Encodes the production config, fixed corpus slices, repeat/median discipline
and the noise floor, so the measurement method does not have to be
rediscovered each time. Runs the corpus prefill test through llama-server and
the llama-bench sweep as a controlled cross-check, with interleaved A/B.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PZz44SLQvTXMyWGio6t9DZ
2026-11-21 19:39:51 +01:00
Aleksander Grygier 427a836734 ui: stage Llama Voice's sources for CMake and Docker UI builds (#32680)
Llama Chat compiles @ggml-org/llama-voice from ui/lib/voice/src, but the
CMake stage and the Docker UI stages copied only ui/lib/ui, so their npm
build could not resolve the voice imports. Stage ui/lib/voice/src next to
Llama UI and track it as an input of the CMake build.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-20 19:42:49 +01:00
Xuejun ZhaiandSigbjørn Skjæret e626db7caa server-bench : fix TypeError in histogram generation (#32518)
* fix(server-bench): prevent float bin-count TypeError in histogram generation

Symptom: benchmark crashes while generating gen_rate histogram.

Root cause: np.ceil(...) returns numpy.float64, and that value was used where an integer count/shape argument is required.

Fix: cast ceil result to int before +1, ensuring bin count is an integer.

Result: histogram path completes and gen_rate.png is generated reliably.

* Update scripts/server-bench.py

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-20 15:16:52 +01:00
Aleksander Grygier 80437488a4 ui: cut Llama Chat's first-load JavaScript by 63%, load diagrams, math and PDFs on demand (#32555)
* ui: import Lucide icons by path in modules that run in Node

Assisted-by: Claude (Opus 5.5 Medium)

* ui: register highlight.js languages once, from lowlight's modules

Assisted-by: Claude (Opus 5.5 Medium)

* ui: load the pdf.js worker and nerdamer as files of their own

Assisted-by: Claude (Opus 5.5 Medium)

* ui: draw Mermaid diagrams with Pele

Assisted-by: Claude (Opus 5.5 Medium)

* ui: pin dependency versions and use one Svelte across the UI packages

Assisted-by: Claude (Opus 5.5 Medium)

* ui: build Llama Chat as split chunks

Assisted-by: Claude (Opus 5.5 Medium)

* ui: load KaTeX and pdf.js on first use

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-19 08:37:35 +01:00
Alessandro de Oliveira Faria (A.K.A.CABELO)andAlessandro de Oliveira Faria 4483b2b987 vendor : update cpp-httplib-0.67.0 [no ci] (#32563)
Co-authored-by: Alessandro de Oliveira Faria (A.K.A.CABELO) <cabelo@multicortex.ai>
2026-11-18 21:00:31 +01:00
Aleksander Grygier 7f106b5459 ui: Llama UI design system: tokens, surfaces and fonts (#32468)
* ui: bring @ggml-org/llama-ui into ui/lib/ui

The design system package (tokens, fonts, surfaces, components and Model UI), so far in the temporary ggml-org/llama-ui
repository at edaf518, moves to its home in ui/lib/ui, built with @sveltejs/package and published as @ggml-org/llama-ui
from here. ui/lib holds the packages, one folder each. Its references to tools/ui now name ui/apps/chat, its
package.json points at this repository, and the UI workflow type-checks and packs it. ui/README.md describes the
layout.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: Llama UI design system: tokens, surfaces and brand

The design system from allozaur/docs/storybook-overhaul: three tiers of
tokens, matte glass surfaces, Llama Sans and Llama Mono, and the base
components on the new tokens (Tabs and GlassKey among them). It also
restyles the stack's features with it, replacing the plain stand-ins they
used until now. DESIGN.md is generated from the tokens. The Storybook that
documents it is the next PR.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: have Llama Chat use @ggml-org/llama-ui for its base components, tokens and fonts

Llama Chat kept its own copy of the design system: 29 base components, the
three tiers of tokens, the surfaces, the theme and the fonts, next to the 12
components ui/lib/ui held. The copy goes, and the library gets what it lacked.

ui/lib/ui
- the 17 base components the app had and the library did not (alert,
  alert-dialog, button-group, checkbox, collapsible, context-menu, dialog,
  drawer, dropdown-menu, empty, hover-card, radio-group, scroll-area,
  skeleton, table, tabs, textarea), with vaul-svelte for the drawer
- the inset-group and selected-row tokens and surfaces
- sources import each other by relative path instead of $lib, which in an
  app points at the app's own src/lib
- theme.css no longer imports tw-animate-css: it is a peer dependency that
  the consumer imports before the theme, as it does Tailwind
- 0.3.0

ui/apps/chat
- imports the base components as @ggml-org/llama-ui/ui/<name> and cn/tv as
  @ggml-org/llama-ui/utils; app.css imports the library's theme.css
- compiles the library from source, with no install or build of it:
  scripts/vite-plugin-llama-ui.ts resolves its entry points and what it
  imports, and svelte.config.js gives TypeScript the same paths
- src/lib/components/ui, styles/tokens, styles/surfaces.css and assets/fonts
  are removed; components.json moves to the library

Builds
- CMake stages ui/lib/ui next to the app, keeping the layout of ui/, and
  rebuilds when a library source changes
- the Dockerfiles copy the library's sources; the Nix derivation takes ui/
  as its source

The built CSS has the same rules in the same order, apart from one
zero-specificity rule and the @font-face rules moving, plus ten utilities
and two tokens that only the library's own components use.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: AGENTS.md places Llama UI in ui/lib/ui

Llama Chat's AGENTS.md says where each part lives: Llama UI is
ui/lib/ui, and Chat imports its tokens, fonts and base components
from there.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: Llama Sans and Llama Mono from llama-brand's latest, and shell commands marked in bash code

The fonts in ui/lib/ui come from llama-brand's latest, with optical sizes
from 10 to 36. Bash code blocks mark shell commands and flags.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: open submenus outside the menu they come from

A submenu was rendered inside its parent menu, so the menu's scroll box
clipped it: the glass menus' backdrop filter, and the filter and transform
of the open animation, make the menu the containing block even for the
fixed-position submenu. Dropdown and context submenus now render in a
portal, like the menus themselves.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: draw submenus like the menu they open from

Submenus had smaller corners than their menus, and context menus had none
of the dropdown's glass. Context-menu parts now match the dropdown's, and
both submenus take the menu's corners and height cap, with their first row
level with the row that opens them, 4px past the menu's edge. Sub-triggers
take the items' radius and pointer.

The submenus' height cap and zoom origin named the dropdown's CSS variables,
which bits-ui does not set on submenus; they now use its --bits-menu-* ones.
The chat group submenu takes the actions menu's z-[999999] layer, as it no
longer renders inside that menu.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: keep a late current-node write off a closed or switched conversation

updateCurrentNode checked the active conversation, awaited the database write, then wrote currNode to whatever was active after it. When the conversation closed during the write it threw, and the reply ended in a streaming error dialog. When another conversation opened, that one got the wrong node.

Assisted-by: Claude (Fable 5.2)

* ui: have Llama Chat use the library's Model UI and app components

Llama Chat still kept its own copies of what the library holds above the
base components. It now takes them from @ggml-org/llama-ui too, and its
copies go.

ui/lib/ui
- ModelId gains what Chat's had: the model's own draft kind, draft
  sidecars with their quant (ModelDraftSidecars, new), aliases, hideName
  and a tooltip, in the same badge order
- entry points for them, like ui/<name>: @ggml-org/llama-ui/app/<group>
  (actions, content, forms, misc) and @ggml-org/llama-ui/models

ui/apps/chat
- ActionIcon, SearchInput, SyntaxHighlightedCode, Logo,
  ModelCapabilityIcons, ModelDownloadProgressBar and ModelDraftSidecars
  come from the library; Chat's copies and the two constants only they
  used are removed
- ModelId and ModelOrgAvatar stay as thin wrappers: ModelId parses the raw
  id and applies the user's display settings, ModelOrgAvatar hides the
  avatar while Hub metadata is off
- the svelte/* TypeScript path falls back to Svelte's types/index.d.ts:
  svelte/reactivity has no file of its own, so a library source importing
  it got its types from ui/lib/ui's own install when there was one

Assisted-by: Claude (Opus 5.5 Medium)

* ui: Llama UI takes the brand's colours and logo from llama-brand

@ggml-org/llama-brand is now a dependency of the library, by the same GitHub
tarball Llama Chat already installs, which now names it @ggml-org/llama-brand
too.

- primitives.css drops the 25 tokens it held copies of (the neutral
  greys, sand, the skies and cinder) and builds on llama-brand's
  css/colors.css instead; the skies' gradients point at llama-brand's.
  Apps import that file before theme.css, as they do Tailwind and
  tw-animate-css: the library's stylesheets import no package, so an app
  that compiles them from source resolves it from its own node_modules
- Logo draws llama-brand's icon, in currentColor, and assets/logo.svg goes
- Llama Chat's TypeScript paths resolve llama-brand from its own
  node_modules, like the library's other dependencies

The fonts stay as the library's WOFF2 copies until llama-brand ships
WOFF2 builds of them.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: have Llama Chat take the library's constants, enums and helpers

Llama Chat redefined most of what the library exports next to its
components. Where the two were the same, Chat now re-exports the library's
from the files that held them, so its own imports stay as they are.

- the library gets entry points for them: @ggml-org/llama-ui/constants,
  @ggml-org/llama-ui/enums and @ggml-org/llama-ui/code, and
  @ggml-org/llama-ui/models carries the model data with the Model UI
- from the library now: the shared CSS classes and code-block constants,
  the byte units, the memory-fit constants and minMemoryTierGb, the bit
  depth labels, the capability and modality icons, labels and order,
  ColorMode, TooltipSide, BooleanString, the model enums, ModelModalities,
  formatParameters, highlightCode and trimCodePadding
- the library's modality order follows Chat's: vision, video, audio
- Chat keeps its own where they differ: UI_DATA_ATTRS and
  ModelDownloadProgress are supersets of the library's (the latter extends
  it now), and formatFileSize counts in 1024s

Assisted-by: Claude (Opus 5.5 Medium)

* ui: give the chat form's add key and model selector a tighter shadow

In light, the two keys inside the chat form, white on its white glass, took
the control elevation, whose soft spread blended into the form. They take a
new elevation-key instead: a 0.5px hairline all round and a 1px drop. In
dark they keep the control elevation. Chosen from the options in
Scrapbook › UI › Composer keys.

- semantic tokens: --elevation-key (light: the hairline and the drop; dark:
  the control elevation)
- component token: --chat-form-key-shadow, for the add key and every model
  selector trigger
- DESIGN.md names it next to the other elevations

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-17 14:39:03 +01:00
Xuan-Son Nguyen 8d1599fca1 chat: add tts_dict for kokoro G2P template (#32397)
* chat: add tts_dict for kokoro G2P template

* more testing

* value_float_t range check

* no json-in-meta, add llama_model_meta_arr_str/_n
2026-11-17 12:52:50 +01:00
Aleksander Grygier 67cf8967eb ui: move Llama Chat to SvelteKit 3 and its built-in service worker (#32463)
* ui: replace deprecated packages and fix npm audit findings

Swap the mdast stub for @types/mdast, move the katex override to 0.19.0,
and override glob, source-map and html-encoding-sniffer under
workbox-build and http-server so installs no longer pull deprecated
packages. Apply the non-breaking npm audit fixes to the lockfile.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: migrate Llama Chat to SvelteKit 3 and its built-in service worker

Move to SvelteKit 3, Vite 8, TypeScript 6 and Storybook 10.6. The
SvelteKit options move from svelte.config.js into sveltekit() in
vite.config.ts, $lib imports become #lib subpath imports from
package.json, and $app/environment becomes $app/env. Modules named
server.* or living under a server/ folder are server-only in SvelteKit 3,
so the llama-server components, store and enums move to llama-server
names.

@vite-pwa/sveltekit and Workbox give way to SvelteKit's own service
worker in src/service-worker, registered by usePwa(), which precaches the
app files and leaves API requests to the network. A small plugin emits
manifest.webmanifest. static/sw.js removes the worker of older builds, and
llama-server serves a self-removing service-worker.js as well as sw.js
when the built-in UI is off or replaced.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: use SvelteKit 3's update detection and Svelte 5's current patterns

A page the service worker doesn't control now learns of a new build
from SvelteKit itself: `updated` from $app/state checks
_app/version.json on navigation, on focus and every hour. It replaces
the version Llama Chat kept in localStorage and compared by hand.
Reload on the update prompt pulls in the new worker and activates it
before reloading, so one click lands on the new build, whether or not
a worker controlled the page.

Following Svelte's current guidance:
- usePwa keeps its state in $state rather than a store, and no effect
  checks `browser`
- nearViewport, autoResizeTextarea and the custom CSS injection are
  attachments ({@attach}) rather than actions (use:)
- class lists use arrays rather than class: directives
- the chat contexts come from createContext, typed, in place of
  setContext/getContext with string keys
- ActionIconCopyToClipboard takes $props() rather than `export let`

Assisted-by: Claude (Opus 5.5 Medium)

* ui: move to ESLint 10

ESLint 9 is out of support. Every plugin the app uses supports 10:
typescript-eslint, eslint-plugin-svelte, perfectionist,
simple-import-sort, eslint-plugin-storybook and eslint-config-prettier.
@eslint/js and @eslint/compat follow.

ESLint 10's recommended rules ask for two things the code now does:
errors rethrown from a catch keep the caught error as their `cause`,
and no variable starts with a value that every path overwrites.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: override minimatch to clear npm audit's last finding

ESLint 10.14, @eslint/config-array 0.23.5 and typescript-eslint 8.72
depend on minimatch 10, which GHSA-g3hv-75wg-m249 (denial of service
via globparts amplification) affects below 11.0.1. Their fixed releases
move to minimatch 11 but are all younger than the repo's 7-day
min-release-age, so an override pins minimatch 11.0.5, the newest the
rule allows. The three only call new Minimatch(), match() and
minimatch(), which minimatch 11 keeps. npm audit now finds nothing.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-17 11:43:36 +01:00
Aleksander Grygier 3a7660371d ui: move the web UI to ui/apps/chat and call it Llama Chat (#32032)
* ui: move tools/ui to ui/apps/chat

The chat app moves to its own root folder, ui/apps/chat, as the first step of splitting the UI into one library
(ui/lib) and the apps built on it (ui/apps). The code is unchanged; every path that pointed at tools/ui now points at
ui/apps/chat: CMake adds it from tools/, and the Dockerfiles, the Nix package, the workflows, the labeler, CODEOWNERS,
.editorconfig, the git hooks, the dev and demo scripts and the docs follow it.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: name Llama Chat's package @ggml-org/llama-chat

It was still called llama-ui, the name of the library that now sits beside it
in ui/lib (@ggml-org/llama-ui). The package stays private; the app's display name
and the llama-ui.zip build artifact are unchanged.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: call the built-in web UI Llama Chat in docs, help text and comments

The app in tools/ui is Llama Chat; Llama UI is the component library it is
built from. Prose that said "Web UI", "web UI" or "WebUI" for the app now
names it: the READMEs, SECURITY.md, the --ui help text, the CMake option
description, one server log line, code comments and the Storybook intro.

Names that others depend on are unchanged: the deprecated --webui flag
aliases, the LLAMA_*_WEBUI CMake and environment names, the webui_settings
field, the storage migration keys and the Nix attributes.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: name tools/ui as a former path of Llama Chat in the code-review skill

Changes under the old tools/ui path, as well as tools/server/webui, still
get the Llama Chat checklist.

Assisted-by: Claude (Opus 5.5 Medium)

* ui: name Llama Chat's path in the security policy's exclusions

Llama Chat no longer lives under tools/server/, so the exclusion listed
under it now names ui/apps/chat. What is covered stays the same.

Assisted-by: Claude (Opus 5.5 Medium)
2026-11-16 17:23:36 +01:00
uvosandSigbjørn Skjæret 900f01ba09 CI/HIP: update fattn symbol names in hip-quality-check (#32331)
* CI/HIP: update fattn symbol names in hip-quality-check

* ignore import-not-found

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
2026-11-15 15:53:22 +01:00
Xuan-Son Nguyen 73f53317eb vendor: sync stb_image (#32305) 2026-11-15 09:37:30 +01:00
Todor BoinovskiandMax Krasnyansky ded522f830 hexagon: add support for SSM_SCAN op (#31921)
* hexagon: add SSM_SCAN Op support

* hex-ssm-scan: fix multi-device spliting

* hex-ssm-scan: remove redundant checks and make kernel use host config

* hex-ssm-scan: remove direct ddr access

* hex-ssm-scan: remove DMA ordering

* hex-ops: update supported ops

* hex-ssm-scan: rewrite the dma pipeline to match other ops (dbl.buf, etc)

* hex-ssm-scan: remove all fallbacks, only dma/vtcm paths

* hex-ssm-scan: add kernel_type to kparams and remove redundant recomputes

* hex-ssm-scan: avoid stalls in the overall pipeline (dma flush between chunks)

* hex-ssm-scan: vectorize softplus

* hex-ssm-scan: optimize hvx loops

* hex-ssm-scan: add HMX support

* hex-ssm-scan: add support for GGML_HEXAGON_SSM_SELECT

* hex-ssm-scan: cleanup kernel names

* hex-ssm-scan: simplify mdev checks

* hex-mdev: improve mdev row-split solver

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-13 12:27:51 -08:00
Max Krasnyansky 4df9b47976 ci: reduce qdc timeouts to avoid stalling pipelines and runners (#32148)
* ci: reduce qdc timeouts to avoid stalling pipelines and runners

* ci: fix flake8 warnings
2026-11-13 11:49:33 -08:00
Georgi Gerganov 6859813352 sync : ggml
[no ci]
2026-11-13 10:37:44 +02:00
Pascal bd2767d06b ci: fix HIP quality check by ignoring the FP8 vec FA kernel spill (#32183)
The D=256 vec flash attention kernel with F8_E4M3 K and V uses 257
VGPRs (255 + 2 spilled) on gfx908, one over the limit of the check.
The spill is a perf issue only, like the Q4_0 D=256 vec kernel
already in the ignore list.

Both variants of the kernel join the ignore list.

[no ci]
2026-11-13 10:25:08 +02:00
Xuan-Son Nguyen f7b886fe6f chat: allow TTS model to provide custom text normalizer via jinja (#32053)
* chat: allow TTS model to provide custom text normalizer via jinja

* rm tts_norm_is_reserved_name
2026-11-11 17:34:55 +01:00
Alessandro de Oliveira Faria (A.K.A.CABELO) b8abe41a50 vendor: update cpp-httplib to 0.66.0 (#32049) 2026-11-11 20:07:30 +08:00
Sigbjørn Skjæret 582eab7186 ci : bump ty to 0.0.100 (#32030)
* bump ty to 0.0.100

* rename rules

* type fixes

* type fix

* proper type fix
2026-11-10 21:54:14 +01:00
Aparna M PandMax Krasnyansky 76a67730ff hexagon: support for automatic loading of libhtp without legacy env vars (#31781)
* hexagon: use absolute skel path

* hexagon: support non-ascii dir and long path in skel-abs path

* hex-libdir: add support for FASTRPC_MANAGE_DSP_LIBRARY_PATH and GGML_HEXAGON_LIBHTP_DIR on all platforms

* hex-libdir: keep extra fwd slash in abs path on Android and Linux

* hex-libpath: always start with absolute path then fallback to session ctl then env

* hex-libdir: add CDSP1,2,3 to the fallback, needed for the IQ10

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-07 16:25:20 -08:00
Max Krasnyansky e9a2a44fe9 hexagon: n_hvx and n_threads cleanup (#31857)
* hex-nhvx: apply nhvx updates to the latest master

* hex-nhvx: set N_HVX and N_THREADS based on the arch to avoid overallocation

* hex-build: add support for enabling HTP_DEBUG via build.py

* hex-trace: move tracing state/helpers into new htp-trace.h header

* hex-dcache: cleanup dcache #defines

* hex-hnvx: minimize the diff by reflowing the macros

* hex-ci: fix python type/linter warnings

* hex-nhvx: consitent logging
2026-11-07 15:03:31 -08:00
e95426f169 llama_moe_cache: run missed experts on CPU, add tuning script (#30966)
* moe-cache : compute part of the misses on the CPU

--moe-cache-cpu-share F computes a share F of the MoE cache misses on the CPU
from the host experts instead of uploading them. The graph builds the expert
FFN twice: on the device from the cache slots, where the misses left to the CPU
read zero slots, and on the CPU from the host experts, where the experts
computed on the device have id -1. The two results are added before the down
bias.

- ggml-cpu: MUL_MAT_ID skips negative ids and writes zero rows
- sched: the copy callback is called for all the inputs of a split, and before
  the host sync when there are no events
- the slot lookup callback reads the inputs of the CPU part together with the
  selected experts, so the CPU part does not wait for the device, and its
  result is copied back asynchronously
- each token uses distinct zero slots, mul_mat_id does not support an expert
  selected twice by a token
- LLAMA_MOE_CACHE_TRACE writes the selected experts of each lookup, for a
  simulator of the cache

* scripts : add moe-cache-tune.py to pick the MoE cache arguments

The script wraps a llama-server command: it starts the server at two cache
sizes and three CPU shares, fits the upload and CPU speeds of the machine, and
replays the routing of the runs through a copy of the cache logic to predict
other sizes and shares. It prints the recommended --moe-cache-mib and
--moe-cache-cpu-share.

docs/moe-cache.md describes the MoE cache arguments and the script.

* add am17an as codeowner for MoE cache

* moe-cache : upload all the misses until the cache is full

With a high --moe-cache-cpu-share, few misses are uploaded, so an empty cache
fills slowly and more misses than needed go to the CPU. The CPU share now
starts once all the slots of a cache group hold an expert. moe-cache-tune.py
replays the same rule.

* add backend sync for async transfer queue

* moe-cache : let the scheduler copy the results of the CPU part

* Fix MSVC compilation

* remove sync change

* llama-graph : extract the CPU share condition into can_cpu_share

* llama-graph : make llm_graph_params::moe_cache const

The graph construction no longer updates the MoE cache directly: the CPU
parts of the layers are accumulated in the graph result and applied to
the cache in llama_context::process_ubatch, after the graph inputs are
set and before the graph is computed.

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (Mac Studio, M5 Ultra, 256GB)

* scripts : replay the ClassLRU policy in moe-cache-tune.py

* moe-cache : pick CPU misses by CRF instead of random (#31853)

* moe-cache : remove the backend logic of the CPU part from the graph

The CPU part reads the host experts, so the scheduler places it on the
CPU without set_tensor_backend. The cache gives no CPU map for batches
that a device would offload. Graphs with a CPU part look up their own
slot map, so the graph does not register the part with the cache.

ggml-backend : the inputs of the next split that are already computed
are copied to host memory before the current split runs, so a CPU split
does not wait for the device split before it. This replaces the stash
of the cache, and the copy callback is again called for host weights
only.

* llama-graph : return the MoE cache slots and CPU ids in a struct

The scheduler prefetch no longer checks that the next split runs on
another backend: the inputs of a split are never on its own backend,
so the input loop already skips that case.

---------

Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Alexey Kopytko <alexey@kopytko.com>
2026-11-08 00:20:58 +08:00
Adrien Gallouët 1416f76aa1 vendor : update cpp-httplib to 0.65.0 (#31731)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-11-05 15:34:18 +01:00
Georgi Gerganov 13ec8f614c sync : ggml 2026-11-04 12:00:37 +02:00
Todor BoinovskiandMax Krasnyansky bee23322f1 hexagon: add LIGHTNING_INDEXER support (#31289)
* hexagon: initial support for LIGHTNING_INDEXER op

* hex-lght-idxr: hmx for prefill

* hex-lght-idxr: q8_0 decode gather transpose, two-block decode compute

* hex-lght-idxr: pipelined hmx path for decode and prefill

* hex-lid: add support for K (f32)

* hex-lid: add support for K (f32)

* hex-lid: add support for q4_0

* hex-lid: remove scalar VTCM reads in the HMX q4_0 K prep and remove vector spills

* hex-lid: host/kernel kparams alighnment and remove sw devices in kernel

* hex-lit: update dma pipeline to use canonical dbl buffering in all paths

* hex-lit: add GGML_HEXAGON_LI_SELECT to force hmx/hvx paths

* hex-lit: add support for dims collapse

* hex-lit: make kparams constistent with other ops

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-03 21:17:30 -08:00
Todor BoinovskiandMax Krasnyansky 20a51f24a8 hexagon: add DSV4_HC_PRE and DSV4_HC_POST ops (#30953)
* hexagon: add support for DSV4_HC_PRE

* hexagon: hsv4-hc-pre plain/gated split kernel and l2 prefetch optimization

* hex-dsv4-hc-pre: double-buffered DMA for prefill

* hex-dsv4-hc-pre: refactor DMA block setup

* hex-dsv4-hc-pre: precompute kernel params

* hex-dsv4-hc-pre: support for n_hc != 4

* hexagon: add support for DSV4_HC_POST

* hex-dsv4-hc-pre: refactor gated PRE prefill DMA to dst writeback after each block

* hex-dsv4-hc-pre: remove in-loop vector spills and redundant device-side checks

* hex-run: add support for test-llama-archs tool

* hex-dsv4: refactoring and optimizations

Remove non-dma paths.
Clean up DMA pipelines and dma_addr_t casts.
Add dims collapsing support.

Tested with test-llama-archs -a qwen4ext and test-backend-ops

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
2026-11-03 08:26:18 -08:00
Aman Gupta 74101b3cda ci : update the HIP VGPR spill ignore list for the kv_rows FA kernel argument (#31550)
The kv_rows argument added to the FA kernels in #29510 changes their mangled names, so the known spills were no longer ignored.
2026-11-03 07:54:47 +02:00
Alessandro de Oliveira Faria (A.K.A.CABELO) f6312ad75e vendor : update miniaudio to 0.11.26 (#31459)
[no ci]
2026-10-31 23:37:20 +01:00
Alessandro de Oliveira Faria (A.K.A.CABELO) 1ab23c33b7 vendor : update cpp-httplib to 0.64.0 (#31400) 2026-10-30 11:49:58 +01:00
uvos 78febbad9d HIP/CUDA: enable sparse attn on CDNA (#31259)
* HIP/CUDA: enable sparse attn on CDNA

* Use __ballot for __ballot_sync on old ROCm versions

* Ignore new large head fattn mma variant
2026-10-29 10:14:11 +02:00
Georgi Gerganov 5d7793d6fc sync : ggml 2026-10-28 12:36:51 +02:00
Mengsheng Wu 6c44591049 snapdragon: switch QDC artifact upload and log download to v2 API (#31240)
Signed-off-by: Mengsheng Wu <mengshengwu@outlook.com>
2026-10-27 18:06:41 +01:00
Max Krasnyansky f0fb5984b0 hexagon: generic dims collapse/reshape for unary, binary and matmul ops (#31071)
* hex-tools: support for --compare option in the profile.py

* hex-unary: improve collapse logic to also do splits if nrows is small

* hex-colapse: handle all colapsing at the host and make it generic

* hex-colapse: remove leftover unary helper
2026-10-25 11:11:48 +02:00
Christian Kastner 707616feb5 ci: Fix --strict mode for API check (#30623)
abi-compliance-checker was running in strict mode unconditionally,
rather than honoring the CLI flag.
2026-10-20 13:38:45 +03:00
Max Krasnyansky 8466395353 hexagon: update to snapdragon-toolchain v0.8 and enable hexagon v85 (#30854)
* hex-build: bump toolchain, enable v85

* hex-build: bump toolchain to v0.8 in the CI and build scripts

* hex-build: add a little python script for swiv needed on some iot devices

* hex-build: fix flake8 warnings

* hex-arch: update version cap to v85

* hex-inspect: flexible TOOLS_ROOT

* hex-build: update to snake_case naming

* hex-build: update leftover old toolchain refs

* hex-target: update build and run scripts to support ubuntu with adb transport
2026-10-19 23:49:59 -07:00
7adf160c41 server: add --connect, allow p2p access to server outside LAN (#28277)
* init

* wip

* wire it up

* make it clear that it's experimental

* add props.connect_code

* nits

* fix windows build

* server: do not advertise a connect code without the tunnel

--connect-code on its own had no effect other than being served in
/props, so the Web UI showed a remote access code that no tunnel was
serving. Clear it when --connect is off, and say so once at startup.

The teardown comment claimed that terminate() closes the child stdout.
It does not: the log thread leaves fgets() only once every writer of
that pipe is gone, so a descendant of llama-connect would block the
join forever. Describe the real contract instead.

Also restore the --reasoning-preserve default in the generated docs.

* Update tools/connect/CMakeLists.txt

Co-authored-by: Adrien Gallouët <angt@huggingface.co>

* server: dial llama-connect on the bound port and a reachable host

Pass ctx_http.port so --port 0 and the GCP override work, pick the
dial host from --host (wildcards map to loopback, IPv6 is bracketed)
and refuse --connect when only unix sockets are bound.

---------

Co-authored-by: Pascal <admin@serveurperso.com>
Co-authored-by: Adrien Gallouët <angt@huggingface.co>
2026-10-20 00:23:14 +02:00
Alessandro de Oliveira Faria (A.K.A.CABELO) 7951f601e8 vendor : update cpp-httplib to 0.62.0 (#30785) 2026-10-19 20:23:23 +08:00
Georgi Gerganov 7db0d3d781 pi : turn .pi/gg into a system prompt addendum (#30762)
.pi/gg/SYSTEM.md repeated the full agent ruleset. Move it to .pi/gg/APPEND_SYSTEM.md and keep only the llama.cpp-specific rules, so it can be appended to a main system prompt with --append-system-prompt.

Drop the .pi/SYSTEM.md symlink setup from wc2wt.sh and pr2wt.sh. Worktrees no longer need it now that the folder holds an addendum instead of a full prompt.

Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB)
2026-10-19 09:11:13 +03:00
Sandro Hanea 83f9389419 ci : added ABI reports as github artifact (#30619) 2026-10-17 11:41:53 +02:00
Max Krasnyansky b797b9ca81 hexagon: tensor repacking rework and support for tiled and flat quantized and fp weights (#30635)
* hex-repack: Remove need_repack hack and streamline the rules -- breaks GET_ROWS tests for K-Quants, will fix in the next round

Add support for GET_ROWS with tiled Q5_K
Add support for raw k-quant tensors

* hex-get-rows: vectorize raw k-quant ukernels

* hex-binary: refactor dma-loop thread functions (no need for macros, etc)

* hex-quant: move/cleanup flat and tiled dequant ukernels

* hex-repack: add support for repacking/tiling F16/F32/BF16 2D weights

* hex-get-rows: remove redundant cast

* hex-repack: proper filtering for BF16 that were leaking through during the tests

* hex-repack: streamling tiling rules (broken MM for now)

* hex-repack: fix mmid after tiling updates

* hex-repack: rename repacked to tiled in all flags/funcs/kernels/vars

* hex-mm: fix valid_rows calc in various kernels

* hex-mm: remove unused hvx_mm_nx_2d_flat

* hex-mm: cleanup and optimize hvx-tiled vec_dot_f16

* hex-mm: another pass at hvx-tiled f16 matmul

* hex-get-rows: optimize small src1

* hex-get-rows: one more pass at recovering get-rows perf regression

* hex-repack: export GGML_HEXAGON_RT_REPACK via run.py

* hex-mm-tiled: fix accuracy regression with hmx-tiled-f16 matmul

* hex-mm: fold tiled_f16 checks into the general flow

* hex-get-rows: remove dead checks for buffer_is_hexagon and tiling during supports_op

* hex-quant: proper tail handling in new dequant ukernels

* hex-get-rows: remove unused code and cleanup support checks

* hex-mm: cleanup ne1 padding/tiling checks

* hex-mm: cleanup fusion checks for tiled tensors

* hex-repack: add a little can_repack helper to reduce duplication

* hex-repack: never repack host buffers

* hex-repack: factor out repack functions into separate .h and .cpp

* hex-tensor: cleanup tensor ifaces for tiled/padded sizes

* hex-repack: update k-quant tiling comments and cleanup #defines

* hex-dcache: factor out l2/dcache dirty tracking and flushing

* hex-tensor: update docs and remove #defs duplication

* hex-tensor: remove unused/incorrect get_data interface

* hex-get-rows: cleanup vtcm layout build

* tests: disable huge MM tests

* hex-dcache: add missing header
2026-10-15 20:30:11 -07:00
Georgi Gerganov a11b6530b5 scripts : add ci-status.sh to list active CI runs (#30586)
Small helper around the GitHub Actions API: lists the workflow runs that are not
finished yet (in_progress, queued, pending, waiting, requested), grouped by status.

Supports selecting runs by status, by branch (server-side, to reduce API traffic)
or by a regex on the branch name, and can cancel the selected runs via --kill.

Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
2026-10-15 16:28:09 +03:00
Georgi Gerganov c0cb53ac4e sync : ggml 2026-10-15 16:20:14 +03:00
Sigbjørn Skjæret 711780ffc7 ci : bump ty to 0.0.88 (#30587)
* bump ty to 0.0.88

* arange -> linspace.tolist() to pass expected type
2026-10-15 14:15:55 +02:00
Adrien Gallouët 3eb101360f vendor : update cpp-httplib to 0.61.0 (#30494)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
2026-10-14 08:28:24 +02:00