- remove build_delta_net_chunking and build_delta_net_autoregressive
- always use the fused gated delta net
- make graph_max_nodes independent of the ubatch size
- drop the fused GDN probes and fused node tags
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
ggml-sycl.dll built with oneAPI 2026.2 imports mkl_intel_thread.3.dll and libimalloc.dll, which are not part of the release package. LoadLibrary fails and the SYCL backend is silently skipped, llama.cpp falls back to CPU.
Adding both DLLs fixes the load: measured on Arc B570 with LFM2.5-8B-A1B Q4_K_M, pp4096 goes from 404 t/s (CPU fallback) to 2232 t/s (SYCL) with the extracted b12244 package.
Assisted-by: GLM (z-ai/glm-5.3-flash)
Co-authored-by: Asahi-Prv <Asahi-Prv@users.noreply.github.com>
* CUDA: re-instantiate graphs when the cluster dims of a kernel change
* CUDA: keep the cluster dims with the graph instance, check them only on Hopper+
Assisted-by: Claude Opus 5.5
* CUDA: use CUDA_CHECK and a helper struct for the graph cluster dims
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* CUDA: add a TODO for the graph cluster dims test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ROCm/HIP: fixed BLAS workspace instead of rocBLAS allocating one per call
Assisted-by: Claude Opus 5
* ROCm/HIP: corrected by only setting the workspace on ROCm 7.0 or newer
* cpu: accumulate flash attention V in F32 in the one-chunk kernel
The one-chunk kernel accumulated an F16 V cache in an F16 running sum
that is only normalized at the end, which overflows past 65504 on long
contexts: inf, then NaN on every following token.
Every V type now accumulates in F32, like the tiled kernel. F16 rows go
through ggml_vec_mad_f16_f32, which converts and multiplies in the same
register (F16C/AVX512, NEON, RVV) with no round trip through a temporary
F32 row, so decode keeps master speed on x86 and loses about 4% at 4k
context on ARM, where master used native F16 FMA.
* cpu: widen the F16 V rows like ggml_vec_dot_f16_f32
ggml_vec_mad_f16_f32 required F16C and FMA, so AVX builds without FMA
fell back to a scalar loop and ran 3x slower than master. It now
follows the guards of ggml_vec_dot_f16_f32: AVX512, then AVX with F16C
through GGML_F32x8_FMA, then NEON and RVV, then the scalar loop.
* additional FA test cases
Assisted-by: Codex
* vulkan: allow dequantized KV optimization with CM2
Enable the KV dequantization optimization with coopmat2, guarded by a heuristic
based on query tokens*GQA ratio.
While doing this we saw very large speedups for certain head sizes, and it was
because those head sizes were incorrectly treated as unaligned. So also fix
those to be treated as aligned, which should help for when the dequant
optimization doesn't apply.
* gqa uses nbq2 for the stride, so we need to check that value for alignment
* tests : use VARS_TO_STR18 for flash attention
Assisted-by: Codex
CodeMirror takes text input through EditContext on Chrome for Android
only, and there the chat editor never brings up the virtual keyboard.
The editor now always takes its input through contenteditable, as on
desktop and iOS.
A press in the input box outside the text also focuses the editor, in
the collapsed form too, so the whole box is a tap target again as the
textarea was.
MMF tiles 32 rows per block, so a BF16 matrix with few rows (e.g. the
qwen4exp hyper-connection projections, 320 rows with K = 10240) runs only
a handful of blocks and leaves most SMs idle at batch sizes 2..8. MMVF
runs a block per row and wins there when K is long enough.
MMVF's cost grows with rows x batch while MMF's stays roughly flat in that
range, so use MMVF only while rows * ne11 * 256 <= nsm * min(K, max_k) and
K >= 1024. The budget scales with the SM count, which keeps GPUs with few
SMs (e.g. the A16) on MMF. Blackwell allows up to 8 columns and
max_k = 6144; older GPUs stop at 5 columns and max_k = 4096.
Add eval cases at 128x2560, 512x2560 and 320x10240 for n = 2, 3, 4, 5 and 8.
* CUDA : add sm70 FlashAttention config case for the 256x256 head
* CUDA : tune sm70 FlashAttention for 256x256 ncols=32
Assisted-by: Codex
---------
Co-authored-by: Jirka Svítil <mistrjirka@users.noreply.github.com>
* cuda : add MMA FlashAttention support for head sizes 40 and 72
Head sizes 40 and 72 (used e.g. in vision towers of Gemma 3/4 and Qwen3-VL) were excluded from every tensor-core dispatch branch and fell back to the tile kernel.
The fp16 MMA fragment covers 16 halfs in the head direction, so the in-kernel geometry is padded to the next multiple of 16 (40 -> 48, 72 -> 80). Register arrays, shared memory tile strides and MMA loop bounds use the padded size, while global memory loads and stores the real one.
Enabled on NVIDIA Turing and newer while Volta and AMD keep the tile kernel. The RDNA/CDNA config rows and the mma_tile_sizes specialisations are needed because the explicit instantiations are compiled for every target of a translation unit.
* tests : add the head size = 40 encoder perf shape
* cuda : address review comments on the head size 40 and 72 kernels
* Initial CUDA WIP version
* Use fp8 intrinsics
* Do W8A16 for FP8 in GEMV case
(base) osimons@ub-osimons:~/lcpp_fp8.git$ ./build-x64-linux-gcc-reldbg/bin/llama-bench -m /mnt/share/gguf/nvidia/Qwen3.6-27B-NVFP4/Qwen3.6-27B-NVFP4_fp8.gguf -p 8196 -ub 512,2048 -d 0,32000 -n 256 --load-mode dio
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 32165 MiB):
Device 0: NVIDIA RTX PRO 4500 Blackwell, compute capability 12.0, VMM: yes, VRAM: 32165 MiB
| model | size | params | backend | ngl | n_ubatch | lm | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | ---------: | --------------: | -------------------: |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | pp8196 | 2781.53 ± 8.48 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | tg256 | 37.56 ± 0.02 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | pp8196 @ d32000 | 1616.31 ± 2.61 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 512 | dio | tg256 @ d32000 | 34.11 ± 0.03 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | pp8196 | 2542.00 ± 0.51 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | tg256 | 37.36 ± 0.02 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | pp8196 @ d32000 | 1762.86 ± 1.33 |
| qwen35 27B NVFP4 | 19.56 GiB | 27.32 B | CUDA | -1 | 2048 | dio | tg256 @ d32000 | 34.09 ± 0.02 |
* Optimize GEMV kernel for FP8
* Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.2
Fixes https://github.com/ggml-org/llama.cpp/pull/28413#discussion_r4044500099
* Remove trailing whitespace
* Add supported for strided copies to/from FP8 in CUDA
* Prepare Q_reg for FP8 FlashAttention in fattn-vec.cuh
* Fix odd-rows for convert/getrows, possible with FP8
* Fix FP8 fallback in cuBlas
* Gate mmvf_f8x2_e4m3_to_bf162 on FP8 availability, i.e. CTK >= 11.8
* Restrict (fused) MMVF to 2*ts alignment, and cublas FP8 fo 16-byte
* Add todo about accumulator/precision honoring
This is beyond the scope of the stacked PR
* Avoid races in fp8_fallback
* Allow Hopper and other BW SKUs for FP8
* Use intrinsic over explicit manual conversions
* Resolve TODO
As per spec, we don't have to preserve signs
* Add commend on cublaslt workspaces
* Use convert_unary_cont_cuda over block dequant in convert.cu
* Use get_rows_cuda_float for fp8 kv cache
* Make FP8 copies work on all IHV's
* More cleanup
* Simplify by falling back to shared cuBLAS helper
* Fuse weight_scale multiplication for FP8 also
* More code-cleanup
* Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.4
Only cvt.rn.bf16x2.e4m3x2 is exposed from 13.2 onwards, so
93e98b9203dfacb85f9ac665f5f0482a8c6b4192 was incorrect fix
* Use F32 activations in fallback mmvf path
* Explicitly link against cublaslt
* Move FP8 conversion intrinsics into ggml_cuda_cast
Cleaner this way
* WIP src1/prec for fp8 in MMVF
* Remove comment
* Keep only f8 x f8 KV cache for now
* Just do FP8 x FP32 x FP32 in MMVF as its fast enough
* Update ggml/src/ggml-cuda/ggml-cuda.cu
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
* Add batched FP8 GEMMS in cublas as suggested by code review
https://github.com/ggml-org/llama.cpp/pull/28413#issuecomment-6676015644
* Extract fp8x2 -> float2 load into helper
* Searapte copy checks to/from fp8
* Compiler massaging to avoid repeated address offsets
---------
Co-authored-by: Aman Gupta <amangupta052@gmail.com>
Modify `ggml_sycl_op_clamp` to use the unary helper function
`ggml_sycl_detail::ggml_sycl_op_unary` to handle non-contigous or
strided views. The fix uses a pattern that matches other stride-safe
unary ops, and is a simpler execution path.
Deleted dead template `unary_op_clamp_kernel`, the flat `clamp` kernel
launcher from element_wise.cpp, and deleted (now) unused
`SYCL_CLAMP_BLOCK_SIZE` from presets.hpp.
The hidden desktop strip relied on inherited `visibility: hidden` to keep
its content from catching clicks. Since the sidebar's sections fold
(#31936), an open section's CollapsibleRegion sets `visibility: visible`
again, and its rows' context-menu triggers carry an inline
`pointer-events: auto`, so they caught clicks meant for the page.
Make the hidden strip's content inert, which no descendant can undo, and
add a client test that hit-tests the hidden sidebar's area.
Fixes#31971
Assisted-by: Claude (Opus 5.5 Medium)
* media : remove the legacy llama0 and llama1 brand assets
The brand assets live in ggml-org/llama.brand; none of these is referenced
in the repository.
Assisted by: Claude (Opus 5.5 Medium)
* readme : use the new llama.cpp cover from llama.brand
Assisted by: Claude (Opus 5.5 Medium)
* ui: show audio in Discover for models whose mmproj has an audio encoder
Discover took audio only from the pipeline tag, so omni models tagged
any-to-any, such as Gemma 4 and LFM 3 Omni, showed vision but never audio,
though their mmproj carries an audio encoder.
Read the encoders from the mmproj's GGUF metadata instead: a ranged read of
its first 64 KB, once per repo, for clip.has_vision_encoder and
clip.has_audio_encoder. The Discover list and details now show audio when
either the pipeline tag or the mmproj says so.
Assisted by: Claude (Opus 5.5 Medium)
* ui: take Discover modalities from the mmproj's encoders in one place
Review fixes for the mmproj header read:
- drop vision for audio-only mmprojs: the converter writes
clip.has_vision_encoder only when true, so a header read in full without
it has no vision encoder; a header cut short leaves vision as it was
- accept only a 206 to the ranged read, so a server that ignores Range
never streams the whole mmproj
- encode the mmproj path in the resolve URL
- derive the list and details modalities with one HuggingFaceService
helper, so both views agree
- point the key comment at tools/mtmd/mtmd-impl.h, where the keys live now
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep a gated repo's mmproj from being read again on every mount
A gated, private or missing mmproj answers 401, 403 or 404 to every read,
so caching nothing made each mount of its Discover row request it again and
log an error. Cache it as unknown instead: no audio, and vision from the
mmproj being there, as before the header was read.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : move Models, MCP and Settings to a sidebar footer
The sidebar's actions split in two: New chat, Search and Projects stay at
its top, and Models, MCP and Settings move to a footer at its bottom, under
a thin rule, in the expanded sidebar and its collapsed strip alike. On a
phone the footer keeps clear of the home indicator, and Models stays
desktop-only as before. The search dialog still lists every action.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : group the sidebar footer in an inset surface
The design system gains inset-group-surface, for rows grouped inside a glass
panel: white on the light glass and a faint tint of the text colour on the
dark one, with the sidebar's own ambient edge, no inner highlight, a faint
wide shade under it and the card radius. It is matte-glass pointed at its
own component tokens, like the sidebar and chat form surfaces, and reads as
part of the panel: no tight contact shadow or solid border like a button,
no inset shade like a field.
The sidebar footer uses it in place of the rule above Models, MCP and
Settings. DESIGN.md and the Surfaces page document it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : give the sidebar's logo button the add button's key look
Whenever the sidebar shows, open or peeking, its logo button takes the same
key as the chat form's add button (matte-key, white in light mode, a light
tint in dark), in place of its own muted fill. While the sidebar is hidden
the logo has no fill and no shadow, and hover and focus never change the
fill. It keeps its round shape and its own size.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : put search in a round key at the top of the phone sidebar
On a phone the sidebar's Search row moves to a round key at the right end
of its top row, across from the logo, with the same key look as the logo
and the chat form's add button. It opens search as the row did. Desktop
keeps the Search row.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : size the phone sidebar for a thumb and list its footer with the actions
On a phone the sidebar's actions and rows are 48px tall at 17px, flush one under
the other with 16px corners, its section headings 15px and grey, and its groups
28px apart. MCP and Settings join the top actions there: the footer is a desktop
pattern.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : mark the selected sidebar row with a tinted inset surface
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the phone sidebar's round keys white on the grey page
Assisted-by: Claude (Opus 5.5 Medium)
* ui : open Settings and MCP as pages on a phone, dialogs elsewhere
On a phone the sidebar opens them at #/settings and #/mcp-servers, with the menu key and a
title like any phone page. Pinned under the status bar, Settings' section tabs fill the
screen above them in the page's colour.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : open a phone sidebar's pages behind it, settled before they slide back
A tap in the phone sidebar now opens its page while the current one is still aside and slides
back once it has rendered, instead of swapping pages mid-slide. The slid page's fixed bars are
pinned again as the page under them changes, the lock ends with the slide itself, and a
phone's new chat shows its form and greeting at once, without the move and entry they have
on a desktop.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: style the sidebar footer group and the selected row with master's classes until the design system lands
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add GlassKey with master's styles for the phone sidebar's round keys
Assisted-by: Claude (Opus 5.5 Medium)
* ui: center the top edge of a new chat's form on the screen
The form rested 42px below the middle up to 2xl and 6px above it past
that; one drop, from the resting card's height, now puts its top edge on
the middle at every width from md up.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: round the sidebar's action rows concentric with the footer group
The sidebar stacks four surfaces: the page, the sidebar, the footer group
inset in it and a row in the group. app.css now names the group's radius
and padding, and derives the row radius from them (16px less 4px), so the
footer's rows run parallel to its corners; the actions at the top of the
sidebar share that radius. The footer moves to its own component, the
README documents the pattern, and a client test holds the rule on the
rendered footer.
A selected action row was bg-accent, the same grey as a hovered row in
light and darker than the footer's fill in dark, so it read as hovered or
not at all; it is now raised and edged, while a hover only tints.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: import demo chats through the Settings page on a phone
A phone opens Settings as a page now, not a dialog, so the demo harness
waited for a dialog that never came and every phone demo with traces
failed; it now imports through whichever of the two opens.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: count the sidebar's rows from their items in the surface test
The test expected three footer rows and two top actions, which the
stacked PRs change as they add Projects, Library and Servers; it now
expects as many rows as the sidebar has items.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a line on Meta+Enter off macOS too
The chat form takes Meta+Enter as Ctrl+Enter and leaves the newline to
the editor, but the editor bound only Mod-Enter, which is Ctrl off
macOS, so on Linux Meta+Enter neither sent nor added a line. This failed
the client test on CI's Linux runners, on master too.
Assisted-by: Claude (Opus 5.5 Medium)
This commit removes the unused ids and type_src0 parameters from
quantize_row_q8_1_cuda.
The motivation for this is that these parameters were required in the
past because this function was used in combination with the function
pointer quantize_cuda_t, which has been now been removed, and this
function no longer needs to adher to the function pointer signature.
Refs: https://github.com/ggml-org/llama.cpp/pull/31884.
* Add FP8 KV-cache support
* Add tests for scaled, FP8 KV-cache to test-backend-ops
* Fix get_*_scale mapping and dequnatize FP8 also before rope shift
* Normalizes cache-scale names independently of projection layout.
While fused KQV is currently impractival for non-shared scales, this is
an improvement for future use regardless
* Enable Gemma 4 Assistant with FP8 by storing and fetching scale ptrs
* Update comment wording
* Move E2E FP8 KV cache tests to test-llama-archs
They belong there as they are not just a test for ggml's backend ops
* Increase support surface, remove seed
* make scaled kv generic
* Fix CI failings for test-kv-cache
https://github.com/ggml-org/llama.cpp/actions/runs/38718763447/job/116367842578?pr=28392
* CI: execute kv-cache-scale tests
* exclude fp8-kv tests from OpenVINO
* Fix linting errors of test_kv_cache_scales
* Disable kv-ffload=false + multi-GPU tests for FP8
See https://github.com/ggml-org/llama.cpp/actions/runs/38952722084/job/117148353699?pr=28413
* Adjust test label for test-kv-cache.cpp
* Fix conflict-resolution mistakes in test-kv-cache.cpp
https://github.com/ggml-org/llama.cpp/actions/runs/40554674463/job/122367426461?pr=28392
* Move KV tests to test-fp8.cpp
https://github.com/ggml-org/llama.cpp/pull/27512#discussion_r4410518158
* Remove tmp file for test-kv-cache-scales.py on Windows as well
* Fix fp8 kv cache test by population kv_source_impl
* Exclude models that consume KV cache directly
* Remove obsolete comment
* Allow BF16 scales as well for KV cache
* Rename *_cache_scale to *_cache_s
https://github.com/ggml-org/llama.cpp/pull/28392#discussion_r4517194999
* Move `is_type_scaled()` to `llama-kv-cache.cpp`
* ui: let a menu action open a submenu
Group and project pickers open as flyouts from a chat's menus. Master's menu
list has no submenus, so bring them back next to its tooltips, with a close
callback for picks made inside them.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : store groups and let conversations join them
Groups live in their own IndexedDB table (schema version 2) with a name.
groupsStore creates, renames and deletes them; deleting a group keeps its
chats, out of any group. A conversation records the group it is sorted
into, and a group picked before the first message is kept pending and
applied when the conversation is created.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : show each group as a sidebar section
Every group is a sidebar section next to Pinned and Recent, folding,
showing its first six chats and dragged into place like them. A chat
dropped on a group joins it and leaves Pinned; dropped on Recent, it
leaves its group. Recent is called Ungrouped once a group exists, and its
menus create a group in a small naming dialog; so does a right-click
anywhere in the list that no row or heading takes, down to the empty space
under the last section. A group's heading starts a new chat in it, and its
menus rename or delete it; deleting keeps its chats, back in Ungrouped.
Assisted-by: Claude (Opus 5.5 Medium)
* ui : move chats into groups from their menus
A chat's menus, in the sidebar and on right-click, move it into a group
or take it out of its group. Desktop menus show a searchable group flyout
whose New group entry names a group in the group dialog and moves the chat
into it; a phone gets a picker dialog instead, which creates a group from
what is typed.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open the group picker from a phone's hold menu
The hold menu is anchored to the row, so it showed the desktop flyout,
which ran off the left edge of the screen and whose New group closed
without naming one. On a phone, Move to group now opens the picker
dialog, as intended.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: open a new chat in a tab of its own
With conversation tabs on, New chat from the sidebar, a group's heading
or the shortcut replaced the open chat's tab. It now adds a tab, as the
tab bar's "+" does, and the open chat keeps its own.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: list a phone's groups under the picker's search
On a phone the picker fills the screen, and its grid spread the rows
over the height, so the list sat in the middle. It is now a column: the
list starts under the search and scrolls in the rest of the screen.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: fix gaps in chat groups found in review
- A phone shows a group heading's New chat and menu, and the Ungrouped
heading's menu, which were reachable only by hover or right-click.
- A pinned chat moved into a group from its menu leaves Pinned, as a
drag onto the group already did, so the move shows.
- Only the sidebar's New chat, a group's New chat and the new-chat
shortcut open a new tab; stepping past the last chat, search and the
models drawer show the new-chat screen in the current tab again.
- An export carries each chat's group, and an import sorts the chat
into the group of that id or name, or creates it; a group the file
does not carry is dropped.
- The group picked with a group's New chat survives leaving the
new-chat tab and coming back.
- A phone's menus skip any submenu in the shared menu list, not only
the group one.
- Deleting a group forgets its section's place and fold.
- Groups load newest first by creation, so a rename no longer moves one
after a reload.
- groupsStore looks groups up in a map and answers which live group a
chat is in, for the three places that checked it by hand.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep a forked chat in its source chat's group
A fork copied the working directory but not the group, so it landed in
Ungrouped.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: move a chat with its whole fork family between groups
Moving a chat into a group, or out of one, takes its parents and its
children along, so a fork tree never splits across sidebar sections. A
pinned member that joins a group leaves Pinned with the rest. Group
sections render the tree with its depth, as Ungrouped does.
* ui: show the group name prompt as a small centered card on phones
The new and rename group prompt opened full screen on a phone, like a
large dialog. It is now built on the alert dialog, as the chat rename
prompt is, so a phone shows it as a card in the middle of the screen.
Its Cancel is a plain button, so Enter in the name field saves.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: space a phone's group sections like Ungrouped
On a phone a group or Pinned section put its first chat right under the
heading, where Ungrouped leaves a row gap. They now leave the same gap,
and the sections sit a little further apart than the rows inside them.
Assisted-by: Claude (Opus 5.5 Medium)
---------
Co-authored-by: Pascal <admin@serveurperso.com>
This commit removes the unused quantize_cuda_t from quantize.cuh.
The motivation for this is that the usage of this typedef as a function
parameter was removed in Commit 74976e1aef
("CUDA: remove -sm row, refactor cuBLAS (#24216)") and there is no longer
any usages of it.
The sampling chains are read on the host after the graph compute, and the
graph addresses are reused across builds with different selected rows: the
token-generation reserve graph has one output row per sequence, while the
actual block draft has n_max rows per sequence. The reserve allocates the
unselected chains without keeping them, so their buffers alias, and the
actual graph then reuses those addresses by node index.
set_outputs only marked the tensors stored for the selected rows, so the
unselected per-(sequence, row) chains were freed and reused. Collect every
chain tensor in t_sampling_all and mark all of them as outputs there.
Reproduces with -np 2 (non-unified KV) and 2 concurrent requests for
draft-dspark, draft-dflash and draft-eagle3; with different prompts one
sequence's acceptance collapses (eagle3 0.15 vs 0.40). After the fix:
dspark 0.71/0.78, dflash 0.56/0.44, eagle3 0.35/0.42.
Assisted-by: dsh:llama.cpp/ggml-org/DeepSeek-V4.1-Flash-1017-GGUF (Mac Studio, M5 Ultra, 256GB)
* ui: expand the chat form into a large editor for longer messages
The chat form expands into a large editor once a message has markdown
beyond plain paragraphs or reaches four lines, and a Maximize button in its
top-right corner toggles it by hand. The expanded form starts around half the
screen height and grows with the text up to the space below the tabs bar,
animating its height by mirroring the editor's content height.
On a new chat the expanded form keeps the collapsed form's top edge and the
greeting only fades out, waiting for a collapse to settle before it fades
back in. In a conversation the messages follow the form as it grows, so the
latest one stays in view above it.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: make the expanded chat form a full-screen sheet on phones
In the expanded editor Enter adds a line and Cmd/Ctrl+Enter sends.
On a phone the expanded form is a full-screen sheet with no corners or
border. The page around it and Safari's bars take its colour, and the
sidebar's logo button and the working-directory chip step aside while it is
open. It moves by relative offsets and a clip on the main thread, so its
edges, text, actions and chip stay in step in every engine, and its content
keeps the collapsed card's places and widths, so nothing reflows when it
settles. Expanding does not focus the editor on a phone.
On a new chat on desktop the form is lifted by its sticky bottom offset
instead of a translate, and the expanded size is built on the collapsed
card's real height, so its top edge holds still through any sequence of
quick toggles.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: hide the phone menu toggle and the sheet's focus edge while expanded
On a phone the menu toggle rides on the page, above the chat form: it now
fades out while the expanded form covers the screen, and comes back when
it collapses. The expanded sheet also drops the glass rim and inner ring,
so focusing the editor no longer outlines the whole screen.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the expanded phone sheet clear of the status bar
The expanded chat form fills the screen on a phone, under the status bar
an installed app draws over the page. Its text now starts below the bar
(pt-safe) and its collapse button sits below it too (top-safe-2.5).
Assisted-by: Claude (Opus 5.5 Medium)
* ui: hide the phone's context gauge while the chat form is expanded
The gauge pinned to the phone's top right sat over the expanded sheet's
collapse key. Like the menu toggle across from it, it now fades out while
the expanded form covers the screen and comes back when it collapses.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: lift the expanded phone sheet's actions above the iOS keyboard
Assisted-by: Claude (Opus 5.5 Medium)
* ui: move the chat form with a short, quick drift, without sliding after it lands
Its position transition came back on before the browser had laid out the new position,
so it slid the form there after the fade; the drift is now 10px each way and quicker.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: style the expanded chat form with master's surfaces until the design system lands
Assisted-by: Claude (Opus 5.5 Medium)
* ui: tighten the expanding chat form after review
- Expand on markdown only in finished lines: a lone `-`, `#` or `1.` on the
line being typed no longer opens the large editor.
- Show "Cmd/Ctrl+Enter to send" by the send button while expanded.
- Move the expansion state, the phone sheet and the held collapsed height
into useChatFormExpansion, and the sheet's geometry into pure functions.
- Drop the working-directory chip's phone animation, which never ran since
the phone chip moved to the + menu.
- Clearing or sending mid-move cancels the sheet's animation.
- A click only continues at the end of the text when it lands below it.
- Reuse ActionIcon for the toggle and INPUT_CLASSES for the card, derive the
centered form's offset from its drop, name the shared easing and fade.
- The greeting owns its fade behind the expanded form.
- MarkdownEditor's view is reactive, so following its content starts once it
exists.
Assisted-by: Claude (Fable 5.2)
* ui: keep the expanded chat form to the message being written
- The form expands on its own only for what is written or pasted, and
another conversation starts collapsed: a restored draft no longer opens
the large editor, or the phone sheet, on arriving at a chat.
- In the expanded editor the pickers open by the line being written, above
it or, short of room, below, instead of above the form's top edge.
- The sidebar's swipe is off while the phone sheet covers the screen.
- The send hint shows only beside the send button, where the row has room.
- The inner area keeps master's focus shadow, except on the phone sheet.
- The card and the form's position honour reduced motion.
- The phone greeting's place accounts for the wrapper it is counted from.
- The sheet's animation tolerates the form going away mid-move.
Assisted-by: Claude (Fable 5.2)
* ui: turn the typed pickers off on phones, and scroll long messages
- On a phone, typing `/` or `@` opens no picker for now: each mobile
browser moves the page its own way as the keyboard opens, so a list by
the form can not stay clear of the text being written. A note marks it
for later.
- The editor scrolls inside the form again: MarkdownEditor's own box sat
between the input and the editor without taking its height, so a message
longer than the form ran under the actions instead of scrolling.
Assisted-by: Claude (Fable 5.2)
* ui: settle the expanding chat form's review findings
- A phone expands the form only by hand: the sheet switching in mid-typing
moves the text under the keyboard.
- The typed pickers are off wherever a finger is the main pointer, not
below a viewport width, so a narrow desktop window keeps them and an
iPad loses them. The prompt picker opened from the + menu floats by the
sheet's actions on a phone.
- A message put back after a failed send expands the form again.
- Only the left button moves the caret from a click below the text.
- The sheet's fill is one token, used by the sheet, the page and Safari's bars.
- The collapsed-height hold is its own module and no longer finds the
editor's elements by class name.
- Shorter comments.
Assisted-by: Claude (Sonnet 5.5)
* ui: keep a one-line message on the chat form's actions row
On a desktop the main chat form is a single pill while its message is one
line: the add button, the text, then the model selector and send. A second
line, or a line that wraps beside the actions, brings back the two rows,
and they stay until the text is shorter than where it wrapped, so the
wider row can not flip it straight back. The editor stays mounted: the
actions join the text's row by CSS alone.
The single row's corners are `--chat-form-single-row-radius` in app.css, a
pill by default; the "Scrapbook / ChatForm single row" story shows the real
form with other radii.
Assisted-by: Claude (Opus 5.5)
* ui: give the one-line chat form 1.25rem corners
Assisted-by: Claude (Opus 5.5)
* ui: pick the working directory from the + menu on desktop too
The chip under the chat form is gone: the + menu offers the working
directory everywhere but a phone's chat, whose header shows it. The menu
item names the directory once one is set, and the picker gains a row to
clear it, which the chip's X used to do.
Assisted-by: Claude (Opus 5.5)
* ui: put the single-row chat form behind a Display setting
"Single-row chat form for short messages", on by default.
Assisted-by: Claude (Opus 5.5)
* ui: send the chat form on Enter, and add a line with Shift or Ctrl/Cmd
Enter sends wherever the caret is, the expanded editor and code blocks
included; Shift+Enter and Ctrl/Cmd+Enter add a line, continuing lists and
quotes. With "Send message on Enter" off the two swap as before. The
expanded editor's send hint and the code-block rule are gone with it.
Assisted-by: Claude (Opus 5.5)
* ui: start the chat form on the row layout it will settle on
Assisted-by: Claude (Opus 5.5)
* ui: show a set working directory under the chat form again
On desktop the chip under the form is back once a directory is set, and
the + menu offers the directory only until then.
Assisted-by: Claude (Opus 5.5)
* ui: add a line on Enter with an on-screen keyboard
A phone's Return has no Shift, so on a touch screen Enter adds a line and
the send button sends, as "Send message on Enter" turned off does; on a
desktop Enter still sends.
Assisted-by: Claude (Opus 5.5)
* cuda : Lowering VMM pool reservation size to 8 GB on GPUs with combined memory due to CUDA out of memory errors
Assisted-by: Gemini Argon
* cuda : Halve VMM pool reservation size and retry on failure
Assisted-by: Gemini Argon
* cuda : only shrink VMM pool on CUDA_ERROR_OUT_OF_MEMORY
Assisted-by: Gemini Argon
* cuda : log warning when VMM pool reserves less than max size
Assisted-by: Gemini Argon
ggml_cuda_graph, ggml_cuda_concurrent_event and ggml_cuda_stream_context
are only used by ggml-cuda.cu, but keeping them in common.cuh meant any
change to them rebuilt every CUDA translation unit. Move them to graph.cuh
and leave forward declarations in common.cuh. The context constructor,
cuda_graph() and any_cuda_graph_*() are now defined in ggml-cuda.cu, and
the stream context is held through a unique_ptr.
Also define ggml_cuda_kernel_can_use_pdl once in ggml-cuda.cu instead of
as a static function in the header, so there is one PDL cache instead of
one per translation unit, and fold the GGML_CUDA_PDL env check into it.
* ui: use a live-preview markdown editor for the chat form input
Replace the contenteditable rich input with a CodeMirror editor that shows
the markdown source of the block being edited and renders every other block
through the MarkdownContent pipeline and stylesheet, so they match chat
messages. Plain text types exactly as before, the value stays the exact
markdown source, and mention links show as atomic badges.
The editor lives in a shared MarkdownEditor component, and the highlight.js
theme loader is shared between MarkdownContent and SyntaxHighlightedCode.
The old rich input, its tokenizer, source history and their tests go away.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: give the production build more memory, note what mdsvex is for
The single-bundle build now includes CodeMirror and runs past Node's default
heap limit on CI runners. Raise it in the build script, so local and CI builds
behave the same on every platform.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep the live preview's blocks while only the cursor moves
Moving the cursor re-walked the syntax tree and rebuilt every block's
decoration. Keep the blocks with the decorations and rebuild only when the
text changes or the cursor enters another block.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: point the chat form story at the editor and drop the engraved placeholder until the design system lands
Assisted-by: Claude (Opus 5.5 Medium)
* ui: name the markdown editor for screen readers
CodeMirror's content is a textbox with no name of its own, so the chat form
input was announced as an unnamed text box, and could not be found by its name
as the old input could. It takes an aria-label, its placeholder by default,
which makes the chat form's "Type a message..." again.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: format the chat form input's classes
Assisted-by: Claude (Opus 5.5 Medium)
* ui: share the markdown styles and diagram drawing between messages and the editor
The editor and MarkdownContent each @imported markdown-content.css into their own
<style>, so the bundle carried two scoped copies of it. Both now render into a
MarkdownSurface, the one element that carries the stylesheet; the few rules for
MarkdownContent's own template elements are global under .markdown-content.
Drawing mermaid and svg blocks and the actions of their header buttons move to
markdown-diagrams.ts, so the editor's rendered blocks can draw diagrams the same
way messages do.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: harden the live preview editor after review
- onChange skips only the editor's own syncs from `value`, marked by an
annotation, instead of reporting only edits that carry a user event
- Enter and the mention badges ask the syntax tree whether they are in code, so
~~~ fences and indented code count, and a closing fence ends a block
- a picked mention is one undoable edit, through replaceText
- rendered blocks draw mermaid and svg diagrams with their header buttons, a $$
math block with blank lines renders as one block, and a link reference
definition stays raw instead of rendering to nothing
- rendered blocks render again when the mention path settings change
- a drag in a rendered block selects its text and copy takes it, across blocks;
a click still opens the block
- the block margins are the ones a chat message renders (each heading is a first
child there), checked by a test; the blank lines after a raw block are its gap
- the tree comes from syntaxTree, without forcing a full parse on every edit
- placeholder, label, extensions, mention widgets and render key follow prop
changes after mount, through compartments
- the editor uses the markdown language alone, which drops the html, css and
javascript parsers markdown() pulls in for html tag completion (-70 KB gzip)
Assisted-by: Claude (Opus 5.5 Medium)
* ui: pin the CodeMirror packages and set the build heap in .npmrc
The new dependencies are pinned like the rest of package.json. The build heap
moves from a node call on vite's bin path to npm's node-options, so the build
script is plain `vite build` again and the setting reaches it on every platform.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: update the live preview block by block and estimate block heights
An edit used to build every block's decorations and widgets again, which the
editor then compared one by one. The previous decorations, and the mention badges
of raw blocks, now move through the edit, and only the blocks whose range, text,
raw or rendered state, or gap changed get new ones; the texts of the blocks an
edit left alone are reused instead of sliced again. A test drives random edits,
cursor moves and focus changes and checks each result against a full build.
Rendered blocks report the height they were last measured at, or a guess from
their text, so CodeMirror's height map for blocks off screen in a long document
stays close and scrolling stays steady.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: skip the editor's redundant syncs, reconfigures and theme reloads
- syncing `value` compares it with the text the editor last reported or synced, so
a keystroke coming back through onChange no longer serializes the document again
- the prop effects reconfigure the editor only when a prop changed, not once more
right after it is created (which rebuilt the whole preview)
- loadHighlightTheme leaves the theme's style element alone while the mode stays,
so a message or an editor mounting no longer restyles the whole page
Tests check that typing in one block leaves the other rendered blocks without a
single DOM mutation, and that mounting reconfigures nothing.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: simplify the live preview to a plain rebuild in three small modules
Rebuilding the decorations of every block on an edit measures the same as
patching only the changed ones (0.09 ms against 0.11 ms per keystroke at
200 blocks, 0.9 ms against 1.1 ms at 2000), and equal widgets keep their
DOM either way, so the incremental update and its test go.
- markdown-blocks.ts collects the document's blocks, markdown-block-widget.ts
renders one, markdown-live-preview.ts decides which ones show raw text
- one bounded cache holds a block's HTML and its measured height
- the render key is a plain option: a new one reconfigures the preview,
which keeps the editor's focus
- MarkdownEditor follows its props with one helper instead of five effects
Assisted-by: Claude (Opus 5.5 Medium)
* ui: keep setext headings raw in the live preview editor
`test` over `---` is a setext heading, so the preview rendered it as an h2
and the underline vanished once the cursor left the block. The editor now
shows a setext heading's text as typed at all times, like a link
reference definition.
Assisted-by: Claude (Opus 5.5 Medium)
* ui: add a setting to turn the live markdown preview off
"Live markdown preview in the message input" (General) is on by default.
Off, the input shows the plain markdown text, with file mention badges
still in place; switching it takes effect without a reload.
Assisted-by: Claude (Opus 5.5 Medium)
normalize_quotes_to_json copies raw control characters verbatim while
converting pythonic tool arguments to JSON, so calls like
python(code='a<LF>b') produce invalid JSON in the arguments returned to
clients; replaying that history then fails hard. Escape C0 control
characters inside quoted strings via escape_json_string_inner.