server: introduce API for serving / loading / unloading multiple models (#17470)

* server: add model management and proxy

* fix compile error

* does this fix windows?

* fix windows build

* use subprocess.h, better logging

* add test

* fix windows

* feat: Model/Router server architecture WIP

* more stable

* fix unsafe pointer

* also allow terminate loading model

* add is_active()

* refactor: Architecture improvements

* tmp apply upstream fix

* address most problems

* address thread safety issue

* address review comment

* add docs (first version)

* address review comment

* feat: Improved UX for model information, modality interactions etc

* chore: update webui build output

* refactor: Use only the message data `model` property for displaying model used info

* chore: update webui build output

* add --models-dir param

* feat: New Model Selection UX WIP

* chore: update webui build output

* feat: Add auto-mic setting

* feat: Attachments UX improvements

* implement LRU

* remove default model path

* better --models-dir

* add env for args

* address review comments

* fix compile

* refactor: Chat Form Submit component

* ad endpoint docs

* Merge remote-tracking branch 'webui/allozaur/server_model_management_v1_2' into xsn/server_model_maagement_v1_2

Co-authored-by: Aleksander <aleksander.grygier@gmail.com>

* feat: Add copy to clipboard to model name in model info dialog

* feat: Model unavailable UI state for model selector

* feat: Chat Form Actions UI logic improvements

* feat: Auto-select model from last assistant response

* chore: update webui build output

* expose args and exit_code in API

* add note

* support extra_args on loading model

* allow reusing args if auto_load

* typo docs

* oai-compat /models endpoint

* cleaner

* address review comments

* feat: Use `model` property for displaying the `repo/model-name` naming format

* refactor: Attachments data

* chore: update webui build output

* refactor: Enum imports

* feat: Improve Model Selector responsiveness

* chore: update webui build output

* refactor: Cleanup

* refactor: Cleanup

* refactor: Formatters

* chore: update webui build output

* refactor: Copy To Clipboard Icon component

* chore: update webui build output

* refactor: Cleanup

* chore: update webui build output

* refactor: UI badges

* chore: update webui build output

* refactor: Cleanup

* refactor: Cleanup

* chore: update webui build output

* add --models-allow-extra-args for security

* nits

* add stdin_file

* fix merge

* fix: Retrieve lost setting after resolving merge conflict

* refactor: DatabaseStore -> DatabaseService

* refactor: Database, Conversations & Chat services + stores architecture improvements (WIP)

* refactor: Remove redundant settings

* refactor: Multi-model business logic WIP

* chore: update webui build output

* feat: Switching models logic for ChatForm or when regenerating messges + modality detection logic

* chore: update webui build output

* fix: Add `untrack` inside chat processing info data logic to prevent infinite effect

* fix: Regenerate

* feat: Remove redundant settigns + rearrange

* fix: Audio attachments

* refactor: Icons

* chore: update webui build output

* feat: Model management and selection features WIP

* chore: update webui build output

* refactor: Improve server properties management

* refactor: Icons

* chore: update webui build output

* feat: Improve model loading/unloading status updates

* chore: update webui build output

* refactor: Improve API header management via utility functions

* remove support for extra args

* set hf_repo/docker_repo as model alias when posible

* refactor: Remove ConversationsService

* refactor: Chat requests abort handling

* refactor: Server store

* tmp webui build

* refactor: Model modality handling

* chore: update webui build output

* refactor: Processing state reactivity

* fix: UI

* refactor: Services/Stores syntax + logic improvements

Refactors components to access stores directly instead of using exported getter functions.

This change centralizes store access and logic, simplifying component code and improving maintainability by reducing the number of exported functions and promoting direct store interaction.

Removes exported getter functions from `chat.svelte.ts`, `conversations.svelte.ts`, `models.svelte.ts` and `settings.svelte.ts`.

* refactor: Architecture cleanup

* feat: Improve statistic badges

* feat: Condition available models based on modality + better model loading strategy & UX

* docs: Architecture documentation

* feat: Update logic for PDF as Image

* add TODO for http client

* refactor: Enhance model info and attachment handling

* chore: update webui build output

* refactor: Components naming

* chore: update webui build output

* refactor: Cleanup

* refactor: DRY `getAttachmentDisplayItems` function + fix UI

* chore: update webui build output

* fix: Modality detection improvement for text-based PDF attachments

* refactor: Cleanup

* docs: Add info comment

* refactor: Cleanup

* re

* refactor: Cleanup

* refactor: Cleanup

* feat: Attachment logic & UI improvements

* refactor: Constants

* feat: Improve UI sidebar background color

* chore: update webui build output

* refactor: Utils imports + move types to `app.d.ts`

* test: Fix Storybook mocks

* chore: update webui build output

* test: Update Chat Form UI tests

* refactor: Tooltip Provider from core layout

* refactor: Tests to separate location

* decouple server_models from server_routes

* test: Move demo test  to tests/server

* refactor: Remove redundant method

* chore: update webui build output

* also route anthropic endpoints

* fix duplicated arg

* fix invalid ptr to shutdown_handler

* server : minor

* rm unused fn

* add ?autoload=true|false query param

* refactor: Remove redundant code

* docs: Update README documentations + architecture & data flow diagrams

* fix: Disable autoload on calling server props for the model

* chore: update webui build output

* fix ubuntu build

* fix: Model status reactivity

* fix: Modality detection for MODEL mode

* chore: update webui build output

---------

Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
This commit is contained in:
Xuan-Son Nguyen
2025-12-01 19:41:04 +01:00
committed by GitHub
co-authored by Aleksander Georgi Gerganov
parent 7733409734
commit ec18edfcba
178 changed files with 11643 additions and 4356 deletions
@@ -0,0 +1,627 @@
import { browser } from '$app/environment';
import { goto } from '$app/navigation';
import { toast } from 'svelte-sonner';
import { DatabaseService } from '$lib/services/database';
import { config } from '$lib/stores/settings.svelte';
import { filterByLeafNodeId, findLeafNode } from '$lib/utils';
import { AttachmentType } from '$lib/enums';
/**
* conversationsStore - Persistent conversation data and lifecycle management
*
* **Terminology - Chat vs Conversation:**
* - **Chat**: The active interaction space with the Chat Completions API. Represents the
* real-time streaming session, loading states, and UI visualization of AI communication.
* Managed by chatStore, a "chat" is ephemeral and exists during active AI interactions.
* - **Conversation**: The persistent database entity storing all messages and metadata.
* A "conversation" survives across sessions, page reloads, and browser restarts.
* It contains the complete message history, branching structure, and conversation metadata.
*
* This store manages all conversation-level data and operations including creation, loading,
* deletion, and navigation. It maintains the list of conversations and the currently active
* conversation with its message history, providing reactive state for UI components.
*
* **Architecture & Relationships:**
* - **conversationsStore** (this class): Persistent conversation data management
* - Manages conversation list and active conversation state
* - Handles conversation CRUD operations via DatabaseService
* - Maintains active message array for current conversation
* - Coordinates branching navigation (currNode tracking)
*
* - **chatStore**: Uses conversation data as context for active AI streaming
* - **DatabaseService**: Low-level IndexedDB storage for conversations and messages
*
* **Key Features:**
* - **Conversation Lifecycle**: Create, load, update, delete conversations
* - **Message Management**: Active message array with branching support
* - **Import/Export**: JSON-based conversation backup and restore
* - **Branch Navigation**: Navigate between message tree branches
* - **Title Management**: Auto-update titles with confirmation dialogs
* - **Reactive State**: Svelte 5 runes for automatic UI updates
*
* **State Properties:**
* - `conversations`: All conversations sorted by last modified
* - `activeConversation`: Currently viewed conversation
* - `activeMessages`: Messages in current conversation path
* - `isInitialized`: Store initialization status
*/
class ConversationsStore {
// ─────────────────────────────────────────────────────────────────────────────
// State
// ─────────────────────────────────────────────────────────────────────────────
/** List of all conversations */
conversations = $state<DatabaseConversation[]>([]);
/** Currently active conversation */
activeConversation = $state<DatabaseConversation | null>(null);
/** Messages in the active conversation (filtered by currNode path) */
activeMessages = $state<DatabaseMessage[]>([]);
/** Whether the store has been initialized */
isInitialized = $state(false);
/** Callback for title update confirmation dialog */
titleUpdateConfirmationCallback?: (currentTitle: string, newTitle: string) => Promise<boolean>;
// ─────────────────────────────────────────────────────────────────────────────
// Modalities
// ─────────────────────────────────────────────────────────────────────────────
/**
* Modalities used in the active conversation.
* Computed from attachments in activeMessages.
* Used to filter available models - models must support all used modalities.
*/
usedModalities: ModelModalities = $derived.by(() => {
return this.calculateModalitiesFromMessages(this.activeMessages);
});
/**
* Calculate modalities from a list of messages.
* Helper method used by both usedModalities and getModalitiesUpToMessage.
*/
private calculateModalitiesFromMessages(messages: DatabaseMessage[]): ModelModalities {
const modalities: ModelModalities = { vision: false, audio: false };
for (const message of messages) {
if (!message.extra) continue;
for (const extra of message.extra) {
if (extra.type === AttachmentType.IMAGE) {
modalities.vision = true;
}
// PDF only requires vision if processed as images
if (extra.type === AttachmentType.PDF) {
const pdfExtra = extra as DatabaseMessageExtraPdfFile;
if (pdfExtra.processedAsImages) {
modalities.vision = true;
}
}
if (extra.type === AttachmentType.AUDIO) {
modalities.audio = true;
}
}
if (modalities.vision && modalities.audio) break;
}
return modalities;
}
/**
* Get modalities used in messages BEFORE the specified message.
* Used for regeneration - only consider context that was available when generating this message.
*/
getModalitiesUpToMessage(messageId: string): ModelModalities {
const messageIndex = this.activeMessages.findIndex((m) => m.id === messageId);
if (messageIndex === -1) {
return this.usedModalities;
}
const messagesBefore = this.activeMessages.slice(0, messageIndex);
return this.calculateModalitiesFromMessages(messagesBefore);
}
constructor() {
if (browser) {
this.initialize();
}
}
// ─────────────────────────────────────────────────────────────────────────────
// Lifecycle
// ─────────────────────────────────────────────────────────────────────────────
/**
* Initializes the conversations store by loading conversations from the database
*/
async initialize(): Promise<void> {
try {
await this.loadConversations();
this.isInitialized = true;
} catch (error) {
console.error('Failed to initialize conversations store:', error);
}
}
/**
* Loads all conversations from the database
*/
async loadConversations(): Promise<void> {
this.conversations = await DatabaseService.getAllConversations();
}
// ─────────────────────────────────────────────────────────────────────────────
// Conversation CRUD
// ─────────────────────────────────────────────────────────────────────────────
/**
* Creates a new conversation and navigates to it
* @param name - Optional name for the conversation
* @returns The ID of the created conversation
*/
async createConversation(name?: string): Promise<string> {
const conversationName = name || `Chat ${new Date().toLocaleString()}`;
const conversation = await DatabaseService.createConversation(conversationName);
this.conversations.unshift(conversation);
this.activeConversation = conversation;
this.activeMessages = [];
await goto(`#/chat/${conversation.id}`);
return conversation.id;
}
/**
* Loads a specific conversation and its messages
* @param convId - The conversation ID to load
* @returns True if conversation was loaded successfully
*/
async loadConversation(convId: string): Promise<boolean> {
try {
const conversation = await DatabaseService.getConversation(convId);
if (!conversation) {
return false;
}
this.activeConversation = conversation;
if (conversation.currNode) {
const allMessages = await DatabaseService.getConversationMessages(convId);
this.activeMessages = filterByLeafNodeId(
allMessages,
conversation.currNode,
false
) as DatabaseMessage[];
} else {
this.activeMessages = await DatabaseService.getConversationMessages(convId);
}
return true;
} catch (error) {
console.error('Failed to load conversation:', error);
return false;
}
}
/**
* Clears the active conversation and messages
* Used when navigating away from chat or starting fresh
*/
clearActiveConversation(): void {
this.activeConversation = null;
this.activeMessages = [];
// Active processing conversation is now managed by chatStore
}
// ─────────────────────────────────────────────────────────────────────────────
// Message Management
// ─────────────────────────────────────────────────────────────────────────────
/**
* Refreshes active messages based on currNode after branch navigation
*/
async refreshActiveMessages(): Promise<void> {
if (!this.activeConversation) return;
const allMessages = await DatabaseService.getConversationMessages(this.activeConversation.id);
if (allMessages.length === 0) {
this.activeMessages = [];
return;
}
const leafNodeId =
this.activeConversation.currNode ||
allMessages.reduce((latest, msg) => (msg.timestamp > latest.timestamp ? msg : latest)).id;
const currentPath = filterByLeafNodeId(allMessages, leafNodeId, false) as DatabaseMessage[];
this.activeMessages.length = 0;
this.activeMessages.push(...currentPath);
}
/**
* Updates the name of a conversation
* @param convId - The conversation ID to update
* @param name - The new name for the conversation
*/
async updateConversationName(convId: string, name: string): Promise<void> {
try {
await DatabaseService.updateConversation(convId, { name });
const convIndex = this.conversations.findIndex((c) => c.id === convId);
if (convIndex !== -1) {
this.conversations[convIndex].name = name;
}
if (this.activeConversation?.id === convId) {
this.activeConversation.name = name;
}
} catch (error) {
console.error('Failed to update conversation name:', error);
}
}
/**
* Updates conversation title with optional confirmation dialog based on settings
* @param convId - The conversation ID to update
* @param newTitle - The new title content
* @param onConfirmationNeeded - Callback when user confirmation is needed
* @returns True if title was updated, false if cancelled
*/
async updateConversationTitleWithConfirmation(
convId: string,
newTitle: string,
onConfirmationNeeded?: (currentTitle: string, newTitle: string) => Promise<boolean>
): Promise<boolean> {
try {
const currentConfig = config();
if (currentConfig.askForTitleConfirmation && onConfirmationNeeded) {
const conversation = await DatabaseService.getConversation(convId);
if (!conversation) return false;
const shouldUpdate = await onConfirmationNeeded(conversation.name, newTitle);
if (!shouldUpdate) return false;
}
await this.updateConversationName(convId, newTitle);
return true;
} catch (error) {
console.error('Failed to update conversation title with confirmation:', error);
return false;
}
}
// ─────────────────────────────────────────────────────────────────────────────
// Navigation
// ─────────────────────────────────────────────────────────────────────────────
/**
* Updates the current node of the active conversation
* @param nodeId - The new current node ID
*/
async updateCurrentNode(nodeId: string): Promise<void> {
if (!this.activeConversation) return;
await DatabaseService.updateCurrentNode(this.activeConversation.id, nodeId);
this.activeConversation.currNode = nodeId;
}
/**
* Updates conversation lastModified timestamp and moves it to top of list
*/
updateConversationTimestamp(): void {
if (!this.activeConversation) return;
const chatIndex = this.conversations.findIndex((c) => c.id === this.activeConversation!.id);
if (chatIndex !== -1) {
this.conversations[chatIndex].lastModified = Date.now();
const updatedConv = this.conversations.splice(chatIndex, 1)[0];
this.conversations.unshift(updatedConv);
}
}
/**
* Navigates to a specific sibling branch by updating currNode and refreshing messages
* @param siblingId - The sibling message ID to navigate to
*/
async navigateToSibling(siblingId: string): Promise<void> {
if (!this.activeConversation) return;
const allMessages = await DatabaseService.getConversationMessages(this.activeConversation.id);
const rootMessage = allMessages.find((m) => m.type === 'root' && m.parent === null);
const currentFirstUserMessage = this.activeMessages.find(
(m) => m.role === 'user' && m.parent === rootMessage?.id
);
const currentLeafNodeId = findLeafNode(allMessages, siblingId);
await DatabaseService.updateCurrentNode(this.activeConversation.id, currentLeafNodeId);
this.activeConversation.currNode = currentLeafNodeId;
await this.refreshActiveMessages();
// Only show title dialog if we're navigating between different first user message siblings
if (rootMessage && this.activeMessages.length > 0) {
const newFirstUserMessage = this.activeMessages.find(
(m) => m.role === 'user' && m.parent === rootMessage.id
);
if (
newFirstUserMessage &&
newFirstUserMessage.content.trim() &&
(!currentFirstUserMessage ||
newFirstUserMessage.id !== currentFirstUserMessage.id ||
newFirstUserMessage.content.trim() !== currentFirstUserMessage.content.trim())
) {
await this.updateConversationTitleWithConfirmation(
this.activeConversation.id,
newFirstUserMessage.content.trim(),
this.titleUpdateConfirmationCallback
);
}
}
}
/**
* Deletes a conversation and all its messages
* @param convId - The conversation ID to delete
*/
async deleteConversation(convId: string): Promise<void> {
try {
await DatabaseService.deleteConversation(convId);
this.conversations = this.conversations.filter((c) => c.id !== convId);
if (this.activeConversation?.id === convId) {
this.activeConversation = null;
this.activeMessages = [];
await goto(`?new_chat=true#/`);
}
} catch (error) {
console.error('Failed to delete conversation:', error);
}
}
// ─────────────────────────────────────────────────────────────────────────────
// Import/Export
// ─────────────────────────────────────────────────────────────────────────────
/**
* Downloads a conversation as JSON file
* @param convId - The conversation ID to download
*/
async downloadConversation(convId: string): Promise<void> {
let conversation: DatabaseConversation | null;
let messages: DatabaseMessage[];
if (this.activeConversation?.id === convId) {
conversation = this.activeConversation;
messages = this.activeMessages;
} else {
conversation = await DatabaseService.getConversation(convId);
if (!conversation) return;
messages = await DatabaseService.getConversationMessages(convId);
}
this.triggerDownload({ conv: conversation, messages });
}
/**
* Exports all conversations with their messages as a JSON file
* @returns The list of exported conversations
*/
async exportAllConversations(): Promise<DatabaseConversation[]> {
const allConversations = await DatabaseService.getAllConversations();
if (allConversations.length === 0) {
throw new Error('No conversations to export');
}
const allData = await Promise.all(
allConversations.map(async (conv) => {
const messages = await DatabaseService.getConversationMessages(conv.id);
return { conv, messages };
})
);
const blob = new Blob([JSON.stringify(allData, null, 2)], { type: 'application/json' });
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = `all_conversations_${new Date().toISOString().split('T')[0]}.json`;
document.body.appendChild(a);
a.click();
document.body.removeChild(a);
URL.revokeObjectURL(url);
toast.success(`All conversations (${allConversations.length}) prepared for download`);
return allConversations;
}
/**
* Imports conversations from a JSON file
* Opens file picker and processes the selected file
* @returns The list of imported conversations
*/
async importConversations(): Promise<DatabaseConversation[]> {
return new Promise((resolve, reject) => {
const input = document.createElement('input');
input.type = 'file';
input.accept = '.json';
input.onchange = async (e) => {
const file = (e.target as HTMLInputElement)?.files?.[0];
if (!file) {
reject(new Error('No file selected'));
return;
}
try {
const text = await file.text();
const parsedData = JSON.parse(text);
let importedData: ExportedConversations;
if (Array.isArray(parsedData)) {
importedData = parsedData;
} else if (
parsedData &&
typeof parsedData === 'object' &&
'conv' in parsedData &&
'messages' in parsedData
) {
importedData = [parsedData];
} else {
throw new Error('Invalid file format');
}
const result = await DatabaseService.importConversations(importedData);
toast.success(`Imported ${result.imported} conversation(s), skipped ${result.skipped}`);
await this.loadConversations();
const importedConversations = (
Array.isArray(importedData) ? importedData : [importedData]
).map((item) => item.conv);
resolve(importedConversations);
} catch (err: unknown) {
const message = err instanceof Error ? err.message : 'Unknown error';
console.error('Failed to import conversations:', err);
toast.error('Import failed', { description: message });
reject(new Error(`Import failed: ${message}`));
}
};
input.click();
});
}
/**
* Gets all messages for a specific conversation
* @param convId - The conversation ID
* @returns Array of messages
*/
async getConversationMessages(convId: string): Promise<DatabaseMessage[]> {
return await DatabaseService.getConversationMessages(convId);
}
/**
* Adds a message to the active messages array
* Used by chatStore when creating new messages
* @param message - The message to add
*/
addMessageToActive(message: DatabaseMessage): void {
this.activeMessages.push(message);
}
/**
* Updates a message at a specific index in active messages
* Creates a new object to trigger Svelte 5 reactivity
* @param index - The index of the message to update
* @param updates - Partial message data to update
*/
updateMessageAtIndex(index: number, updates: Partial<DatabaseMessage>): void {
if (index !== -1 && this.activeMessages[index]) {
// Create new object to trigger Svelte 5 reactivity
this.activeMessages[index] = { ...this.activeMessages[index], ...updates };
}
}
/**
* Finds the index of a message in active messages
* @param messageId - The message ID to find
* @returns The index of the message, or -1 if not found
*/
findMessageIndex(messageId: string): number {
return this.activeMessages.findIndex((m) => m.id === messageId);
}
/**
* Removes messages from active messages starting at an index
* @param startIndex - The index to start removing from
*/
sliceActiveMessages(startIndex: number): void {
this.activeMessages = this.activeMessages.slice(0, startIndex);
}
/**
* Removes a message from active messages by index
* @param index - The index to remove
* @returns The removed message or undefined
*/
removeMessageAtIndex(index: number): DatabaseMessage | undefined {
if (index !== -1) {
return this.activeMessages.splice(index, 1)[0];
}
return undefined;
}
/**
* Triggers file download in browser
* @param data - The data to download
* @param filename - Optional filename for the download
*/
private triggerDownload(data: ExportedConversations, filename?: string): void {
const conversation =
'conv' in data ? data.conv : Array.isArray(data) ? data[0]?.conv : undefined;
if (!conversation) {
console.error('Invalid data: missing conversation');
return;
}
const conversationName = conversation.name?.trim() || '';
const truncatedSuffix = conversationName
.toLowerCase()
.replace(/[^a-z0-9]/gi, '_')
.replace(/_+/g, '_')
.substring(0, 20);
const downloadFilename = filename || `conversation_${conversation.id}_${truncatedSuffix}.json`;
const blob = new Blob([JSON.stringify(data, null, 2)], { type: 'application/json' });
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = downloadFilename;
document.body.appendChild(a);
a.click();
document.body.removeChild(a);
URL.revokeObjectURL(url);
}
// ─────────────────────────────────────────────────────────────────────────────
// Utilities
// ─────────────────────────────────────────────────────────────────────────────
/**
* Sets the callback function for title update confirmations
* @param callback - Function to call when confirmation is needed
*/
setTitleUpdateConfirmationCallback(
callback: (currentTitle: string, newTitle: string) => Promise<boolean>
): void {
this.titleUpdateConfirmationCallback = callback;
}
}
export const conversationsStore = new ConversationsStore();
export const conversations = () => conversationsStore.conversations;
export const activeConversation = () => conversationsStore.activeConversation;
export const activeMessages = () => conversationsStore.activeMessages;
export const isConversationsInitialized = () => conversationsStore.isInitialized;
export const usedModalities = () => conversationsStore.usedModalities;