{"topic":{"id":15623,"title":"本站当前使用的软件的详细解析","slug":"topic","categoryId":4,"createdAt":"2026-07-15T14:24:38.094Z","bumpedAt":"2026-07-16T13:55:52.183Z","views":1264,"postsCount":4,"highestPostNumber":4,"likeCount":0,"closed":false,"pinned":false,"pinnedGlobally":false,"pinnedAt":null,"pinnedUntil":null,"isBanner":false,"excerpt":"前言本站使用的软件也运行了好多天了，目前各种小bug不断，但是好在没有什么致命的问题。为了帮助用户更好地理解本软件的开发情况，所以写一篇文章，介绍有哪些功能，数据是怎么迁移的。设计理念简单，好用侧重点：易用 > 易开发维护 > 拥有各种功能实现的功能本软件的设计，高度受Discourse启发。但不谋求实现Discourse的全部功能。 Murmur: Modern Discussion Forum murmur…","deletedAt":null,"isAnonymous":false,"category":{"id":4,"name":"谈笑风生","slug":"general","color":"25AAE2"},"author":{"id":4668,"username":"IO","name":null,"avatarUrl":"/uploads/image/86/f2/86f267c0aaa73835a999b7cc3a7e7a701605574eda9a2c2a56f5ddbbba64d1a9.avif"},"tags":[]},"posts":[{"id":72582,"postNumber":1,"raw":"## 前言\r\n本站使用的软件也运行了好多天了，目前各种小bug不断，但是好在没有什么致命的问题。\r\n为了帮助用户更好地理解本软件的开发情况，所以写一篇文章，介绍有哪些功能，数据是怎么迁移的。\r\n\r\n## 设计理念\r\n简单，好用\r\n侧重点：易用 > 易开发维护 > 拥有各种功能\r\n\r\n## 实现的功能\r\n本软件的设计，高度受[Discourse](https://meta.discourse.org/)启发。但不谋求实现Discourse的全部功能。\r\n\r\n\r\n# Murmur: Modern Discussion Forum\r\n\r\n`murmur` is a high-performance, features-rich, Discourse/NimForum-style discussion forum. It is engineered with a hybrid **SSR-First (Server-Side Rendering)** architecture, delivering instant page loads and optimal SEO crawlers accessibility, while layering on real-time interactive widgets as progressive client-side enhancements.\r\n\r\n---\r\n\r\n## 🛠️ The Tech Stack\r\n\r\nMurmur's architecture is divided into a high-performance web core, a relational database, and a specialized asset caching layer:\r\n\r\n- **Web Core**: [SvelteKit 2](https://kit.svelte.dev/) utilizing [Svelte 5 Runes](https://svelte.dev/blog/runes) for robust state management.\r\n- **Database & ORM**: Drizzle ORM managing SQLite (`better-sqlite3` driver) for synchronous operations.\r\n- **Asset Performance**: A custom caching proxy written in Rust using Axum (`uploads-proxy`).\r\n- **Search Subsystem**: SQLite FTS5 index combined with the `sqlite-vec` extension for vector operations, and `@node-rs/jieba` CJK tokenizer.\r\n- **Real-time Features**: Server-Sent Events (SSE) for pub/sub notifications, and Yjs text structures for wiki collaboration.\r\n- **Styling**: TailwindCSS v4 with Flowbite Svelte.\r\n\r\n---\r\n\r\n## 🏛️ System Architecture Preview\r\n\r\nMurmur is designed to maximize speed by avoiding clientside rendering overhead during initial page loads.\r\n\r\n```mermaid\r\ngraph TD\r\n Client[Browser Client]\r\n SvelteKit[SvelteKit Node.js Server]\r\n SQLite[(SQLite DB / local.db)]\r\n Proxy[Rust Uploads-Proxy]\r\n Upstream[Upstream Assets Server]\r\n LLM[LLM API / Ollama / Gemini / OpenAI / Anthropic]\r\n\r\n Client -->|1. HTTP Page Request / SSE Stream| SvelteKit\r\n Client -->|2. HTTP Uploaded Files| Proxy\r\n Proxy -->|3. Cache Miss Fetch| Upstream\r\n Proxy -->|4. Save locally| Disk[(Local Uploads)]\r\n SvelteKit -->|5. Synchronous DB Queries| SQLite\r\n SvelteKit -->|6. Vectors & Generation| LLM\r\n```\r\n\r\n1. **Server-Side Rendering**: SvelteKit processes the request, queries the SQLite database synchronously through the repository layer, compiles the page into static HTML, and responds.\r\n2. **Client Hydration**: The browser displays the HTML, downloads the JavaScript bundles, and hydrates components (e.g. Composer, Likes, Chat).\r\n3. **Live Updates**: An SSE connection is established to stream live updates (new posts, edits) to active tabs.\r\n4. **Asset Offloading**: The Rust `uploads-proxy` handles asset queries. If a file is cached locally on disk, it is served; otherwise, the proxy downloads it from the upstream server with retry logic, saves it locally, and serves it.\r\n\r\n---\r\n\r\n## 📚 Documentation Index\r\n\r\nBelow is the directory map of the Murmur system documentation. Click on any link to view detailed guides:\r\n\r\n### ⚙️ [System Architecture & Scaling](file:///f/murmur/doc/architecture.md)\r\n\r\n- Detailed breakdown of the request-response lifecycle and hydration.\r\n- Rust `uploads-proxy` microservice design (thundering herd protection, atomic writes).\r\n- Horizontal scaling recommendations (reconciliation, moving from memory SSE to Redis pub/sub, object storage).\r\n\r\n# Murmur: System Architecture\r\n\r\nThis document describes the high-level architecture of Murmur, including the frontend framework, backend data flow, integration with external services, and horizontal scaling patterns.\r\n\r\n---\r\n\r\n## Technical Stack\r\n\r\nMurmur is built using a modern, lightweight JavaScript stack, layered with a high-performance Rust microservice:\r\n\r\n- **Frontend / Meta-framework**: SvelteKit 2 using Svelte 5 (Runes mode).\r\n- **Styling**: TailwindCSS v4 (CSS-first, no `tailwind.config.js`) & Flowbite Svelte components.\r\n- **Database**: SQLite (`better-sqlite3` driver) wrapped with Drizzle ORM.\r\n- **Search Vector calculations**: SQLite extension `sqlite-vec`.\r\n- **Search tokenization**: `@node-rs/jieba` for CJK languages.\r\n- **Real-Time Communication**: Server-Sent Events (SSE).\r\n- **Collaboration**: Yjs text types synced stateless-ly over HTTP.\r\n- **Assets Proxy**: Custom Axum-based Rust microservice (`uploads-proxy`).\r\n\r\n---\r\n\r\n## Component Layout & Data Flow\r\n\r\nMurmur leverages Server-Side Rendering (SSR) by default to deliver HTML instantly to crawlers and users. Clientside Svelte components progressively hydrate the page to enable real-time messaging, infinite scroll, and rich editing.\r\n\r\n```mermaid\r\ngraph TD\r\n Client[Browser Client]\r\n SvelteKit[SvelteKit Node.js Server]\r\n SQLite[(SQLite DB / local.db)]\r\n Proxy[Rust Uploads-Proxy]\r\n Upstream[Upstream Assets Server]\r\n LLM[LLM API / Ollama / Gemini / OpenAI / Anthropic]\r\n\r\n Client -->|1. HTTP / SSE| SvelteKit\r\n Client -->|2. HTTP Static Assets| Proxy\r\n Proxy -->|3. Fetch cache miss| Upstream\r\n Proxy -->|4. Save locally| Disk[(Local Uploads)]\r\n SvelteKit -->|5. Synchronous Queries| SQLite\r\n SvelteKit -->|6. Vectors & Generation| LLM\r\n```\r\n\r\n### 1. Request Lifecycle\r\n\r\n1. The browser requests a page (e.g., `/t/slug/id`).\r\n2. SvelteKit's routing resolves the path and calls the server-side load function (`+page.server.ts`), which queries the database synchronously via the repository layer (`src/lib/server/repo.ts`).\r\n3. SvelteKit renders the page as static HTML (SSR-first) and returns it.\r\n4. The client browser parses the HTML, displays it immediately, and fetches Svelte bundles to hydrate active widgets (e.g., the composer, emoji picker, likes).\r\n5. The client initiates a Server-Sent Events (SSE) connection to listen to live updates.\r\n\r\n---\r\n\r\n## Node.js & Rust uploads-proxy Integration\r\n\r\nUser-uploaded attachments and legacy images are cached locally on disk. To prevent heavy Node.js threads from blocking on large file IO operations:\r\n\r\n- **Node.js Server**: Serves core HTML/JS payloads and handles API mutations.\r\n- **Rust uploads-proxy**: Serves files located under `/uploads/`. It listens on `127.0.0.1:4119` and handles file lookups.\r\n - **Cache Hit**: Serves the local file immediately using high-speed async file streaming via `tokio-util`.\r\n - **Cache Miss**: Downloads the asset from the configured upstream server (attempts up to 3 times, with a 2-second sleep duration between attempts), saves it locally, and serves it to the user.\r\n - **Thundering Herd Coalesce**: Employs a mutex and `tokio::sync::Notify` to block duplicate concurrent requests for the same missing asset, ensuring only one download request hits the upstream server.\r\n\r\n---\r\n\r\n## Real-Time Engine (SSE Pub/Sub)\r\n\r\nMurmur uses a single-process in-memory pub/sub pattern implemented in [events.ts](file:///f/murmur/src/lib/server/events.ts):\r\n\r\n- Route handlers or repository mutations call `publish(topicId, event)`.\r\n- The Server-Sent Events endpoint [events/+server.ts](file:///f/murmur/src/routes/t/[slug]/[id]/events/+server.ts) subscribes to updates for a specific topic.\r\n- It streams JSON-LD chunk lines to the browser's `EventSource` instance.\r\n- The UI handles the event reactively to append new posts, update counts, or delete posts without page reloads.\r\n\r\n---\r\n\r\n## Horizontal Scaling Strategy\r\n\r\nBecause SQLite and the SSE pub/sub system operate within a single process, horizontal scaling (running multiple instances of the Node.js process) requires:\r\n\r\n1. **Database Layer**: Replace the local `better-sqlite3` instance with a replicated SQLite setup (such as LiteFS) or migrate Drizzle configuration to PostgreSQL.\r\n2. **Pub/Sub Layer**: Replace the in-memory singleton in [events.ts](file:///f/murmur/src/lib/server/events.ts) with a Redis or RabbitMQ pub/sub server. Route connections to the pub/sub interface so that messages publish across all Node server processes.\r\n3. **Uploads caching**: Ensure the uploads-proxy instances point to a shared directory (e.g., NFS, GlusterFS) or replace the local filesystem storage with an Object Store (like AWS S3) by modifying the SvelteKit file upload API.\r\n\r\n### 🗄️ [Data Model & Database Schema](file:///f/murmur/doc/data-model.md)\r\n\r\n- Relational schema definitions for SQLite.\r\n- Data design conventions (auto-increments, ISO-8601 timestamps, soft-deletions).\r\n- Database tables, relationships, and indices optimization.\r\n\r\n# Murmur: Data Model & Database Schema\r\n\r\nThis document details the database schema, SQLite conventions, and tables managed by Drizzle ORM in Murmur. The database schema is located in [schema.ts](file:///f/murmur/src/lib/server/db/schema.ts).\r\n\r\n---\r\n\r\n## Data Layer Conventions\r\n\r\nTo simplify imports, backups, and exports from active Discourse installations, Murmur maintains the following database patterns:\r\n\r\n1. **Integer Auto-increment Primary Keys**: All tables use auto-increment integers. The system user is seeded with ID `-1` (matching Discourse conventions).\r\n2. **ISO-8601 Timestamps**: Timestamps are stored as text (e.g. `2026-07-15T22:00:00.000Z`) rather than SQLite Unix integers. They default to the current time via `$defaultFn(() => new Date().toISOString())`. This allows lexicographical sorting and matches Discourse CSV files.\r\n3. **Booleans**: Stored as integers (`0` or `1`) using Drizzle's `integer(..., { mode: 'boolean' })`.\r\n4. **Post Rendering**: Posts store both the raw Markdown text (`raw`) and the cooked sanitized HTML (`cooked`). Rendering is done via `{@html cooked}` without dynamic clientside compilation.\r\n5. **Soft Deletions**: Deleting topics or posts updates a `deletedAt` ISO timestamp column instead of deleting database rows.\r\n\r\n---\r\n\r\n## Database Tables\r\n\r\nBelow is an overview of the Drizzle tables:\r\n\r\n### 1. `users`\r\n\r\nRepresents forum members, administrators, and moderators.\r\n\r\n- `id`: Integer primary key (Auto-increment).\r\n- `username`: Unique username (Text).\r\n- `name`: Full display name (Text, nullable).\r\n- `avatarUrl`: URL of the user's avatar.\r\n- `email`: User email address.\r\n- `githubId`: Bound GitHub ID for SSO.\r\n- `admin` / `moderator`: Role toggles (Booleans).\r\n- `trustLevel`: Trust system ranking (`0` to `4`).\r\n- `silencedUntil`: ISO timestamp until which the user is banned from posting.\r\n- `signatureRaw` / `signatureCooked`: Markdown/HTML for post signatures.\r\n- `chatGlobalMuted`: Toggle to silence all chat badge notifications (Boolean).\r\n\r\n### 2. `sessions` & `ssoSyncTickets`\r\n\r\nAuthentication sessions matching Oslo specifications.\r\n\r\n- `sessions.id`: SHA-256 hash of the session token.\r\n- `sessions.userId`: References `users.id` (on cascade delete).\r\n- `sessions.expiresAt`: Expiration timestamp.\r\n\r\n### 3. `categories` & `categoryGroups`\r\n\r\nDiscussion categories which support nesting.\r\n\r\n- `categories.id`: Integer primary key.\r\n- `categories.slug`: Unique URL slug.\r\n- `categories.parentCategoryId`: Self-referential FK targeting the parent category.\r\n- `categories.readRestricted`: If true, restricts read permissions.\r\n- `categoryGroups`: Maps `categoryId` to `groupId` and defines `permissionType` (Read, Create, Reply).\r\n\r\n### 4. `topics` & `posts`\r\n\r\nForum threads and post items.\r\n\r\n- `topics.userId`: Thread creator.\r\n- `topics.bumpedAt`: ISO timestamp updated on every reply to drive sorting.\r\n- `topics.closed` / `topics.pinned`: State flags.\r\n- `posts.topicId`: Parent thread.\r\n- `posts.postNumber`: 1-based sequential position within the topic (OP is `1`).\r\n- `posts.raw` / `posts.cooked`: Markdown/cooked text.\r\n- `posts.replyToPostNumber`: Identifies the post being replied to.\r\n- `posts.wiki`: Toggle indicating if the post is editable by non-creators.\r\n\r\n### 5. `postRevisions`\r\n\r\nHistory of edits made outside the grace period.\r\n\r\n- `postId`: References `posts.id`.\r\n- `userId`: Author of the edit.\r\n- `modifications`: JSON string of modified properties (e.g. `raw`, `cooked`, `title`).\r\n- `number`: Version sequence number.\r\n- `hidden`: If true, hides modifications from normal users.\r\n\r\n### 6. `postEmbeddings` & `topicRecommendations`\r\n\r\nVector semantic search embeddings and precomputed suggestions.\r\n\r\n- `postEmbeddings.postId`: References `posts.id`.\r\n- `postEmbeddings.embedding`: Raw vector float buffer (`Float32Array`).\r\n- `topicRecommendations.topicId`: References `topics.id`.\r\n- `topicRecommendations.recommendations`: JSON string list of recommended topic IDs.\r\n\r\n### 7. `chatChannels`, `chatChannelMembers` & `chatMessages`\r\n\r\nReal-time messaging channels.\r\n\r\n- `chatMessages.channelId`: Parent channel.\r\n- `chatMessages.message` / `chatMessages.cooked`: Raw/cooked text.\r\n- `chatChannelMembers.muted`: Individual channel muting preference.\r\n\r\n---\r\n\r\n## Schema DDL & Indices\r\n\r\nDrizzle automatically creates indices to optimize query performance and prevent database locks:\r\n\r\n- `posts_topic_number_idx` on `(posts.topicId, posts.postNumber)`.\r\n- `topics_category_bumped_idx` on `(topics.categoryId, topics.bumpedAt)`.\r\n- `likes_post_user_unq` unique index on `(likes.postId, likes.userId)`.\r\n- `topic_recommendations` and `post_embeddings` map primary keys to their parent components to enable swift joins.\r\n\r\n### ✍️ [Reply Composer, Drafts & Uploads](file:///f/murmur/doc/features/composer.md)\r\n\r\n- Composer UI controls (height adjustments, split ratios, fullscreen mode).\r\n- Local storage autosave and server-side syncing database logic.\r\n- Formatting toolbar helpers, emoji picker, and attachments uploads.\r\n- Kate-style whole-line keyboard shortcuts (`Ctrl+X` / `Ctrl+C`).\r\n\r\n# Composer UI, Attachments & Draft Syncing\r\n\r\nThis document describes the reply composer UI, clientside formatting shortcuts, the draft autosave engine, and the multi-file attachment pipeline.\r\n\r\n---\r\n\r\n## Composer Layout & Controls\r\n\r\nThe composer component [Composer.svelte](file:///f/murmur/src/lib/components/Composer.svelte) is modeled after the Discourse editor. It resides in a sticky drawer at the bottom of the screen.\r\n\r\n```\r\n+-------------------------------------------------------------------+\r\n| [v] Drag Handle (Height) |\r\n+-------------------------------------------------------------------+\r\n| Title / Action Summary [ ] Full [ _ ] Min |\r\n+-------------------------------------------------------------------+\r\n| [ B ] [ I ] [ Link ] [ Quote ] [ Tt v ] [ + v ] [ Undo ] [ Redo ]|\r\n+-------------------------------------------------------------------+\r\n| Textarea Input (Markdown) | Live HTML Preview |\r\n| | |\r\n| | |\r\n| |<---> Vertical Split Resizer |\r\n+-------------------------------------------------------------------+\r\n```\r\n\r\n### Resizing Features\r\n\r\n- **Height Resizing**: Users can click and drag the top border handle. The chosen height is written to `localStorage`.\r\n- **Split ratio Resizing**: A vertical split bar sits between the editor and preview panels. Dragging it updates the split percentage dynamically.\r\n- **Fullscreen mode**: Toggles the composer to fill the viewport.\r\n- **Minimization drawer**: Minimizes the composer into a small, floating status bar at the bottom right.\r\n\r\n---\r\n\r\n## Rich Formatting Toolbar & Emojis\r\n\r\n### 1. Heading Selector (`Tt`)\r\n\r\nInserts heading markdown (`#` to `####`) at the cursor selection.\r\n\r\n### 2. Actions Selector (`+`)\r\n\r\n- **Hide Details**: Inserts a folding BBCode detail block:\r\n ```markdown\r\n [details=\"Summary\"]\r\n content\r\n [/details]\r\n ```\r\n- **Footnote**: Inserts inline footnote syntax (`^[footnote]`).\r\n- **LaTeX Math**: Inserts standard math symbols wrapper (`$x^2$`).\r\n- **Mermaid Flowchart**: Inserts standard Mermaid graphs template.\r\n- **Table / Date**: Generates formatted markdown tables or inserts local timestamps.\r\n\r\n### 3. Emoji Picker\r\n\r\nA custom Svelte 5 component [EmojiPicker.svelte](file:///f/murmur/src/lib/components/EmojiPicker.svelte) displays emoji categories. It supports text-based query filtering and inserts the selected emoji at the cursor.\r\n\r\n---\r\n\r\n## Kate-Like Whole Line Editing\r\n\r\nWhen typing inside the editor, Murmur supports KDE Kate-style shortcuts:\r\n\r\n- **Ctrl+C (Copy)**: If no characters are selected (collapsed selection), copies the entire line containing the cursor (including the newline character).\r\n- **Ctrl+X (Cut)**: If no selection is active, cuts the entire line and moves lines below it up.\r\n- **Cursor Repositioning**: After cutting, the cursor automatically snaps to the start of the next line, saving keystrokes.\r\n\r\nThis logic is bound to the `handleKeyDown` function in the composer. It interacts directly with the browser's clipboard API and forces a tick update on Svelte's reactive states to maintain history.\r\n\r\n---\r\n\r\n## Draft Autosave & Database Synchronization\r\n\r\nTo prevent data loss, the composer synchronizes drafts using a hybrid approach:\r\n\r\n```mermaid\r\nsequenceDiagram\r\n participant Client as Browser Textarea\r\n participant Local as localStorage Cache\r\n participant API as /api/drafts/sync\r\n participant DB as SQLite drafts Table\r\n\r\n Client->>Local: Save drafts instantly on keystroke (Keydown)\r\n Client->>API: Periodic debounced save (Every 5 seconds)\r\n API->>DB: Write draft using compound key (userId + key)\r\n Note over API, DB: Last-write-wins merging by timestamp\r\n```\r\n\r\n- **LocalStorage**: Serves as the primary cache to save text instantaneously.\r\n- **Database Synchronization**: Every 5 seconds (debounced), the client makes an HTTP POST request to `/api/drafts/sync`. It pushes drafts to the SQLite database, ensuring edits are preserved across devices.\r\n- **Draft Disposal**: Submitting a post deletes both the local storage key and the database row.\r\n\r\n---\r\n\r\n## Clipboard Paste & Attachment Pipeline\r\n\r\nThe upload interface is triggered by clicking the paperclip icon or pasting an image directly from the clipboard into the editor:\r\n\r\n1. **Upload Initiation**: An async file reader creates a unique placeholder: `[Uploading image.png...]` at the current cursor position.\r\n2. **Chromium List safeguard**: Before beginning async processing, the file list is cloned into a static array. This prevents Chromium browsers from resetting the active input files when the input element value is cleared.\r\n3. **API Upload**: The file is sent via multipart POST to `/api/upload`. The endpoint validates files against allowed formats (e.g. `jpg`, `png`, `gif`, `svg`).\r\n4. **Link Replacement**: Once the API returns the permanent URL, the placeholder is replaced with the correct Markdown markup: `![filename|widthxheight](url)`.\r\n\r\n\r\n## Codeblocks\r\n* auto-detection of codeblocks based on the first line of the codeblock.\r\n* mermaid flowcharts.\r\n* toggle line numbers\r\n* copy and full-screen buttons\r\n\r\n### 🔍 [Full-Text & Semantic Search (RAG)](file:///f/murmur/doc/features/search-and-rag.md)\r\n\r\n- SQLite FTS5 full-text indexing with Porter stemming.\r\n- CJK search query segmentation using Jieba tokenizer.\r\n- `sqlite-vec` semantic vector distance search pipeline.\r\n- RAG AI chatbot integration and context permission safeguards.\r\n\r\n# Full-Text & Semantic Search (RAG)\r\n\r\nThis document describes the search subsystem in Murmur, including full-text indexing, CJK segmentation, vector semantic calculations, and the RAG (Retrieval-Augmented Generation) chatbot.\r\n\r\n---\r\n\r\n## 1. Full-Text Search Index (FTS5)\r\n\r\nFull-text search is powered by SQLite's FTS5 extension. During database initialization in [index.ts](file:///f/murmur/src/lib/server/db/index.ts), Murmur configures the virtual table `search_index` with Porter stemming to enable English word matching (e.g. \"runs\" matches \"running\"):\r\n\r\n```sql\r\nCREATE VIRTUAL TABLE IF NOT EXISTS search_index USING fts5(\r\n title,\r\n content,\r\n topic_id UNINDEXED,\r\n post_id UNINDEXED,\r\n post_number UNINDEXED,\r\n user_id UNINDEXED,\r\n category_id UNINDEXED,\r\n created_at UNINDEXED,\r\n tokenize = 'porter unicode61'\r\n);\r\n```\r\n\r\n### CJK Query Segmentation (Jieba)\r\n\r\nStandard FTS5 tokenizers do not split Chinese, Japanese, or Korean (CJK) characters into individual words because they lack space boundaries.\r\n\r\nTo solve this, Murmur intercepts search requests in [repo.ts](file:///f/murmur/src/lib/server/repo.ts) and parses queries using `Jieba` (`@node-rs/jieba`):\r\n\r\n1. **RegEx Splitting**: The query is split into CJK and non-CJK blocks.\r\n2. **Segmentation**: CJK blocks are parsed using `jieba.cutAll()` to extract words.\r\n3. **FTS5 Wrapping**: Extracted tokens are wrapped in double quotes and separated by spaces, converting the search into an implicit AND condition (e.g. `学习 Svelte` becomes `\"学习\" \"Svelte\"`).\r\n\r\n---\r\n\r\n## 2. Semantic Search (sqlite-vec)\r\n\r\nSemantic search finds matches based on conceptual similarity rather than exact keyword matches. It uses the `sqlite-vec` extension to run vector operations inside SQLite.\r\n\r\n- **Storage**: Embeddings are stored as raw binary BLOBs (`Float32Array`) in the `post_embeddings` table to prevent thread blocking (converting JSON strings to arrays in JavaScript is CPU intensive).\r\n- **Startup Migration**: On startup, Murmur checks the database. If it finds legacy JSON string representations in the embeddings table, it converts them to binary buffers in batches of 500 rows.\r\n- **Query Resolution**: Employs a cosine similarity calculation in SQLite:\r\n ```sql\r\n SELECT post_id, vec_distance_cosine(embedding, ?) AS distance\r\n FROM post_embeddings\r\n ORDER BY distance ASC\r\n LIMIT 20;\r\n ```\r\n\r\n---\r\n\r\n## 3. RAG Chatbot Integration\r\n\r\nWhen a user runs a search query, Murmur displays a RAG chatbot assistant panel next to the search results:\r\n\r\n```\r\n+-----------------------------------------------------------------+\r\n| Search Results | 🤖 Chatbot Reply |\r\n|------------------------------------|----------------------------|\r\n| 1. Topic: How to run Murmur | To run Murmur, execute: |\r\n| By user1 - \"Run pnpm dev...\" | `pnpm dev`. Refer to |\r\n| | /t/murmur/1 for details. |\r\n| 2. Topic: Production Setup | |\r\n| By user2 - \"Compile build...\" | [ AI thinking indicator ] |\r\n+-----------------------------------------------------------------+\r\n```\r\n\r\n### LLM Streaming client\r\n\r\nThe server client in [chatLlm.ts](file:///f/murmur/src/lib/server/chatLlm.ts) streams responses from different providers:\r\n\r\n- **Ollama**: Parses line-by-line NDJSON chunks.\r\n- **Gemini / OpenAI / Anthropic**: Parses Server-Sent Events (SSE) data streams.\r\n\r\n### SSE Stream Synthesis\r\n\r\nWhen a search is requested, the search API endpoint [stream/+server.ts](file:///f/murmur/src/routes/api/search/stream/+server.ts) initiates an SSE stream:\r\n\r\n1. It queries FTS5 and semantic databases for matches.\r\n2. Results are formatted as Markdown context blocks.\r\n3. It asks the LLM to summarize the findings and include links to the matched topics (e.g. `/t/slug/id/post_number`).\r\n4. It streams response tokens to the client browser via `ai_answer_chunk` events, which are compiled and rendered in the UI using `markdown-it`.\r\n\r\n---\r\n\r\n## 4. Privacy & Permission Filtering\r\n\r\nTo prevent information leaks (e.g., exposing private messages or restricted categories in search results or AI summaries):\r\n\r\n- **Permission Checking**: The search service retrieves category permission rules via `getPermittedCategoryIds(user)`.\r\n- **Pre-filtering**: Topics and posts belonging to restricted categories are filtered out of the results **before** the data is cached or sent to the LLM.\r\n- **RAG Security**: The LLM context contains only the results the user has permission to read.\r\n\r\n### ✍️ [Revisions & Wiki Editing](file:///f/murmur/doc/features/revisions-and-wiki.md)\r\n\r\n- Post edit revisions, grace period overrides, and modifications storage.\r\n- Myers split-diff comparison modal and revision rollbacks.\r\n- Wiki post permissions and stateless Yjs collaborative sync protocol.\r\n\r\n# Revisions, Wiki Posts & Yjs Collaborative Editing\r\n\r\nThis document explains the version control system for post edits, wiki post privileges, and the collaborative Yjs document synchronization engine.\r\n\r\n---\r\n\r\n## 1. Post Revisions System\r\n\r\nMurmur tracks the history of changes made to posts and topic details (titles and categories) over time.\r\n\r\n### Grace Period & Versioning\r\n\r\n- **Editing Grace Period**: Managed by the `editing_grace_period` site setting. If a user edits a post within this threshold (e.g. 5 minutes) after creation, modifications are overwritten without creating a new revision.\r\n- **Revision Capture**: Edits made after the grace period increment the post's `version` (and `publicVersion`) and save the differences to `post_revisions`.\r\n- **Modifications Storage**: Differences are stored as JSON-serialized change objects containing the previous values of modified fields:\r\n ```json\r\n {\r\n \t\"raw\": [\"Old post raw markdown\", \"New post raw markdown\"],\r\n \t\"title\": [\"Old topic title\", \"New topic title\"]\r\n }\r\n ```\r\n\r\n---\r\n\r\n## 2. Visual Diff Comparison Modal\r\n\r\nWhen a post contains revisions, a clickable edit counter (e.g., `✍️ 2`) is displayed next to the post's timestamp. Clicking it opens the [PostRevisionsModal.svelte](file:///f/murmur/src/lib/components/PostRevisionsModal.svelte) component:\r\n\r\n- **Myers Diff Algorithm**: Compares line-by-line differences in raw markdown or HTML tags.\r\n- **Cooked HTML Preview**: Toggles to render the formatted HTML output.\r\n- **Reversion**: Users with edit permissions can click the \"Revert\" button to roll back the post to a chosen revision.\r\n- **Visibility controls**: Staff members (administrators and moderators) can toggle the visibility of individual revisions to hide sensitive information.\r\n\r\n---\r\n\r\n## 3. Wiki Posts\r\n\r\nWiki posts allow collaborative editing by members of the community rather than just the author.\r\n\r\n- **Wiki Toggle**: Staff members can convert any post into a wiki post.\r\n- **Permissions**: Admins can configure the minimum user group or trust level required to edit wiki posts (defaults to staff and `trust_level_2` users).\r\n- **Indicators**: Displays a wiki badge next to the post to show users they have permission to edit it.\r\n\r\n---\r\n\r\n## 4. Stateless Yjs HTTP Sync Protocol\r\n\r\nTo enable concurrent collaborative editing without keeping persistent WebSocket connections open, Murmur uses a stateless HTTP synchronization protocol:\r\n\r\n```mermaid\r\nsequenceDiagram\r\n participant Client as Browser Editor\r\n participant Server as /api/posts/[id]/wiki-sync\r\n participant DB as SQLite DB\r\n\r\n Client->>Server: Send stateVector (base64) & update (base64)\r\n Server->>DB: Fetch post yjsState BLOB\r\n Server->>Server: Apply client update to Y.Doc\r\n Server->>Server: Compute diff using client stateVector\r\n Server->>DB: Save updated Y.Doc buffer to yjs_state\r\n Server->>Client: Return update (base64) & merged text\r\n```\r\n\r\n### Protocol Steps\r\n\r\n1. **Initiation**: The editor initializes a Yjs document (`Y.Doc`) and extracts the text content.\r\n2. **Synchronization Request**: The client sends an HTTP POST request to `/api/posts/[id]/wiki-sync` containing its local Yjs state vector and updates, encoded in base64.\r\n3. **Server Merge**:\r\n - The server loads the post's current `yjsState` from SQLite.\r\n - It instantiates a server-side `Y.Doc` and applies the database state.\r\n - It applies the base64 update received from the client.\r\n4. **Calculations**: The server computes the missing updates for the client using the client's state vector.\r\n5. **DB Update**: If the merged text differs from the post's current content, the server runs `editPost()` to update `raw`/`cooked` columns and saves the new Yjs state buffer.\r\n6. **Response**: The server returns the missing updates and the updated text to the client.\r\n\r\n\r\n### 💬 [Real-Time Chat & Badges](file:///f/murmur/doc/features/chat.md)\r\n\r\n- Public channels, direct message groups, and message management.\r\n- Svelte 5 runes unread badge calculation and global/channel muting.\r\n\r\n# Real-Time Chat & Unread Badge system\r\n\r\nThis document describes the real-time chat subsystem in Murmur, including channels, user privileges, message edits, and the Svelte 5 runes unread badge system.\r\n\r\n---\r\n\r\n## 1. Chat Channels & DM Rooms\r\n\r\nMurmur includes a built-in real-time chat workspace. It supports two channel types:\r\n\r\n- **Public Channels**: General channels accessible by all registered members.\r\n- **Direct Messages (DMs)**: Private chat rooms created between two or more users.\r\n\r\n---\r\n\r\n## 2. Real-Time Message Flow\r\n\r\nThe chat interface streams updates in real time using Server-Sent Events (SSE):\r\n\r\n```mermaid\r\nsequenceDiagram\r\n participant ClientA as Sender Browser\r\n participant API as /api/chat/messages\r\n participant DB as SQLite DB\r\n participant SSE as SSE Stream Channel\r\n participant ClientB as Receiver Browser\r\n\r\n ClientA->>API: POST send message\r\n API->>DB: Write message to chat_messages\r\n API->>SSE: Publish message event\r\n SSE->>ClientB: Stream JSON event\r\n Note over ClientB: Append message to UI\r\n```\r\n\r\n### Message Modifications\r\n\r\nUsers can edit or delete their own messages after they are sent. Moderators and administrators can edit or delete any message. Modifying a message updates the database and publishes an update event over SSE to sync all active clients.\r\n\r\n---\r\n\r\n## 3. Svelte 5 Runes Unread Badge System\r\n\r\nTo manage notification badges dynamically across the app, Murmur uses a client-side global state store powered by Svelte 5 runes:\r\n\r\n- **State Store**: Defined in a client-safe state module. It tracks active channels, members, read states, and global mute preferences.\r\n- **Dynamic Badge calculations**: Unread counts are computed using `$derived` values:\r\n ```typescript\r\n // Simplified logic\r\n const unreadCount = $derived(() => {\r\n \tif (globalMuted) return 0;\r\n \treturn channels\r\n \t\t.filter((c) => !c.muted)\r\n \t\t.reduce((sum, c) => sum + (c.messagesCount - c.lastReadMessageId), 0);\r\n });\r\n ```\r\n- **Real-Time Updates**: When a new message event is received via the SSE connection, the client-side store increments the channel's message count. The UI updates the unread badge automatically.\r\n- **Read State Sync**: Reading a channel updates the client-side state and sends an API call to save the `lastReadMessageId` in the database, syncing the read state across devices.\r\n\r\n---\r\n\r\n## 4. Muting Preferences\r\n\r\nTo prevent notification fatigue, users can configure their alert preferences:\r\n\r\n- **Channel Muting**: Users can mute individual channels. Muted channels do not trigger badge alerts or red dot indicators.\r\n- **Global Muting**: Users can enable global muting in their profile settings. This silences all chat notifications across the site.\r\n\r\n### 💻 [Developer Guide & Conventions](file:///f/murmur/doc/developer-guide.md)\r\n\r\n- Coding standards (runes, TailwindCSS v4, synchronous queries, types).\r\n- File-based routing structure.\r\n- ESLint and Prettier code formatting standards.\r\n\r\n# Developer Guide & Conventions\r\n\r\nThis document outlines the coding standards, repository layout, and development guidelines for developers working on the Murmur project.\r\n\r\n---\r\n\r\n## 1. Project Conventions\r\n\r\n### Svelte 5 Runes Mode\r\n\r\n- Murmur uses Svelte 5 runes. Use `$props()`, `$state()`, `$derived()`, and `$effect()` for state management.\r\n- Do not use legacy Svelte APIs (such as `export let`, `$:`, or `store.subscribe`).\r\n- The compiler options are inlined in the `vite.config.ts` file under the Svelte plugin configurations.\r\n\r\n### Database Layer\r\n\r\n- SQLite database operations run synchronously.\r\n- **Do not await database queries**: Drizzle queries use `.all()`, `.get()`, and `.run()` synchronously.\r\n- Implement database mutations in [repo.ts](file:///f/murmur/src/lib/server/repo.ts). Do not write inline database queries in routes or page endpoints.\r\n\r\n### Code Separation\r\n\r\n- **Server-Only Code**: Any logic placed under `src/lib/server/**` is excluded from browser bundles.\r\n- **Shared Types**: Clientside modules cannot import server code. Shared structures must be defined in [types.ts](file:///f/murmur/src/lib/types.ts).\r\n\r\n### Styling & CSS (TailwindCSS v4)\r\n\r\n- TailwindCSS v4 is integrated as a CSS-first design tool. There is no `tailwind.config.js`.\r\n- Configurations, theme overrides, and custom fonts are managed in [layout.css](file:///f/murmur/src/routes/layout.css) using CSS variables.\r\n\r\n---\r\n\r\n## 2. Directory Routing Structure\r\n\r\nThe routing layer uses standard SvelteKit file-based routing under `src/routes/`:\r\n\r\n- `/` : Landing page displaying latest topics.\r\n- `/c/[slug]/[id]` : Topics filtered by category.\r\n- `/t/[slug]/[id]` : Topic thread displaying posts and reply forms. Includes SSE streaming listeners.\r\n- `/new` : Full-page composer workspace.\r\n- `/login` : Authentication page (GitHub SSO & dev login).\r\n- `/admin/settings` : Dashboard for administrators to configure site settings.\r\n- `/sitemap.xml` : Dynamic sitemap generation for web crawlers.\r\n\r\n---\r\n\r\n## 3. Formatting Standards\r\n\r\nCode styling is enforced using Prettier and ESLint:\r\n\r\n- **Indent**: Tabs.\r\n- **Quotes**: Single quotes (`'`).\r\n- **Trailing Commas**: None.\r\n- **Line Width**: 100 characters.\r\n\r\nVerify formatting using the formatting scripts before submitting changes:\r\n\r\n```sh\r\npnpm lint # Checks linting and formatting compliance\r\npnpm format # Automatically formats files\r\n```\r\n\r\n### 🛠️ [Operations & Maintenance Scripts](file:///f/murmur/doc/scripts.md)\r\n\r\n- Standalone operational scripts under `scripts/` (import tools, reindexing, post rebaking, and embedding calculations).\r\n\r\n# Data Migrations & Operations Scripts\r\n\r\nThis document details the standalone command-line scripts located in the `/scripts/` folder. These scripts are run using `tsx` (e.g. `pnpm tsx scripts/rebake-posts.ts`).\r\n\r\n---\r\n\r\n## 1. Data Imports & Exports\r\n\r\n### [import_discourse_csv.ts](file:///f/murmur/scripts/import_discourse_csv.ts)\r\n\r\nImports forum databases from a Discourse CSV export.\r\n\r\n- **Workflow**: Parses user profiles, categories, topics, and posts.\r\n- **Parsing**: Translates Discourse emojis, system formatting, and image layouts into Murmur equivalents.\r\n- **Rebaking**: Automatically runs the HTML cooking function on all imported posts to populate the `cooked` column.\r\n\r\n### [import_post_revisions.ts](file:///f/murmur/scripts/import_post_revisions.ts)\r\n\r\nReads historical edit logs from a Discourse CSV export and populates the `postRevisions` table.\r\n\r\n- **Workflow**: Converts Discourse YAML modifications into JSON and links them to their respective post IDs.\r\n- **Recalculation**: Adjusts `version` and `publicVersion` sequences on affected posts.\r\n\r\n---\r\n\r\n## 2. Topic recommendations & Embeddings\r\n\r\n### [embed-posts.ts](file:///f/murmur/scripts/embed-posts.ts)\r\n\r\nGenerates vector float embeddings for search calculations.\r\n\r\n- **Workflow**: Identifies posts that do not have embeddings in the `post_embeddings` table.\r\n- **Generation**: Calls the configured LLM API (e.g. Ollama, OpenAI) to generate vector embeddings.\r\n- **Persistence**: Writes vector embeddings to the database as binary buffers.\r\n\r\n### [generate-related-topics.ts](file:///f/murmur/scripts/generate-related-topics.ts)\r\n\r\nComputes topic similarity recommendations.\r\n\r\n- **Workflow**: Calculates vector distances between topic posts.\r\n- **Storage**: Saves the top 50 matches for each topic as a JSON list in `topic_recommendations`.\r\n\r\n---\r\n\r\n## 3. Maintenance & Optimizations\r\n\r\n### [rebake-posts.ts](file:///f/murmur/scripts/rebake-posts.ts)\r\n\r\nRe-processes post content using the Markdown compiler.\r\n\r\n- **Usage**: Run this script after modifying markdown parser rules, BBCode preprocessors, or HTML sanitization options to update all compiled posts.\r\n\r\n### [rehash-compress-replace.ts](file:///f/murmur/scripts/rehash-compress-replace.ts)\r\n\r\nCompresses and optimizes uploaded assets.\r\n\r\n- **Workflow**: Compresses images on disk, updates their URLs, and updates file paths in raw and cooked post content.\r\n\r\n### [reindex.ts](file:///f/murmur/scripts/reindex.ts)\r\n\r\nRebuilds the virtual FTS5 search index.\r\n\r\n- **Workflow**: Wipes `search_index` and indexes all posts in the database.\r\n- **CJK splitting**: Uses Jieba to segment Chinese, Japanese, and Korean content.\r\n\r\n---\r\n\r\n## ⚡ Quick Start\r\n\r\n### 1. Prerequisite Installations\r\n\r\nEnsure you have Node.js 26+ and pnpm 11+ installed. If you plan to run the uploads proxy or semantic search, make sure `cargo` (Rust toolchain) and SQLite are available.\r\n\r\n### 2. Development Setup\r\n\r\nClone the repository and install dependencies:\r\n\r\n```sh\r\npnpm install\r\n```\r\n\r\nConfigure your environment variables:\r\n\r\n```sh\r\ncp .env.example .env\r\n```\r\n\r\nModify `.env` to define:\r\n\r\n- `DATABASE_URL`: Path to your database file (defaults to `local.db`).\r\n- `ORIGIN`: The server domain (e.g. `http://localhost:5173`).\r\n- `GITHUB_CLIENT_ID` / `GITHUB_CLIENT_SECRET`: OAuth credentials.\r\n\r\nInitialize and seed the database with mock categories, users, and topics:\r\n\r\n```sh\r\npnpm db:push\r\npnpm db:seed\r\n```\r\n\r\nStart the Vite development server:\r\n\r\n```sh\r\npnpm dev\r\n```\r\n\r\nOpen `/login` and use the **Dev Login** button to authenticate as any user without configuring GitHub OAuth.\r\n\r\n---\r\n\r\n## 📁 Repository Layout\r\n\r\n- `src/routes/` - SvelteKit routing layer (pages, endpoints, SSE streams).\r\n- `src/lib/` - Shared code, UI components, and state modules.\r\n- `src/lib/server/` - Server-side modules (database pools, auth Oslo, markdown cooker, search caches).\r\n- `scripts/` - Standalone tools (data imports, re-indexing, embedding pre-calculations).\r\n- `uploads-proxy/` - Rust Axum asset cache proxy server.\r\n- `doc/` - System documentation markdown files.","cooked":"

前言

本站使用的软件也运行了好多天了，目前各种小bug不断，但是好在没有什么致命的问题。
\n为了帮助用户更好地理解本软件的开发情况，所以写一篇文章，介绍有哪些功能，数据是怎么迁移的。

设计理念

简单，好用
\n侧重点：易用 > 易开发维护 > 拥有各种功能

实现的功能

本软件的设计，高度受Discourse启发。但不谋求实现Discourse的全部功能。

Murmur: Modern Discussion Forum

murmur is a high-performance, features-rich, Discourse/NimForum-style discussion forum. It is engineered with a hybrid SSR-First (Server-Side Rendering) architecture, delivering instant page loads and optimal SEO crawlers accessibility, while layering on real-time interactive widgets as progressive client-side enhancements.

🛠️ The Tech Stack

Murmur's architecture is divided into a high-performance web core, a relational database, and a specialized asset caching layer:

Web Core: SvelteKit 2 utilizing Svelte 5 Runes for robust state management.
Database & ORM: Drizzle ORM managing SQLite (better-sqlite3 driver) for synchronous operations.
Asset Performance: A custom caching proxy written in Rust using Axum (uploads-proxy).
Search Subsystem: SQLite FTS5 index combined with the sqlite-vec extension for vector operations, and @node-rs/jieba CJK tokenizer.
Real-time Features: Server-Sent Events (SSE) for pub/sub notifications, and Yjs text structures for wiki collaboration.
Styling: TailwindCSS v4 with Flowbite Svelte.

🏛️ System Architecture Preview

Murmur is designed to maximize speed by avoiding clientside rendering overhead during initial page loads.

graph TD\n    Client[Browser Client]\n    SvelteKit[SvelteKit Node.js Server]\n    SQLite[(SQLite DB / local.db)]\n    Proxy[Rust Uploads-Proxy]\n    Upstream[Upstream Assets Server]\n    LLM[LLM API / Ollama / Gemini / OpenAI / Anthropic]\n\n    Client -->|1. HTTP Page Request / SSE Stream| SvelteKit\n    Client -->|2. HTTP Uploaded Files| Proxy\n    Proxy -->|3. Cache Miss Fetch| Upstream\n    Proxy -->|4. Save locally| Disk[(Local Uploads)]\n    SvelteKit -->|5. Synchronous DB Queries| SQLite\n    SvelteKit -->|6. Vectors & Generation| LLM\n

Server-Side Rendering: SvelteKit processes the request, queries the SQLite database synchronously through the repository layer, compiles the page into static HTML, and responds.
Client Hydration: The browser displays the HTML, downloads the JavaScript bundles, and hydrates components (e.g. Composer, Likes, Chat).
Live Updates: An SSE connection is established to stream live updates (new posts, edits) to active tabs.
Asset Offloading: The Rust uploads-proxy handles asset queries. If a file is cached locally on disk, it is served; otherwise, the proxy downloads it from the upstream server with retry logic, saves it locally, and serves it.

📚 Documentation Index

Below is the directory map of the Murmur system documentation. Click on any link to view detailed guides:

⚙️ [System Architecture & Scaling](file:///f/murmur/doc/architecture.md)

Detailed breakdown of the request-response lifecycle and hydration.
Rust uploads-proxy microservice design (thundering herd protection, atomic writes).
Horizontal scaling recommendations (reconciliation, moving from memory SSE to Redis pub/sub, object storage).

Murmur: System Architecture

This document describes the high-level architecture of Murmur, including the frontend framework, backend data flow, integration with external services, and horizontal scaling patterns.

Technical Stack

Murmur is built using a modern, lightweight JavaScript stack, layered with a high-performance Rust microservice:

Frontend / Meta-framework: SvelteKit 2 using Svelte 5 (Runes mode).
Styling: TailwindCSS v4 (CSS-first, no tailwind.config.js) & Flowbite Svelte components.
Database: SQLite (better-sqlite3 driver) wrapped with Drizzle ORM.
Search Vector calculations: SQLite extension sqlite-vec.
Search tokenization: @node-rs/jieba for CJK languages.
Real-Time Communication: Server-Sent Events (SSE).
Collaboration: Yjs text types synced stateless-ly over HTTP.
Assets Proxy: Custom Axum-based Rust microservice (uploads-proxy).

Component Layout & Data Flow

Murmur leverages Server-Side Rendering (SSR) by default to deliver HTML instantly to crawlers and users. Clientside Svelte components progressively hydrate the page to enable real-time messaging, infinite scroll, and rich editing.

graph TD\n    Client[Browser Client]\n    SvelteKit[SvelteKit Node.js Server]\n    SQLite[(SQLite DB / local.db)]\n    Proxy[Rust Uploads-Proxy]\n    Upstream[Upstream Assets Server]\n    LLM[LLM API / Ollama / Gemini / OpenAI / Anthropic]\n\n    Client -->|1. HTTP / SSE| SvelteKit\n    Client -->|2. HTTP Static Assets| Proxy\n    Proxy -->|3. Fetch cache miss| Upstream\n    Proxy -->|4. Save locally| Disk[(Local Uploads)]\n    SvelteKit -->|5. Synchronous Queries| SQLite\n    SvelteKit -->|6. Vectors & Generation| LLM\n

1. Request Lifecycle

The browser requests a page (e.g., /t/slug/id).
SvelteKit's routing resolves the path and calls the server-side load function (+page.server.ts), which queries the database synchronously via the repository layer (src/lib/server/repo.ts).
SvelteKit renders the page as static HTML (SSR-first) and returns it.
The client browser parses the HTML, displays it immediately, and fetches Svelte bundles to hydrate active widgets (e.g., the composer, emoji picker, likes).
The client initiates a Server-Sent Events (SSE) connection to listen to live updates.

Node.js & Rust uploads-proxy Integration

User-uploaded attachments and legacy images are cached locally on disk. To prevent heavy Node.js threads from blocking on large file IO operations:

Node.js Server: Serves core HTML/JS payloads and handles API mutations.
Rust uploads-proxy: Serves files located under /uploads/. It listens on 127.0.0.1:4119 and handles file lookups.\n
- Cache Hit: Serves the local file immediately using high-speed async file streaming via tokio-util.
- Cache Miss: Downloads the asset from the configured upstream server (attempts up to 3 times, with a 2-second sleep duration between attempts), saves it locally, and serves it to the user.
- Thundering Herd Coalesce: Employs a mutex and tokio::sync::Notify to block duplicate concurrent requests for the same missing asset, ensuring only one download request hits the upstream server.
\n

Real-Time Engine (SSE Pub/Sub)

Murmur uses a single-process in-memory pub/sub pattern implemented in [events.ts](file:///f/murmur/src/lib/server/events.ts):

Route handlers or repository mutations call publish(topicId, event).
The Server-Sent Events endpoint [events/+server.ts](file:///f/murmur/src/routes/t/[slug]/[id]/events/+server.ts) subscribes to updates for a specific topic.
It streams JSON-LD chunk lines to the browser's EventSource instance.
The UI handles the event reactively to append new posts, update counts, or delete posts without page reloads.

Horizontal Scaling Strategy

Because SQLite and the SSE pub/sub system operate within a single process, horizontal scaling (running multiple instances of the Node.js process) requires:

Database Layer: Replace the local better-sqlite3 instance with a replicated SQLite setup (such as LiteFS) or migrate Drizzle configuration to PostgreSQL.
Pub/Sub Layer: Replace the in-memory singleton in [events.ts](file:///f/murmur/src/lib/server/events.ts) with a Redis or RabbitMQ pub/sub server. Route connections to the pub/sub interface so that messages publish across all Node server processes.
Uploads caching: Ensure the uploads-proxy instances point to a shared directory (e.g., NFS, GlusterFS) or replace the local filesystem storage with an Object Store (like AWS S3) by modifying the SvelteKit file upload API.

🗄️ [Data Model & Database Schema](file:///f/murmur/doc/data-model.md)

Relational schema definitions for SQLite.
Data design conventions (auto-increments, ISO-8601 timestamps, soft-deletions).
Database tables, relationships, and indices optimization.

Murmur: Data Model & Database Schema

This document details the database schema, SQLite conventions, and tables managed by Drizzle ORM in Murmur. The database schema is located in [schema.ts](file:///f/murmur/src/lib/server/db/schema.ts).

Data Layer Conventions

To simplify imports, backups, and exports from active Discourse installations, Murmur maintains the following database patterns:

Integer Auto-increment Primary Keys: All tables use auto-increment integers. The system user is seeded with ID -1 (matching Discourse conventions).
ISO-8601 Timestamps: Timestamps are stored as text (e.g. 2026-07-15T22:00:00.000Z) rather than SQLite Unix integers. They default to the current time via $defaultFn(() => new Date().toISOString()). This allows lexicographical sorting and matches Discourse CSV files.
Booleans: Stored as integers (0 or 1) using Drizzle's integer(..., { mode: 'boolean' }).
Post Rendering: Posts store both the raw Markdown text (raw) and the cooked sanitized HTML (cooked). Rendering is done via {@html cooked} without dynamic clientside compilation.
Soft Deletions: Deleting topics or posts updates a deletedAt ISO timestamp column instead of deleting database rows.

Database Tables

Below is an overview of the Drizzle tables:

1. `users`

Represents forum members, administrators, and moderators.

id: Integer primary key (Auto-increment).
username: Unique username (Text).
name: Full display name (Text, nullable).
avatarUrl: URL of the user's avatar.
email: User email address.
githubId: Bound GitHub ID for SSO.
admin / moderator: Role toggles (Booleans).
trustLevel: Trust system ranking (0 to 4).
silencedUntil: ISO timestamp until which the user is banned from posting.
signatureRaw / signatureCooked: Markdown/HTML for post signatures.
chatGlobalMuted: Toggle to silence all chat badge notifications (Boolean).

2. `sessions` & `ssoSyncTickets`

Authentication sessions matching Oslo specifications.

sessions.id: SHA-256 hash of the session token.
sessions.userId: References users.id (on cascade delete).
sessions.expiresAt: Expiration timestamp.

3. `categories` & `categoryGroups`

Discussion categories which support nesting.

categories.id: Integer primary key.
categories.slug: Unique URL slug.
categories.parentCategoryId: Self-referential FK targeting the parent category.
categories.readRestricted: If true, restricts read permissions.
categoryGroups: Maps categoryId to groupId and defines permissionType (Read, Create, Reply).

4. `topics` & `posts`

Forum threads and post items.

topics.userId: Thread creator.
topics.bumpedAt: ISO timestamp updated on every reply to drive sorting.
topics.closed / topics.pinned: State flags.
posts.topicId: Parent thread.
posts.postNumber: 1-based sequential position within the topic (OP is 1).
posts.raw / posts.cooked: Markdown/cooked text.
posts.replyToPostNumber: Identifies the post being replied to.
posts.wiki: Toggle indicating if the post is editable by non-creators.

5. `postRevisions`

History of edits made outside the grace period.

postId: References posts.id.
userId: Author of the edit.
modifications: JSON string of modified properties (e.g. raw, cooked, title).
number: Version sequence number.
hidden: If true, hides modifications from normal users.

6. `postEmbeddings` & `topicRecommendations`

Vector semantic search embeddings and precomputed suggestions.

postEmbeddings.postId: References posts.id.
postEmbeddings.embedding: Raw vector float buffer (Float32Array).
topicRecommendations.topicId: References topics.id.
topicRecommendations.recommendations: JSON string list of recommended topic IDs.

7. `chatChannels`, `chatChannelMembers` & `chatMessages`

Real-time messaging channels.

chatMessages.channelId: Parent channel.
chatMessages.message / chatMessages.cooked: Raw/cooked text.
chatChannelMembers.muted: Individual channel muting preference.

Schema DDL & Indices

Drizzle automatically creates indices to optimize query performance and prevent database locks:

posts_topic_number_idx on (posts.topicId, posts.postNumber).
topics_category_bumped_idx on (topics.categoryId, topics.bumpedAt).
likes_post_user_unq unique index on (likes.postId, likes.userId).
topic_recommendations and post_embeddings map primary keys to their parent components to enable swift joins.

✍️ [Reply Composer, Drafts & Uploads](file:///f/murmur/doc/features/composer.md)

Composer UI controls (height adjustments, split ratios, fullscreen mode).
Local storage autosave and server-side syncing database logic.
Formatting toolbar helpers, emoji picker, and attachments uploads.
Kate-style whole-line keyboard shortcuts (Ctrl+X / Ctrl+C).

Composer UI, Attachments & Draft Syncing

This document describes the reply composer UI, clientside formatting shortcuts, the draft autosave engine, and the multi-file attachment pipeline.

Composer Layout & Controls

The composer component [Composer.svelte](file:///f/murmur/src/lib/components/Composer.svelte) is modeled after the Discourse editor. It resides in a sticky drawer at the bottom of the screen.

+-------------------------------------------------------------------+\n|  [v] Drag Handle (Height)                                         |\n+-------------------------------------------------------------------+\n|  Title / Action Summary                      [ ] Full [ _ ] Min  |\n+-------------------------------------------------------------------+\n|  [ B ] [ I ] [ Link ] [ Quote ] [ Tt v ] [ + v ] [ Undo ] [ Redo ]|\n+-------------------------------------------------------------------+\n|  Textarea Input (Markdown)         |  Live HTML Preview           |\n|                                    |                              |\n|                                    |                              |\n|                                    |<---> Vertical Split Resizer  |\n+-------------------------------------------------------------------+\n

Resizing Features

Height Resizing: Users can click and drag the top border handle. The chosen height is written to localStorage.
Split ratio Resizing: A vertical split bar sits between the editor and preview panels. Dragging it updates the split percentage dynamically.
Fullscreen mode: Toggles the composer to fill the viewport.
Minimization drawer: Minimizes the composer into a small, floating status bar at the bottom right.

Rich Formatting Toolbar & Emojis

1. Heading Selector (`Tt`)

Inserts heading markdown (# to ####) at the cursor selection.

2. Actions Selector (`+`)

Hide Details: Inserts a folding BBCode detail block:

<details><summary>Summary</summary>\n\n\ncontent\n\n\n

\n\n ```\n- **Footnote**: Inserts inline footnote syntax (`^[footnote]`).\n- **LaTeX Math**: Inserts standard math symbols wrapper (`$x^2## 前言\n本站使用的软件也运行了好多天了，目前各种小bug不断，但是好在没有什么致命的问题。\n为了帮助用户更好地理解本软件的开发情况，所以写一篇文章，介绍有哪些功能，数据是怎么迁移的。\n

设计理念

简单，好用
\n侧重点：易用 > 易开发维护 > 拥有各种功能

实现的功能

本软件的设计，高度受Discourse启发。但不谋求实现Discourse的全部功能。

Murmur: Modern Discussion Forum

🛠️ The Tech Stack

Murmur's architecture is divided into a high-performance web core, a relational database, and a specialized asset caching layer:

Web Core: SvelteKit 2 utilizing Svelte 5 Runes for robust state management.
Database & ORM: Drizzle ORM managing SQLite (better-sqlite3 driver) for synchronous operations.
Asset Performance: A custom caching proxy written in Rust using Axum (uploads-proxy).
Search Subsystem: SQLite FTS5 index combined with the sqlite-vec extension for vector operations, and @node-rs/jieba CJK tokenizer.
Real-time Features: Server-Sent Events (SSE) for pub/sub notifications, and Yjs text structures for wiki collaboration.
Styling: TailwindCSS v4 with Flowbite Svelte.

🏛️ System Architecture Preview

Murmur is designed to maximize speed by avoiding clientside rendering overhead during initial page loads.

graph TD\n    Client[Browser Client]\n    SvelteKit[SvelteKit Node.js Server]\n    SQLite[(SQLite DB / local.db)]\n    Proxy[Rust Uploads-Proxy]\n    Upstream[Upstream Assets Server]\n    LLM[LLM API / Ollama / Gemini / OpenAI / Anthropic]\n\n    Client -->|1. HTTP Page Request / SSE Stream| SvelteKit\n    Client -->|2. HTTP Uploaded Files| Proxy\n    Proxy -->|3. Cache Miss Fetch| Upstream\n    Proxy -->|4. Save locally| Disk[(Local Uploads)]\n    SvelteKit -->|5. Synchronous DB Queries| SQLite\n    SvelteKit -->|6. Vectors & Generation| LLM\n

Server-Side Rendering: SvelteKit processes the request, queries the SQLite database synchronously through the repository layer, compiles the page into static HTML, and responds.
Client Hydration: The browser displays the HTML, downloads the JavaScript bundles, and hydrates components (e.g. Composer, Likes, Chat).
Live Updates: An SSE connection is established to stream live updates (new posts, edits) to active tabs.
Asset Offloading: The Rust uploads-proxy handles asset queries. If a file is cached locally on disk, it is served; otherwise, the proxy downloads it from the upstream server with retry logic, saves it locally, and serves it.

📚 Documentation Index

Below is the directory map of the Murmur system documentation. Click on any link to view detailed guides:

⚙️ [System Architecture & Scaling](file:///f/murmur/doc/architecture.md)

Detailed breakdown of the request-response lifecycle and hydration.
Rust uploads-proxy microservice design (thundering herd protection, atomic writes).
Horizontal scaling recommendations (reconciliation, moving from memory SSE to Redis pub/sub, object storage).

Murmur: System Architecture

This document describes the high-level architecture of Murmur, including the frontend framework, backend data flow, integration with external services, and horizontal scaling patterns.

Technical Stack

Murmur is built using a modern, lightweight JavaScript stack, layered with a high-performance Rust microservice:

Frontend / Meta-framework: SvelteKit 2 using Svelte 5 (Runes mode).
Styling: TailwindCSS v4 (CSS-first, no tailwind.config.js) & Flowbite Svelte components.
Database: SQLite (better-sqlite3 driver) wrapped with Drizzle ORM.
Search Vector calculations: SQLite extension sqlite-vec.
Search tokenization: @node-rs/jieba for CJK languages.
Real-Time Communication: Server-Sent Events (SSE).
Collaboration: Yjs text types synced stateless-ly over HTTP.
Assets Proxy: Custom Axum-based Rust microservice (uploads-proxy).

Component Layout & Data Flow

graph TD\n    Client[Browser Client]\n    SvelteKit[SvelteKit Node.js Server]\n    SQLite[(SQLite DB / local.db)]\n    Proxy[Rust Uploads-Proxy]\n    Upstream[Upstream Assets Server]\n    LLM[LLM API / Ollama / Gemini / OpenAI / Anthropic]\n\n    Client -->|1. HTTP / SSE| SvelteKit\n    Client -->|2. HTTP Static Assets| Proxy\n    Proxy -->|3. Fetch cache miss| Upstream\n    Proxy -->|4. Save locally| Disk[(Local Uploads)]\n    SvelteKit -->|5. Synchronous Queries| SQLite\n    SvelteKit -->|6. Vectors & Generation| LLM\n

1. Request Lifecycle

The browser requests a page (e.g., /t/slug/id).
SvelteKit's routing resolves the path and calls the server-side load function (+page.server.ts), which queries the database synchronously via the repository layer (src/lib/server/repo.ts).
SvelteKit renders the page as static HTML (SSR-first) and returns it.
The client browser parses the HTML, displays it immediately, and fetches Svelte bundles to hydrate active widgets (e.g., the composer, emoji picker, likes).
The client initiates a Server-Sent Events (SSE) connection to listen to live updates.

Node.js & Rust uploads-proxy Integration

User-uploaded attachments and legacy images are cached locally on disk. To prevent heavy Node.js threads from blocking on large file IO operations:

Node.js Server: Serves core HTML/JS payloads and handles API mutations.
Rust uploads-proxy: Serves files located under /uploads/. It listens on 127.0.0.1:4119 and handles file lookups.\n
- Cache Hit: Serves the local file immediately using high-speed async file streaming via tokio-util.
- Cache Miss: Downloads the asset from the configured upstream server (attempts up to 3 times, with a 2-second sleep duration between attempts), saves it locally, and serves it to the user.
- Thundering Herd Coalesce: Employs a mutex and tokio::sync::Notify to block duplicate concurrent requests for the same missing asset, ensuring only one download request hits the upstream server.
\n

Real-Time Engine (SSE Pub/Sub)

Murmur uses a single-process in-memory pub/sub pattern implemented in [events.ts](file:///f/murmur/src/lib/server/events.ts):

Route handlers or repository mutations call publish(topicId, event).
The Server-Sent Events endpoint [events/+server.ts](file:///f/murmur/src/routes/t/[slug]/[id]/events/+server.ts) subscribes to updates for a specific topic.
It streams JSON-LD chunk lines to the browser's EventSource instance.
The UI handles the event reactively to append new posts, update counts, or delete posts without page reloads.

Horizontal Scaling Strategy

Because SQLite and the SSE pub/sub system operate within a single process, horizontal scaling (running multiple instances of the Node.js process) requires:

Database Layer: Replace the local better-sqlite3 instance with a replicated SQLite setup (such as LiteFS) or migrate Drizzle configuration to PostgreSQL.
Pub/Sub Layer: Replace the in-memory singleton in [events.ts](file:///f/murmur/src/lib/server/events.ts) with a Redis or RabbitMQ pub/sub server. Route connections to the pub/sub interface so that messages publish across all Node server processes.
Uploads caching: Ensure the uploads-proxy instances point to a shared directory (e.g., NFS, GlusterFS) or replace the local filesystem storage with an Object Store (like AWS S3) by modifying the SvelteKit file upload API.

🗄️ [Data Model & Database Schema](file:///f/murmur/doc/data-model.md)

Relational schema definitions for SQLite.
Data design conventions (auto-increments, ISO-8601 timestamps, soft-deletions).
Database tables, relationships, and indices optimization.

Murmur: Data Model & Database Schema

Data Layer Conventions

To simplify imports, backups, and exports from active Discourse installations, Murmur maintains the following database patterns:

Integer Auto-increment Primary Keys: All tables use auto-increment integers. The system user is seeded with ID -1 (matching Discourse conventions).
ISO-8601 Timestamps: Timestamps are stored as text (e.g. 2026-07-15T22:00:00.000Z) rather than SQLite Unix integers. They default to the current time via $defaultFn(() => new Date().toISOString()). This allows lexicographical sorting and matches Discourse CSV files.
Booleans: Stored as integers (0 or 1) using Drizzle's integer(..., { mode: 'boolean' }).
Post Rendering: Posts store both the raw Markdown text (raw) and the cooked sanitized HTML (cooked). Rendering is done via {@html cooked} without dynamic clientside compilation.
Soft Deletions: Deleting topics or posts updates a deletedAt ISO timestamp column instead of deleting database rows.

Database Tables

Below is an overview of the Drizzle tables:

1. `users`

Represents forum members, administrators, and moderators.

id: Integer primary key (Auto-increment).
username: Unique username (Text).
name: Full display name (Text, nullable).
avatarUrl: URL of the user's avatar.
email: User email address.
githubId: Bound GitHub ID for SSO.
admin / moderator: Role toggles (Booleans).
trustLevel: Trust system ranking (0 to 4).
silencedUntil: ISO timestamp until which the user is banned from posting.
signatureRaw / signatureCooked: Markdown/HTML for post signatures.
chatGlobalMuted: Toggle to silence all chat badge notifications (Boolean).

2. `sessions` & `ssoSyncTickets`

Authentication sessions matching Oslo specifications.

sessions.id: SHA-256 hash of the session token.
sessions.userId: References users.id (on cascade delete).
sessions.expiresAt: Expiration timestamp.

3. `categories` & `categoryGroups`

Discussion categories which support nesting.

categories.id: Integer primary key.
categories.slug: Unique URL slug.
categories.parentCategoryId: Self-referential FK targeting the parent category.
categories.readRestricted: If true, restricts read permissions.
categoryGroups: Maps categoryId to groupId and defines permissionType (Read, Create, Reply).

4. `topics` & `posts`

Forum threads and post items.

topics.userId: Thread creator.
topics.bumpedAt: ISO timestamp updated on every reply to drive sorting.
topics.closed / topics.pinned: State flags.
posts.topicId: Parent thread.
posts.postNumber: 1-based sequential position within the topic (OP is 1).
posts.raw / posts.cooked: Markdown/cooked text.
posts.replyToPostNumber: Identifies the post being replied to.
posts.wiki: Toggle indicating if the post is editable by non-creators.

5. `postRevisions`

History of edits made outside the grace period.

postId: References posts.id.
userId: Author of the edit.
modifications: JSON string of modified properties (e.g. raw, cooked, title).
number: Version sequence number.
hidden: If true, hides modifications from normal users.

6. `postEmbeddings` & `topicRecommendations`

Vector semantic search embeddings and precomputed suggestions.

postEmbeddings.postId: References posts.id.
postEmbeddings.embedding: Raw vector float buffer (Float32Array).
topicRecommendations.topicId: References topics.id.
topicRecommendations.recommendations: JSON string list of recommended topic IDs.

7. `chatChannels`, `chatChannelMembers` & `chatMessages`

Real-time messaging channels.

chatMessages.channelId: Parent channel.
chatMessages.message / chatMessages.cooked: Raw/cooked text.
chatChannelMembers.muted: Individual channel muting preference.

Schema DDL & Indices

Drizzle automatically creates indices to optimize query performance and prevent database locks:

posts_topic_number_idx on (posts.topicId, posts.postNumber).
topics_category_bumped_idx on (topics.categoryId, topics.bumpedAt).
likes_post_user_unq unique index on (likes.postId, likes.userId).
topic_recommendations and post_embeddings map primary keys to their parent components to enable swift joins.

✍️ [Reply Composer, Drafts & Uploads](file:///f/murmur/doc/features/composer.md)

Composer UI controls (height adjustments, split ratios, fullscreen mode).
Local storage autosave and server-side syncing database logic.
Formatting toolbar helpers, emoji picker, and attachments uploads.
Kate-style whole-line keyboard shortcuts (Ctrl+X / Ctrl+C).

Composer UI, Attachments & Draft Syncing

This document describes the reply composer UI, clientside formatting shortcuts, the draft autosave engine, and the multi-file attachment pipeline.

Composer Layout & Controls

The composer component [Composer.svelte](file:///f/murmur/src/lib/components/Composer.svelte) is modeled after the Discourse editor. It resides in a sticky drawer at the bottom of the screen.

+-------------------------------------------------------------------+\n|  [v] Drag Handle (Height)                                         |\n+-------------------------------------------------------------------+\n|  Title / Action Summary                      [ ] Full [ _ ] Min  |\n+-------------------------------------------------------------------+\n|  [ B ] [ I ] [ Link ] [ Quote ] [ Tt v ] [ + v ] [ Undo ] [ Redo ]|\n+-------------------------------------------------------------------+\n|  Textarea Input (Markdown)         |  Live HTML Preview           |\n|                                    |                              |\n|                                    |                              |\n|                                    |<---> Vertical Split Resizer  |\n+-------------------------------------------------------------------+\n

Resizing Features

Height Resizing: Users can click and drag the top border handle. The chosen height is written to localStorage.
Split ratio Resizing: A vertical split bar sits between the editor and preview panels. Dragging it updates the split percentage dynamically.
Fullscreen mode: Toggles the composer to fill the viewport.
Minimization drawer: Minimizes the composer into a small, floating status bar at the bottom right.

Rich Formatting Toolbar & Emojis

1. Heading Selector (`Tt`)

Inserts heading markdown (# to ####) at the cursor selection.

2. Actions Selector (`+`)

Hide Details: Inserts a folding BBCode detail block:

<details><summary>Summary</summary>\n\n\ncontent\n\n\n

\n\n ```\n- **Footnote**: Inserts inline footnote syntax (`^[footnote]`).\n- **LaTeX Math**: Inserts standard math symbols wrapper ().\n- **Mermaid Flowchart**: Inserts standard Mermaid graphs template.\n- **Table / Date**: Generates formatted markdown tables or inserts local timestamps.\n

3. Emoji Picker

A custom Svelte 5 component [EmojiPicker.svelte](file:///f/murmur/src/lib/components/EmojiPicker.svelte) displays emoji categories. It supports text-based query filtering and inserts the selected emoji at the cursor.

Kate-Like Whole Line Editing

When typing inside the editor, Murmur supports KDE Kate-style shortcuts:

Ctrl+C (Copy): If no characters are selected (collapsed selection), copies the entire line containing the cursor (including the newline character).
Ctrl+X (Cut): If no selection is active, cuts the entire line and moves lines below it up.
Cursor Repositioning: After cutting, the cursor automatically snaps to the start of the next line, saving keystrokes.

This logic is bound to the handleKeyDown function in the composer. It interacts directly with the browser's clipboard API and forces a tick update on Svelte's reactive states to maintain history.

Draft Autosave & Database Synchronization

To prevent data loss, the composer synchronizes drafts using a hybrid approach:

sequenceDiagram\n    participant Client as Browser Textarea\n    participant Local as localStorage Cache\n    participant API as /api/drafts/sync\n    participant DB as SQLite drafts Table\n\n    Client->>Local: Save drafts instantly on keystroke (Keydown)\n    Client->>API: Periodic debounced save (Every 5 seconds)\n    API->>DB: Write draft using compound key (userId + key)\n    Note over API, DB: Last-write-wins merging by timestamp\n

LocalStorage: Serves as the primary cache to save text instantaneously.
Database Synchronization: Every 5 seconds (debounced), the client makes an HTTP POST request to /api/drafts/sync. It pushes drafts to the SQLite database, ensuring edits are preserved across devices.
Draft Disposal: Submitting a post deletes both the local storage key and the database row.

Clipboard Paste & Attachment Pipeline

The upload interface is triggered by clicking the paperclip icon or pasting an image directly from the clipboard into the editor:

Upload Initiation: An async file reader creates a unique placeholder: [Uploading image.png...] at the current cursor position.
Chromium List safeguard: Before beginning async processing, the file list is cloned into a static array. This prevents Chromium browsers from resetting the active input files when the input element value is cleared.
API Upload: The file is sent via multipart POST to /api/upload. The endpoint validates files against allowed formats (e.g. jpg, png, gif, svg).
Link Replacement: Once the API returns the permanent URL, the placeholder is replaced with the correct Markdown markup: ![filename|widthxheight](url).

Codeblocks

auto-detection of codeblocks based on the first line of the codeblock.
mermaid flowcharts.
toggle line numbers
copy and full-screen buttons

🔍 [Full-Text & Semantic Search (RAG)](file:///f/murmur/doc/features/search-and-rag.md)

SQLite FTS5 full-text indexing with Porter stemming.
CJK search query segmentation using Jieba tokenizer.
sqlite-vec semantic vector distance search pipeline.
RAG AI chatbot integration and context permission safeguards.

Full-Text & Semantic Search (RAG)

This document describes the search subsystem in Murmur, including full-text indexing, CJK segmentation, vector semantic calculations, and the RAG (Retrieval-Augmented Generation) chatbot.

1. Full-Text Search Index (FTS5)

Full-text search is powered by SQLite's FTS5 extension. During database initialization in [index.ts](file:///f/murmur/src/lib/server/db/index.ts), Murmur configures the virtual table search_index with Porter stemming to enable English word matching (e.g. \"runs\" matches \"running\"):

CREATE VIRTUAL TABLE IF NOT EXISTS search_index USING fts5(\n    title,\n    content,\n    topic_id UNINDEXED,\n    post_id UNINDEXED,\n    post_number UNINDEXED,\n    user_id UNINDEXED,\n    category_id UNINDEXED,\n    created_at UNINDEXED,\n    tokenize = 'porter unicode61'\n);\n

CJK Query Segmentation (Jieba)

Standard FTS5 tokenizers do not split Chinese, Japanese, or Korean (CJK) characters into individual words because they lack space boundaries.

To solve this, Murmur intercepts search requests in [repo.ts](file:///f/murmur/src/lib/server/repo.ts) and parses queries using Jieba (@node-rs/jieba):

RegEx Splitting: The query is split into CJK and non-CJK blocks.
Segmentation: CJK blocks are parsed using jieba.cutAll() to extract words.
FTS5 Wrapping: Extracted tokens are wrapped in double quotes and separated by spaces, converting the search into an implicit AND condition (e.g. 学习 Svelte becomes \"学习\" \"Svelte\").

2. Semantic Search (sqlite-vec)

Semantic search finds matches based on conceptual similarity rather than exact keyword matches. It uses the sqlite-vec extension to run vector operations inside SQLite.

Storage: Embeddings are stored as raw binary BLOBs (Float32Array) in the post_embeddings table to prevent thread blocking (converting JSON strings to arrays in JavaScript is CPU intensive).
Startup Migration: On startup, Murmur checks the database. If it finds legacy JSON string representations in the embeddings table, it converts them to binary buffers in batches of 500 rows.

Query Resolution: Employs a cosine similarity calculation in SQLite:

SELECT post_id, vec_distance_cosine(embedding, ?) AS distance\nFROM post_embeddings\nORDER BY distance ASC\nLIMIT 20;\n

3. RAG Chatbot Integration

When a user runs a search query, Murmur displays a RAG chatbot assistant panel next to the search results:

+-----------------------------------------------------------------+\n| Search Results                     |  🤖 Chatbot Reply          |\n|------------------------------------|----------------------------|\n| 1. Topic: How to run Murmur        |  To run Murmur, execute:   |\n|    By user1 - \"Run pnpm dev...\"    |  `pnpm dev`. Refer to      |\n|                                    |  /t/murmur/1 for details.  |\n| 2. Topic: Production Setup         |                            |\n|    By user2 - \"Compile build...\"   |  [ AI thinking indicator ] |\n+-----------------------------------------------------------------+\n

LLM Streaming client

The server client in [chatLlm.ts](file:///f/murmur/src/lib/server/chatLlm.ts) streams responses from different providers:

Ollama: Parses line-by-line NDJSON chunks.
Gemini / OpenAI / Anthropic: Parses Server-Sent Events (SSE) data streams.

SSE Stream Synthesis

When a search is requested, the search API endpoint [stream/+server.ts](file:///f/murmur/src/routes/api/search/stream/+server.ts) initiates an SSE stream:

It queries FTS5 and semantic databases for matches.
Results are formatted as Markdown context blocks.
It asks the LLM to summarize the findings and include links to the matched topics (e.g. /t/slug/id/post_number).
It streams response tokens to the client browser via ai_answer_chunk events, which are compiled and rendered in the UI using markdown-it.

4. Privacy & Permission Filtering

To prevent information leaks (e.g., exposing private messages or restricted categories in search results or AI summaries):

Permission Checking: The search service retrieves category permission rules via getPermittedCategoryIds(user).
Pre-filtering: Topics and posts belonging to restricted categories are filtered out of the results before the data is cached or sent to the LLM.
RAG Security: The LLM context contains only the results the user has permission to read.

✍️ [Revisions & Wiki Editing](file:///f/murmur/doc/features/revisions-and-wiki.md)

Post edit revisions, grace period overrides, and modifications storage.
Myers split-diff comparison modal and revision rollbacks.
Wiki post permissions and stateless Yjs collaborative sync protocol.

Revisions, Wiki Posts & Yjs Collaborative Editing

This document explains the version control system for post edits, wiki post privileges, and the collaborative Yjs document synchronization engine.

1. Post Revisions System

Murmur tracks the history of changes made to posts and topic details (titles and categories) over time.

Grace Period & Versioning

Editing Grace Period: Managed by the editing_grace_period site setting. If a user edits a post within this threshold (e.g. 5 minutes) after creation, modifications are overwritten without creating a new revision.
Revision Capture: Edits made after the grace period increment the post's version (and publicVersion) and save the differences to post_revisions.

Modifications Storage: Differences are stored as JSON-serialized change objects containing the previous values of modified fields:

{\n\t\"raw\": [\"Old post raw markdown\", \"New post raw markdown\"],\n\t\"title\": [\"Old topic title\", \"New topic title\"]\n}\n

2. Visual Diff Comparison Modal

When a post contains revisions, a clickable edit counter (e.g., ✍️ 2) is displayed next to the post's timestamp. Clicking it opens the [PostRevisionsModal.svelte](file:///f/murmur/src/lib/components/PostRevisionsModal.svelte) component:

Myers Diff Algorithm: Compares line-by-line differences in raw markdown or HTML tags.
Cooked HTML Preview: Toggles to render the formatted HTML output.
Reversion: Users with edit permissions can click the \"Revert\" button to roll back the post to a chosen revision.
Visibility controls: Staff members (administrators and moderators) can toggle the visibility of individual revisions to hide sensitive information.

3. Wiki Posts

Wiki posts allow collaborative editing by members of the community rather than just the author.

Wiki Toggle: Staff members can convert any post into a wiki post.
Permissions: Admins can configure the minimum user group or trust level required to edit wiki posts (defaults to staff and trust_level_2 users).
Indicators: Displays a wiki badge next to the post to show users they have permission to edit it.

4. Stateless Yjs HTTP Sync Protocol

To enable concurrent collaborative editing without keeping persistent WebSocket connections open, Murmur uses a stateless HTTP synchronization protocol:

sequenceDiagram\n    participant Client as Browser Editor\n    participant Server as /api/posts/[id]/wiki-sync\n    participant DB as SQLite DB\n\n    Client->>Server: Send stateVector (base64) & update (base64)\n    Server->>DB: Fetch post yjsState BLOB\n    Server->>Server: Apply client update to Y.Doc\n    Server->>Server: Compute diff using client stateVector\n    Server->>DB: Save updated Y.Doc buffer to yjs_state\n    Server->>Client: Return update (base64) & merged text\n

Protocol Steps

Initiation: The editor initializes a Yjs document (Y.Doc) and extracts the text content.
Synchronization Request: The client sends an HTTP POST request to /api/posts/[id]/wiki-sync containing its local Yjs state vector and updates, encoded in base64.
Server Merge:\n
- The server loads the post's current yjsState from SQLite.
- It instantiates a server-side Y.Doc and applies the database state.
- It applies the base64 update received from the client.
\n
Calculations: The server computes the missing updates for the client using the client's state vector.
DB Update: If the merged text differs from the post's current content, the server runs editPost() to update raw/cooked columns and saves the new Yjs state buffer.
Response: The server returns the missing updates and the updated text to the client.

💬 [Real-Time Chat & Badges](file:///f/murmur/doc/features/chat.md)

Public channels, direct message groups, and message management.
Svelte 5 runes unread badge calculation and global/channel muting.

Real-Time Chat & Unread Badge system

This document describes the real-time chat subsystem in Murmur, including channels, user privileges, message edits, and the Svelte 5 runes unread badge system.

1. Chat Channels & DM Rooms

Murmur includes a built-in real-time chat workspace. It supports two channel types:

Public Channels: General channels accessible by all registered members.
Direct Messages (DMs): Private chat rooms created between two or more users.

2. Real-Time Message Flow

The chat interface streams updates in real time using Server-Sent Events (SSE):

sequenceDiagram\n    participant ClientA as Sender Browser\n    participant API as /api/chat/messages\n    participant DB as SQLite DB\n    participant SSE as SSE Stream Channel\n    participant ClientB as Receiver Browser\n\n    ClientA->>API: POST send message\n    API->>DB: Write message to chat_messages\n    API->>SSE: Publish message event\n    SSE->>ClientB: Stream JSON event\n    Note over ClientB: Append message to UI\n

Message Modifications

Users can edit or delete their own messages after they are sent. Moderators and administrators can edit or delete any message. Modifying a message updates the database and publishes an update event over SSE to sync all active clients.

3. Svelte 5 Runes Unread Badge System

To manage notification badges dynamically across the app, Murmur uses a client-side global state store powered by Svelte 5 runes:

State Store: Defined in a client-safe state module. It tracks active channels, members, read states, and global mute preferences.

Dynamic Badge calculations: Unread counts are computed using $derived values:

// Simplified logic\nconst unreadCount = $derived(() => {\n\tif (globalMuted) return 0;\n\treturn channels\n\t\t.filter((c) => !c.muted)\n\t\t.reduce((sum, c) => sum + (c.messagesCount - c.lastReadMessageId), 0);\n});\n

Real-Time Updates: When a new message event is received via the SSE connection, the client-side store increments the channel's message count. The UI updates the unread badge automatically.
Read State Sync: Reading a channel updates the client-side state and sends an API call to save the lastReadMessageId in the database, syncing the read state across devices.

4. Muting Preferences

To prevent notification fatigue, users can configure their alert preferences:

Channel Muting: Users can mute individual channels. Muted channels do not trigger badge alerts or red dot indicators.
Global Muting: Users can enable global muting in their profile settings. This silences all chat notifications across the site.

💻 [Developer Guide & Conventions](file:///f/murmur/doc/developer-guide.md)

Coding standards (runes, TailwindCSS v4, synchronous queries, types).
File-based routing structure.
ESLint and Prettier code formatting standards.

Developer Guide & Conventions

This document outlines the coding standards, repository layout, and development guidelines for developers working on the Murmur project.

1. Project Conventions

Svelte 5 Runes Mode

Murmur uses Svelte 5 runes. Use $props(), $state(), $derived(), and $effect() for state management.
Do not use legacy Svelte APIs (such as export let, $:, or store.subscribe).
The compiler options are inlined in the vite.config.ts file under the Svelte plugin configurations.

Database Layer

SQLite database operations run synchronously.
Do not await database queries: Drizzle queries use .all(), .get(), and .run() synchronously.
Implement database mutations in [repo.ts](file:///f/murmur/src/lib/server/repo.ts). Do not write inline database queries in routes or page endpoints.

Code Separation

Server-Only Code: Any logic placed under src/lib/server/** is excluded from browser bundles.
Shared Types: Clientside modules cannot import server code. Shared structures must be defined in [types.ts](file:///f/murmur/src/lib/types.ts).

Styling & CSS (TailwindCSS v4)

TailwindCSS v4 is integrated as a CSS-first design tool. There is no tailwind.config.js.
Configurations, theme overrides, and custom fonts are managed in [layout.css](file:///f/murmur/src/routes/layout.css) using CSS variables.

2. Directory Routing Structure

The routing layer uses standard SvelteKit file-based routing under src/routes/:

/ : Landing page displaying latest topics.
/c/[slug]/[id] : Topics filtered by category.
/t/[slug]/[id] : Topic thread displaying posts and reply forms. Includes SSE streaming listeners.
/new : Full-page composer workspace.
/login : Authentication page (GitHub SSO & dev login).
/admin/settings : Dashboard for administrators to configure site settings.
/sitemap.xml : Dynamic sitemap generation for web crawlers.

3. Formatting Standards

Code styling is enforced using Prettier and ESLint:

Indent: Tabs.
Quotes: Single quotes (').
Trailing Commas: None.
Line Width: 100 characters.

Verify formatting using the formatting scripts before submitting changes:

pnpm lint      # Checks linting and formatting compliance\npnpm format    # Automatically formats files\n

🛠️ [Operations & Maintenance Scripts](file:///f/murmur/doc/scripts.md)

Standalone operational scripts under scripts/ (import tools, reindexing, post rebaking, and embedding calculations).

Data Migrations & Operations Scripts

This document details the standalone command-line scripts located in the /scripts/ folder. These scripts are run using tsx (e.g. pnpm tsx scripts/rebake-posts.ts).

1. Data Imports & Exports

[import_discourse_csv.ts](file:///f/murmur/scripts/import_discourse_csv.ts)

Imports forum databases from a Discourse CSV export.

Workflow: Parses user profiles, categories, topics, and posts.
Parsing: Translates Discourse emojis, system formatting, and image layouts into Murmur equivalents.
Rebaking: Automatically runs the HTML cooking function on all imported posts to populate the cooked column.

[import_post_revisions.ts](file:///f/murmur/scripts/import_post_revisions.ts)

Reads historical edit logs from a Discourse CSV export and populates the postRevisions table.

Workflow: Converts Discourse YAML modifications into JSON and links them to their respective post IDs.
Recalculation: Adjusts version and publicVersion sequences on affected posts.

2. Topic recommendations & Embeddings

[embed-posts.ts](file:///f/murmur/scripts/embed-posts.ts)

Generates vector float embeddings for search calculations.

Workflow: Identifies posts that do not have embeddings in the post_embeddings table.
Generation: Calls the configured LLM API (e.g. Ollama, OpenAI) to generate vector embeddings.
Persistence: Writes vector embeddings to the database as binary buffers.

[generate-related-topics.ts](file:///f/murmur/scripts/generate-related-topics.ts)

Computes topic similarity recommendations.

Workflow: Calculates vector distances between topic posts.
Storage: Saves the top 50 matches for each topic as a JSON list in topic_recommendations.

3. Maintenance & Optimizations

[rebake-posts.ts](file:///f/murmur/scripts/rebake-posts.ts)

Re-processes post content using the Markdown compiler.

Usage: Run this script after modifying markdown parser rules, BBCode preprocessors, or HTML sanitization options to update all compiled posts.

[rehash-compress-replace.ts](file:///f/murmur/scripts/rehash-compress-replace.ts)

Compresses and optimizes uploaded assets.

Workflow: Compresses images on disk, updates their URLs, and updates file paths in raw and cooked post content.

[reindex.ts](file:///f/murmur/scripts/reindex.ts)

Rebuilds the virtual FTS5 search index.

Workflow: Wipes search_index and indexes all posts in the database.
CJK splitting: Uses Jieba to segment Chinese, Japanese, and Korean content.

⚡ Quick Start

1. Prerequisite Installations

Ensure you have Node.js 26+ and pnpm 11+ installed. If you plan to run the uploads proxy or semantic search, make sure cargo (Rust toolchain) and SQLite are available.

2. Development Setup

Clone the repository and install dependencies:

pnpm install\n

Configure your environment variables:

cp .env.example .env\n

Modify .env to define:

DATABASE_URL: Path to your database file (defaults to local.db).
ORIGIN: The server domain (e.g. http://localhost:5173).
GITHUB_CLIENT_ID / GITHUB_CLIENT_SECRET: OAuth credentials.

Initialize and seed the database with mock categories, users, and topics:

pnpm db:push\npnpm db:seed\n

Start the Vite development server:

pnpm dev\n

Open /login and use the Dev Login button to authenticate as any user without configuring GitHub OAuth.

📁 Repository Layout

src/routes/ - SvelteKit routing layer (pages, endpoints, SSE streams).
src/lib/ - Shared code, UI components, and state modules.
src/lib/server/ - Server-side modules (database pools, auth Oslo, markdown cooker, search caches).
scripts/ - Standalone tools (data imports, re-indexing, embedding pre-calculations).
uploads-proxy/ - Rust Axum asset cache proxy server.
doc/ - System documentation markdown files.

\n","createdAt":"2026-07-15T14:24:38.094Z","updatedAt":"2026-07-15T14:27:34.999Z","replyToPostNumber":null,"replyCount":1,"likeCount":3,"wordCount":4945,"liked":false,"bookmarked":false,"polls":[],"reactions":[{"reaction":"👍","count":5,"users":[{"id":2902,"username":"1313","name":"ᐕ)ﾉ"},{"id":5351,"username":"Xrystal-line","name":"羽流弦ノ迹"},{"id":5905,"username":"PipaQinse233","name":"PipaQinse233"},{"id":7364,"username":"青年杰出贡献上交大門人","name":"紫金港第一皮革<・)))><<"},{"id":7830,"username":"stateless_trash_can","name":"无状态垃圾桶"}],"hasReacted":false},{"reaction":"❤️","count":3,"users":[{"id":7830,"username":"stateless_trash_can","name":"无状态垃圾桶"},{"id":2872,"username":"芙蓉湖第一树莓果","name":"RaspberryPi"},{"id":2902,"username":"1313","name":"ᐕ)ﾉ"}],"hasReacted":false}],"wiki":false,"version":1,"publicVersion":1,"deletedAt":null,"isAnonymous":false,"user":{"id":4668,"username":"IO","name":null,"avatarUrl":"/uploads/image/86/f2/86f267c0aaa73835a999b7cc3a7e7a701605574eda9a2c2a56f5ddbbba64d1a9.avif","admin":true,"moderator":false,"signatureRaw":"","signatureCooked":null}},{"id":72587,"postNumber":2,"raw":"管理员辛苦了 :wave_gif:","cooked":"

管理员辛苦了 $\":wave_gif:\"$

\n","createdAt":"2026-07-15T15:45:52.840Z","updatedAt":"2026-07-15T15:45:52.840Z","replyToPostNumber":null,"replyCount":0,"likeCount":1,"wordCount":7,"liked":false,"bookmarked":false,"polls":[],"reactions":[{"reaction":"👍","count":1,"users":[{"id":2872,"username":"芙蓉湖第一树莓果","name":"RaspberryPi"}],"hasReacted":false},{"reaction":"plus1","count":1,"users":[{"id":7364,"username":"青年杰出贡献上交大門人","name":"紫金港第一皮革<・)))><<"}],"hasReacted":false},{"reaction":"❤️","count":1,"users":[{"id":2872,"username":"芙蓉湖第一树莓果","name":"RaspberryPi"}],"hasReacted":false}],"wiki":false,"version":1,"publicVersion":1,"deletedAt":null,"isAnonymous":false,"user":{"id":7830,"username":"stateless_trash_can","name":"无状态垃圾桶","avatarUrl":"/uploads/image/f4/69/f469375c27cd72c6970425e94b999be00fed8fe6234b977b14f5bdb65da24b4e.avif","admin":false,"moderator":false,"signatureRaw":"

","signatureCooked":"

"}},{"id":72596,"postNumber":3,"raw":"有好几天上不来门，是我网络问题吗:cat_mouse:","cooked":"

有好几天上不来门，是我网络问题吗 $\":cat_mouse:\"$

\n","createdAt":"2026-07-16T06:52:08.323Z","updatedAt":"2026-07-16T06:52:08.323Z","replyToPostNumber":1,"replyCount":1,"likeCount":0,"wordCount":17,"liked":false,"bookmarked":false,"polls":[],"reactions":[],"wiki":false,"version":1,"publicVersion":1,"deletedAt":null,"isAnonymous":false,"user":{"id":8181,"username":"Aicnal_Liueic","name":null,"avatarUrl":"https://d1ebt9pet72xei.cloudfront.net/user_avatar/id.discourse.com/aicnal_liueic/200/16809_2.png","admin":false,"moderator":false,"signatureRaw":null,"signatureCooked":null}},{"id":72597,"postNumber":4,"raw":"[xjtu.app](https://xjtu.app)和[xjtu.men](https://xjtu.men)域名换着用，我经常好几个节点都无法访问.app，但是.men正常。","cooked":"

xjtu.app和xjtu.men域名换着用，我经常好几个节点都无法访问.app，但是.men正常。

\n","createdAt":"2026-07-16T13:55:52.183Z","updatedAt":"2026-07-16T13:55:52.183Z","replyToPostNumber":3,"replyCount":0,"likeCount":1,"wordCount":29,"liked":false,"bookmarked":false,"polls":[],"reactions":[{"reaction":"👍","count":1,"users":[{"id":8181,"username":"Aicnal_Liueic","name":null}],"hasReacted":false},{"reaction":"❤️","count":1,"users":[{"id":5905,"username":"PipaQinse233","name":"PipaQinse233"}],"hasReacted":false}],"wiki":false,"version":1,"publicVersion":1,"deletedAt":null,"isAnonymous":false,"user":{"id":4668,"username":"IO","name":null,"avatarUrl":"/uploads/image/86/f2/86f267c0aaa73835a999b7cc3a7e7a701605574eda9a2c2a56f5ddbbba64d1a9.avif","admin":true,"moderator":false,"signatureRaw":"","signatureCooked":null}}],"hasMore":false,"lastReadPostNumber":1,"hasUrlPostNumber":true}

前言

设计理念

实现的功能

Murmur: Modern Discussion Forum

🛠️ The Tech Stack

🏛️ System Architecture Preview

📚 Documentation Index

⚙️ [System Architecture & Scaling](file:///f/murmur/doc/architecture.md)

Murmur: System Architecture

Technical Stack

Component Layout & Data Flow

1. Request Lifecycle

Node.js & Rust uploads-proxy Integration

Real-Time Engine (SSE Pub/Sub)

Horizontal Scaling Strategy

🗄️ [Data Model & Database Schema](file:///f/murmur/doc/data-model.md)

Murmur: Data Model & Database Schema

Data Layer Conventions

Database Tables

1. users

2. sessions & ssoSyncTickets

3. categories & categoryGroups

4. topics & posts

5. postRevisions

6. postEmbeddings & topicRecommendations

7. chatChannels, chatChannelMembers & chatMessages

Schema DDL & Indices

✍️ [Reply Composer, Drafts & Uploads](file:///f/murmur/doc/features/composer.md)

Composer UI, Attachments & Draft Syncing

Composer Layout & Controls

Resizing Features

Rich Formatting Toolbar & Emojis

1. Heading Selector (Tt)

2. Actions Selector (+)

设计理念

实现的功能

Murmur: Modern Discussion Forum

🛠️ The Tech Stack

🏛️ System Architecture Preview

📚 Documentation Index

⚙️ [System Architecture & Scaling](file:///f/murmur/doc/architecture.md)

Murmur: System Architecture

Technical Stack

Component Layout & Data Flow

1. Request Lifecycle

Node.js & Rust uploads-proxy Integration

Real-Time Engine (SSE Pub/Sub)

Horizontal Scaling Strategy

🗄️ [Data Model & Database Schema](file:///f/murmur/doc/data-model.md)

Murmur: Data Model & Database Schema

Data Layer Conventions

Database Tables

1. users

2. sessions & ssoSyncTickets

3. categories & categoryGroups

4. topics & posts

5. postRevisions

6. postEmbeddings & topicRecommendations

7. chatChannels, chatChannelMembers & chatMessages

Schema DDL & Indices

✍️ [Reply Composer, Drafts & Uploads](file:///f/murmur/doc/features/composer.md)

Composer UI, Attachments & Draft Syncing

Composer Layout & Controls

Resizing Features

Rich Formatting Toolbar & Emojis

1. Heading Selector (Tt)

2. Actions Selector (+)

3. Emoji Picker

Kate-Like Whole Line Editing

Draft Autosave & Database Synchronization

Clipboard Paste & Attachment Pipeline

Codeblocks

🔍 [Full-Text & Semantic Search (RAG)](file:///f/murmur/doc/features/search-and-rag.md)

Full-Text & Semantic Search (RAG)

1. Full-Text Search Index (FTS5)

CJK Query Segmentation (Jieba)

2. Semantic Search (sqlite-vec)

3. RAG Chatbot Integration

LLM Streaming client

SSE Stream Synthesis

1. `users`

2. `sessions` & `ssoSyncTickets`

3. `categories` & `categoryGroups`

4. `topics` & `posts`

5. `postRevisions`

6. `postEmbeddings` & `topicRecommendations`

7. `chatChannels`, `chatChannelMembers` & `chatMessages`

1. Heading Selector (`Tt`)

2. Actions Selector (`+`)

1. `users`

2. `sessions` & `ssoSyncTickets`

3. `categories` & `categoryGroups`

4. `topics` & `posts`

5. `postRevisions`

6. `postEmbeddings` & `topicRecommendations`

7. `chatChannels`, `chatChannelMembers` & `chatMessages`

1. Heading Selector (`Tt`)

2. Actions Selector (`+`)