Livrare LOT 1 - Didi
This commit is contained in:
commit
5380c3fc63
990 changed files with 133308 additions and 0 deletions
611
ai_platform/modules/didi_brain/AUDIT.md
Normal file
611
ai_platform/modules/didi_brain/AUDIT.md
Normal file
|
|
@ -0,0 +1,611 @@
|
|||
# AUDIT COMPLET — Atomic Knowledge Base
|
||||
|
||||
> Audit generat prin analiză paralelă cu 6 agenți specializați.
|
||||
> Versiune cod auditată: `1.19.2` (clone GitHub `kenforthewin/atomic`)
|
||||
> Data: 2026-04-10
|
||||
|
||||
---
|
||||
|
||||
## 0. CE ESTE ATOMIC (TL;DR)
|
||||
|
||||
**Atomic** este un **personal knowledge base** care transformă note Markdown ("atoms") într-un **graf semantic AI-augmented**. Fiecare notă este automat:
|
||||
|
||||
1. **Chunked** (bucățită markdown-aware)
|
||||
2. **Embedded** (vectorizată prin LLM)
|
||||
3. **Tagged** (taguri ierarhice extrase de LLM)
|
||||
4. **Linked** (edges semantice către alte atoms similare)
|
||||
|
||||
Pe baza acestui graf oferă: **căutare semantică**, **wiki articles auto-sintetizate cu citații**, **canvas vizual interactiv**, **chat agentic RAG**, **sincronizare offline**.
|
||||
|
||||
Rulează ca:
|
||||
- **App desktop Tauri** (macOS/Linux/Windows) — sidecar care pornește atomic-server local
|
||||
- **Server headless Docker/Fly.io** — REST + WebSocket + MCP
|
||||
- **iOS native SwiftUI** — client thin HTTP
|
||||
- **Browser extension** (Web Clipper) — capturi web
|
||||
- **Plugin Obsidian** — sincronizare vault
|
||||
- **Bot Discord** — capturi mesaje/threads
|
||||
- **MCP server** — expune knowledge base la Claude Desktop & alte AI tools
|
||||
|
||||
---
|
||||
|
||||
## 1. ARHITECTURA GENERALĂ — "Core + Thin Wrappers"
|
||||
|
||||
```
|
||||
┌────────────────────┐
|
||||
│ atomic-core │ ← TOATĂ logica de business
|
||||
│ (Rust crate, no │ (no actix/no tauri deps)
|
||||
│ framework deps) │
|
||||
└─────────┬──────────┘
|
||||
┌───────────────┼─────────────────┐
|
||||
▼ ▼ ▼
|
||||
┌────────────┐ ┌──────────────┐ ┌───────────────┐
|
||||
│ src-tauri │ │atomic-server │ │ mcp-bridge │
|
||||
│ (sidecar │ │(REST+WS+MCP) │ │ (stdio→HTTP │
|
||||
│ launcher) │ │ actix-web │ │ pentru Claude│
|
||||
└─────┬──────┘ └──────┬───────┘ │ Desktop) │
|
||||
│ │ └───────┬───────┘
|
||||
▼ ▼ ▼
|
||||
┌──────────────────────────────┐ ┌──────────────┐
|
||||
│ React UI (TypeScript+Vite) │ │ MCP clients │
|
||||
│ (folosește același │ │ (Claude etc) │
|
||||
│ HttpTransport în desktop │ └──────────────┘
|
||||
│ și browser) │
|
||||
└──────────────────────────────┘
|
||||
+
|
||||
┌──────────────┐ ┌─────────────┐ ┌─────────────┐
|
||||
│ iOS app │ │ Browser ext │ │ Discord bot │
|
||||
│ SwiftUI │ │ (MV3) │ │ Obsidian │
|
||||
└──────────────┘ └─────────────┘ └─────────────┘
|
||||
```
|
||||
|
||||
**Principiu cheie:** `atomic-core` este transport-agnostic. Toate evenimentele async sunt livrate prin **callback-uri** (`Fn(EmbeddingEvent)`, `Fn(ChatEvent)`). Fiecare wrapper traduce callback-uri în mecanismul lui (Tauri `app_handle.emit`, sau actix `broadcast::Sender → WebSocket`).
|
||||
|
||||
---
|
||||
|
||||
## 2. WORKSPACE — STRUCTURA REPO
|
||||
|
||||
```
|
||||
atomic/
|
||||
├── Cargo.toml # Workspace Rust (5 crate-uri)
|
||||
├── package.json # Frontend npm (versiune 1.19.2)
|
||||
├── crates/
|
||||
│ ├── atomic-core/ # ~16.4k linii Rust — toată logica
|
||||
│ ├── atomic-server/ # ~7.5k linii — actix-web wrapper
|
||||
│ ├── mcp-bridge/ # ~285 linii — bridge stdio↔HTTP MCP
|
||||
│ └── atomic-cloud/ # Control plane SaaS (Stripe + Fly.io)
|
||||
├── src-tauri/ # Tauri v2 desktop launcher
|
||||
├── src/ # React 18 + TS + Tailwind v4 + Zustand
|
||||
├── ios/ # SwiftUI + XcodeGen (Swift 6, iOS 17+)
|
||||
├── extension/ # Manifest V3 Web Clipper
|
||||
├── plugins/
|
||||
│ ├── discord/ # discord.js 14 bot
|
||||
│ └── obsidian-plugin/ # Obsidian plugin (TS)
|
||||
├── scripts/ # 15 scripturi Node (build, import, reset)
|
||||
├── docker/ # nginx, supervisord, litestream
|
||||
├── docs/ # 5 spec markdown (planuri arhitecturale)
|
||||
├── docker-compose.{yml,dev,test,build}.yml
|
||||
├── Dockerfile # Multi-stage: server, web, all-in-one
|
||||
├── server.dockerfile / web.dockerfile
|
||||
└── fly.toml.example
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. `atomic-core` — INIMA SISTEMULUI
|
||||
|
||||
### 3.1 Module principale
|
||||
|
||||
| Modul | Linii | Rol |
|
||||
|---|---:|---|
|
||||
| `lib.rs` | 3,399 | Facade `AtomicCore`, orchestrare toate operațiile |
|
||||
| `db.rs` | 916 | Init SQLite, migrații v0→v10, PRAGMA tuning, schema |
|
||||
| `manager.rs` | 511 | Multi-database manager (lazy-load instance per DB) |
|
||||
| `models.rs` | 696 | `Atom`, `Tag`, `WikiArticle`, `ChatMessage`, `SemanticEdge` etc. |
|
||||
| `registry.rs` | 863 | Multi-database registry (cross-DB settings/tokens) |
|
||||
| `tokens.rs` | 430 | API tokens (SHA-256 hash + revocable) |
|
||||
| `settings.rs` | 191 | Key-value settings (provider, modele, thresholds) |
|
||||
| `executor.rs` | 39 | Background runtime (4 worker threads) + semafoare concurrență |
|
||||
| `projection.rs` | 287 | PCA/t-SNE pentru canvas (cache poziții) |
|
||||
| `error.rs` | 70 | `AtomicCoreError` enum |
|
||||
|
||||
### 3.2 Pipeline de procesare atom (fire-and-forget)
|
||||
|
||||
```
|
||||
POST /api/atoms (sau core.create_atom)
|
||||
│
|
||||
▼ caller primește atom-ul instant (saved)
|
||||
│
|
||||
│ background, prin executor + callback Fn(EmbeddingEvent):
|
||||
│
|
||||
├─[1]─► CHUNKING (markdown-aware)
|
||||
│ respectă code blocks, headers, paragraphs
|
||||
│ token-aware via tiktoken-rs
|
||||
│
|
||||
├─[2]─► EMBEDDING (provider configurat)
|
||||
│ OpenRouter / Ollama / OpenAI-compat
|
||||
│ inserează în vec_chunks (sqlite-vec virtual table)
|
||||
│ → emit EmbeddingEvent::Started/Complete/Failed
|
||||
│
|
||||
├─[3]─► AUTO-TAGGING (LLM structured outputs)
|
||||
│ categorii: Topics, People, Locations,
|
||||
│ Organizations, Events
|
||||
│ → emit TaggingComplete/Failed/Skipped
|
||||
│
|
||||
├─[4]─► SEMANTIC EDGES (cosine similarity)
|
||||
│ threshold: 0.5 default → tabela semantic_edges
|
||||
│
|
||||
└─[5]─► WIKI INCREMENTAL UPDATE (dacă tagged)
|
||||
LLM integrează atom-ul nou în wiki article-ul existent
|
||||
```
|
||||
|
||||
**Praguri (thresholds):**
|
||||
- Similaritate edges/related atoms: **0.5**
|
||||
- Semantic search & wiki chunk selection: **0.3**
|
||||
- Formula: `similarity = 1.0 - (distance² / 2.0)` (din Euclidean al sqlite-vec pe vectori normalizați)
|
||||
|
||||
### 3.3 Storage abstraction (SQLite + Postgres)
|
||||
|
||||
`atomic-core` are un trait `StorageBackend` cu **două implementări**:
|
||||
- **SQLite** (`storage/sqlite/`) — default, prin `rusqlite 0.32` (bundled) + `sqlite-vec 0.1.6` (vector search)
|
||||
- **Postgres** (`storage/postgres/`) — feature-gated, prin `sqlx 0.8` + `pgvector 0.4`
|
||||
|
||||
Dispatch macro `dispatch!` rulează 111 metode peste backend-uri. Există un plan documentat (`docs/plan-async-migration.md`) pentru a face întregul `AtomicCore` async-native (acum Postgres face sync→async bridge prin `PG_RUNTIME.block_on`).
|
||||
|
||||
### 3.4 Schema bază date (data DB-uri)
|
||||
|
||||
Tabele cheie:
|
||||
- `atoms` — content, source_url, embedding_status, tagging_status, created/updated_at
|
||||
- `atom_chunks` — chunks per atom (cu offset-uri în text)
|
||||
- `vec_chunks` — **virtual table sqlite-vec** (vector index)
|
||||
- `tags` — ierarhie (parent_id), category
|
||||
- `atom_tags` — many-to-many
|
||||
- `semantic_edges` — pereche atom_a, atom_b, similarity
|
||||
- `atom_clusters` — rezultate clustering
|
||||
- `atom_positions` — poziții persistate canvas
|
||||
- `wiki_articles` — content + metadata per tag
|
||||
- `wiki_proposals` — (M1+) human-in-the-loop updates
|
||||
- `conversations`, `chat_messages` — chat agentic
|
||||
- FTS virtual table pentru keyword search
|
||||
|
||||
`registry.db` (separat): `settings`, `api_tokens`, `databases` (UUIDs + nume).
|
||||
|
||||
### 3.5 AI Provider abstraction
|
||||
|
||||
Trait-uri în `providers/traits.rs`:
|
||||
- `EmbeddingProvider` — generare batch embeddings
|
||||
- `LlmProvider` — chat completions
|
||||
- `StreamingLlmProvider` — streaming + tool calling
|
||||
|
||||
Implementări:
|
||||
- `providers/openrouter/` — cloud, OAuth flow, modele separate per capability (embedding/tagging/wiki/chat)
|
||||
- `providers/ollama/` — local, **auto-discovery modele**
|
||||
- `providers/openai_compat/` — generic (Azure OpenAI, Groq, Together, etc.)
|
||||
|
||||
Selecția runtime: factory întoarce `Arc<dyn Trait>` în funcție de setting `provider_type`.
|
||||
|
||||
### 3.6 Wiki synthesis
|
||||
|
||||
- **Generation**: LLM produce articol din atomii unui tag, cu citații inline (`[atom_id]`)
|
||||
- **Centroid mode** (`wiki/centroid.rs`): selecție chunks după centroid embedding al tagului
|
||||
- **Agentic mode** (`wiki/agentic.rs`): agent care pune query-uri pe baza de cunoștințe
|
||||
- **Section operations** (`wiki/section_ops.rs`): `WikiSectionOp::{NoChange, AppendToSection, ReplaceSection, InsertSection}` — LLM emite *operații* peste secțiuni, **nu rewrite total** → diff-uri review-abile (M1 wiki proposals)
|
||||
- **Versioning**: tabela `wiki_versions` păstrează istoric
|
||||
|
||||
### 3.7 Chat agentic / RAG
|
||||
|
||||
- Conversații pot fi **scoped la tag-uri**
|
||||
- Agent are tool-uri: search semantic, read atom, citation
|
||||
- Streaming via `ChatEvent::{Delta, ToolStart, ToolComplete, Complete, CanvasAction, Error}`
|
||||
- `ChatCanvasAction` permite agentului să declanșeze acțiuni vizuale în UI
|
||||
|
||||
### 3.8 Ingestion URL & RSS & Obsidian
|
||||
|
||||
- `ingest/fetch.rs` — HTTP GET cu reqwest
|
||||
- `ingest/extract.rs` — HTML → Markdown via `dom_smoothie 0.15`
|
||||
- `ingest/obsidian.rs` — vault Obsidian, păstrează `[[wikilinks]]`, extrage tags din foldere + YAML frontmatter
|
||||
- RSS — `feed-rs 2.3`, polling configurabil per feed (default 60s)
|
||||
|
||||
---
|
||||
|
||||
## 4. `atomic-server` — REST + WebSocket + MCP
|
||||
|
||||
### 4.1 Stack
|
||||
- **actix-web 4.9** + actix-cors + actix-ws
|
||||
- **rmcp 0.15** + rmcp-actix-web 0.11 (MCP Streamable HTTP)
|
||||
- **utoipa 5** + utoipa-scalar (OpenAPI auto-generat la `/api/docs`)
|
||||
- **clap 4** (CLI cu subcomenzi)
|
||||
- **tokio broadcast channel** (256 buffer) pentru events
|
||||
|
||||
### 4.2 ~78 Endpoints REST (grupate)
|
||||
|
||||
| Domeniu | # | Exemple |
|
||||
|---|---:|---|
|
||||
| **Atoms** | 10 | `GET/POST/PUT/DELETE /api/atoms`, `/api/atoms/bulk`, `/api/atoms/{id}/embedding-status`, `/api/atoms/by-source-url`, `/api/atoms/sources` |
|
||||
| **Tags** | 5 | `GET/POST/PUT/DELETE /api/tags`, `/api/tags/{id}/children` |
|
||||
| **Search** | 2 | `POST /api/search` (modes: keyword/semantic/hybrid), `GET /api/atoms/{id}/similar` |
|
||||
| **Wiki** | 13 | `/api/wiki/{tag_id}` cu generate, update, versions, suggestions, proposal/{accept,dismiss}, related, links |
|
||||
| **Embeddings pipeline** | 8 | process-pending, process-tagging, retry/{atom_id}, reembed-all, reset-stuck, status |
|
||||
| **Canvas** | 5 | positions GET/PUT, atoms-with-embeddings, level, global (PCA proj) |
|
||||
| **Graph** | 3 | edges, neighborhood/{atom_id}, rebuild-edges |
|
||||
| **Clustering** | 3 | compute, get clusters, connection-counts |
|
||||
| **Chat** | 8 | conversations CRUD, scope add/remove tags, send-message |
|
||||
| **Settings** | 5 | get/set, test-openrouter, test-openai-compat, models, embedding-models |
|
||||
| **Databases** | 7 | list, create, rename, delete, activate, set-default, stats |
|
||||
| **Feeds (RSS)** | 6 | list/create/get/update/delete, poll |
|
||||
| **Ingest/Import** | 3 | ingest URL (single/batch), import Obsidian vault |
|
||||
| **Ollama** | 5 | test, models (all/embedding/llm), provider verify |
|
||||
| **Auth tokens** | 3 | create, list, revoke |
|
||||
| **Setup** | 2 | status, claim instance |
|
||||
| **OAuth 2.0 (MCP)** | 7 | `.well-known/*`, `/oauth/{register,authorize,token}` cu PKCE + DCR |
|
||||
| **Utils & logs** | 3 | sqlite-vec check, compact-tags, logs export |
|
||||
| **Public** | 2 | `/health`, `/api/docs/openapi.json` |
|
||||
|
||||
### 4.3 WebSocket
|
||||
|
||||
- Endpoint: `GET /ws?token=<api_token>`
|
||||
- Subscribe la `tokio::sync::broadcast::Sender<ServerEvent>` (buffer 256)
|
||||
- `ServerEvent` enum (~25 variante) include:
|
||||
- **Embedding pipeline**: `EmbeddingStarted/Complete/Failed`, `TaggingComplete/Failed/Skipped`, `BatchProgress`
|
||||
- **Atom lifecycle**: `AtomCreated`
|
||||
- **Ingestion**: `IngestionFetchStarted/Complete/Failed/Skipped`, `IngestionComplete/Failed`
|
||||
- **Feeds**: `FeedPollComplete/Failed`
|
||||
- **Chat streaming**: `ChatStreamDelta`, `ChatToolStart/Complete`, `ChatComplete`, `ChatCanvasAction`, `ChatError`
|
||||
- **Import**: `ImportProgress`
|
||||
|
||||
### 4.4 MCP endpoint `/mcp`
|
||||
|
||||
- **Transport**: Streamable HTTP (stateful, SSE keep-alive 30s)
|
||||
- **Auth**: `McpAuth` middleware → 401 cu `WWW-Authenticate` care indică spre `/.well-known/oauth-protected-resource` (Claude.ai compatible)
|
||||
- **Multi-DB**: `?db=<uuid>` în query
|
||||
- **Tools expuse:**
|
||||
1. `semantic_search(query, limit)` — hybrid search
|
||||
2. `read_atom(atom_id, limit, offset)` — paginated 500 lines
|
||||
3. `create_atom(content, source_url)` — broadcastează `AtomCreated`
|
||||
4. `update_atom(atom_id, content, source_url)`
|
||||
|
||||
### 4.5 Auth
|
||||
|
||||
- **API tokens**: SHA-256 hash în DB, prefix de 8 chars pentru lookup rapid
|
||||
- `BearerAuth` middleware pe `/api/*`, `McpAuth` pe `/mcp`
|
||||
- `last_used_at` updated fire-and-forget
|
||||
- **OAuth 2.0 + PKCE + DCR** (1139 linii în `routes/oauth.rs`):
|
||||
- Dynamic Client Registration → genere client_secret (32 bytes random, base64url)
|
||||
- Authorization endpoint cu consent HTML
|
||||
- Code → 5 min expiration, hash storage
|
||||
- Token exchange cu PKCE S256 verification
|
||||
- Folosit pentru Claude.ai remote MCP
|
||||
|
||||
### 4.6 CLI
|
||||
```bash
|
||||
atomic-server [--data-dir PATH] serve --port 8080 --bind 127.0.0.1 \
|
||||
--public-url https://... --storage sqlite|postgres
|
||||
atomic-server token create --name "..."
|
||||
atomic-server token list
|
||||
atomic-server token revoke <id>
|
||||
```
|
||||
|
||||
### 4.7 Startup behavior
|
||||
1. Init logging (tracing + ring buffer 1000 entries pentru `/api/logs`)
|
||||
2. `DatabaseManager::new()` (SQLite sau Postgres)
|
||||
3. Migrate legacy tokens
|
||||
4. Create broadcast channel(256)
|
||||
5. **Recovery**: reset atoms blocate în `processing`, process pending embeddings/tagging
|
||||
6. **Spawn RSS poll loop** (60s tick, all DBs)
|
||||
7. Bind HTTP server (4 workers), CORS permissive
|
||||
8. Graceful shutdown → `PRAGMA optimize`
|
||||
|
||||
---
|
||||
|
||||
## 5. `mcp-bridge` — stdio↔HTTP
|
||||
|
||||
Binary mic (~285 linii) care se compilează cross-platform și e embedded ca **sidecar Tauri** + distribuit pentru Claude Desktop.
|
||||
|
||||
**Flow:**
|
||||
1. Citește JSON-RPC din **stdin**
|
||||
2. POST la `http://127.0.0.1:44380/mcp` cu header `Mcp-Protocol-Version: 2025-03-26`
|
||||
3. Captează `mcp-session-id` din răspunsul `initialize`, îl folosește pe requesturile următoare
|
||||
4. Parsează SSE (`text/event-stream`) și emite linii data: ca JSON-RPC pe **stdout**
|
||||
5. Timeout HTTP: **300s** (operații AI lungi)
|
||||
|
||||
Env vars: `ATOMIC_HOST` (127.0.0.1), `ATOMIC_PORT` (44380).
|
||||
|
||||
Config Claude Desktop:
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"atomic": { "url": "http://localhost:44380/mcp" }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. `atomic-cloud` — Control Plane SaaS
|
||||
|
||||
Aplicație **Actix separată** care gestionează hosting managed pe Fly.io.
|
||||
|
||||
**Module:**
|
||||
- `clients/fly.rs` — Fly Machines API (create_app, allocate_ips, machines, volumes, delete_app)
|
||||
- `clients/stripe.rs` — Checkout sessions + webhook HMAC verify
|
||||
- `clients/mailgun.rs` — magic link delivery
|
||||
- `routes/checkout.rs` — POST /api/checkout creează Stripe session + Fly app
|
||||
- `routes/webhooks.rs` — `customer.subscription.{created,updated,deleted}`
|
||||
- `routes/instances.rs` — start/stop/restart/billing_portal (auth via management_token)
|
||||
- `routes/admin.rs` — list instances, stats (MRR estimate), rollout image
|
||||
- `routes/auth.rs` — magic link send/verify
|
||||
- `jobs.rs` — **cleanup background**: după 30 zile cancel → `fly.delete_app()` (machines + volumes + IPs)
|
||||
|
||||
**Modele Postgres:** `Customer`, `Subscription`, `Instance`, `Event`.
|
||||
|
||||
**Env vars necesare:** `DATABASE_URL`, `STRIPE_SECRET_KEY`, `STRIPE_WEBHOOK_SECRET`, `STRIPE_PRICE_ID`, `FLY_API_TOKEN`, `FLY_ORG`, `FLY_REGION`, `BASE_DOMAIN`, `MAILGUN_API_KEY/DOMAIN/FROM`, `ADMIN_API_KEY`, `PUBLIC_URL`.
|
||||
|
||||
---
|
||||
|
||||
## 7. `src-tauri` — Desktop wrapper
|
||||
|
||||
- **Tauri v2** + plugins: `opener`, `dialog`, `shell`, `fs`
|
||||
- Pornește `atomic-server` ca **sidecar process** (binar pre-built în `src-tauri/binaries/{target_triple}/`)
|
||||
- Expune o singură comandă IPC: `get_local_server_config()` → `{ url, token }`
|
||||
- Frontend folosește apoi același `HttpTransport` ca în browser → conectează la sidecar
|
||||
- Pe exit, Tauri kill-uiește sidecar-ul
|
||||
|
||||
**Database location** auto:
|
||||
- macOS: `~/Library/Application Support/com.atomic.app/`
|
||||
- Linux: `~/.local/share/com.atomic.app/`
|
||||
- Windows: `%APPDATA%/com.atomic.app/`
|
||||
|
||||
---
|
||||
|
||||
## 8. Frontend React (`src/`)
|
||||
|
||||
### 8.1 Stack
|
||||
- **React 18** + TypeScript strict
|
||||
- **Vite 6** + Tailwind CSS v4 (`@tailwindcss/vite` + `@tailwindcss/typography`)
|
||||
- **Zustand 5** (state management modular)
|
||||
- **CodeMirror 6** (`@uiw/react-codemirror`, lang-markdown, theme-one-dark)
|
||||
- **react-markdown** + remark-gfm
|
||||
- **Sigma.js v3** + `@sigma/edge-curve` + **Graphology** (canvas graph)
|
||||
- **d3-force v3** (physics simulation)
|
||||
- **react-zoom-pan-pinch**, `@tanstack/react-virtual`
|
||||
- **sonner** (toasts), **diff** (diff view), **qrcode**
|
||||
|
||||
### 8.2 Stores Zustand (`src/stores/`)
|
||||
`atoms`, `tags`, `ui`, `settings`, `wiki`, `chat`, `databases`, `canvas`, `embedding-progress`.
|
||||
|
||||
`ui` store: tag filter selectat, drawer state, view mode (canvas/grid/list — persistat în localStorage), search query.
|
||||
|
||||
### 8.3 Componente principale (`src/components/`)
|
||||
- `atoms/` — `AtomCard`, `AtomGrid`, `AtomList`, `AtomViewer` (markdown render + search), `AtomReader`, `RelatedAtoms`, `FilterBar`
|
||||
- `canvas/` — `CanvasView`, `SigmaCanvas`, `HierarchicalCanvas`, `LocalGraphView`, `MiniGraphPreview`, `ClusterBubble`, `ClusterVisualization`, `ConnectionLines`, `CanvasControls`, `CanvasBreadcrumb`, `useForceSimulation` hooks
|
||||
- `wiki/` — `WikiArticleContent`, `CitationPopover`, proposal diff view
|
||||
- `chat/` — `ChatMessage`, conversations list
|
||||
- `onboarding/` — wizard cu `AIProviderStep`, OpenRouter OAuth callback
|
||||
- `settings/`, `tags/`, `search/`
|
||||
|
||||
### 8.4 Transport abstraction
|
||||
Interfața `Transport { invoke(name, args), subscribe(event, handler) }`.
|
||||
|
||||
Singura implementare: `HttpTransport` (folosită și în desktop, și în browser):
|
||||
- Mapează nume comandă → HTTP spec (method, path, body/query transform) printr-un **command map**
|
||||
- WebSocket pentru events
|
||||
- În Tauri: înainte cheamă `get_local_server_config` via Tauri IPC → primește URL+token sidecar, apoi totul e HTTP
|
||||
|
||||
Codul React e **transport-unaware**.
|
||||
|
||||
### 8.5 Vite config (`vite.config.ts`)
|
||||
- Desktop: fără proxy, stub-uri Tauri în `src/lib/stubs/tauri-*.ts`
|
||||
- Web (`VITE_BUILD_TARGET=web`): proxy `/api`, `/health`, `/ws` → `http://127.0.0.1:8080`
|
||||
|
||||
### 8.6 Design system
|
||||
Dark theme inspirat Obsidian:
|
||||
- Backgrounds: `#1e1e1e` / `#252525` / `#2d2d2d`
|
||||
- Accent purple: `#7c3aed`
|
||||
- Text: white / `#8c8c8c`
|
||||
- 3-panel layout: tag tree stânga (fix) | main view (canvas/grid/list) | right drawer (editor/viewer/wiki/chat)
|
||||
|
||||
---
|
||||
|
||||
## 9. iOS App (`ios/`)
|
||||
|
||||
- **SwiftUI** + Swift 6 strict concurrency, iOS 17.0+
|
||||
- **XcodeGen** (`project.yml` → `.xcodeproj` regenerat)
|
||||
- **Bundle**: `com.atomic.mobile` + extension `com.atomic.mobile.share` (Share Extension)
|
||||
- **App Group**: `group.com.atomic.mobile` (shared credentials)
|
||||
- **Dependencies** (SPM): `MarkdownUI 2.4`, `Runestone 0.5`, `TreeSitterLanguages`
|
||||
|
||||
**Fișiere cheie:**
|
||||
- `AtomicApp.swift` — entry point, QR scanner setup
|
||||
- `APIClient.swift` — `@Observable` HTTP client (Bearer + opțional `X-Atomic-Database`)
|
||||
- `AtomStore.swift` — state cu DiskCache (atoms, tags) + OfflineQueue
|
||||
- `Models.swift` — Codable: `Atom`, `AtomSummary`, `SearchResult`, `TagWithCount`, `DatabaseInfo`
|
||||
- `OfflineQueue.swift` — `Documents/pending_atoms.json`
|
||||
- `SharedConfig.swift` — UserDefaults suite pentru extensie
|
||||
- `Theme.swift` — culori match cu desktop
|
||||
|
||||
**Endpoint-uri folosite:** `/api/atoms` (CRUD + sources), `/api/tags`, `/api/search`, `/api/databases/{activate}`.
|
||||
|
||||
---
|
||||
|
||||
## 10. Browser Extension (`extension/`)
|
||||
|
||||
- **Manifest V3** (Chrome/Edge/Brave)
|
||||
- **Permisiuni**: activeTab, scripting, contextMenus, storage, notifications, alarms
|
||||
|
||||
**Structură:**
|
||||
- `background/service-worker.js` — capture flow, queue offline, sync alarm 30s, badge status
|
||||
- `content/content-script.js` — extragere via **Readability** (Mozilla) + **Turndown** (HTML→Markdown), suportă full page și selection
|
||||
- `popup/` — toolbar UI cu Capture Page / Selection / Sync Now
|
||||
- `options/` — server URL + API token + Test Connection
|
||||
- `lib/config.js` — `chrome.storage.local` cu cheia `serverConfig`
|
||||
|
||||
**Endpoint-uri:** `POST /api/atoms`, `GET /health`, `GET /api/atoms?limit=1` (test).
|
||||
|
||||
**Offline queue** în `chrome.storage.local`, drain la fiecare alarm tick (30s).
|
||||
|
||||
---
|
||||
|
||||
## 11. Plugins externe
|
||||
|
||||
### 11.1 Discord bot (`plugins/discord/`)
|
||||
- **discord.js 14.16** + **better-sqlite3** + **yaml**
|
||||
- Slash commands: `/atomic-subscribe`, `/atomic-unsubscribe`, `/atomic-config`, `/atomic-save`, `/atomic-search`, `/atomic-status`
|
||||
- **Reaction-based capture**: emoji custom (sau fallback Unicode) pe orice mesaj → ingestie
|
||||
- **Settle window** (default 300s): debounce per channel pentru thread-uri active (resetat la fiecare nou mesaj)
|
||||
- Suportă **Forum channels** + **Voice/Stage** + **Threads** (cu fetch full history)
|
||||
- **NormalizedMessage** structure unifică text/forum/voice/dm
|
||||
- Local SQLite tables: `dedup_index` (guild:channel:message → atom_id), `channel_configs`
|
||||
- Templates de formatting per tip (single message / thread / forum post)
|
||||
- Tag resolution cu cache + auto-create pentru taguri ierarhice (`discord/<channel>`)
|
||||
|
||||
### 11.2 Obsidian plugin (`plugins/obsidian-plugin/`)
|
||||
- **TypeScript** + Obsidian API (`requestUrl`)
|
||||
- Comenzi: `semantic-search`, `sync-current-note`, `sync-vault`, `toggle-auto-sync`, `open-similar-notes`, `open-wiki`
|
||||
- 2 sidebar views: **SimilarView** (related notes), **WikiView** (browse wiki articles)
|
||||
- **Sync engine**:
|
||||
- File watcher cu debounce configurabil (default 2s)
|
||||
- Hash SHA-256 pe content → skip dacă neschimbat
|
||||
- Source URL: `obsidian://VaultName/path/to/note.md`
|
||||
- Handle: create/modify/delete/rename
|
||||
- State în `plugin-data.json`: `Map<path, {atomId, contentHash, lastSynced}>`
|
||||
- Settings: server URL, token, vault name, auto-sync, debounce, folder→tags, delete-on-remove, exclude patterns
|
||||
|
||||
---
|
||||
|
||||
## 12. Scripturi Node (`scripts/`)
|
||||
|
||||
| Script | Rol |
|
||||
|---|---|
|
||||
| `dev-server.js` | Pornește atomic-server + Vite simultan; opțional `--postgres` (docker pgvector) |
|
||||
| `build-server.js` | Compile `atomic-server` → `src-tauri/binaries/{target}/` |
|
||||
| `build-mcp-bridge.js` | Compile `mcp-bridge` → același folder pentru sidecar Tauri |
|
||||
| `build-release.js` | Bump versiune + git tag + tauri build cross-platform + GitHub upload |
|
||||
| `import/obsidian.js` | Import vault Obsidian (folders→tags, YAML frontmatter, dedup, dry-run) |
|
||||
| `import-rss.js` | Import feed RSS (turndown HTML→MD) |
|
||||
| `import-wikipedia.js` | Crawl Wikipedia (BFS din 3 domenii: Computing/Philosophy/History), 100ms rate limit |
|
||||
| `stress-test-wikipedia.js` | Volume test 1000+ articole |
|
||||
| `stress-test-summaries.js` | Lightweight: Wikipedia summary endpoint, concurrency 10 |
|
||||
| `reset-database.js` | Drop completă a DB |
|
||||
| `reset-tags.js` | Reset tags + remark atoms pentru re-tagging |
|
||||
| `reset-chunks.js` | Șterge chunks/embeddings/edges/positions, păstrează atoms |
|
||||
| `drop-database.js` | Delete persistent (registry + databases) |
|
||||
| `open-in-db-browser.sh` | Quick sqlite3 inspection |
|
||||
|
||||
---
|
||||
|
||||
## 13. Docker & Deployment
|
||||
|
||||
### 13.1 Dockerfile multi-stage
|
||||
1. **planner** — `cargo-chef prepare` (cache deps)
|
||||
2. **rust-builder** — mold linker, `cargo chef cook`, build `atomic-server` cu profile `server`
|
||||
3. **frontend-builder** — `npm ci` + `vite build` web
|
||||
4. **server** target — `debian:bookworm-slim`, EXPOSE 8080, ENTRYPOINT atomic-server
|
||||
5. **web** target — `nginx:1.28`, copy dist-web
|
||||
6. **all-in-one** target — supervisord rulează atomic-server (127.0.0.1:8080) + nginx (8081), VOLUME /data — folosit pentru Fly.io single-machine
|
||||
|
||||
### 13.2 docker-compose.yml (production)
|
||||
Servicii:
|
||||
- `server` (ghcr.io/kenforthewin/atomic-server)
|
||||
- `web` (ghcr.io/kenforthewin/atomic-web)
|
||||
- `proxy` (nginx :8080→80) cu config în `docker/nginx.conf`
|
||||
- `litestream` (profile backup) — replicare DB → S3 (sync 10s, snapshot 1h)
|
||||
- volume `atomic-data:/data`
|
||||
|
||||
### 13.3 nginx config
|
||||
- `/api/` → server (proxy_buffering off pentru SSE, read_timeout 300s)
|
||||
- `/ws` → server (Upgrade headers, 86400s timeout)
|
||||
- `/.well-known/`, `/oauth/`, `/mcp` → server
|
||||
- `/` → web frontend
|
||||
- `/assets/` → cache 1y immutable
|
||||
- SPA fallback `try_files $uri /index.html`
|
||||
|
||||
### 13.4 docker-compose.dev.yml
|
||||
- `pgvector/pgvector:pg16` pe portul 5434, healthcheck `pg_isready`
|
||||
- Folosit de `npm run dev:server:pg` și `npm run db:reset:pg`
|
||||
|
||||
### 13.5 Fly.io (`fly.toml.example`)
|
||||
- Target build: `all-in-one`
|
||||
- Mount volume `atomic_data → /data`
|
||||
- Internal port 8081 (nginx)
|
||||
- Auto-stop suspend, min 0 machines
|
||||
- Health check `/health` 30s
|
||||
- VM `shared-cpu-1x` 512MB
|
||||
|
||||
---
|
||||
|
||||
## 14. Documentația din `docs/`
|
||||
|
||||
| Fișier | Subiect |
|
||||
|---|---|
|
||||
| `foreign-keys.md` | Plan de a activa `PRAGMA foreign_keys` (acum off). 6 probleme + 6 pași: virtual table cleanup, tranzacții, validare tag IDs, stale positions, wiki migration on tag merge. |
|
||||
| `llm-wiki-gist-analysis.md` | Comparație cu gist-ul Karpathy "LLM Wiki". Top idee nouă: **Lint pass** (flag contradictions, orphan pages) — distinctive feature. |
|
||||
| `plan-async-migration.md` | Plan în 6 pași pentru a face `AtomicCore` async-native (eliminarea sync→async bridge la Postgres prin `PG_RUNTIME.block_on`). |
|
||||
| `url-ingestion-improvements.md` | Roadmap inspirat din Obsidian Web Clipper / Defuddle: wire metadata în columns, auto published_at, site-specific extractors (AI chats prioritate), MathML/footnotes/callouts. |
|
||||
| `wiki-proposal-loop-plan.md` | Spec arhitectural M1+M2+M3 pentru **wiki updates ca propuneri review-abile** în loc de mutații directe. Tabela `wiki_proposals`, `WikiSectionOp`, dirty set, quiet window, supersede budget, daily caps. |
|
||||
|
||||
---
|
||||
|
||||
## 15. Tech stack — TABEL FINAL
|
||||
|
||||
| Layer | Tehnologii |
|
||||
|---|---|
|
||||
| **Core (Rust)** | rusqlite 0.32 (bundled) + sqlite-vec 0.1.6, sqlx 0.8 + pgvector 0.4, tokio 1, reqwest 0.12, tiktoken-rs 0.6, pulldown-cmark 0.12, dom_smoothie 0.15, feed-rs 2.3, sha2, uuid, chrono, tracing |
|
||||
| **Server** | actix-web 4.9, actix-cors, actix-ws, rmcp 0.15 (MCP Streamable HTTP), utoipa 5 + scalar, clap 4 |
|
||||
| **Desktop** | Tauri v2 + plugins (opener, dialog, shell, fs) |
|
||||
| **Frontend** | React 18, TypeScript strict, Vite 6, Tailwind v4, Zustand 5, CodeMirror 6, react-markdown + remark-gfm, Sigma.js 3 + Graphology, d3-force 3, react-zoom-pan-pinch, @tanstack/react-virtual, sonner, diff, qrcode |
|
||||
| **iOS** | SwiftUI, Swift 6, iOS 17+, MarkdownUI, Runestone, TreeSitterLanguages, XcodeGen |
|
||||
| **Extension** | Manifest V3, Mozilla Readability, Turndown |
|
||||
| **Discord** | discord.js 14.16, better-sqlite3, yaml |
|
||||
| **Obsidian** | Obsidian API, Web Crypto SHA-256 |
|
||||
| **Cloud (SaaS)** | actix-web, sqlx postgres, Stripe API, Fly Machines API, Mailgun, hmac+sha2 (webhook verify) |
|
||||
| **Containere** | Docker multi-stage (cargo-chef + mold), nginx, supervisord, litestream backup → S3, pgvector/pgvector:pg16 dev |
|
||||
| **Deploy** | Fly.io single-machine (volume), Docker Compose self-host, GHCR images |
|
||||
| **AI providers** | OpenRouter (cloud, OAuth), Ollama (local, auto-discover), OpenAI-compatible (Azure/Groq/Together) |
|
||||
| **MCP** | Streamable HTTP server + stdio bridge pentru Claude Desktop, OAuth 2.0 + PKCE + DCR pentru Claude.ai remote |
|
||||
|
||||
---
|
||||
|
||||
## 16. Integrări externe (efective)
|
||||
|
||||
| Integrare | Cum |
|
||||
|---|---|
|
||||
| **OpenRouter** | OAuth callback `public/oauth/openrouter-callback.html`, modele separate per capability, settings persistate |
|
||||
| **Ollama** | Auto-discovery via `/api/ollama/models` |
|
||||
| **OpenAI-compatible** | Base URL + API key generic |
|
||||
| **Stripe** | Cloud only — checkout sessions + webhook signed (HMAC-SHA256) |
|
||||
| **Fly.io Machines API** | Cloud only — provision/start/stop/destroy machines + volumes + IPs |
|
||||
| **Mailgun** | Cloud only — magic link auth |
|
||||
| **Claude Desktop / Claude.ai** | MCP `/mcp` (Streamable HTTP) + `mcp-bridge` (stdio) + OAuth 2.0 PKCE pentru remote |
|
||||
| **GitHub Releases** | Auto-upload via `build-release.js` |
|
||||
| **Wikipedia REST API** | Doar import scripts |
|
||||
| **Obsidian** | Plugin oficial + import script CLI |
|
||||
| **Discord API** | discord.js gateway + REST |
|
||||
| **S3 / R2** | Litestream backup opțional |
|
||||
|
||||
---
|
||||
|
||||
## 17. Tehnical debt & roadmap (din docs)
|
||||
|
||||
1. **FK constraints OFF** → plan în `foreign-keys.md` (cleanup virtual tables, tranzacții, validare)
|
||||
2. **AtomicCore sync→async bridge la Postgres** → plan în `plan-async-migration.md` (6 pași)
|
||||
3. **Wiki proposals M1/M2/M3** → spec în `wiki-proposal-loop-plan.md` (M1 manual ready, M2 background scheduler planificat)
|
||||
4. **URL ingestion** → metadata wiring, published_at extraction, site-specific extractors (AI chat transcripts prioritate)
|
||||
5. **Lint pass** (contradicții/orfani) → idee nouă din analiza gist Karpathy
|
||||
|
||||
---
|
||||
|
||||
## 18. CONCLUZIE
|
||||
|
||||
**Atomic** este o arhitectură excepțional curat designed:
|
||||
- **Single source of truth** în `atomic-core` (16k+ linii Rust, zero framework deps)
|
||||
- **Storage abstraction completă** SQLite ↔ Postgres
|
||||
- **AI provider abstraction** plug-and-play (3 implementări trait-based)
|
||||
- **Wrappers thin** pentru fiecare transport: Tauri (sidecar IPC), actix-web (REST/WS/MCP), mcp-bridge (stdio)
|
||||
- **Frontend transport-unaware** (același cod în desktop și browser)
|
||||
- **Pipeline async fire-and-forget** cu callback eventing → broadcast → WebSocket
|
||||
- **Multi-database** cu registry separat
|
||||
- **API token + OAuth 2.0 PKCE + DCR** pentru securitate enterprise
|
||||
- **6 platforme client**: desktop, web, iOS, Chrome ext, Discord, Obsidian
|
||||
- **Deployment flexibil**: standalone binary, Docker Compose, Fly.io single-machine, SaaS managed (atomic-cloud)
|
||||
- **Documentație tehnică matură** (5 spec docs cu planuri executabile)
|
||||
|
||||
Este un proiect **production-ready** cu o roadmap clară de tehnical debt și features noi.
|
||||
Loading…
Add table
Add a link
Reference in a new issue