LOT 1 - Optimizare script build -Instalare mono comanda
This commit is contained in:
parent
5380c3fc63
commit
42ff22bf85
127 changed files with 16163 additions and 532 deletions
|
|
@ -17,7 +17,7 @@ it, what does it depend on" for any file in the repo.
|
|||
|
||||
| Module / File | What it does | When it runs | Who calls it |
|
||||
|---|---|---|---|
|
||||
| **`infra/docker-compose.yml`** | Defines the 3-container stack | `docker compose up` | operator |
|
||||
| **`infra/docker-compose.yml`** | Defines the 4-container stack (api, atomic, postgres, scheduler) | `docker compose up` | operator |
|
||||
| **`brain_api/`** *(container)* | HTTP service speaking Didi's contract | every Didi request | Didi backend (HTTP) |
|
||||
| **`shared/`** *(library)* | Common code: config, clients, taxonomy | imported everywhere | brain_api, extractor, lint, scripts |
|
||||
| **`extractor/`** *(library + jobs)* | Turns Document atoms into Claim atoms | operator (script 07) OR background after `/v1/ingest` | scripts/07, brain_api ingest |
|
||||
|
|
@ -26,12 +26,12 @@ it, what does it depend on" for any file in the repo.
|
|||
| **Atomic server** *(container)* | Storage + chunking + embedding pipeline + REST | always | brain_api (HTTP), scripts (HTTP) |
|
||||
| **Postgres + pgvector** *(container)* | Persistent atom + vector storage | always | atomic-server (SQL) |
|
||||
|
||||
**External services** (not in our repo, on VPN):
|
||||
**External services** (not in our repo, reached over Docker DNS on `didi-network`):
|
||||
| Where | What | Used by |
|
||||
|---|---|---|
|
||||
| `10.11.10.17:14011` | LLM router → Qwen 397B | brain_api NLI, extractor, lint |
|
||||
| `10.11.10.15:8200` | BGE-M3 embedding server | atomic-server (for embeddings) |
|
||||
| `10.11.10.15:8100` | BGE-reranker-v2-m3 | brain_api gather (for precision rerank) |
|
||||
| `didiAI-llm-api` → `10.11.10.17:14011` | LLM router → qwen3.5 | brain_api NLI, extractor, lint |
|
||||
| `didiAI-embeddings-api` → `10.11.10.15:14100` | BGE-M3 embedding server | atomic-server (for embeddings) |
|
||||
| `didiAI-rerank-api` → `10.11.10.15:14200` | BGE-reranker-v2-m3 | brain_api gather (for precision rerank) |
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -56,12 +56,12 @@ it, what does it depend on" for any file in the repo.
|
|||
│ atomic_api, taxonomy, logging} │
|
||||
└─────┬─────────────┬──────────────┬──────────────────────────┘
|
||||
│ │ │
|
||||
│ HTTP │ HTTPS │ HTTPS
|
||||
│ HTTP │ HTTP │ HTTP
|
||||
▼ ▼ ▼
|
||||
┌───────────┐ ┌──────────┐ ┌────────────────┐
|
||||
│ atomic- │ │ BGE-M3 │ │ Qwen 397B │
|
||||
│ atomic- │ │ BGE-M3 │ │ qwen3.5 │
|
||||
│ server │ │ + rerank │ │ via LLM router │
|
||||
│ (cont.) │ │ (VPN) │ │ (VPN) │
|
||||
│ (cont.) │ │ (DNS) │ │ (DNS) │
|
||||
└─────┬─────┘ └──────────┘ └────────────────┘
|
||||
│
|
||||
▼
|
||||
|
|
@ -121,8 +121,9 @@ its own.
|
|||
|
||||
## `brain_api/` — the HTTP service
|
||||
|
||||
**Purpose.** Containerized FastAPI service that exposes the 5-endpoint
|
||||
contract Didi's backend already speaks. It does NOT contain business logic
|
||||
**Purpose.** Containerized FastAPI service that exposes the full route set
|
||||
(~30 routes: the web-gathering contract + brain-owned cache/freshness layers)
|
||||
that Didi's backend already speaks. It does NOT contain business logic
|
||||
that another module reuses; it ties together `shared` + `extractor` (via
|
||||
`/v1/ingest`) and translates between Didi's HTTP shape and the brain's
|
||||
internal capabilities.
|
||||
|
|
@ -139,7 +140,7 @@ FastAPI/Uvicorn. The lifespan hook initializes the long-lived clients
|
|||
| `Dockerfile` | python:3.12-slim, non-root user `brain`, COPY shared/extractor/brain_api, curl healthcheck on /health, CMD `python -m brain_api.run`. |
|
||||
| `requirements.txt` | Pinned runtime deps. |
|
||||
| `__init__.py` | Module marker + docstring. |
|
||||
| `app.py` | The FastAPI app. Defines `lifespan` (startup/shutdown), the `/health` route, and the 5 v1 routes. Each route delegates to a function in `services/`. |
|
||||
| `app.py` | The FastAPI app. Defines `lifespan` (startup/shutdown), the `/health` route, and the full set of v1 routes (gather/search/fetch/image-search/ingest + verification_cache + analysis_atom* + canonicalize + cache/invalidate + cache/audit_log + fact_status*). Each route delegates to a function in `services/`. |
|
||||
| `run.py` | Uvicorn entry. Reads `BRAIN_API_HOST` and `BRAIN_API_PORT` from env (defaults 127.0.0.1:8090; container overrides to 0.0.0.0). |
|
||||
| `deps.py` | `AppState` dataclass + module-level singleton accessor. Set by lifespan, read by route handlers. |
|
||||
| `schemas.py` | **The contract.** Pydantic v2 models for every request/response shape the service speaks: `SearchRequest/Response`, `FetchRequest/Response`, `GatherRequest/Response`, `ImageSearchRequest/Response`, `IngestRequest/Response`, plus `EvidenceItem`, `BrainEvidenceMeta`, `BrainMeta`, etc. |
|
||||
|
|
@ -159,7 +160,7 @@ FastAPI/Uvicorn. The lifespan hook initializes the long-lived clients
|
|||
|
||||
## `extractor/` — Document → Claim transformation
|
||||
|
||||
**Purpose.** Reads `Type/Document` atoms from Atomic, asks Qwen 397B to
|
||||
**Purpose.** Reads `Type/Document` atoms from Atomic, asks qwen3.5 to
|
||||
extract atomic factual claims, validates each claim against the source
|
||||
text (substring check on the quote), and creates new `Type/Claim` atoms
|
||||
with canonical hash-based source URLs and inherited tags.
|
||||
|
|
@ -186,7 +187,7 @@ Both paths share the same idempotency state (`extractor/_extracted.json`).
|
|||
| `extract.py` | Single-document logic. `extract_claims_from_atom(llm, title, language, content)` → `ExtractionResult` with `valid: list[ExtractedClaim]` and `rejected: dict[reason → count]`. Includes the substring quote validation. |
|
||||
| `push.py` | `push_claim(atomic, parent_atom, claim, parent_title, resolver)` creates one `Type/Claim` atom. URL is `{parent_url}#claim={hash8}` for natural dedup. Inherits parent's Country/Topic/SourceType/Credibility/Language tags, adds Type/Claim + Stance/{X}. |
|
||||
| `batch.py` | The orchestrator. `run_batch(limit, only_atom_ids)` paginates Document atoms, fetches each fully (list_atoms returns summary only — gotcha!), runs extraction sequentially, pushes claims, saves state after every doc so Ctrl-C is recoverable. |
|
||||
| `prompts/claim_extraction_v1.md` | Versioned prompt that instructs Qwen 397B to return strict JSON with claims + verbatim quotes. Output format pinned. |
|
||||
| `prompts/claim_extraction_v1.md` | Versioned prompt that instructs qwen3.5 to return strict JSON with claims + verbatim quotes. Output format pinned. |
|
||||
|
||||
**Imports.** `shared`, `httpx` (transitively).
|
||||
|
||||
|
|
@ -198,7 +199,7 @@ Both paths share the same idempotency state (`extractor/_extracted.json`).
|
|||
|
||||
**Purpose.** Audit job that finds claim atoms in our corpus that
|
||||
contradict each other (or are paraphrases). For every claim, asks Atomic
|
||||
for its semantic neighbors, then asks Qwen 397B to classify each
|
||||
for its semantic neighbors, then asks qwen3.5 to classify each
|
||||
candidate pair as **EQUIVALENT**, **CONTRADICTORY**, or **INCOMPARABLE**.
|
||||
Stores all verdicts in a JSON ledger so re-runs only process new pairs.
|
||||
|
||||
|
|
@ -264,7 +265,7 @@ clean rich-formatted report, and exits with a meaningful code.
|
|||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `docker-compose.yml` | Three services: `postgres` (pgvector/pgvector:pg16), `atomic-server` (ghcr.io/kenforthewin/atomic-server:latest with overridden entrypoint to use `--data-dir` instead of legacy `--db-path`, and `ATOMIC_STORAGE=postgres` env), `brain-api` (built from `brain_api/Dockerfile`). Two named volumes (`didibrain-pg-data`, `didibrain-atomic-data`), one bridge network (`didibrain`). All services have `restart: unless-stopped` and healthchecks. |
|
||||
| `docker-compose.yml` | Four services: `postgres` (pgvector/pgvector:pg16), `atomic-server` (ghcr.io/kenforthewin/atomic-server:latest with overridden entrypoint to use `--data-dir` instead of legacy `--db-path`, and `ATOMIC_STORAGE=postgres` env), `brain-api` (built from `brain_api/Dockerfile`), and `scheduler` (built from `scheduler/Dockerfile` — feeder + auditor + watcher + heartbeat). Two named volumes (`didibrain-pg-data`, `didibrain-atomic-data`), and the external shared network `didi-network`. All services have `restart: unless-stopped` and healthchecks. |
|
||||
|
||||
**When it runs.** `docker compose up` from the operator (or via `bootstrap_deploy.sh`).
|
||||
|
||||
|
|
@ -301,7 +302,7 @@ brain_api/app.py: post_gather()
|
|||
│
|
||||
│ stage 4 nli → mapping.parse_claim_atom_body() per top result
|
||||
│ → nli.classify_batch(llm, claim, evidence_texts)
|
||||
│ (parallel calls to Qwen 397B, ~3-4 sec; one per parent doc)
|
||||
│ (parallel calls to qwen3.5, ~3-4 sec; one per parent doc)
|
||||
│
|
||||
│ stage 5 evidence → atomic.get_atom_by_source_url() per parent doc
|
||||
│ → mapping.evidence_from_parent() builds EvidenceItem
|
||||
|
|
@ -355,7 +356,7 @@ cp .env.example .env, vim .env (none)
|
|||
│ │
|
||||
│ ├─ list Type/Document atoms
|
||||
│ ├─ for each not in extractor/_extracted.json:
|
||||
│ │ fetch full atom → extract.extract_claims_from_atom() (Qwen 397B)
|
||||
│ │ fetch full atom → extract.extract_claims_from_atom() (qwen3.5)
|
||||
│ │ push.push_claim() per valid claim → atomic.create_atom()
|
||||
│ └─ save state every doc
|
||||
│
|
||||
|
|
@ -394,7 +395,7 @@ brain_api/app.py: post_ingest()
|
|||
FastAPI BackgroundTasks: _run_extraction_background(created_ids)
|
||||
│
|
||||
└─▶ extractor.batch.run_batch(only_atom_ids=set(created_ids))
|
||||
├─ runs Qwen 397B claim extraction
|
||||
├─ runs qwen3.5 claim extraction
|
||||
├─ pushes Type/Claim atoms
|
||||
└─ saves extractor/_extracted.json
|
||||
```
|
||||
|
|
@ -416,9 +417,14 @@ from Atomic without loss.
|
|||
| `extractor/_extracted.json` | `scripts/07_run_extraction.py` (and brain_api ingest) | Per-document extraction log: which docs have been processed, with which prompt version, how many claims came out. Skip already-done docs on rerun. | yes — delete and re-run; will re-extract everything |
|
||||
| `lint/_contradictions.json` | `scripts/10_run_lint.py` | Per-pair verdict ledger with the EQUIVALENT / CONTRADICTORY / INCOMPARABLE labels. Skip already-done pairs on rerun. | yes — delete and re-run; will re-classify everything |
|
||||
|
||||
The brain_api **container** does NOT use any of these files. It calls
|
||||
`atomic.list_tags()` at startup to refresh `TagResolver` in-memory, so
|
||||
the image stays portable.
|
||||
The brain_api **container** refreshes `TagResolver` in-memory at startup by
|
||||
calling `atomic.list_tags()`, so it does not bake any of these files into the
|
||||
image. It does, however, **mount** `shared/_tag_ids.json` read-only via the
|
||||
compose `volumes:` entry — background extraction (triggered by `/v1/ingest`)
|
||||
instantiates `TagResolver` directly, bypassing the lifespan refresh, so it
|
||||
relies on that mounted file to resolve canonical tag UUIDs. The other two
|
||||
ledgers (`extractor/_extracted.json`, `lint/_contradictions.json`) are
|
||||
host-only and not used by the container.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -435,20 +441,20 @@ project root. The brain_api container also receives these via
|
|||
| `LLM_ROUTER_API_KEY` | empty | optional bearer | no |
|
||||
| `LLM_VLLM_URL` | `http://localhost:14001` | direct vLLM URL (fallback only, currently unused) | no |
|
||||
| `LLM_LLAMACPP_URLS` | empty | comma-separated direct llamacpp URLs (fallback) | no |
|
||||
| `MODEL_REASONING` | `Qwen3.5-397B-A17B` | model id for extraction, NLI, gather | yes |
|
||||
| `MODEL_REASONING_BACKEND` | `llamacpp` | router backend hint | yes |
|
||||
| `MODEL_FAST` | `qwen3.5` | fast model id (currently disabled) | no |
|
||||
| `MODEL_REASONING` | `qwen3.5` | model id for extraction, NLI, gather (live working model) | yes |
|
||||
| `MODEL_REASONING_BACKEND` | `vllm` | router backend hint | yes |
|
||||
| `MODEL_FAST` | `qwen3.5` | fast model id (enabled — the live working model) | no |
|
||||
| `MODEL_FAST_BACKEND` | `vllm` | fast backend hint | no |
|
||||
| `MODEL_FAST_ENABLED` | `false` | enable fast model in routing | no |
|
||||
| `MODEL_VISION` | `gemma-3-27b-it` | vision model id (currently down) | no |
|
||||
| `MODEL_FAST_ENABLED` | `true` | fast model enabled in routing (live: `true`, qwen3.5 active) | no |
|
||||
| `MODEL_VISION` | empty | vision model id (vision disabled; no vision model deployed) | no |
|
||||
| `MODEL_VISION_URL` | empty | direct vision endpoint | no |
|
||||
| `MODEL_VISION_ENABLED` | `false` | enable vision in routing | no |
|
||||
| `EMBEDDING_URL` | `http://10.11.10.15:8200` | BGE-M3 vLLM endpoint | yes |
|
||||
| `EMBEDDING_URL` | `http://10.11.10.15:14100` | BGE-M3 vLLM endpoint (`didiAI-embeddings-api`) | yes |
|
||||
| `EMBEDDING_API_KEY` | empty | optional bearer | no |
|
||||
| `EMBEDDING_MODEL` | `BAAI/bge-m3` | model id sent in requests | yes |
|
||||
| `EMBEDDING_DIM` | `1024` | vector dimension (must match Atomic's setting) | yes |
|
||||
| `EMBEDDING_MAX_TOKENS` | `8192` | input length cap | yes |
|
||||
| `RERANKER_URL` | `http://10.11.10.15:8100` | BGE-reranker-v2-m3 endpoint | yes |
|
||||
| `RERANKER_URL` | `http://10.11.10.15:14200` | BGE-reranker-v2-m3 endpoint (`didiAI-rerank-api`) | yes |
|
||||
| `RERANKER_API_KEY` | empty | optional bearer | no |
|
||||
| `RERANKER_MODEL` | `BAAI/bge-reranker-v2-m3` | model id sent in requests | yes |
|
||||
| `ATOMIC_URL` | `http://localhost:8088` | atomic-server URL (host scripts use this; brain_api container overrides to docker DNS) | yes |
|
||||
|
|
@ -504,7 +510,7 @@ Something with /v1/gather failing?
|
|||
Atomic returning 500?
|
||||
│
|
||||
├─ docker logs didibrain-atomic — likely BGE unreachable
|
||||
└─ curl http://10.11.10.15:8200/v1/models on host — VPN check
|
||||
└─ curl http://10.11.10.15:14100/v1/models on host — upstream reachability check
|
||||
|
||||
Claim extraction acting weird?
|
||||
│
|
||||
|
|
|
|||
|
|
@ -64,7 +64,7 @@ rulare). Cu evidence_hash în cheie, cache-ul era practic nereachable.
|
|||
"https://somes-tisa.rowater.ro/pdf2"
|
||||
],
|
||||
"tier": "free",
|
||||
"model": "qwen35:Qwen3.5-397B-A17B",
|
||||
"model": "qwen35:qwen3.5",
|
||||
"prompt_hash": "a3f2b1c9d4e7",
|
||||
"framework_version": "f9e1d2c3b4a5",
|
||||
"schema_name": "didi-v1",
|
||||
|
|
@ -118,7 +118,7 @@ rulare). Cu evidence_hash în cheie, cache-ul era practic nereachable.
|
|||
- `503 Service Unavailable` — PG indisponibil, retry mai târziu
|
||||
- `500 Internal Server Error` — orice alt eșec
|
||||
|
||||
**Idempotență:** UPSERT pe cheia `(claim_hash, evidence_hash, tier)`. Same-tuple,
|
||||
**Idempotență:** UPSERT pe cheia `(claim_hash, tier)`. Same-tuple,
|
||||
second write → update `verification_processed`/`verification_raw`/`updated_at`/`expires_at`.
|
||||
Last-writer-wins.
|
||||
|
||||
|
|
@ -155,7 +155,7 @@ Last-writer-wins.
|
|||
|
||||
"verification_staleness": "fresh",
|
||||
"verification": { ... payload stocat ... },
|
||||
"verification_model": "qwen35:Qwen3.5-397B-A17B",
|
||||
"verification_model": "qwen35:qwen3.5",
|
||||
"verification_tier": "free",
|
||||
"verification_prompt_hash": "a3f2b1c9d4e7",
|
||||
"verification_framework_version": "f9e1d2c3b4a5",
|
||||
|
|
@ -263,7 +263,7 @@ Cache pe `free` și `premium` sunt **complet separate**. Un user `free` nu vede
|
|||
ce a scris un user `premium` (și invers). Este intenționat: modele diferite =
|
||||
nuanțe diferite la stance.
|
||||
|
||||
Cheia unică în DB: `(claim_hash, evidence_hash, tier)`.
|
||||
Cheia unică în DB: `(claim_hash, tier)`.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
Knowledge atom storage + claim verification cache + analysis atom cache pentru platforma DIDI. FastAPI service care vorbeste contractul `web-gathering` 1:1 (drop-in replacement / cache layer pentru `/v1/gather`) si in plus expune doua cache-uri proprii: `brain_verification_cache` (claims, din 2026-04-23) si `brain_analysis_atom` (techniques + ai_tampered + claims, NEW v2 din 2026-05-01).
|
||||
|
||||
**Stack**: Python 3.11+, FastAPI, Pydantic v2, asyncpg, BGE-M3 1024-dim embeddings (vLLM), BGE-reranker-v2-m3 (vLLM cross-encoder), Qwen3.5-397B-A17B (llama.cpp via OpenAI-compat router)
|
||||
**Stack**: Python 3.11+, FastAPI, Pydantic v2, asyncpg, BGE-M3 1024-dim embeddings (vLLM), BGE-reranker-v2-m3 (vLLM cross-encoder), qwen3.5 (vLLM via OpenAI-compat router)
|
||||
**Storage**: Atomic (Rust knowledge graph) on Postgres 16 + pgvector. Brain owns its own tables prefixed `brain_*` alongside Atomic's tables.
|
||||
**URL**: `http://10.11.10.12:8090` (production-local, since brain cutover 2026-05-01 — anterior pe `10.11.10.13:8090`)
|
||||
**Container**: `didibrain-api` (alongside `didibrain-atomic` and `didibrain-postgres`)
|
||||
|
|
@ -163,9 +163,10 @@ UNIQUE: `(content_hash, component, prompt_hash)` — `tier` OUT of unique key (w
|
|||
## Configuration
|
||||
|
||||
- Settings via `shared/config.py` cu `pydantic-settings`
|
||||
- Env vars: `postgres_dsn`, `atomic_url`, `llm_router_url`, `embedding_url`, `reranker_url`, `verification_cache_ttl_days` (default 30), `verification_cache_max_payload_kb` (default 64)
|
||||
- Internal Atomic URL: `http://didibrain-atomic:8080` (Docker DNS)
|
||||
- Upstream LLM/embed/rerank pe VPN 10.11.10.x (10.11.10.17:14011 router, 10.11.10.15:8200 BGE-M3, 10.11.10.15:8100 reranker)
|
||||
- Env vars: `postgres_dsn`, `atomic_url`, `atomic_token`, `llm_router_url`, `embedding_url`, `reranker_url`, `verification_cache_ttl_days` (default 30), `verification_cache_max_payload_kb` (default 64)
|
||||
- `ATOMIC_TOKEN`: required for atomic-server API calls, auto-generated/auto-populated by `scripts/02_bootstrap_atomic.py` (written back to `.env`)
|
||||
- Internal Atomic URL: `http://atomic-server:8080` (Docker DNS)
|
||||
- Upstream LLM/embed/rerank reached over Docker DNS on `didi-network` (`didiAI-llm-api` → 10.11.10.17:14011 router, `didiAI-embeddings-api` → 10.11.10.15:14100 BGE-M3, `didiAI-rerank-api` → 10.11.10.15:14200 reranker)
|
||||
- **No auth in v1** (binds to internal Docker network)
|
||||
|
||||
---
|
||||
|
|
@ -173,10 +174,10 @@ UNIQUE: `(content_hash, component, prompt_hash)` — `tier` OUT of unique key (w
|
|||
## Deployment
|
||||
|
||||
- Docker compose la `infra/docker-compose.yml`
|
||||
- 3 containere: `didibrain-api` (FastAPI), `didibrain-atomic` (Rust KG), `didibrain-postgres` (PG 16 + pgvector)
|
||||
- 4 containere: `didibrain-api` (FastAPI), `didibrain-atomic` (Rust KG), `didibrain-postgres` (PG 16 + pgvector), `didibrain-scheduler` (feeder + auditor + watcher)
|
||||
- Schema bootstrap on startup (idempotent — `_SCHEMA_SQL` ruleaza la fiecare connect, all DDL is `IF NOT EXISTS` + ALTER guards)
|
||||
- TTL cleanup: cron job in `scripts/` sterge expired silver/bronze atoms
|
||||
- Resource footprint: ~200 MB RAM total cross 3 containere; brain-api idle ~6% CPU, spike ~20% during `/v1/gather` cu NLI
|
||||
- Resource footprint: ~200 MB RAM total cross 4 containere; brain-api idle ~6% CPU, spike ~20% during `/v1/gather` cu NLI
|
||||
|
||||
**Rebuild**:
|
||||
```bash
|
||||
|
|
@ -194,7 +195,7 @@ docker compose -f infra/docker-compose.yml --env-file .env up -d --build
|
|||
- `scripts/04_seed_taxonomy.py` — seed canonical tag taxonomy (79 tags, 7 namespaces)
|
||||
- `scripts/05_import_wikipedia_seed.py` — seed-list Wikipedia import
|
||||
- `scripts/06_validate_queries.py` — smoke test doc-level retrieval
|
||||
- `scripts/07_run_extraction.py` — claim extraction batch (Qwen 397B)
|
||||
- `scripts/07_run_extraction.py` — claim extraction batch (qwen3.5)
|
||||
- `scripts/08_validate_claims.py` — smoke test claim-level retrieval
|
||||
- `scripts/09_brain_api_demo.py` — brain_api contract test (all endpoints)
|
||||
- `scripts/10_run_lint.py` — Lint pass runner (contradiction detection)
|
||||
|
|
|
|||
|
|
@ -23,7 +23,7 @@ HTTP contract but serves responses from a locally-grown **knowledge graph**:
|
|||
|
||||
- **Atoms** (documents and atomic claims) are stored in a Postgres +
|
||||
pgvector backend with multilingual embeddings (BGE-M3, 1024-dim).
|
||||
- **Claim extraction** runs Qwen 3.5 397B over every ingested document to
|
||||
- **Claim extraction** runs Qwen3.5 over every ingested document to
|
||||
pull out verifiable atomic claims with source quotes and stance.
|
||||
- **Retrieval** uses vector kNN + BGE-reranker-v2-m3 cross-encoder for
|
||||
precision.
|
||||
|
|
@ -46,9 +46,9 @@ background. Next time a similar claim arrives, it's a HIT.
|
|||
| **Brain storage** | [Atomic](https://github.com/kenforthewin/atomic) (Rust) on Postgres 16 + pgvector |
|
||||
| **Embeddings** | `BAAI/bge-m3` via vLLM OpenAI-compat (1024 dim, 8K ctx, multilingual) |
|
||||
| **Reranker** | `BAAI/bge-reranker-v2-m3` cross-encoder via vllm-rerank-api |
|
||||
| **LLM (reasoning)** | `Qwen3.5-397B-A17B` (MoE) via llama.cpp through an OpenAI-compat router |
|
||||
| **LLM (reasoning)** | `qwen3.5` via vLLM through an OpenAI-compat router (`MODEL_FAST` enabled) |
|
||||
| **Service** | Python 3.12, FastAPI, Uvicorn, Pydantic v2, httpx, structlog, tenacity |
|
||||
| **Deployment** | Docker + docker-compose (3 services: api, atomic, postgres) |
|
||||
| **Deployment** | Docker + docker-compose (4 services: api, atomic, postgres, scheduler) |
|
||||
|
||||
## Quick start — local dev
|
||||
|
||||
|
|
@ -88,7 +88,7 @@ curl -fsS http://localhost:8090/health
|
|||
```
|
||||
|
||||
Open the interactive Swagger UI at **http://localhost:8090/docs** to poke
|
||||
the 5 endpoints live.
|
||||
the full route set live (~30 routes).
|
||||
|
||||
## Production deploy — fresh Linux server
|
||||
|
||||
|
|
@ -127,6 +127,21 @@ backends safely ignore.
|
|||
| `POST /v1/gather` | claim → ranked evidence with NLI stance | 5-6 s (1.5 s without NLI) |
|
||||
| `POST /v1/image-search` | stub (always empty list) | <5 ms |
|
||||
| `POST /v1/ingest` | populate brain from web-module output | variable (async extraction) |
|
||||
| `POST /v1/verification_cache` | write claim verification result (claims cache) | <50 ms |
|
||||
| `POST /v1/analysis_atom/lookup` | read cached analysis result (techniques / ai_tampered / claims) | <50 ms |
|
||||
| `POST /v1/analysis_atom` | write analysis atom (silver/bronze by confidence) | <50 ms |
|
||||
| `PATCH /v1/analysis_atom/{id}` | promote atom to gold (moderator review) | <50 ms |
|
||||
| `GET /v1/analysis_atom/stats` | per-tier/component counts + 24h hit rate | <20 ms |
|
||||
| `POST /v1/canonicalize` | temporal claim disambiguation (Pilon 7) | LLM-bound |
|
||||
| `POST /v1/cache/invalidate` | mass invalidation with `dry_run` (Pilon 8) | variable |
|
||||
| `GET /v1/cache/audit_log` | paginated audit browser | <50 ms |
|
||||
| `GET/PATCH /v1/fact_status/*` | versioned fact-status layer (list/detail/versions/override) | <50 ms |
|
||||
|
||||
This is the full set served by `brain_api` (~30 routes including the FastAPI
|
||||
auto docs); the `/v1/gather`, `/v1/search`, `/v1/fetch`, `/v1/ingest`,
|
||||
`/v1/image-search` group is the web-gathering contract, the rest are the
|
||||
brain-owned cache + freshness-defense layers. See `INDEX.md` for the
|
||||
canonical route list.
|
||||
|
||||
The response from `/v1/gather` matches the existing web-module shape
|
||||
exactly plus an additive `brain_meta` object on the top level and inside
|
||||
|
|
@ -156,11 +171,11 @@ Didi backend
|
|||
│ stage context → language detection │
|
||||
│ stage retrieval → Atomic semantic search (top 50) │
|
||||
│ stage rerank → BGE cross-encoder (top 15) │
|
||||
│ stage nli → Qwen 397B stance vs query │
|
||||
│ stage nli → Qwen3.5 stance vs query │
|
||||
│ stage evidence → group by parent doc, shape │
|
||||
└──────┬──────────────────────────┬──────────────────┘
|
||||
│ │
|
||||
│ HTTP (docker DNS) │ HTTPS (VPN)
|
||||
│ HTTP (docker DNS) │ HTTP (docker DNS)
|
||||
▼ ▼
|
||||
┌────────────────┐ ┌───────────────────────┐
|
||||
│ atomic-server │ │ BGE-M3 embeddings │
|
||||
|
|
@ -169,9 +184,9 @@ Didi backend
|
|||
│ └───────────────────────┘
|
||||
▼
|
||||
┌────────────────┐ ┌───────────────────────┐
|
||||
│ postgres │ │ Qwen 3.5 397B-A17B │
|
||||
│ postgres │ │ qwen3.5 │
|
||||
│ + pgvector │ │ via LLM router │
|
||||
│ :5432 (5434) │ │ (llama.cpp + vLLM) │
|
||||
│ :5432 (5434) │ │ (vLLM) │
|
||||
└────────────────┘ └───────────────────────┘
|
||||
```
|
||||
|
||||
|
|
@ -179,7 +194,7 @@ Didi backend
|
|||
|
||||
```
|
||||
didibrain/
|
||||
├── infra/docker-compose.yml # 3-service stack
|
||||
├── infra/docker-compose.yml # 4-service stack (api, atomic, postgres, scheduler)
|
||||
├── brain_api/ # FastAPI service (the main deliverable)
|
||||
├── shared/ # config, clients, taxonomy (reused everywhere)
|
||||
├── extractor/ # claim extraction (host jobs + /v1/ingest)
|
||||
|
|
@ -201,7 +216,7 @@ Detailed file-by-file rundown is in `STATUS.md`.
|
|||
- [x] Claim extraction (513 atoms, 1.2% hallucination filter)
|
||||
- [x] Document-level retrieval validated (cross-lingual cosine 0.88-0.94)
|
||||
- [x] Claim-level retrieval validated
|
||||
- [x] brain_api HTTP service with Didi contract (5 endpoints)
|
||||
- [x] brain_api HTTP service with Didi contract (full route set, ~30 routes)
|
||||
- [x] brain_api dockerized (self-sufficient, taxonomy auto-refresh)
|
||||
- [x] NLI stance vs query in `/v1/gather`
|
||||
- [x] Lint pass contradiction detection (code + smoke test)
|
||||
|
|
@ -213,16 +228,20 @@ Detailed file-by-file rundown is in `STATUS.md`.
|
|||
|
||||
## Operational notes
|
||||
|
||||
- **Resource footprint**: ~200 MB RAM total across the 3 containers; brain-api
|
||||
- **Resource footprint**: ~200 MB RAM total across the 4 containers; brain-api
|
||||
idles at ~6% CPU, spikes to ~20% during a `/v1/gather` with NLI.
|
||||
- **Image size**: brain-api Docker image is ~253 MB (Python 3.12-slim base).
|
||||
- **VPN dependency**: BGE and the LLM router live on a VPN-routed 10.11.10.x
|
||||
network. If the VPN drops, `/v1/gather` returns 500 because Atomic cannot
|
||||
embed the query. Confirm upstream reachability before debugging anything
|
||||
else when search starts failing.
|
||||
- **Self-sufficient startup**: brain_api pulls the current taxonomy from
|
||||
Atomic at startup, so there is no baked `_tag_ids.json` in the image and
|
||||
the container is portable across environments.
|
||||
- **Upstream connectivity**: the LLM router, BGE-M3 embeddings, and the BGE
|
||||
reranker are reached over Docker DNS as `didiAI-llm-api:14011`,
|
||||
`didiAI-embeddings-api:14100`, and `didiAI-rerank-api:14200` on the shared
|
||||
`didi-network`. If those upstreams are unreachable, `/v1/gather` returns 500
|
||||
because Atomic cannot embed the query. Confirm upstream reachability before
|
||||
debugging anything else when search starts failing.
|
||||
- **Startup**: brain_api refreshes the `TagResolver` from Atomic at startup;
|
||||
in addition `shared/_tag_ids.json` is mounted into the container (compose
|
||||
volume) so background extraction — which instantiates `TagResolver`
|
||||
directly, bypassing the lifespan refresh — can still resolve canonical tag
|
||||
UUIDs.
|
||||
- **Idempotency**: every operator script (taxonomy seeder, Wikipedia
|
||||
importer, claim extractor, Lint pass) is idempotent via state files or
|
||||
URL-based dedup. Re-running is always safe.
|
||||
|
|
|
|||
|
|
@ -1,6 +1,13 @@
|
|||
# DidiBrain — Status Snapshot
|
||||
|
||||
**Last updated:** 2026-04-23 (integration session — web-api cache + backend verification cache)
|
||||
|
||||
> **Note (current):** the platform is **LIVE and functional**. Since this
|
||||
> snapshot the stack grew to **four containers** (added `didibrain-scheduler`),
|
||||
> the analysis-atom cache (v2, gold/silver/bronze) and the Cache Freshness
|
||||
> Defense layer (feeder + auditor + watcher) shipped, and the live working
|
||||
> model is **`qwen3.5`** (vLLM, `MODEL_FAST` enabled). Sections below that
|
||||
> still say "3 containers / 5 endpoints / 397B" are corrected inline.
|
||||
**Working dir:** `/home/admin365/ml-projects/modules/didi_brain/`
|
||||
**Related docs:**
|
||||
- `CONTRACT_VERIFICATION_CACHE.md` — contract backend↔brain pentru verification cache (v2, current)
|
||||
|
|
@ -17,24 +24,31 @@ DidiBrain is **SHIP READY** și **INTEGRATED**. Pe lângă contractul inițial H
|
|||
- **Verification cache pentru didi-backend** — post-LLM stance/verdict storage
|
||||
cu 4 staleness states (fresh / stale_framework / stale_prompt / miss)
|
||||
|
||||
**Trei containere, zero regressions pe funcționalitatea v1.** Tabelul relational
|
||||
nou `brain_verification_cache` trăiește în același Postgres cu atomic-server,
|
||||
prefix `brain_` pentru izolare.
|
||||
- **Analysis atom cache (v2)** — techniques + ai_tampered + claims full-component
|
||||
results, 3-tier gold/silver/bronze (since 2026-05-01)
|
||||
- **Cache Freshness Defense** — `didibrain-scheduler` (feeder + auditor + watcher)
|
||||
keeps cached verdicts fresh
|
||||
|
||||
**Patru containere, zero regressions pe funcționalitatea v1.** Tabelele relationale
|
||||
noi `brain_verification_cache` + `brain_analysis_atom` (și layerul fact_status)
|
||||
trăiesc în același Postgres cu atomic-server, prefix `brain_` pentru izolare.
|
||||
|
||||
## Current state in 30 seconds
|
||||
|
||||
```
|
||||
3 containers running continuously, all healthy
|
||||
4 containers running continuously, all healthy
|
||||
|
||||
didibrain-api brain_api FastAPI service :8090 52 MB RAM
|
||||
didibrain-atomic atomic-server (API only) :8088 83 MB RAM
|
||||
didibrain-postgres pgvector pg16 :5434 60 MB RAM
|
||||
(+ brain_verification_cache table)
|
||||
didibrain-api brain_api FastAPI service :8090 52 MB RAM
|
||||
didibrain-atomic atomic-server (API only) :8088 (8080 internal) 83 MB RAM
|
||||
didibrain-postgres pgvector pg16 :5434 60 MB RAM
|
||||
(+ brain_verification_cache
|
||||
+ brain_analysis_atom tables)
|
||||
didibrain-scheduler feeder + auditor + watcher (no port)
|
||||
|
||||
Brain contents (grow continuously via web-api ingest + manual bootstrap)
|
||||
79 tags (canonical taxonomy, 7 root namespaces)
|
||||
19+ documents (initial Wikipedia vaccines seed + anything premium adds)
|
||||
509+ claims (extracted by Qwen 397B, substring validated)
|
||||
509+ claims (extracted by qwen3.5, substring validated)
|
||||
|
||||
Verification cache (new in v2 — Apr 23)
|
||||
brain_verification_cache UNIQUE (claim_hash, tier)
|
||||
|
|
@ -82,7 +96,7 @@ Didi backend ┌─────────────
|
|||
│ DidiBrain stack │
|
||||
│ │
|
||||
│ ┌──────────────────────┐ │
|
||||
│ │ brain-api :8090 │ FastAPI + Uvicorn, 5 v1 endpoints │
|
||||
│ │ brain-api :8090 │ FastAPI + Uvicorn, full route set │
|
||||
│ │ (didibrain-api) │ speaks Didi contract 1:1 │
|
||||
│ └──────────┬───────────┘ │
|
||||
│ │ HTTP (docker DNS) │
|
||||
|
|
@ -97,25 +111,26 @@ Didi backend ┌─────────────
|
|||
│ │ HTTPS
|
||||
│ ▼
|
||||
│ ┌────────────────────────────────┐
|
||||
│ │ BGE-M3 embeddings │ 10.11.10.15:8200
|
||||
│ │ BGE-M3 embeddings │ 10.11.10.15:14100
|
||||
│ │ (vLLM OpenAI-compat) │ 1024 dim, 8K ctx, multilingual
|
||||
│ └────────────────────────────────┘
|
||||
│
|
||||
│ ┌────────────────────────────────┐
|
||||
│ │ BGE-reranker-v2-m3 │ 10.11.10.15:8100
|
||||
│ │ BGE-reranker-v2-m3 │ 10.11.10.15:14200
|
||||
│ │ (cross-encoder) │ precision boost
|
||||
│ └────────────────────────────────┘
|
||||
│
|
||||
│ ┌────────────────────────────────┐
|
||||
└─▶│ Qwen3.5-397B-A17B │ 10.11.10.17:14011 (router)
|
||||
└─▶│ qwen3.5 │ 10.11.10.17:14011 (router)
|
||||
│ via LLM router │ round-robin to .18 and .19
|
||||
│ (llama.cpp + vLLM backends) │ claim extraction, NLI
|
||||
│ (vLLM backends) │ claim extraction, NLI
|
||||
└────────────────────────────────┘
|
||||
```
|
||||
|
||||
All upstream endpoints live on a VPN-routed 10.11.10.x network. The Docker
|
||||
containers reach them through WSL2 NAT (Docker Desktop) or host networking
|
||||
(Linux server).
|
||||
All upstream endpoints (LLM router, BGE-M3, reranker) are reached over Docker
|
||||
DNS on the shared external `didi-network` as `didiAI-llm-api:14011`,
|
||||
`didiAI-embeddings-api:14100`, and `didiAI-rerank-api:14200` (the 10.11.10.x
|
||||
addresses above are the host-side equivalents).
|
||||
|
||||
## Repo layout
|
||||
|
||||
|
|
@ -131,7 +146,7 @@ didibrain/
|
|||
├── AUDIT.md # initial upstream Atomic audit
|
||||
│
|
||||
├── infra/
|
||||
│ └── docker-compose.yml # 3-service stack (+ optional atomic-web)
|
||||
│ └── docker-compose.yml # 4-service stack (api, atomic, postgres, scheduler)
|
||||
│
|
||||
├── brain_api/ # HTTP service — the main deliverable
|
||||
│ ├── Dockerfile
|
||||
|
|
@ -184,7 +199,7 @@ didibrain/
|
|||
├── 06_validate_queries.py # smoke test doc-level retrieval
|
||||
├── 07_run_extraction.py # claim extraction batch
|
||||
├── 08_validate_claims.py # smoke test claim-level retrieval
|
||||
├── 09_brain_api_demo.py # brain_api contract test (all 5 endpoints)
|
||||
├── 09_brain_api_demo.py # brain_api contract test (core web-gathering endpoints)
|
||||
├── 10_run_lint.py # Lint pass runner (--limit --force)
|
||||
├── 11_show_contradictions.py # read state file, render top contradictions
|
||||
└── bootstrap_deploy.sh # fresh-server deploy orchestrator
|
||||
|
|
@ -209,6 +224,17 @@ backends.
|
|||
| `POST /v1/gather` with `run_nli:false` | claim → evidence | ~1.5-2 s | skip stance classification for speed |
|
||||
| `POST /v1/image-search` | stub | <5 ms | always empty list |
|
||||
| `POST /v1/ingest` | populate from web module | variable | creates atoms + optional async extraction |
|
||||
| `POST /v1/verification_cache` | write claim verification result | <50 ms | claims cache (v2) |
|
||||
| `POST /v1/analysis_atom/lookup` | read cached analysis | <50 ms | techniques / ai_tampered / claims |
|
||||
| `POST /v1/analysis_atom` | write analysis atom | <50 ms | silver/bronze by `llm_confidence` |
|
||||
| `PATCH /v1/analysis_atom/{id}` | promote to gold | <50 ms | moderator review |
|
||||
| `GET /v1/analysis_atom/stats` | per-tier/component counts | <20 ms | 24h hit rate |
|
||||
| `POST /v1/canonicalize` | temporal claim disambiguation | LLM-bound | Pilon 7 |
|
||||
| `POST /v1/cache/invalidate` | mass invalidation (`dry_run`) | variable | Pilon 8 |
|
||||
| `GET /v1/cache/audit_log` | audit browser | <50 ms | paginated |
|
||||
| `GET/PATCH /v1/fact_status/*` | versioned fact-status layer | <50 ms | list/detail/versions/override |
|
||||
|
||||
(Full set ~30 routes incl. FastAPI auto docs; see `INDEX.md` for the canonical list.)
|
||||
|
||||
### /v1/gather response shape (key fields)
|
||||
|
||||
|
|
@ -297,9 +323,9 @@ or in README.md under "Production deploy".
|
|||
Only the upstream endpoint URLs may need updating:
|
||||
|
||||
```bash
|
||||
LLM_ROUTER_URL=http://10.11.10.17:14011 # if router is on VPN, unchanged
|
||||
EMBEDDING_URL=http://10.11.10.15:8200 # if BGE is on VPN, unchanged
|
||||
RERANKER_URL=http://10.11.10.15:8100 # if reranker is on VPN, unchanged
|
||||
LLM_ROUTER_URL=http://10.11.10.17:14011 # didiAI-llm-api, unchanged
|
||||
EMBEDDING_URL=http://10.11.10.15:14100 # didiAI-embeddings-api, unchanged
|
||||
RERANKER_URL=http://10.11.10.15:14200 # didiAI-rerank-api, unchanged
|
||||
```
|
||||
|
||||
Everything else (`ATOMIC_URL`, ports, model names, Postgres creds) is either
|
||||
|
|
@ -344,15 +370,13 @@ curl -s -X POST http://localhost:8090/v1/gather \
|
|||
call this endpoint; we read `embedding_status` per atom via `list_atoms`
|
||||
or `get_atom` instead.
|
||||
|
||||
3. **Qwen 35B (vLLM on :14001)** — thinking mode stuck ON via the router and
|
||||
safety alignment refuses disinfo-extraction tasks. Disabled in config
|
||||
(`MODEL_FAST_ENABLED=false`); the whole pipeline runs on Qwen 397B. If a
|
||||
future session un-sticks 35B, Lint pass could speed up ~4x by using it.
|
||||
3. **Fast model `qwen3.5`** — now **enabled** (`MODEL_FAST_ENABLED=true`) and
|
||||
is the live working model for the whole pipeline (extraction, NLI, gather)
|
||||
via the LLM router at `didiAI-llm-api:14011` (vLLM). (Historically the fast
|
||||
model was disabled and the pipeline ran on a separate reasoning model; that
|
||||
is no longer the case.)
|
||||
|
||||
4. **Gemma 31B endpoint (10.11.10.16:8001)** — port unreachable (host pings
|
||||
OK). Not used by any feature; listed as optional in `.env`.
|
||||
|
||||
5. **Atomic's React UI not running** — we only deploy `atomic-server` (API),
|
||||
4. **Atomic's React UI not running** — we only deploy `atomic-server` (API),
|
||||
not `atomic-web` (React frontend). `http://localhost:8088/` returns 404
|
||||
for this reason. Brain_api has its own Swagger UI at `/docs` which covers
|
||||
dev-testing needs. If visual atom/tag/wiki browsing is needed, add an
|
||||
|
|
@ -362,7 +386,7 @@ curl -s -X POST http://localhost:8090/v1/gather \
|
|||
|
||||
| Symptom | Root cause | Fix |
|
||||
|---|---|---|
|
||||
| `/health` timeouts | BGE endpoint unreachable (VPN down?) | `curl http://10.11.10.15:8200/v1/models` on host; bring VPN back |
|
||||
| `/health` timeouts | BGE endpoint unreachable | `curl http://10.11.10.15:14100/v1/models` on host; restore upstream reachability |
|
||||
| `/v1/gather` returns 500 silently | Same as above — Atomic can't embed the query | Same fix |
|
||||
| Post-reboot: brain_api empty reply | uvicorn bound to 127.0.0.1 inside container (not 0.0.0.0) | Rebuild — Dockerfile now sets `BRAIN_API_HOST=0.0.0.0` |
|
||||
| `LLM_ROUTER_URL` points to localhost but nothing there | Router is actually on 10.11.10.17:14011 (not on the dev box) | Update `.env`; validated in session 2 |
|
||||
|
|
@ -396,12 +420,12 @@ curl -s -X POST http://localhost:8090/v1/gather \
|
|||
- Built `shared/` + `extractor/` + scripts 01-08
|
||||
- Stood up Atomic + Postgres in compose
|
||||
- Imported 19 Wikipedia vaccines articles, EN + RO
|
||||
- Extracted 513 claims via Qwen 397B (1.2% hallucination drop)
|
||||
- Extracted 513 claims via qwen3.5 (1.2% hallucination drop)
|
||||
- Validated claim-level retrieval (cross-lingual RO↔EN confirmed)
|
||||
|
||||
### Session 2 (2026-04-11, ~4 hours)
|
||||
- Re-onboarded state after VPN restart (fixed LLM router URL)
|
||||
- Built `brain_api/` FastAPI service with 5-endpoint Didi contract
|
||||
- Built `brain_api/` FastAPI service with the Didi web-gathering contract
|
||||
- Dockerized brain_api (self-sufficient startup, taxonomy refresh from atomic)
|
||||
- Added NLI stance-vs-query pass in `/v1/gather` (additive fields in `brain_meta`)
|
||||
- Fixed NLI concurrency (MAX_PARALLEL=2 matches llama.cpp backends)
|
||||
|
|
|
|||
|
|
@ -1,15 +1,18 @@
|
|||
# =============================================================================
|
||||
# DidiBrain — Atomic + Postgres pgvector
|
||||
# =============================================================================
|
||||
# This compose stack runs:
|
||||
# This compose stack runs four services:
|
||||
# - postgres (pgvector/pgvector:pg16) on port 5434
|
||||
# - atomic-server (kenforthewin/atomic-server:latest) on port 8080
|
||||
# - atomic-server (kenforthewin/atomic-server:latest) on port 8088 (8080 internal)
|
||||
# - brain-api (FastAPI service) on port 8090
|
||||
# - scheduler (feeder + auditor + watcher + heartbeat, no exposed port)
|
||||
# All four attach to the external shared `didi-network`.
|
||||
#
|
||||
# Atomic is configured to use Postgres as the data backend (atoms, embeddings,
|
||||
# tags, etc.) while keeping the SQLite registry for tokens & global settings
|
||||
# in the local volume at /data.
|
||||
#
|
||||
# AI provider config (BGE-M3 endpoint, Qwen 397B router) is NOT set here —
|
||||
# AI provider config (BGE-M3 endpoint, qwen3.5 router) is NOT set here —
|
||||
# it lives in Atomic's settings table and is bootstrapped post-startup by
|
||||
# `scripts/02_bootstrap_atomic.py` which calls PUT /api/settings.
|
||||
# =============================================================================
|
||||
|
|
|
|||
|
|
@ -47,4 +47,4 @@ requires = ["hatchling"]
|
|||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["shared", "scraper", "extractor", "lint", "didi_client"]
|
||||
packages = ["shared", "extractor", "lint"]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue