Livrare LOT 1 - Didi

ai_platform/modules/didi_brain/ARCHITECTURE.md

@@ -0,0 +1,645 @@
+# DidiBrain — Architecture & "Who Does What"
+
+This is the **codebase rosetta stone**. If you read this top-to-bottom you
+should be able to answer "what does file X do, when does it run, who calls
+it, what does it depend on" for any file in the repo.
+
+> **Reading order if you're new**:
+> 1. `README.md` — what DidiBrain is in 2 minutes
+> 2. **this file** — full mental model of the codebase (15 minutes)
+> 3. `STATUS.md` — operational state, deployment, troubleshooting
+> 4. `brain_api/schemas.py` — the literal HTTP contract with Didi
+> 5. `brain_api/services/gather.py` — the main pipeline in 200 lines
+
+---
+
+## TL;DR — "WHO DOES WHAT" in one table
+
+| Module / File | What it does | When it runs | Who calls it |
+|---|---|---|---|
+| **`infra/docker-compose.yml`** | Defines the 3-container stack | `docker compose up` | operator |
+| **`brain_api/`** *(container)* | HTTP service speaking Didi's contract | every Didi request | Didi backend (HTTP) |
+| **`shared/`** *(library)* | Common code: config, clients, taxonomy | imported everywhere | brain_api, extractor, lint, scripts |
+| **`extractor/`** *(library + jobs)* | Turns Document atoms into Claim atoms | operator (script 07) OR background after `/v1/ingest` | scripts/07, brain_api ingest |
+| **`lint/`** *(library + job)* | Detects contradictions across the corpus | operator overnight (script 10) | scripts/10, scripts/11 |
+| **`scripts/`** *(operator CLI)* | One-shot maintenance + bootstrap commands | operator types `python scripts/XX_*.py` | human |
+| **Atomic server** *(container)* | Storage + chunking + embedding pipeline + REST | always | brain_api (HTTP), scripts (HTTP) |
+| **Postgres + pgvector** *(container)* | Persistent atom + vector storage | always | atomic-server (SQL) |
+
+**External services** (not in our repo, on VPN):
+| Where | What | Used by |
+|---|---|---|
+| `10.11.10.17:14011` | LLM router → Qwen 397B | brain_api NLI, extractor, lint |
+| `10.11.10.15:8200` | BGE-M3 embedding server | atomic-server (for embeddings) |
+| `10.11.10.15:8100` | BGE-reranker-v2-m3 | brain_api gather (for precision rerank) |
+
+---
+
+## The big picture in one diagram
+
+```
+                          ┌─────────────────────────────────┐
+                          │       Didi backend              │
+                          │    (somewhere else)             │
+                          └─────┬───────────────────────────┘
+                                │ POST /v1/gather, /v1/search,
+                                │ /v1/fetch, /v1/ingest, ...
+                                ▼
+   ┌────────────────────────────────────────────────────────────┐
+   │ brain_api (container :8090)                                 │
+   │  ┌──────────────────────────────────────────────────────┐   │
+   │  │ FastAPI app (app.py)                                 │   │
+   │  │   routes → services/{gather, search, fetch, ingest, │   │
+   │  │            mapping, nli}                            │   │
+   │  └──────────────────────────────────────────────────────┘   │
+   │  imports → shared/{config, llm_client, embedding_client,    │
+   │            atomic_api, taxonomy, logging}                   │
+   └─────┬─────────────┬──────────────┬──────────────────────────┘
+         │             │              │
+         │ HTTP        │ HTTPS        │ HTTPS
+         ▼             ▼              ▼
+   ┌───────────┐ ┌──────────┐ ┌────────────────┐
+   │  atomic-  │ │ BGE-M3   │ │ Qwen 397B      │
+   │  server   │ │ + rerank │ │ via LLM router │
+   │  (cont.)  │ │ (VPN)    │ │ (VPN)          │
+   └─────┬─────┘ └──────────┘ └────────────────┘
+         │
+         ▼
+   ┌───────────────┐
+   │ postgres      │
+   │ + pgvector    │
+   │ (container)   │
+   └───────────────┘
+
+Operator scripts run on the host, talk over HTTP:
+
+   ┌─────────────────────┐
+   │ host venv           │       ┌──────────┐
+   │ scripts/01..11.py   │──HTTP─▶│ brain    │
+   │ + bootstrap.sh      │       │ stack    │
+   └─────────────────────┘       └──────────┘
+```
+
+---
+
+# Module deep-dives
+
+Each module section answers:
+
+1. **Purpose** — one paragraph
+2. **When it runs** — request-time / startup / background / operator
+3. **Files inside** — what each does
+4. **Imports** — what it depends on
+5. **Imported by** — who depends on it
+
+## `shared/` — the library every other module imports
+
+**Purpose.** Pure-Python utilities and clients with no HTTP surface of their
+own. Single source of truth for config, logging, all upstream clients, and
+the canonical tag taxonomy. Everything else in the repo treats this as a
+read-only library.
+
+**When it runs.** Imported at process startup; no long-lived behavior of
+its own.
+
+**Files inside.**
+
+| File | Role |
+|---|---|
+| `config.py` | Pydantic Settings singleton. Reads `.env` + env vars, validates types, exposes `settings.X` everywhere. Defines `LlmRole` enum and `model_for(role)` for routing. |
+| `logging.py` | Sets up structlog once per process. Forces UTF-8 stdout (Windows fix), silences noisy `httpx`/`httpcore` loggers. Call `setup_logging()` once in `main()`. |
+| `llm_client.py` | Async OpenAI-compat HTTP client targeting `LLM_ROUTER_URL`. Exposes `chat`, `chat_text`, `chat_json`, `chat_label`. tenacity-based retries on HTTP/timeout errors. |
+| `embedding_client.py` | Async client for BGE-M3 embeddings (`/v1/embeddings`) and BGE reranker (`/v1/rerank`). Exposes `embed`, `embed_one`, `rerank`. Has the `cosine()` helper. |
+| `atomic_api.py` | Typed REST client for atomic-server. Wraps the endpoints we actually use: `health`, `settings` get/set, `atoms` CRUD + by-source-url, `tags` list/create, `search`, `find_similar`. Returns dataclasses (`SearchHit`, `AtomSummary`) or raw dicts. |
+| `taxonomy.py` | Defines the canonical `TAXONOMY` tree (~80 paths), exposes `walk()`, `all_paths()`, `build_path_map_from_tags()`, and the `TagResolver` class for `path → uuid` lookup. Also `load_from_mapping()` for in-memory refresh (used by brain_api at startup). |
+
+**Imports.** Standard library + `httpx` + `pydantic` + `pydantic-settings` + `structlog` + `tenacity` + `rich`. Nothing internal.
+
+**Imported by.** `brain_api/`, `extractor/`, `lint/`, every script in `scripts/`.
+
+---
+
+## `brain_api/` — the HTTP service
+
+**Purpose.** Containerized FastAPI service that exposes the 5-endpoint
+contract Didi's backend already speaks. It does NOT contain business logic
+that another module reuses; it ties together `shared` + `extractor` (via
+`/v1/ingest`) and translates between Didi's HTTP shape and the brain's
+internal capabilities.
+
+**When it runs.** Always. One container per deployment, lifespan-managed by
+FastAPI/Uvicorn. The lifespan hook initializes the long-lived clients
+(`AtomicClient`, `EmbeddingClient`, `LlmClient`) and refreshes the
+`TagResolver` from atomic-server at startup so the image is portable.
+
+**Files inside.**
+
+| File | Role |
+|---|---|
+| `Dockerfile` | python:3.12-slim, non-root user `brain`, COPY shared/extractor/brain_api, curl healthcheck on /health, CMD `python -m brain_api.run`. |
+| `requirements.txt` | Pinned runtime deps. |
+| `__init__.py` | Module marker + docstring. |
+| `app.py` | The FastAPI app. Defines `lifespan` (startup/shutdown), the `/health` route, and the 5 v1 routes. Each route delegates to a function in `services/`. |
+| `run.py` | Uvicorn entry. Reads `BRAIN_API_HOST` and `BRAIN_API_PORT` from env (defaults 127.0.0.1:8090; container overrides to 0.0.0.0). |
+| `deps.py` | `AppState` dataclass + module-level singleton accessor. Set by lifespan, read by route handlers. |
+| `schemas.py` | **The contract.** Pydantic v2 models for every request/response shape the service speaks: `SearchRequest/Response`, `FetchRequest/Response`, `GatherRequest/Response`, `ImageSearchRequest/Response`, `IngestRequest/Response`, plus `EvidenceItem`, `BrainEvidenceMeta`, `BrainMeta`, etc. |
+| `prompts/nli_v1.md` | Versioned prompt template for the stance-vs-query NLI step in `/v1/gather`. |
+| `services/mapping.py` | Pure functions converting atomic atoms ↔ Didi's `EvidenceItem` / `FetchedPage` / `SearchResultItem`. Includes `parse_claim_atom_body()` (used by lint too) and the credibility-tag → score mapping. |
+| `services/gather.py` | The 5-stage pipeline for `/v1/gather`: context → retrieval → rerank → NLI → evidence build. Returns a `GatherResponse`. |
+| `services/search.py` | Lighter version of gather: just semantic search across queries, dedup by parent doc, return `SearchResponse`. |
+| `services/fetch.py` | Look up atoms by URL via `atomic.get_atom_by_source_url`. Returns `FetchResponse` with HIT pages and `failed_urls=[{url, "not_in_brain"}]` for misses. |
+| `services/ingest.py` | Accept `IngestRequest` (typically a web-module result), create `Type/Document` atoms with inferred tags, optionally schedule background extraction via FastAPI `BackgroundTasks`. |
+| `services/nli.py` | The stance-vs-query LLM call. `classify_one()` for a single (claim, evidence) pair, `classify_batch()` for parallel with `MAX_PARALLEL=2` (matches llamacpp backend concurrency). |
+
+**Imports.** `shared`, `extractor.batch` (lazily, only when ingest spawns extraction), `fastapi`, `uvicorn`.
+
+**Imported by.** Nothing — it's the leaf consumer. Didi backend talks to it over HTTP, not Python imports.
+
+---
+
+## `extractor/` — Document → Claim transformation
+
+**Purpose.** Reads `Type/Document` atoms from Atomic, asks Qwen 397B to
+extract atomic factual claims, validates each claim against the source
+text (substring check on the quote), and creates new `Type/Claim` atoms
+with canonical hash-based source URLs and inherited tags.
+
+**When it runs.** Two paths:
+
+1. **Operator-triggered**: `python scripts/07_run_extraction.py` runs
+   `extractor.batch.run_batch()` over every Document atom that hasn't been
+   processed at the current prompt version.
+2. **Background after ingest**: when `POST /v1/ingest` is called with
+   `run_extraction: true` (the default), the brain_api's ingest service
+   schedules a `BackgroundTasks` callback that calls
+   `extractor.batch.run_batch(only_atom_ids=...)` on just the atoms that
+   were just created. Fire-and-forget, doesn't block the HTTP response.
+
+Both paths share the same idempotency state (`extractor/_extracted.json`).
+
+**Files inside.**
+
+| File | Role |
+|---|---|
+| `__init__.py` | Module marker. |
+| `_state.py` | `DocExtractionRecord` dataclass + `ExtractionState` class with `has`, `get`, `upsert`, `save`, `_load`. JSON file at `extractor/_extracted.json`, host-only. |
+| `extract.py` | Single-document logic. `extract_claims_from_atom(llm, title, language, content)` → `ExtractionResult` with `valid: list[ExtractedClaim]` and `rejected: dict[reason → count]`. Includes the substring quote validation. |
+| `push.py` | `push_claim(atomic, parent_atom, claim, parent_title, resolver)` creates one `Type/Claim` atom. URL is `{parent_url}#claim={hash8}` for natural dedup. Inherits parent's Country/Topic/SourceType/Credibility/Language tags, adds Type/Claim + Stance/{X}. |
+| `batch.py` | The orchestrator. `run_batch(limit, only_atom_ids)` paginates Document atoms, fetches each fully (list_atoms returns summary only — gotcha!), runs extraction sequentially, pushes claims, saves state after every doc so Ctrl-C is recoverable. |
+| `prompts/claim_extraction_v1.md` | Versioned prompt that instructs Qwen 397B to return strict JSON with claims + verbatim quotes. Output format pinned. |
+
+**Imports.** `shared`, `httpx` (transitively).
+
+**Imported by.** `scripts/07_run_extraction.py`, `brain_api/services/ingest.py`.
+
+---
+
+## `lint/` — cross-corpus contradiction detection
+
+**Purpose.** Audit job that finds claim atoms in our corpus that
+contradict each other (or are paraphrases). For every claim, asks Atomic
+for its semantic neighbors, then asks Qwen 397B to classify each
+candidate pair as **EQUIVALENT**, **CONTRADICTORY**, or **INCOMPARABLE**.
+Stores all verdicts in a JSON ledger so re-runs only process new pairs.
+
+**When it runs.** Operator-triggered, typically as an overnight job
+(~1 pair / 1.5s × ~6000 pairs for 513 atoms ≈ 4 hours). Re-runs are
+incremental thanks to the ledger.
+
+**Files inside.**
+
+| File | Role |
+|---|---|
+| `__init__.py` | Module marker. |
+| `_state.py` | `PairVerdict` dataclass, `LintState` class (loads/saves the ledger atomically via tmp+rename). `pair_hash(a, b)` is canonical (sorted) so (A,B) and (B,A) hash the same. `LintStats` for per-run counters. |
+| `detector.py` | Single-pair classification. `classify_pair(llm, claim_a, claim_b)` → `PairClassification(label, confidence, error)`. Never raises; falls through to INCOMPARABLE on any error so the batch keeps going. |
+| `pairs.py` | `generate_candidates(atomic, source_atom_ids)` walks each source atom, calls `atomic.find_similar` with `MIN_PAIR_SIMILARITY=0.55`, filters to Type/Claim neighbors only, returns canonical de-duplicated pairs. |
+| `runner.py` | The orchestrator. `run_lint_pass(limit_atoms, force)` does the full flow: load atom ids → generate candidates → drop already-evaluated → bulk-fetch full atom bodies → classify with `MAX_PARALLEL=2` → save ledger every 25 verdicts → ALWAYS save on exit (even crash). |
+| `reporter.py` | Pretty printing with `rich`: `render_stats(stats)`, `render_contradictions(state, top_n, min_confidence)`, `render_equivalents(state)`. |
+| `prompts/pair_nli_v1.md` | Versioned prompt for pair classification. Same-language and cross-language are judged identically. |
+
+**Imports.** `shared`, `brain_api.services.mapping` (for `parse_claim_atom_body`).
+
+**Imported by.** `scripts/10_run_lint.py`, `scripts/11_show_contradictions.py`.
+
+---
+
+## `scripts/` — operator CLI tools
+
+**Purpose.** One-shot maintenance and bootstrap commands that operators
+run from the host venv. Each is self-contained, idempotent, prints a
+clean rich-formatted report, and exits with a meaningful code.
+
+**When it runs.** Operator types `python scripts/XX_*.py`.
+
+**Files inside.** All numbered for execution order during a fresh deploy.
+
+| Script | What it does | Idempotent? | Typical runtime |
+|---|---|---|---|
+| `01_sanity_full.py` | Validates upstream stack: pings router, lists models, runs JSON extraction + NLI + Romanian + cross-lingual + embed cosine + reranker. Writes a JSON report. | yes | ~10 s |
+| `02_bootstrap_atomic.py` | Waits for atomic-server, claims the instance via `/api/setup/claim` if needed, configures `provider=openai_compat` pointed at BGE-M3, disables `auto_tagging_enabled`, patches `ATOMIC_TOKEN` into `.env`. | yes | ~5 s |
+| `03_sanity_atomic.py` | End-to-end brain check: create test atom → wait for embedding → semantic search retrieves it → cleanup. Catches "atomic + Postgres + BGE-M3 wired" issues. | yes (cleanup deletes the atom) | ~5-10 s |
+| `04_seed_taxonomy.py` | Reads `shared.taxonomy.TAXONOMY`, creates any missing tags in atomic-server preserving parent_id structure, writes `shared/_tag_ids.json`. | yes | ~10 s |
+| `05_import_wikipedia_seed.py` | Imports a curated seed list of Wikipedia articles (currently vaccines EN+RO) via the MediaWiki action API, dedupes by canonical URL, creates Type/Document atoms with proper tags. | yes (URL dedup) | ~30 s for 19 articles |
+| `06_validate_queries.py` | 4 doc-level queries against `/api/search` with reranker, prints top hits. Sanity that retrieval works on the corpus. | yes (read-only) | ~5-10 s |
+| `07_run_extraction.py` | Calls `extractor.batch.run_batch()`. With `--limit N` for smoke tests. | yes (state file) | ~1 min/document |
+| `08_validate_claims.py` | 4 claim-level queries, splits doc/claim hits, reranks claims. Sanity that claim-level retrieval works. | yes | ~10-20 s |
+| `09_brain_api_demo.py` | Hits all 5 brain_api endpoints with realistic inputs, validates schema compliance and HIT/MISS cache_status. | yes | ~30 s |
+| `10_run_lint.py` | Calls `lint.runner.run_lint_pass()`. With `--limit N` to cap source atoms, `--force` to re-evaluate cached pairs. | yes (ledger) | ~30 s × N atoms |
+| `11_show_contradictions.py` | READ ONLY. Loads `lint/_contradictions.json`, renders top contradictions and (with `--equivalents`) paraphrase clusters. | yes | <1 s |
+| `bootstrap_deploy.sh` | Bash orchestrator for fresh-server deploy. Preflight → compose build+up → wait healthy → venv → 02 → 04 → optional 05+07 → smoke test. Idempotent, color-coded output. | yes | ~5 min + 15 min if importing corpus |
+
+**Imports.** Each script `sys.path.insert`s the project root then imports
+`shared`, `brain_api`, `extractor`, or `lint` as needed.
+
+**Imported by.** Nothing. They are leaf entry points.
+
+---
+
+## `infra/` — containerization
+
+**Purpose.** Defines the deployment topology.
+
+**Files inside.**
+
+| File | Role |
+|---|---|
+| `docker-compose.yml` | Three services: `postgres` (pgvector/pgvector:pg16), `atomic-server` (ghcr.io/kenforthewin/atomic-server:latest with overridden entrypoint to use `--data-dir` instead of legacy `--db-path`, and `ATOMIC_STORAGE=postgres` env), `brain-api` (built from `brain_api/Dockerfile`). Two named volumes (`didibrain-pg-data`, `didibrain-atomic-data`), one bridge network (`didibrain`). All services have `restart: unless-stopped` and healthchecks. |
+
+**When it runs.** `docker compose up` from the operator (or via `bootstrap_deploy.sh`).
+
+---
+
+# Data flow — three concrete scenarios
+
+The codebase has exactly three control flows worth memorizing.
+
+## Scenario 1 — request time: Didi calls `POST /v1/gather`
+
+```
+Didi backend
+   │
+   │ POST http://brain-host:8090/v1/gather
+   │ {"claim": "vaccinurile cauzeaza autism", "max_evidence": 5, "run_nli": true}
+   ▼
+brain_api/app.py: post_gather()
+   │
+   ├─▶ deps.get_state()  (returns shared AppState with clients)
+   │
+   ├─▶ services/gather.py: gather(req, atomic, embed, llm, resolver)
+   │
+   │      stage 1 context  → mapping.detect_language_simple(claim)
+   │                         (~0 ms)
+   │
+   │      stage 2 retrieval → atomic.search(claim, mode="semantic", limit=100)
+   │                         → filter to Type/Claim only
+   │                         (~300 ms; atomic embeds query via BGE-M3 internally)
+   │
+   │      stage 3 rerank   → atomic.get_atom(...) for top-15 claim atoms
+   │                         → embed.rerank(claim, claim_texts)
+   │                         (~1000 ms; calls reranker via BGE cross-encoder)
+   │
+   │      stage 4 nli      → mapping.parse_claim_atom_body() per top result
+   │                         → nli.classify_batch(llm, claim, evidence_texts)
+   │                         (parallel calls to Qwen 397B, ~3-4 sec; one per parent doc)
+   │
+   │      stage 5 evidence → atomic.get_atom_by_source_url() per parent doc
+   │                         → mapping.evidence_from_parent() builds EvidenceItem
+   │                         (~400 ms parallel)
+   │
+   ├─▶ Returns GatherResponse with brain_meta.cache_status set based on top
+   │   relevance score: >=0.6 HIT, 0.3-0.6 PARTIAL, <0.3 MISS.
+   │
+   ▼
+Didi backend receives JSON, branches on cache_status.
+```
+
+**Total**: ~5-6 sec with NLI, ~1.5 sec without.
+
+## Scenario 2 — operator time: fresh-server deployment
+
+```
+operator                                   what runs
+═══════                                    ══════════
+ssh server, cd ~/didibrain                 (none)
+cp .env.example .env, vim .env             (none)
+./scripts/bootstrap_deploy.sh
+   │
+   ├─ preflight                            docker, python3, curl available
+   ├─ docker compose build brain-api       brain_api/Dockerfile builds the image
+   ├─ docker compose up -d                 starts postgres, atomic-server, brain-api
+   ├─ wait_healthy x3                      polls docker inspect for "healthy"
+   ├─ python3 -m venv .venv                local Python venv
+   ├─ pip install ...                      operator deps in venv
+   ├─ python scripts/02_bootstrap_atomic.py
+   │    │
+   │    ├─ atomic.health()                 verify atomic responding
+   │    ├─ atomic.setup_status()           is the instance claimed?
+   │    ├─ atomic._request POST /api/setup/claim   if not, claim and get token
+   │    ├─ patch_env_token()               write ATOMIC_TOKEN= back to .env
+   │    └─ atomic.set_setting() x8         provider=openai_compat, BGE URLs, dim, etc.
+   │
+   ├─ python scripts/04_seed_taxonomy.py
+   │    │
+   │    ├─ atomic.list_tags()              what's already there
+   │    ├─ for each path in TAXONOMY: atomic.create_tag() if missing
+   │    └─ resolver.save()                 write shared/_tag_ids.json (host-only)
+   │
+   ├─ python scripts/05_import_wikipedia_seed.py    (optional, on by default)
+   │    │
+   │    ├─ for each title in SEED_EN + SEED_RO:
+   │    │    fetch_article() → atomic.create_atom() with proper tag_ids
+   │    └─ Atomic chunks + embeds in background via BGE-M3
+   │
+   ├─ python scripts/07_run_extraction.py           (optional, on by default)
+   │    │
+   │    ├─ list Type/Document atoms
+   │    ├─ for each not in extractor/_extracted.json:
+   │    │    fetch full atom → extract.extract_claims_from_atom() (Qwen 397B)
+   │    │    push.push_claim() per valid claim → atomic.create_atom()
+   │    └─ save state every doc
+   │
+   └─ smoke test: curl /health + curl /v1/gather, parse cache_status
+```
+
+## Scenario 3 — background time: Didi falls back to web module, posts result back
+
+```
+Didi backend
+   │
+   │ POST /v1/gather → cache_status=MISS
+   │
+   ├─ Didi backend calls live web module (expensive)
+   │
+   │ POST /v1/ingest with the web-module result
+   ▼
+brain_api/app.py: post_ingest()
+   │
+   ├─▶ services/ingest.py: ingest(req, atomic, resolver, background)
+   │
+   │      for each evidence in req.evidence:
+   │         atomic.get_atom_by_source_url()  (dedup check)
+   │         if not exists:
+   │            evidence_to_markdown(evidence)
+   │            build_tag_ids_for_evidence()  (Type/Document + Credibility from
+   │                                           credibility_score + Language)
+   │            atomic.create_atom()           (synchronous)
+   │
+   │      if req.run_extraction and created_ids:
+   │         background.add_task(_run_extraction_background, created_ids)
+   │
+   ├─▶ Returns IngestResponse immediately (extraction is fire-and-forget)
+   │
+   ▼
+   FastAPI BackgroundTasks: _run_extraction_background(created_ids)
+   │
+   └─▶ extractor.batch.run_batch(only_atom_ids=set(created_ids))
+          ├─ runs Qwen 397B claim extraction
+          ├─ pushes Type/Claim atoms
+          └─ saves extractor/_extracted.json
+```
+
+**Effect**: next time Didi asks the same claim, brain returns HIT and Didi
+doesn't need to call the web module.
+
+---
+
+# State files reference
+
+DidiBrain has **three host-only JSON state files** (gitignored). They are
+all idempotency ledgers — never primary data — so they can be regenerated
+from Atomic without loss.
+
+| File | Owner | Purpose | Recreatable? |
+|---|---|---|---|
+| `shared/_tag_ids.json` | `scripts/04_seed_taxonomy.py` | Maps canonical paths like `Country/Romania` → atom tag UUIDs. Used by every script that creates atoms. | yes — re-run script 04 |
+| `extractor/_extracted.json` | `scripts/07_run_extraction.py` (and brain_api ingest) | Per-document extraction log: which docs have been processed, with which prompt version, how many claims came out. Skip already-done docs on rerun. | yes — delete and re-run; will re-extract everything |
+| `lint/_contradictions.json` | `scripts/10_run_lint.py` | Per-pair verdict ledger with the EQUIVALENT / CONTRADICTORY / INCOMPARABLE labels. Skip already-done pairs on rerun. | yes — delete and re-run; will re-classify everything |
+
+The brain_api **container** does NOT use any of these files. It calls
+`atomic.list_tags()` at startup to refresh `TagResolver` in-memory, so
+the image stays portable.
+
+---
+
+# Environment variables — full table
+
+All loaded by `shared/config.py` via Pydantic Settings from `.env` at the
+project root. The brain_api container also receives these via
+`env_file: ../.env` in docker-compose, with `ATOMIC_URL` overridden to
+`http://atomic-server:8080` (Docker DNS).
+
+| Var | Default | What it controls | Required? |
+|---|---|---|---|
+| `LLM_ROUTER_URL` | `http://localhost:14011` | URL of the unified LLM router (vLLM + llamacpp behind it) | yes (set to `10.11.10.17:14011` in our setup) |
+| `LLM_ROUTER_API_KEY` | empty | optional bearer | no |
+| `LLM_VLLM_URL` | `http://localhost:14001` | direct vLLM URL (fallback only, currently unused) | no |
+| `LLM_LLAMACPP_URLS` | empty | comma-separated direct llamacpp URLs (fallback) | no |
+| `MODEL_REASONING` | `Qwen3.5-397B-A17B` | model id for extraction, NLI, gather | yes |
+| `MODEL_REASONING_BACKEND` | `llamacpp` | router backend hint | yes |
+| `MODEL_FAST` | `qwen3.5` | fast model id (currently disabled) | no |
+| `MODEL_FAST_BACKEND` | `vllm` | fast backend hint | no |
+| `MODEL_FAST_ENABLED` | `false` | enable fast model in routing | no |
+| `MODEL_VISION` | `gemma-3-27b-it` | vision model id (currently down) | no |
+| `MODEL_VISION_URL` | empty | direct vision endpoint | no |
+| `MODEL_VISION_ENABLED` | `false` | enable vision in routing | no |
+| `EMBEDDING_URL` | `http://10.11.10.15:8200` | BGE-M3 vLLM endpoint | yes |
+| `EMBEDDING_API_KEY` | empty | optional bearer | no |
+| `EMBEDDING_MODEL` | `BAAI/bge-m3` | model id sent in requests | yes |
+| `EMBEDDING_DIM` | `1024` | vector dimension (must match Atomic's setting) | yes |
+| `EMBEDDING_MAX_TOKENS` | `8192` | input length cap | yes |
+| `RERANKER_URL` | `http://10.11.10.15:8100` | BGE-reranker-v2-m3 endpoint | yes |
+| `RERANKER_API_KEY` | empty | optional bearer | no |
+| `RERANKER_MODEL` | `BAAI/bge-reranker-v2-m3` | model id sent in requests | yes |
+| `ATOMIC_URL` | `http://localhost:8088` | atomic-server URL (host scripts use this; brain_api container overrides to docker DNS) | yes |
+| `ATOMIC_TOKEN` | empty | API token, set by `02_bootstrap_atomic.py` | yes (auto-populated) |
+| `POSTGRES_USER` | `atomic` | Postgres user | yes |
+| `POSTGRES_PASSWORD` | `atomic_dev_changeme` | Postgres password (override on server!) | yes |
+| `POSTGRES_DB` | `atomic` | Postgres database name | yes |
+| `POSTGRES_PORT` | `5434` | host-side port mapping | yes |
+| `LOG_LEVEL` | `INFO` | structlog filter level | no |
+
+**Container-only env vars set inside the Dockerfile** (not in `.env`):
+
+| Var | Set to | Why |
+|---|---|---|
+| `BRAIN_API_HOST` | `0.0.0.0` | so Docker port mapping reaches uvicorn |
+| `BRAIN_API_PORT` | `8090` | container internal port |
+| `PYTHONIOENCODING` | `utf-8` | UTF-8 logging on Linux base image |
+
+---
+
+# Versioned prompts
+
+Three prompts. Each has its own version number. Bumping a version means
+re-running the relevant batch (extractor or lint) to re-process what's
+been done.
+
+| Prompt file | Used by | Output | Version |
+|---|---|---|---|
+| `extractor/prompts/claim_extraction_v1.md` | `extractor.extract.extract_claims_from_atom` | JSON list of `{claim, quote, stance, confidence}` from a document | v1 |
+| `brain_api/prompts/nli_v1.md` | `brain_api.services.nli.classify_one` | JSON `{label: SUPPORTS/CONTRADICTS/NEUTRAL, confidence}` | v1 |
+| `lint/prompts/pair_nli_v1.md` | `lint.detector.classify_pair` | JSON `{label: EQUIVALENT/CONTRADICTORY/INCOMPARABLE, confidence}` | v1 |
+
+To bump: copy the file, rename to `_v2.md`, update the constant
+`PROMPT_VERSION = "v2"` in the Python file, re-run the relevant batch.
+The state files (`extractor/_extracted.json`, `lint/_contradictions.json`)
+include `prompt_version` per record, so re-runs only re-process items at
+the old version.
+
+---
+
+# When something breaks — start here
+
+```
+Something with /v1/gather failing?
+   │
+   ├─ check brain_api/app.py route handler — it just delegates
+   ├─ → services/gather.py — the 5-stage pipeline
+   ├─ → services/mapping.py — atom shape conversion
+   └─ → shared/atomic_api.py + embedding_client.py + llm_client.py — upstream calls
+       │
+       └─ run scripts/01_sanity_full.py first to verify upstream stack
+
+Atomic returning 500?
+   │
+   ├─ docker logs didibrain-atomic — likely BGE unreachable
+   └─ curl http://10.11.10.15:8200/v1/models on host — VPN check
+
+Claim extraction acting weird?
+   │
+   ├─ extractor/_extracted.json — see what's been processed
+   ├─ extractor/extract.py — substring quote validation may be rejecting
+   └─ extractor/prompts/claim_extraction_v1.md — prompt drift?
+
+Lint pass slow / hanging?
+   │
+   ├─ lint/runner.py MAX_PARALLEL — must match llamacpp backend count (we have 2)
+   ├─ lint/detector.py PER_CALL_TIMEOUT_S — currently 30s
+   └─ lint/_contradictions.json saved every 25 — Ctrl-C safe to interrupt
+
+Atomic doesn't have my tag?
+   │
+   ├─ shared/taxonomy.py — is the path defined in TAXONOMY?
+   ├─ shared/_tag_ids.json — is it in the cache?
+   └─ run scripts/04_seed_taxonomy.py to refresh both
+```
+
+For deeper troubleshooting see `STATUS.md` "Troubleshooting" section.
+
+---
+
+# Verification Cache (added 2026-04-23)
+
+## What
+
+A relational layer **alongside** the knowledge graph (Atomic/atoms) that stores
+the LLM verification result produced by didi-backend after it runs NLI + verdict
+analysis on evidence. This lets backend skip its expensive LLM call on
+subsequent requests for the same `(claim, tier)` pair.
+
+**Backend owns the prompt + model + status logic.** Brain is a pure cache: it
+stores the opaque JSON blob backend produces, and returns it 1:1 on read.
+Zero prompt sync between the two sides.
+
+## Table: `brain_verification_cache`
+
+Lives in the **same Postgres instance atomic-server uses** — we share the DB
+connection with `brain_` prefix for trivial isolation. Asyncpg pool connects
+directly (doesn't go through atomic-server).
+
+Column summary:
+
+| Column | Purpose |
+|---|---|
+| `claim_hash` | `sha256(normalize_claim(claim_text))` — normalization strips diacritics, lowercases, collapses whitespace, trims trailing `.?!` |
+| `tier` | `'free'` or `'premium'` — **separate cache per tier** (different LLMs used) |
+| `evidence_hash` | `sha256(sorted(lowercased_urls))` — **metadata only**, not part of the key |
+| `evidence_urls` | jsonb array of URLs the verification was written for |
+| `prompt_hash` | Backend's current prompt hash; used for staleness detection on read |
+| `framework_version` | Backend's current thresholds hash; used for staleness on read |
+| `verification_processed` | jsonb — UI-ready payload (status + sources + reasoning) |
+| `verification_raw` | jsonb — raw LLM output; lets backend recompute when framework changes |
+| `expires_at` | TTL, default 30 days from write |
+
+**Unique key:** `(claim_hash, tier)`. UPSERT last-wins.
+
+Why not `(claim_hash, evidence_hash, tier)`? Because the URLs backend has at
+write-time rarely match what brain's `/v1/gather` returns at read-time
+(brain uses its own semantic retrieval, which may rank differently than
+backend's original source list). Including evidence in the key made the
+cache unreachable. v2 drops it.
+
+## Flow
+
+### Write (backend → brain, fire-and-forget)
+
+```
+backend runs its LLM verification call
+     │
+     ▼
+POST /v1/verification_cache
+  claim, evidence_urls, tier,
+  model, prompt_hash, framework_version,
+  verification_raw, verification_processed
+     │
+     ▼
+brain normalizes + hashes + UPSERTs row (claim_hash, tier)
+     │
+     ▼
+200 OK (cached: true, claim_hash, evidence_hash, expires_at)
+```
+
+### Read (via /v1/gather extension)
+
+```
+backend wants to check if verification exists for this claim
+     │
+     ▼
+POST /v1/gather
+  claim, include_verification: true, tier,
+  prompt_hash, framework_version
+     │
+     ▼
+brain runs its normal gather (retrieval + rerank + evidence)
+     │
+     ▼
+brain queries brain_verification_cache (claim_hash, tier)
+     │
+     ▼
+returns GatherResponse with brain_meta.verification_staleness:
+  - miss           → null; backend runs LLM + writes cache
+  - stale_prompt   → null; backend runs LLM + writes cache
+  - stale_framework→ returns verification_raw; backend recomputes status locally (zero LLM)
+  - fresh          → returns verification_processed; direct to UI
+```
+
+See `CONTRACT_VERIFICATION_CACHE.md` for the full field-by-field spec.
+
+## Files
+
+- `brain_api/db.py` — asyncpg pool + idempotent schema migrations (runs in lifespan)
+- `brain_api/services/verification_cache.py` — normalize, hash, upsert, lookup, freshness decider
+- `brain_api/schemas.py` — `GatherRequest` extension, `VerificationCacheWriteRequest/Response`, `BrainMeta` extension
+- `brain_api/app.py` — `POST /v1/verification_cache` endpoint wired in lifespan
+- `brain_api/services/gather.py` — cache lookup after evidence build, attach to `brain_meta`
+
+---
+
+# Glossary
+
+| Term | Meaning |
+|---|---|
+| **atom** | The unit of storage in Atomic. Markdown content + metadata + tags. Two flavors: `Type/Document` (a full source article) and `Type/Claim` (an atomic factual claim extracted from a document). |
+| **Atomic** | The Rust knowledge-graph backend (kenforthewin/atomic). DidiBrain uses its REST API. Runs as `didibrain-atomic` container. |
+| **brain_api** | Our FastAPI service. The HTTP face DidiBrain shows to Didi backend. |
+| **chunk** | A piece of an atom's content split for embedding. Atomic does this internally; we never see chunks directly. |
+| **HIT / PARTIAL / MISS** | `brain_meta.cache_status` returned by `/v1/gather`. Tells Didi backend whether the brain had relevant knowledge or it should fall back to web module. |
+| **Lint pass** | Background contradiction-detection job. Reads existing claim atoms, finds contradictory pairs cross-corpus. |
+| **NLI** | Natural Language Inference. The "decide if A supports / contradicts / is neutral toward B" classification. We do it twice: stance-vs-query (gather) and pair-equivalence (lint). |
+| **prompt_version** | A string like "v1" pinned per prompt file. State files record it so we can bump prompts and re-process selectively. |
+| **reranker** | Cross-encoder (BGE-reranker-v2-m3). Takes (query, document) pairs and outputs a precise relevance score. Used after embedding retrieval to refine top-K. |
+| **stance_in_source** | What the source itself asserts about a claim (ASSERTS/REPORTS/REFUTES/QUESTIONS/NEUTRAL). Set during extraction. |
+| **stance_vs_query** | What an evidence claim says about Didi user's input claim (SUPPORTS/CONTRADICTS/NEUTRAL). Set during gather's NLI stage. |
+| **TagResolver** | Helper that maps canonical paths like `Country/Romania` to Atomic tag UUIDs. Loaded from `_tag_ids.json` (scripts) or refreshed from atomic at startup (brain_api container). |
+| **TAXONOMY** | The canonical tree in `shared/taxonomy.py`. Single source of truth for what tags exist. |