645 lines
35 KiB
Markdown
645 lines
35 KiB
Markdown
# DidiBrain — Architecture & "Who Does What"
|
||
|
||
This is the **codebase rosetta stone**. If you read this top-to-bottom you
|
||
should be able to answer "what does file X do, when does it run, who calls
|
||
it, what does it depend on" for any file in the repo.
|
||
|
||
> **Reading order if you're new**:
|
||
> 1. `README.md` — what DidiBrain is in 2 minutes
|
||
> 2. **this file** — full mental model of the codebase (15 minutes)
|
||
> 3. `STATUS.md` — operational state, deployment, troubleshooting
|
||
> 4. `brain_api/schemas.py` — the literal HTTP contract with Didi
|
||
> 5. `brain_api/services/gather.py` — the main pipeline in 200 lines
|
||
|
||
---
|
||
|
||
## TL;DR — "WHO DOES WHAT" in one table
|
||
|
||
| Module / File | What it does | When it runs | Who calls it |
|
||
|---|---|---|---|
|
||
| **`infra/docker-compose.yml`** | Defines the 3-container stack | `docker compose up` | operator |
|
||
| **`brain_api/`** *(container)* | HTTP service speaking Didi's contract | every Didi request | Didi backend (HTTP) |
|
||
| **`shared/`** *(library)* | Common code: config, clients, taxonomy | imported everywhere | brain_api, extractor, lint, scripts |
|
||
| **`extractor/`** *(library + jobs)* | Turns Document atoms into Claim atoms | operator (script 07) OR background after `/v1/ingest` | scripts/07, brain_api ingest |
|
||
| **`lint/`** *(library + job)* | Detects contradictions across the corpus | operator overnight (script 10) | scripts/10, scripts/11 |
|
||
| **`scripts/`** *(operator CLI)* | One-shot maintenance + bootstrap commands | operator types `python scripts/XX_*.py` | human |
|
||
| **Atomic server** *(container)* | Storage + chunking + embedding pipeline + REST | always | brain_api (HTTP), scripts (HTTP) |
|
||
| **Postgres + pgvector** *(container)* | Persistent atom + vector storage | always | atomic-server (SQL) |
|
||
|
||
**External services** (not in our repo, on VPN):
|
||
| Where | What | Used by |
|
||
|---|---|---|
|
||
| `10.11.10.17:14011` | LLM router → Qwen 397B | brain_api NLI, extractor, lint |
|
||
| `10.11.10.15:8200` | BGE-M3 embedding server | atomic-server (for embeddings) |
|
||
| `10.11.10.15:8100` | BGE-reranker-v2-m3 | brain_api gather (for precision rerank) |
|
||
|
||
---
|
||
|
||
## The big picture in one diagram
|
||
|
||
```
|
||
┌─────────────────────────────────┐
|
||
│ Didi backend │
|
||
│ (somewhere else) │
|
||
└─────┬───────────────────────────┘
|
||
│ POST /v1/gather, /v1/search,
|
||
│ /v1/fetch, /v1/ingest, ...
|
||
▼
|
||
┌────────────────────────────────────────────────────────────┐
|
||
│ brain_api (container :8090) │
|
||
│ ┌──────────────────────────────────────────────────────┐ │
|
||
│ │ FastAPI app (app.py) │ │
|
||
│ │ routes → services/{gather, search, fetch, ingest, │ │
|
||
│ │ mapping, nli} │ │
|
||
│ └──────────────────────────────────────────────────────┘ │
|
||
│ imports → shared/{config, llm_client, embedding_client, │
|
||
│ atomic_api, taxonomy, logging} │
|
||
└─────┬─────────────┬──────────────┬──────────────────────────┘
|
||
│ │ │
|
||
│ HTTP │ HTTPS │ HTTPS
|
||
▼ ▼ ▼
|
||
┌───────────┐ ┌──────────┐ ┌────────────────┐
|
||
│ atomic- │ │ BGE-M3 │ │ Qwen 397B │
|
||
│ server │ │ + rerank │ │ via LLM router │
|
||
│ (cont.) │ │ (VPN) │ │ (VPN) │
|
||
└─────┬─────┘ └──────────┘ └────────────────┘
|
||
│
|
||
▼
|
||
┌───────────────┐
|
||
│ postgres │
|
||
│ + pgvector │
|
||
│ (container) │
|
||
└───────────────┘
|
||
|
||
Operator scripts run on the host, talk over HTTP:
|
||
|
||
┌─────────────────────┐
|
||
│ host venv │ ┌──────────┐
|
||
│ scripts/01..11.py │──HTTP─▶│ brain │
|
||
│ + bootstrap.sh │ │ stack │
|
||
└─────────────────────┘ └──────────┘
|
||
```
|
||
|
||
---
|
||
|
||
# Module deep-dives
|
||
|
||
Each module section answers:
|
||
|
||
1. **Purpose** — one paragraph
|
||
2. **When it runs** — request-time / startup / background / operator
|
||
3. **Files inside** — what each does
|
||
4. **Imports** — what it depends on
|
||
5. **Imported by** — who depends on it
|
||
|
||
## `shared/` — the library every other module imports
|
||
|
||
**Purpose.** Pure-Python utilities and clients with no HTTP surface of their
|
||
own. Single source of truth for config, logging, all upstream clients, and
|
||
the canonical tag taxonomy. Everything else in the repo treats this as a
|
||
read-only library.
|
||
|
||
**When it runs.** Imported at process startup; no long-lived behavior of
|
||
its own.
|
||
|
||
**Files inside.**
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `config.py` | Pydantic Settings singleton. Reads `.env` + env vars, validates types, exposes `settings.X` everywhere. Defines `LlmRole` enum and `model_for(role)` for routing. |
|
||
| `logging.py` | Sets up structlog once per process. Forces UTF-8 stdout (Windows fix), silences noisy `httpx`/`httpcore` loggers. Call `setup_logging()` once in `main()`. |
|
||
| `llm_client.py` | Async OpenAI-compat HTTP client targeting `LLM_ROUTER_URL`. Exposes `chat`, `chat_text`, `chat_json`, `chat_label`. tenacity-based retries on HTTP/timeout errors. |
|
||
| `embedding_client.py` | Async client for BGE-M3 embeddings (`/v1/embeddings`) and BGE reranker (`/v1/rerank`). Exposes `embed`, `embed_one`, `rerank`. Has the `cosine()` helper. |
|
||
| `atomic_api.py` | Typed REST client for atomic-server. Wraps the endpoints we actually use: `health`, `settings` get/set, `atoms` CRUD + by-source-url, `tags` list/create, `search`, `find_similar`. Returns dataclasses (`SearchHit`, `AtomSummary`) or raw dicts. |
|
||
| `taxonomy.py` | Defines the canonical `TAXONOMY` tree (~80 paths), exposes `walk()`, `all_paths()`, `build_path_map_from_tags()`, and the `TagResolver` class for `path → uuid` lookup. Also `load_from_mapping()` for in-memory refresh (used by brain_api at startup). |
|
||
|
||
**Imports.** Standard library + `httpx` + `pydantic` + `pydantic-settings` + `structlog` + `tenacity` + `rich`. Nothing internal.
|
||
|
||
**Imported by.** `brain_api/`, `extractor/`, `lint/`, every script in `scripts/`.
|
||
|
||
---
|
||
|
||
## `brain_api/` — the HTTP service
|
||
|
||
**Purpose.** Containerized FastAPI service that exposes the 5-endpoint
|
||
contract Didi's backend already speaks. It does NOT contain business logic
|
||
that another module reuses; it ties together `shared` + `extractor` (via
|
||
`/v1/ingest`) and translates between Didi's HTTP shape and the brain's
|
||
internal capabilities.
|
||
|
||
**When it runs.** Always. One container per deployment, lifespan-managed by
|
||
FastAPI/Uvicorn. The lifespan hook initializes the long-lived clients
|
||
(`AtomicClient`, `EmbeddingClient`, `LlmClient`) and refreshes the
|
||
`TagResolver` from atomic-server at startup so the image is portable.
|
||
|
||
**Files inside.**
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `Dockerfile` | python:3.12-slim, non-root user `brain`, COPY shared/extractor/brain_api, curl healthcheck on /health, CMD `python -m brain_api.run`. |
|
||
| `requirements.txt` | Pinned runtime deps. |
|
||
| `__init__.py` | Module marker + docstring. |
|
||
| `app.py` | The FastAPI app. Defines `lifespan` (startup/shutdown), the `/health` route, and the 5 v1 routes. Each route delegates to a function in `services/`. |
|
||
| `run.py` | Uvicorn entry. Reads `BRAIN_API_HOST` and `BRAIN_API_PORT` from env (defaults 127.0.0.1:8090; container overrides to 0.0.0.0). |
|
||
| `deps.py` | `AppState` dataclass + module-level singleton accessor. Set by lifespan, read by route handlers. |
|
||
| `schemas.py` | **The contract.** Pydantic v2 models for every request/response shape the service speaks: `SearchRequest/Response`, `FetchRequest/Response`, `GatherRequest/Response`, `ImageSearchRequest/Response`, `IngestRequest/Response`, plus `EvidenceItem`, `BrainEvidenceMeta`, `BrainMeta`, etc. |
|
||
| `prompts/nli_v1.md` | Versioned prompt template for the stance-vs-query NLI step in `/v1/gather`. |
|
||
| `services/mapping.py` | Pure functions converting atomic atoms ↔ Didi's `EvidenceItem` / `FetchedPage` / `SearchResultItem`. Includes `parse_claim_atom_body()` (used by lint too) and the credibility-tag → score mapping. |
|
||
| `services/gather.py` | The 5-stage pipeline for `/v1/gather`: context → retrieval → rerank → NLI → evidence build. Returns a `GatherResponse`. |
|
||
| `services/search.py` | Lighter version of gather: just semantic search across queries, dedup by parent doc, return `SearchResponse`. |
|
||
| `services/fetch.py` | Look up atoms by URL via `atomic.get_atom_by_source_url`. Returns `FetchResponse` with HIT pages and `failed_urls=[{url, "not_in_brain"}]` for misses. |
|
||
| `services/ingest.py` | Accept `IngestRequest` (typically a web-module result), create `Type/Document` atoms with inferred tags, optionally schedule background extraction via FastAPI `BackgroundTasks`. |
|
||
| `services/nli.py` | The stance-vs-query LLM call. `classify_one()` for a single (claim, evidence) pair, `classify_batch()` for parallel with `MAX_PARALLEL=2` (matches llamacpp backend concurrency). |
|
||
|
||
**Imports.** `shared`, `extractor.batch` (lazily, only when ingest spawns extraction), `fastapi`, `uvicorn`.
|
||
|
||
**Imported by.** Nothing — it's the leaf consumer. Didi backend talks to it over HTTP, not Python imports.
|
||
|
||
---
|
||
|
||
## `extractor/` — Document → Claim transformation
|
||
|
||
**Purpose.** Reads `Type/Document` atoms from Atomic, asks Qwen 397B to
|
||
extract atomic factual claims, validates each claim against the source
|
||
text (substring check on the quote), and creates new `Type/Claim` atoms
|
||
with canonical hash-based source URLs and inherited tags.
|
||
|
||
**When it runs.** Two paths:
|
||
|
||
1. **Operator-triggered**: `python scripts/07_run_extraction.py` runs
|
||
`extractor.batch.run_batch()` over every Document atom that hasn't been
|
||
processed at the current prompt version.
|
||
2. **Background after ingest**: when `POST /v1/ingest` is called with
|
||
`run_extraction: true` (the default), the brain_api's ingest service
|
||
schedules a `BackgroundTasks` callback that calls
|
||
`extractor.batch.run_batch(only_atom_ids=...)` on just the atoms that
|
||
were just created. Fire-and-forget, doesn't block the HTTP response.
|
||
|
||
Both paths share the same idempotency state (`extractor/_extracted.json`).
|
||
|
||
**Files inside.**
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `__init__.py` | Module marker. |
|
||
| `_state.py` | `DocExtractionRecord` dataclass + `ExtractionState` class with `has`, `get`, `upsert`, `save`, `_load`. JSON file at `extractor/_extracted.json`, host-only. |
|
||
| `extract.py` | Single-document logic. `extract_claims_from_atom(llm, title, language, content)` → `ExtractionResult` with `valid: list[ExtractedClaim]` and `rejected: dict[reason → count]`. Includes the substring quote validation. |
|
||
| `push.py` | `push_claim(atomic, parent_atom, claim, parent_title, resolver)` creates one `Type/Claim` atom. URL is `{parent_url}#claim={hash8}` for natural dedup. Inherits parent's Country/Topic/SourceType/Credibility/Language tags, adds Type/Claim + Stance/{X}. |
|
||
| `batch.py` | The orchestrator. `run_batch(limit, only_atom_ids)` paginates Document atoms, fetches each fully (list_atoms returns summary only — gotcha!), runs extraction sequentially, pushes claims, saves state after every doc so Ctrl-C is recoverable. |
|
||
| `prompts/claim_extraction_v1.md` | Versioned prompt that instructs Qwen 397B to return strict JSON with claims + verbatim quotes. Output format pinned. |
|
||
|
||
**Imports.** `shared`, `httpx` (transitively).
|
||
|
||
**Imported by.** `scripts/07_run_extraction.py`, `brain_api/services/ingest.py`.
|
||
|
||
---
|
||
|
||
## `lint/` — cross-corpus contradiction detection
|
||
|
||
**Purpose.** Audit job that finds claim atoms in our corpus that
|
||
contradict each other (or are paraphrases). For every claim, asks Atomic
|
||
for its semantic neighbors, then asks Qwen 397B to classify each
|
||
candidate pair as **EQUIVALENT**, **CONTRADICTORY**, or **INCOMPARABLE**.
|
||
Stores all verdicts in a JSON ledger so re-runs only process new pairs.
|
||
|
||
**When it runs.** Operator-triggered, typically as an overnight job
|
||
(~1 pair / 1.5s × ~6000 pairs for 513 atoms ≈ 4 hours). Re-runs are
|
||
incremental thanks to the ledger.
|
||
|
||
**Files inside.**
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `__init__.py` | Module marker. |
|
||
| `_state.py` | `PairVerdict` dataclass, `LintState` class (loads/saves the ledger atomically via tmp+rename). `pair_hash(a, b)` is canonical (sorted) so (A,B) and (B,A) hash the same. `LintStats` for per-run counters. |
|
||
| `detector.py` | Single-pair classification. `classify_pair(llm, claim_a, claim_b)` → `PairClassification(label, confidence, error)`. Never raises; falls through to INCOMPARABLE on any error so the batch keeps going. |
|
||
| `pairs.py` | `generate_candidates(atomic, source_atom_ids)` walks each source atom, calls `atomic.find_similar` with `MIN_PAIR_SIMILARITY=0.55`, filters to Type/Claim neighbors only, returns canonical de-duplicated pairs. |
|
||
| `runner.py` | The orchestrator. `run_lint_pass(limit_atoms, force)` does the full flow: load atom ids → generate candidates → drop already-evaluated → bulk-fetch full atom bodies → classify with `MAX_PARALLEL=2` → save ledger every 25 verdicts → ALWAYS save on exit (even crash). |
|
||
| `reporter.py` | Pretty printing with `rich`: `render_stats(stats)`, `render_contradictions(state, top_n, min_confidence)`, `render_equivalents(state)`. |
|
||
| `prompts/pair_nli_v1.md` | Versioned prompt for pair classification. Same-language and cross-language are judged identically. |
|
||
|
||
**Imports.** `shared`, `brain_api.services.mapping` (for `parse_claim_atom_body`).
|
||
|
||
**Imported by.** `scripts/10_run_lint.py`, `scripts/11_show_contradictions.py`.
|
||
|
||
---
|
||
|
||
## `scripts/` — operator CLI tools
|
||
|
||
**Purpose.** One-shot maintenance and bootstrap commands that operators
|
||
run from the host venv. Each is self-contained, idempotent, prints a
|
||
clean rich-formatted report, and exits with a meaningful code.
|
||
|
||
**When it runs.** Operator types `python scripts/XX_*.py`.
|
||
|
||
**Files inside.** All numbered for execution order during a fresh deploy.
|
||
|
||
| Script | What it does | Idempotent? | Typical runtime |
|
||
|---|---|---|---|
|
||
| `01_sanity_full.py` | Validates upstream stack: pings router, lists models, runs JSON extraction + NLI + Romanian + cross-lingual + embed cosine + reranker. Writes a JSON report. | yes | ~10 s |
|
||
| `02_bootstrap_atomic.py` | Waits for atomic-server, claims the instance via `/api/setup/claim` if needed, configures `provider=openai_compat` pointed at BGE-M3, disables `auto_tagging_enabled`, patches `ATOMIC_TOKEN` into `.env`. | yes | ~5 s |
|
||
| `03_sanity_atomic.py` | End-to-end brain check: create test atom → wait for embedding → semantic search retrieves it → cleanup. Catches "atomic + Postgres + BGE-M3 wired" issues. | yes (cleanup deletes the atom) | ~5-10 s |
|
||
| `04_seed_taxonomy.py` | Reads `shared.taxonomy.TAXONOMY`, creates any missing tags in atomic-server preserving parent_id structure, writes `shared/_tag_ids.json`. | yes | ~10 s |
|
||
| `05_import_wikipedia_seed.py` | Imports a curated seed list of Wikipedia articles (currently vaccines EN+RO) via the MediaWiki action API, dedupes by canonical URL, creates Type/Document atoms with proper tags. | yes (URL dedup) | ~30 s for 19 articles |
|
||
| `06_validate_queries.py` | 4 doc-level queries against `/api/search` with reranker, prints top hits. Sanity that retrieval works on the corpus. | yes (read-only) | ~5-10 s |
|
||
| `07_run_extraction.py` | Calls `extractor.batch.run_batch()`. With `--limit N` for smoke tests. | yes (state file) | ~1 min/document |
|
||
| `08_validate_claims.py` | 4 claim-level queries, splits doc/claim hits, reranks claims. Sanity that claim-level retrieval works. | yes | ~10-20 s |
|
||
| `09_brain_api_demo.py` | Hits all 5 brain_api endpoints with realistic inputs, validates schema compliance and HIT/MISS cache_status. | yes | ~30 s |
|
||
| `10_run_lint.py` | Calls `lint.runner.run_lint_pass()`. With `--limit N` to cap source atoms, `--force` to re-evaluate cached pairs. | yes (ledger) | ~30 s × N atoms |
|
||
| `11_show_contradictions.py` | READ ONLY. Loads `lint/_contradictions.json`, renders top contradictions and (with `--equivalents`) paraphrase clusters. | yes | <1 s |
|
||
| `bootstrap_deploy.sh` | Bash orchestrator for fresh-server deploy. Preflight → compose build+up → wait healthy → venv → 02 → 04 → optional 05+07 → smoke test. Idempotent, color-coded output. | yes | ~5 min + 15 min if importing corpus |
|
||
|
||
**Imports.** Each script `sys.path.insert`s the project root then imports
|
||
`shared`, `brain_api`, `extractor`, or `lint` as needed.
|
||
|
||
**Imported by.** Nothing. They are leaf entry points.
|
||
|
||
---
|
||
|
||
## `infra/` — containerization
|
||
|
||
**Purpose.** Defines the deployment topology.
|
||
|
||
**Files inside.**
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `docker-compose.yml` | Three services: `postgres` (pgvector/pgvector:pg16), `atomic-server` (ghcr.io/kenforthewin/atomic-server:latest with overridden entrypoint to use `--data-dir` instead of legacy `--db-path`, and `ATOMIC_STORAGE=postgres` env), `brain-api` (built from `brain_api/Dockerfile`). Two named volumes (`didibrain-pg-data`, `didibrain-atomic-data`), one bridge network (`didibrain`). All services have `restart: unless-stopped` and healthchecks. |
|
||
|
||
**When it runs.** `docker compose up` from the operator (or via `bootstrap_deploy.sh`).
|
||
|
||
---
|
||
|
||
# Data flow — three concrete scenarios
|
||
|
||
The codebase has exactly three control flows worth memorizing.
|
||
|
||
## Scenario 1 — request time: Didi calls `POST /v1/gather`
|
||
|
||
```
|
||
Didi backend
|
||
│
|
||
│ POST http://brain-host:8090/v1/gather
|
||
│ {"claim": "vaccinurile cauzeaza autism", "max_evidence": 5, "run_nli": true}
|
||
▼
|
||
brain_api/app.py: post_gather()
|
||
│
|
||
├─▶ deps.get_state() (returns shared AppState with clients)
|
||
│
|
||
├─▶ services/gather.py: gather(req, atomic, embed, llm, resolver)
|
||
│
|
||
│ stage 1 context → mapping.detect_language_simple(claim)
|
||
│ (~0 ms)
|
||
│
|
||
│ stage 2 retrieval → atomic.search(claim, mode="semantic", limit=100)
|
||
│ → filter to Type/Claim only
|
||
│ (~300 ms; atomic embeds query via BGE-M3 internally)
|
||
│
|
||
│ stage 3 rerank → atomic.get_atom(...) for top-15 claim atoms
|
||
│ → embed.rerank(claim, claim_texts)
|
||
│ (~1000 ms; calls reranker via BGE cross-encoder)
|
||
│
|
||
│ stage 4 nli → mapping.parse_claim_atom_body() per top result
|
||
│ → nli.classify_batch(llm, claim, evidence_texts)
|
||
│ (parallel calls to Qwen 397B, ~3-4 sec; one per parent doc)
|
||
│
|
||
│ stage 5 evidence → atomic.get_atom_by_source_url() per parent doc
|
||
│ → mapping.evidence_from_parent() builds EvidenceItem
|
||
│ (~400 ms parallel)
|
||
│
|
||
├─▶ Returns GatherResponse with brain_meta.cache_status set based on top
|
||
│ relevance score: >=0.6 HIT, 0.3-0.6 PARTIAL, <0.3 MISS.
|
||
│
|
||
▼
|
||
Didi backend receives JSON, branches on cache_status.
|
||
```
|
||
|
||
**Total**: ~5-6 sec with NLI, ~1.5 sec without.
|
||
|
||
## Scenario 2 — operator time: fresh-server deployment
|
||
|
||
```
|
||
operator what runs
|
||
═══════ ══════════
|
||
ssh server, cd ~/didibrain (none)
|
||
cp .env.example .env, vim .env (none)
|
||
./scripts/bootstrap_deploy.sh
|
||
│
|
||
├─ preflight docker, python3, curl available
|
||
├─ docker compose build brain-api brain_api/Dockerfile builds the image
|
||
├─ docker compose up -d starts postgres, atomic-server, brain-api
|
||
├─ wait_healthy x3 polls docker inspect for "healthy"
|
||
├─ python3 -m venv .venv local Python venv
|
||
├─ pip install ... operator deps in venv
|
||
├─ python scripts/02_bootstrap_atomic.py
|
||
│ │
|
||
│ ├─ atomic.health() verify atomic responding
|
||
│ ├─ atomic.setup_status() is the instance claimed?
|
||
│ ├─ atomic._request POST /api/setup/claim if not, claim and get token
|
||
│ ├─ patch_env_token() write ATOMIC_TOKEN= back to .env
|
||
│ └─ atomic.set_setting() x8 provider=openai_compat, BGE URLs, dim, etc.
|
||
│
|
||
├─ python scripts/04_seed_taxonomy.py
|
||
│ │
|
||
│ ├─ atomic.list_tags() what's already there
|
||
│ ├─ for each path in TAXONOMY: atomic.create_tag() if missing
|
||
│ └─ resolver.save() write shared/_tag_ids.json (host-only)
|
||
│
|
||
├─ python scripts/05_import_wikipedia_seed.py (optional, on by default)
|
||
│ │
|
||
│ ├─ for each title in SEED_EN + SEED_RO:
|
||
│ │ fetch_article() → atomic.create_atom() with proper tag_ids
|
||
│ └─ Atomic chunks + embeds in background via BGE-M3
|
||
│
|
||
├─ python scripts/07_run_extraction.py (optional, on by default)
|
||
│ │
|
||
│ ├─ list Type/Document atoms
|
||
│ ├─ for each not in extractor/_extracted.json:
|
||
│ │ fetch full atom → extract.extract_claims_from_atom() (Qwen 397B)
|
||
│ │ push.push_claim() per valid claim → atomic.create_atom()
|
||
│ └─ save state every doc
|
||
│
|
||
└─ smoke test: curl /health + curl /v1/gather, parse cache_status
|
||
```
|
||
|
||
## Scenario 3 — background time: Didi falls back to web module, posts result back
|
||
|
||
```
|
||
Didi backend
|
||
│
|
||
│ POST /v1/gather → cache_status=MISS
|
||
│
|
||
├─ Didi backend calls live web module (expensive)
|
||
│
|
||
│ POST /v1/ingest with the web-module result
|
||
▼
|
||
brain_api/app.py: post_ingest()
|
||
│
|
||
├─▶ services/ingest.py: ingest(req, atomic, resolver, background)
|
||
│
|
||
│ for each evidence in req.evidence:
|
||
│ atomic.get_atom_by_source_url() (dedup check)
|
||
│ if not exists:
|
||
│ evidence_to_markdown(evidence)
|
||
│ build_tag_ids_for_evidence() (Type/Document + Credibility from
|
||
│ credibility_score + Language)
|
||
│ atomic.create_atom() (synchronous)
|
||
│
|
||
│ if req.run_extraction and created_ids:
|
||
│ background.add_task(_run_extraction_background, created_ids)
|
||
│
|
||
├─▶ Returns IngestResponse immediately (extraction is fire-and-forget)
|
||
│
|
||
▼
|
||
FastAPI BackgroundTasks: _run_extraction_background(created_ids)
|
||
│
|
||
└─▶ extractor.batch.run_batch(only_atom_ids=set(created_ids))
|
||
├─ runs Qwen 397B claim extraction
|
||
├─ pushes Type/Claim atoms
|
||
└─ saves extractor/_extracted.json
|
||
```
|
||
|
||
**Effect**: next time Didi asks the same claim, brain returns HIT and Didi
|
||
doesn't need to call the web module.
|
||
|
||
---
|
||
|
||
# State files reference
|
||
|
||
DidiBrain has **three host-only JSON state files** (gitignored). They are
|
||
all idempotency ledgers — never primary data — so they can be regenerated
|
||
from Atomic without loss.
|
||
|
||
| File | Owner | Purpose | Recreatable? |
|
||
|---|---|---|---|
|
||
| `shared/_tag_ids.json` | `scripts/04_seed_taxonomy.py` | Maps canonical paths like `Country/Romania` → atom tag UUIDs. Used by every script that creates atoms. | yes — re-run script 04 |
|
||
| `extractor/_extracted.json` | `scripts/07_run_extraction.py` (and brain_api ingest) | Per-document extraction log: which docs have been processed, with which prompt version, how many claims came out. Skip already-done docs on rerun. | yes — delete and re-run; will re-extract everything |
|
||
| `lint/_contradictions.json` | `scripts/10_run_lint.py` | Per-pair verdict ledger with the EQUIVALENT / CONTRADICTORY / INCOMPARABLE labels. Skip already-done pairs on rerun. | yes — delete and re-run; will re-classify everything |
|
||
|
||
The brain_api **container** does NOT use any of these files. It calls
|
||
`atomic.list_tags()` at startup to refresh `TagResolver` in-memory, so
|
||
the image stays portable.
|
||
|
||
---
|
||
|
||
# Environment variables — full table
|
||
|
||
All loaded by `shared/config.py` via Pydantic Settings from `.env` at the
|
||
project root. The brain_api container also receives these via
|
||
`env_file: ../.env` in docker-compose, with `ATOMIC_URL` overridden to
|
||
`http://atomic-server:8080` (Docker DNS).
|
||
|
||
| Var | Default | What it controls | Required? |
|
||
|---|---|---|---|
|
||
| `LLM_ROUTER_URL` | `http://localhost:14011` | URL of the unified LLM router (vLLM + llamacpp behind it) | yes (set to `10.11.10.17:14011` in our setup) |
|
||
| `LLM_ROUTER_API_KEY` | empty | optional bearer | no |
|
||
| `LLM_VLLM_URL` | `http://localhost:14001` | direct vLLM URL (fallback only, currently unused) | no |
|
||
| `LLM_LLAMACPP_URLS` | empty | comma-separated direct llamacpp URLs (fallback) | no |
|
||
| `MODEL_REASONING` | `Qwen3.5-397B-A17B` | model id for extraction, NLI, gather | yes |
|
||
| `MODEL_REASONING_BACKEND` | `llamacpp` | router backend hint | yes |
|
||
| `MODEL_FAST` | `qwen3.5` | fast model id (currently disabled) | no |
|
||
| `MODEL_FAST_BACKEND` | `vllm` | fast backend hint | no |
|
||
| `MODEL_FAST_ENABLED` | `false` | enable fast model in routing | no |
|
||
| `MODEL_VISION` | `gemma-3-27b-it` | vision model id (currently down) | no |
|
||
| `MODEL_VISION_URL` | empty | direct vision endpoint | no |
|
||
| `MODEL_VISION_ENABLED` | `false` | enable vision in routing | no |
|
||
| `EMBEDDING_URL` | `http://10.11.10.15:8200` | BGE-M3 vLLM endpoint | yes |
|
||
| `EMBEDDING_API_KEY` | empty | optional bearer | no |
|
||
| `EMBEDDING_MODEL` | `BAAI/bge-m3` | model id sent in requests | yes |
|
||
| `EMBEDDING_DIM` | `1024` | vector dimension (must match Atomic's setting) | yes |
|
||
| `EMBEDDING_MAX_TOKENS` | `8192` | input length cap | yes |
|
||
| `RERANKER_URL` | `http://10.11.10.15:8100` | BGE-reranker-v2-m3 endpoint | yes |
|
||
| `RERANKER_API_KEY` | empty | optional bearer | no |
|
||
| `RERANKER_MODEL` | `BAAI/bge-reranker-v2-m3` | model id sent in requests | yes |
|
||
| `ATOMIC_URL` | `http://localhost:8088` | atomic-server URL (host scripts use this; brain_api container overrides to docker DNS) | yes |
|
||
| `ATOMIC_TOKEN` | empty | API token, set by `02_bootstrap_atomic.py` | yes (auto-populated) |
|
||
| `POSTGRES_USER` | `atomic` | Postgres user | yes |
|
||
| `POSTGRES_PASSWORD` | `atomic_dev_changeme` | Postgres password (override on server!) | yes |
|
||
| `POSTGRES_DB` | `atomic` | Postgres database name | yes |
|
||
| `POSTGRES_PORT` | `5434` | host-side port mapping | yes |
|
||
| `LOG_LEVEL` | `INFO` | structlog filter level | no |
|
||
|
||
**Container-only env vars set inside the Dockerfile** (not in `.env`):
|
||
|
||
| Var | Set to | Why |
|
||
|---|---|---|
|
||
| `BRAIN_API_HOST` | `0.0.0.0` | so Docker port mapping reaches uvicorn |
|
||
| `BRAIN_API_PORT` | `8090` | container internal port |
|
||
| `PYTHONIOENCODING` | `utf-8` | UTF-8 logging on Linux base image |
|
||
|
||
---
|
||
|
||
# Versioned prompts
|
||
|
||
Three prompts. Each has its own version number. Bumping a version means
|
||
re-running the relevant batch (extractor or lint) to re-process what's
|
||
been done.
|
||
|
||
| Prompt file | Used by | Output | Version |
|
||
|---|---|---|---|
|
||
| `extractor/prompts/claim_extraction_v1.md` | `extractor.extract.extract_claims_from_atom` | JSON list of `{claim, quote, stance, confidence}` from a document | v1 |
|
||
| `brain_api/prompts/nli_v1.md` | `brain_api.services.nli.classify_one` | JSON `{label: SUPPORTS/CONTRADICTS/NEUTRAL, confidence}` | v1 |
|
||
| `lint/prompts/pair_nli_v1.md` | `lint.detector.classify_pair` | JSON `{label: EQUIVALENT/CONTRADICTORY/INCOMPARABLE, confidence}` | v1 |
|
||
|
||
To bump: copy the file, rename to `_v2.md`, update the constant
|
||
`PROMPT_VERSION = "v2"` in the Python file, re-run the relevant batch.
|
||
The state files (`extractor/_extracted.json`, `lint/_contradictions.json`)
|
||
include `prompt_version` per record, so re-runs only re-process items at
|
||
the old version.
|
||
|
||
---
|
||
|
||
# When something breaks — start here
|
||
|
||
```
|
||
Something with /v1/gather failing?
|
||
│
|
||
├─ check brain_api/app.py route handler — it just delegates
|
||
├─ → services/gather.py — the 5-stage pipeline
|
||
├─ → services/mapping.py — atom shape conversion
|
||
└─ → shared/atomic_api.py + embedding_client.py + llm_client.py — upstream calls
|
||
│
|
||
└─ run scripts/01_sanity_full.py first to verify upstream stack
|
||
|
||
Atomic returning 500?
|
||
│
|
||
├─ docker logs didibrain-atomic — likely BGE unreachable
|
||
└─ curl http://10.11.10.15:8200/v1/models on host — VPN check
|
||
|
||
Claim extraction acting weird?
|
||
│
|
||
├─ extractor/_extracted.json — see what's been processed
|
||
├─ extractor/extract.py — substring quote validation may be rejecting
|
||
└─ extractor/prompts/claim_extraction_v1.md — prompt drift?
|
||
|
||
Lint pass slow / hanging?
|
||
│
|
||
├─ lint/runner.py MAX_PARALLEL — must match llamacpp backend count (we have 2)
|
||
├─ lint/detector.py PER_CALL_TIMEOUT_S — currently 30s
|
||
└─ lint/_contradictions.json saved every 25 — Ctrl-C safe to interrupt
|
||
|
||
Atomic doesn't have my tag?
|
||
│
|
||
├─ shared/taxonomy.py — is the path defined in TAXONOMY?
|
||
├─ shared/_tag_ids.json — is it in the cache?
|
||
└─ run scripts/04_seed_taxonomy.py to refresh both
|
||
```
|
||
|
||
For deeper troubleshooting see `STATUS.md` "Troubleshooting" section.
|
||
|
||
---
|
||
|
||
# Verification Cache (added 2026-04-23)
|
||
|
||
## What
|
||
|
||
A relational layer **alongside** the knowledge graph (Atomic/atoms) that stores
|
||
the LLM verification result produced by didi-backend after it runs NLI + verdict
|
||
analysis on evidence. This lets backend skip its expensive LLM call on
|
||
subsequent requests for the same `(claim, tier)` pair.
|
||
|
||
**Backend owns the prompt + model + status logic.** Brain is a pure cache: it
|
||
stores the opaque JSON blob backend produces, and returns it 1:1 on read.
|
||
Zero prompt sync between the two sides.
|
||
|
||
## Table: `brain_verification_cache`
|
||
|
||
Lives in the **same Postgres instance atomic-server uses** — we share the DB
|
||
connection with `brain_` prefix for trivial isolation. Asyncpg pool connects
|
||
directly (doesn't go through atomic-server).
|
||
|
||
Column summary:
|
||
|
||
| Column | Purpose |
|
||
|---|---|
|
||
| `claim_hash` | `sha256(normalize_claim(claim_text))` — normalization strips diacritics, lowercases, collapses whitespace, trims trailing `.?!` |
|
||
| `tier` | `'free'` or `'premium'` — **separate cache per tier** (different LLMs used) |
|
||
| `evidence_hash` | `sha256(sorted(lowercased_urls))` — **metadata only**, not part of the key |
|
||
| `evidence_urls` | jsonb array of URLs the verification was written for |
|
||
| `prompt_hash` | Backend's current prompt hash; used for staleness detection on read |
|
||
| `framework_version` | Backend's current thresholds hash; used for staleness on read |
|
||
| `verification_processed` | jsonb — UI-ready payload (status + sources + reasoning) |
|
||
| `verification_raw` | jsonb — raw LLM output; lets backend recompute when framework changes |
|
||
| `expires_at` | TTL, default 30 days from write |
|
||
|
||
**Unique key:** `(claim_hash, tier)`. UPSERT last-wins.
|
||
|
||
Why not `(claim_hash, evidence_hash, tier)`? Because the URLs backend has at
|
||
write-time rarely match what brain's `/v1/gather` returns at read-time
|
||
(brain uses its own semantic retrieval, which may rank differently than
|
||
backend's original source list). Including evidence in the key made the
|
||
cache unreachable. v2 drops it.
|
||
|
||
## Flow
|
||
|
||
### Write (backend → brain, fire-and-forget)
|
||
|
||
```
|
||
backend runs its LLM verification call
|
||
│
|
||
▼
|
||
POST /v1/verification_cache
|
||
claim, evidence_urls, tier,
|
||
model, prompt_hash, framework_version,
|
||
verification_raw, verification_processed
|
||
│
|
||
▼
|
||
brain normalizes + hashes + UPSERTs row (claim_hash, tier)
|
||
│
|
||
▼
|
||
200 OK (cached: true, claim_hash, evidence_hash, expires_at)
|
||
```
|
||
|
||
### Read (via /v1/gather extension)
|
||
|
||
```
|
||
backend wants to check if verification exists for this claim
|
||
│
|
||
▼
|
||
POST /v1/gather
|
||
claim, include_verification: true, tier,
|
||
prompt_hash, framework_version
|
||
│
|
||
▼
|
||
brain runs its normal gather (retrieval + rerank + evidence)
|
||
│
|
||
▼
|
||
brain queries brain_verification_cache (claim_hash, tier)
|
||
│
|
||
▼
|
||
returns GatherResponse with brain_meta.verification_staleness:
|
||
- miss → null; backend runs LLM + writes cache
|
||
- stale_prompt → null; backend runs LLM + writes cache
|
||
- stale_framework→ returns verification_raw; backend recomputes status locally (zero LLM)
|
||
- fresh → returns verification_processed; direct to UI
|
||
```
|
||
|
||
See `CONTRACT_VERIFICATION_CACHE.md` for the full field-by-field spec.
|
||
|
||
## Files
|
||
|
||
- `brain_api/db.py` — asyncpg pool + idempotent schema migrations (runs in lifespan)
|
||
- `brain_api/services/verification_cache.py` — normalize, hash, upsert, lookup, freshness decider
|
||
- `brain_api/schemas.py` — `GatherRequest` extension, `VerificationCacheWriteRequest/Response`, `BrainMeta` extension
|
||
- `brain_api/app.py` — `POST /v1/verification_cache` endpoint wired in lifespan
|
||
- `brain_api/services/gather.py` — cache lookup after evidence build, attach to `brain_meta`
|
||
|
||
---
|
||
|
||
# Glossary
|
||
|
||
| Term | Meaning |
|
||
|---|---|
|
||
| **atom** | The unit of storage in Atomic. Markdown content + metadata + tags. Two flavors: `Type/Document` (a full source article) and `Type/Claim` (an atomic factual claim extracted from a document). |
|
||
| **Atomic** | The Rust knowledge-graph backend (kenforthewin/atomic). DidiBrain uses its REST API. Runs as `didibrain-atomic` container. |
|
||
| **brain_api** | Our FastAPI service. The HTTP face DidiBrain shows to Didi backend. |
|
||
| **chunk** | A piece of an atom's content split for embedding. Atomic does this internally; we never see chunks directly. |
|
||
| **HIT / PARTIAL / MISS** | `brain_meta.cache_status` returned by `/v1/gather`. Tells Didi backend whether the brain had relevant knowledge or it should fall back to web module. |
|
||
| **Lint pass** | Background contradiction-detection job. Reads existing claim atoms, finds contradictory pairs cross-corpus. |
|
||
| **NLI** | Natural Language Inference. The "decide if A supports / contradicts / is neutral toward B" classification. We do it twice: stance-vs-query (gather) and pair-equivalence (lint). |
|
||
| **prompt_version** | A string like "v1" pinned per prompt file. State files record it so we can bump prompts and re-process selectively. |
|
||
| **reranker** | Cross-encoder (BGE-reranker-v2-m3). Takes (query, document) pairs and outputs a precise relevance score. Used after embedding retrieval to refine top-K. |
|
||
| **stance_in_source** | What the source itself asserts about a claim (ASSERTS/REPORTS/REFUTES/QUESTIONS/NEUTRAL). Set during extraction. |
|
||
| **stance_vs_query** | What an evidence claim says about Didi user's input claim (SUPPORTS/CONTRADICTS/NEUTRAL). Set during gather's NLI stage. |
|
||
| **TagResolver** | Helper that maps canonical paths like `Country/Romania` to Atomic tag UUIDs. Loaded from `_tag_ids.json` (scripts) or refreshed from atomic at startup (brain_api container). |
|
||
| **TAXONOMY** | The canonical tree in `shared/taxonomy.py`. Single source of truth for what tags exist. |
|