Dezvoltari Evotech 42ff22bf85 LOT 1 - Optimizare script build -Instalare mono comanda

2026-06-27 06:42:02 -07:00

36 KiB

Raw Permalink Blame History

DidiBrain — Architecture & "Who Does What"

This is the codebase rosetta stone. If you read this top-to-bottom you should be able to answer "what does file X do, when does it run, who calls it, what does it depend on" for any file in the repo.

Reading order if you're new:

README.md — what DidiBrain is in 2 minutes

this file — full mental model of the codebase (15 minutes)

STATUS.md — operational state, deployment, troubleshooting

brain_api/schemas.py — the literal HTTP contract with Didi

brain_api/services/gather.py — the main pipeline in 200 lines

TL;DR — "WHO DOES WHAT" in one table

Module / File	What it does	When it runs	Who calls it
`infra/docker-compose.yml`	Defines the 4-container stack (api, atomic, postgres, scheduler)	`docker compose up`	operator
`brain_api/` (container)	HTTP service speaking Didi's contract	every Didi request	Didi backend (HTTP)
`shared/` (library)	Common code: config, clients, taxonomy	imported everywhere	brain_api, extractor, lint, scripts
`extractor/` (library + jobs)	Turns Document atoms into Claim atoms	operator (script 07) OR background after `/v1/ingest`	scripts/07, brain_api ingest
`lint/` (library + job)	Detects contradictions across the corpus	operator overnight (script 10)	scripts/10, scripts/11
`scripts/` (operator CLI)	One-shot maintenance + bootstrap commands	operator types `python scripts/XX_*.py`	human
Atomic server (container)	Storage + chunking + embedding pipeline + REST	always	brain_api (HTTP), scripts (HTTP)
Postgres + pgvector (container)	Persistent atom + vector storage	always	atomic-server (SQL)

External services (not in our repo, reached over Docker DNS on didi-network):

Where	What	Used by
`didiAI-llm-api` → `10.11.10.17:14011`	LLM router → qwen3.5	brain_api NLI, extractor, lint
`didiAI-embeddings-api` → `10.11.10.15:14100`	BGE-M3 embedding server	atomic-server (for embeddings)
`didiAI-rerank-api` → `10.11.10.15:14200`	BGE-reranker-v2-m3	brain_api gather (for precision rerank)

The big picture in one diagram

                          ┌─────────────────────────────────┐
                          │       Didi backend              │
                          │    (somewhere else)             │
                          └─────┬───────────────────────────┘
                                │ POST /v1/gather, /v1/search,
                                │ /v1/fetch, /v1/ingest, ...
                                ▼
   ┌────────────────────────────────────────────────────────────┐
   │ brain_api (container :8090)                                 │
   │  ┌──────────────────────────────────────────────────────┐   │
   │  │ FastAPI app (app.py)                                 │   │
   │  │   routes → services/{gather, search, fetch, ingest, │   │
   │  │            mapping, nli}                            │   │
   │  └──────────────────────────────────────────────────────┘   │
   │  imports → shared/{config, llm_client, embedding_client,    │
   │            atomic_api, taxonomy, logging}                   │
   └─────┬─────────────┬──────────────┬──────────────────────────┘
         │             │              │
         │ HTTP        │ HTTP         │ HTTP
         ▼             ▼              ▼
   ┌───────────┐ ┌──────────┐ ┌────────────────┐
   │  atomic-  │ │ BGE-M3   │ │ qwen3.5        │
   │  server   │ │ + rerank │ │ via LLM router │
   │  (cont.)  │ │ (DNS)    │ │ (DNS)          │
   └─────┬─────┘ └──────────┘ └────────────────┘
         │
         ▼
   ┌───────────────┐
   │ postgres      │
   │ + pgvector    │
   │ (container)   │
   └───────────────┘

Operator scripts run on the host, talk over HTTP:

   ┌─────────────────────┐
   │ host venv           │       ┌──────────┐
   │ scripts/01..11.py   │──HTTP─▶│ brain    │
   │ + bootstrap.sh      │       │ stack    │
   └─────────────────────┘       └──────────┘

Module deep-dives

Each module section answers:

Purpose — one paragraph
When it runs — request-time / startup / background / operator
Files inside — what each does
Imports — what it depends on
Imported by — who depends on it

`shared/` — the library every other module imports

Purpose. Pure-Python utilities and clients with no HTTP surface of their own. Single source of truth for config, logging, all upstream clients, and the canonical tag taxonomy. Everything else in the repo treats this as a read-only library.

When it runs. Imported at process startup; no long-lived behavior of its own.

Files inside.

File	Role
`config.py`	Pydantic Settings singleton. Reads `.env` + env vars, validates types, exposes `settings.X` everywhere. Defines `LlmRole` enum and `model_for(role)` for routing.
`logging.py`	Sets up structlog once per process. Forces UTF-8 stdout (Windows fix), silences noisy `httpx`/`httpcore` loggers. Call `setup_logging()` once in `main()`.
`llm_client.py`	Async OpenAI-compat HTTP client targeting `LLM_ROUTER_URL`. Exposes `chat`, `chat_text`, `chat_json`, `chat_label`. tenacity-based retries on HTTP/timeout errors.
`embedding_client.py`	Async client for BGE-M3 embeddings (`/v1/embeddings`) and BGE reranker (`/v1/rerank`). Exposes `embed`, `embed_one`, `rerank`. Has the `cosine()` helper.
`atomic_api.py`	Typed REST client for atomic-server. Wraps the endpoints we actually use: `health`, `settings` get/set, `atoms` CRUD + by-source-url, `tags` list/create, `search`, `find_similar`. Returns dataclasses (`SearchHit`, `AtomSummary`) or raw dicts.
`taxonomy.py`	Defines the canonical `TAXONOMY` tree (~80 paths), exposes `walk()`, `all_paths()`, `build_path_map_from_tags()`, and the `TagResolver` class for `path → uuid` lookup. Also `load_from_mapping()` for in-memory refresh (used by brain_api at startup).

Imports. Standard library + httpx + pydantic + pydantic-settings + structlog + tenacity + rich. Nothing internal.

Imported by. brain_api/, extractor/, lint/, every script in scripts/.

`brain_api/` — the HTTP service

Purpose. Containerized FastAPI service that exposes the full route set (~30 routes: the web-gathering contract + brain-owned cache/freshness layers) that Didi's backend already speaks. It does NOT contain business logic that another module reuses; it ties together shared + extractor (via /v1/ingest) and translates between Didi's HTTP shape and the brain's internal capabilities.

When it runs. Always. One container per deployment, lifespan-managed by FastAPI/Uvicorn. The lifespan hook initializes the long-lived clients (AtomicClient, EmbeddingClient, LlmClient) and refreshes the TagResolver from atomic-server at startup so the image is portable.

Files inside.

File	Role
`Dockerfile`	python:3.12-slim, non-root user `brain`, COPY shared/extractor/brain_api, curl healthcheck on /health, CMD `python -m brain_api.run`.
`requirements.txt`	Pinned runtime deps.
`__init__.py`	Module marker + docstring.
`app.py`	The FastAPI app. Defines `lifespan` (startup/shutdown), the `/health` route, and the full set of v1 routes (gather/search/fetch/image-search/ingest + verification_cache + analysis_atom* + canonicalize + cache/invalidate + cache/audit_log + fact_status*). Each route delegates to a function in `services/`.
`run.py`	Uvicorn entry. Reads `BRAIN_API_HOST` and `BRAIN_API_PORT` from env (defaults 127.0.0.1:8090; container overrides to 0.0.0.0).
`deps.py`	`AppState` dataclass + module-level singleton accessor. Set by lifespan, read by route handlers.
`schemas.py`	The contract. Pydantic v2 models for every request/response shape the service speaks: `SearchRequest/Response`, `FetchRequest/Response`, `GatherRequest/Response`, `ImageSearchRequest/Response`, `IngestRequest/Response`, plus `EvidenceItem`, `BrainEvidenceMeta`, `BrainMeta`, etc.
`prompts/nli_v1.md`	Versioned prompt template for the stance-vs-query NLI step in `/v1/gather`.
`services/mapping.py`	Pure functions converting atomic atoms ↔ Didi's `EvidenceItem` / `FetchedPage` / `SearchResultItem`. Includes `parse_claim_atom_body()` (used by lint too) and the credibility-tag → score mapping.
`services/gather.py`	The 5-stage pipeline for `/v1/gather`: context → retrieval → rerank → NLI → evidence build. Returns a `GatherResponse`.
`services/search.py`	Lighter version of gather: just semantic search across queries, dedup by parent doc, return `SearchResponse`.
`services/fetch.py`	Look up atoms by URL via `atomic.get_atom_by_source_url`. Returns `FetchResponse` with HIT pages and `failed_urls=[{url, "not_in_brain"}]` for misses.
`services/ingest.py`	Accept `IngestRequest` (typically a web-module result), create `Type/Document` atoms with inferred tags, optionally schedule background extraction via FastAPI `BackgroundTasks`.
`services/nli.py`	The stance-vs-query LLM call. `classify_one()` for a single (claim, evidence) pair, `classify_batch()` for parallel with `MAX_PARALLEL=2` (matches llamacpp backend concurrency).

Imports. shared, extractor.batch (lazily, only when ingest spawns extraction), fastapi, uvicorn.

Imported by. Nothing — it's the leaf consumer. Didi backend talks to it over HTTP, not Python imports.

`extractor/` — Document → Claim transformation

Purpose. Reads Type/Document atoms from Atomic, asks qwen3.5 to extract atomic factual claims, validates each claim against the source text (substring check on the quote), and creates new Type/Claim atoms with canonical hash-based source URLs and inherited tags.

When it runs. Two paths:

Operator-triggered: python scripts/07_run_extraction.py runs extractor.batch.run_batch() over every Document atom that hasn't been processed at the current prompt version.
Background after ingest: when POST /v1/ingest is called with run_extraction: true (the default), the brain_api's ingest service schedules a BackgroundTasks callback that calls extractor.batch.run_batch(only_atom_ids=...) on just the atoms that were just created. Fire-and-forget, doesn't block the HTTP response.

Both paths share the same idempotency state (extractor/_extracted.json).

Files inside.

File	Role
`__init__.py`	Module marker.
`_state.py`	`DocExtractionRecord` dataclass + `ExtractionState` class with `has`, `get`, `upsert`, `save`, `_load`. JSON file at `extractor/_extracted.json`, host-only.
`extract.py`	Single-document logic. `extract_claims_from_atom(llm, title, language, content)` → `ExtractionResult` with `valid: list[ExtractedClaim]` and `rejected: dict[reason → count]`. Includes the substring quote validation.
`push.py`	`push_claim(atomic, parent_atom, claim, parent_title, resolver)` creates one `Type/Claim` atom. URL is `{parent_url}#claim={hash8}` for natural dedup. Inherits parent's Country/Topic/SourceType/Credibility/Language tags, adds Type/Claim + Stance/{X}.
`batch.py`	The orchestrator. `run_batch(limit, only_atom_ids)` paginates Document atoms, fetches each fully (list_atoms returns summary only — gotcha!), runs extraction sequentially, pushes claims, saves state after every doc so Ctrl-C is recoverable.
`prompts/claim_extraction_v1.md`	Versioned prompt that instructs qwen3.5 to return strict JSON with claims + verbatim quotes. Output format pinned.

Imports. shared, httpx (transitively).

Imported by. scripts/07_run_extraction.py, brain_api/services/ingest.py.

`lint/` — cross-corpus contradiction detection

Purpose. Audit job that finds claim atoms in our corpus that contradict each other (or are paraphrases). For every claim, asks Atomic for its semantic neighbors, then asks qwen3.5 to classify each candidate pair as EQUIVALENT, CONTRADICTORY, or INCOMPARABLE. Stores all verdicts in a JSON ledger so re-runs only process new pairs.

When it runs. Operator-triggered, typically as an overnight job (~1 pair / 1.5s × ~6000 pairs for 513 atoms ≈ 4 hours). Re-runs are incremental thanks to the ledger.

Files inside.

File	Role
`__init__.py`	Module marker.
`_state.py`	`PairVerdict` dataclass, `LintState` class (loads/saves the ledger atomically via tmp+rename). `pair_hash(a, b)` is canonical (sorted) so (A,B) and (B,A) hash the same. `LintStats` for per-run counters.
`detector.py`	Single-pair classification. `classify_pair(llm, claim_a, claim_b)` → `PairClassification(label, confidence, error)`. Never raises; falls through to INCOMPARABLE on any error so the batch keeps going.
`pairs.py`	`generate_candidates(atomic, source_atom_ids)` walks each source atom, calls `atomic.find_similar` with `MIN_PAIR_SIMILARITY=0.55`, filters to Type/Claim neighbors only, returns canonical de-duplicated pairs.
`runner.py`	The orchestrator. `run_lint_pass(limit_atoms, force)` does the full flow: load atom ids → generate candidates → drop already-evaluated → bulk-fetch full atom bodies → classify with `MAX_PARALLEL=2` → save ledger every 25 verdicts → ALWAYS save on exit (even crash).
`reporter.py`	Pretty printing with `rich`: `render_stats(stats)`, `render_contradictions(state, top_n, min_confidence)`, `render_equivalents(state)`.
`prompts/pair_nli_v1.md`	Versioned prompt for pair classification. Same-language and cross-language are judged identically.

Imports. shared, brain_api.services.mapping (for parse_claim_atom_body).

Imported by. scripts/10_run_lint.py, scripts/11_show_contradictions.py.

`scripts/` — operator CLI tools

Purpose. One-shot maintenance and bootstrap commands that operators run from the host venv. Each is self-contained, idempotent, prints a clean rich-formatted report, and exits with a meaningful code.

When it runs. Operator types python scripts/XX_*.py.

Files inside. All numbered for execution order during a fresh deploy.

Script	What it does	Idempotent?	Typical runtime
`01_sanity_full.py`	Validates upstream stack: pings router, lists models, runs JSON extraction + NLI + Romanian + cross-lingual + embed cosine + reranker. Writes a JSON report.	yes	~10 s
`02_bootstrap_atomic.py`	Waits for atomic-server, claims the instance via `/api/setup/claim` if needed, configures `provider=openai_compat` pointed at BGE-M3, disables `auto_tagging_enabled`, patches `ATOMIC_TOKEN` into `.env`.	yes	~5 s
`03_sanity_atomic.py`	End-to-end brain check: create test atom → wait for embedding → semantic search retrieves it → cleanup. Catches "atomic + Postgres + BGE-M3 wired" issues.	yes (cleanup deletes the atom)	~5-10 s
`04_seed_taxonomy.py`	Reads `shared.taxonomy.TAXONOMY`, creates any missing tags in atomic-server preserving parent_id structure, writes `shared/_tag_ids.json`.	yes	~10 s
`05_import_wikipedia_seed.py`	Imports a curated seed list of Wikipedia articles (currently vaccines EN+RO) via the MediaWiki action API, dedupes by canonical URL, creates Type/Document atoms with proper tags.	yes (URL dedup)	~30 s for 19 articles
`06_validate_queries.py`	4 doc-level queries against `/api/search` with reranker, prints top hits. Sanity that retrieval works on the corpus.	yes (read-only)	~5-10 s
`07_run_extraction.py`	Calls `extractor.batch.run_batch()`. With `--limit N` for smoke tests.	yes (state file)	~1 min/document
`08_validate_claims.py`	4 claim-level queries, splits doc/claim hits, reranks claims. Sanity that claim-level retrieval works.	yes	~10-20 s
`09_brain_api_demo.py`	Hits all 5 brain_api endpoints with realistic inputs, validates schema compliance and HIT/MISS cache_status.	yes	~30 s
`10_run_lint.py`	Calls `lint.runner.run_lint_pass()`. With `--limit N` to cap source atoms, `--force` to re-evaluate cached pairs.	yes (ledger)	~30 s × N atoms
`11_show_contradictions.py`	READ ONLY. Loads `lint/_contradictions.json`, renders top contradictions and (with `--equivalents`) paraphrase clusters.	yes	<1 s
`bootstrap_deploy.sh`	Bash orchestrator for fresh-server deploy. Preflight → compose build+up → wait healthy → venv → 02 → 04 → optional 05+07 → smoke test. Idempotent, color-coded output.	yes	~5 min + 15 min if importing corpus

Imports. Each script sys.path.inserts the project root then imports shared, brain_api, extractor, or lint as needed.

Imported by. Nothing. They are leaf entry points.

`infra/` — containerization

Purpose. Defines the deployment topology.

Files inside.

File Role

docker-compose.yml Four services: postgres (pgvector/pgvector:pg16), atomic-server (ghcr.io/kenforthewin/atomic-server:latest with overridden entrypoint to use --data-dir instead of legacy --db-path, and ATOMIC_STORAGE=postgres env), brain-api (built from brain_api/Dockerfile), and scheduler (built from scheduler/Dockerfile — feeder + auditor + watcher + heartbeat). Two named volumes (didibrain-pg-data, didibrain-atomic-data), and the external shared network didi-network. All services have restart: unless-stopped and healthchecks.

File	Role
`docker-compose.yml`	Four services: `postgres` (pgvector/pgvector:pg16), `atomic-server` (ghcr.io/kenforthewin/atomic-server:latest with overridden entrypoint to use `--data-dir` instead of legacy `--db-path`, and `ATOMIC_STORAGE=postgres` env), `brain-api` (built from `brain_api/Dockerfile`), and `scheduler` (built from `scheduler/Dockerfile` — feeder + auditor + watcher + heartbeat). Two named volumes (`didibrain-pg-data`, `didibrain-atomic-data`), and the external shared network `didi-network`. All services have `restart: unless-stopped` and healthchecks.

When it runs. docker compose up from the operator (or via bootstrap_deploy.sh).

Data flow — three concrete scenarios

The codebase has exactly three control flows worth memorizing.

Scenario 1 — request time: Didi calls `POST /v1/gather`

Didi backend
   │
   │ POST http://brain-host:8090/v1/gather
   │ {"claim": "vaccinurile cauzeaza autism", "max_evidence": 5, "run_nli": true}
   ▼
brain_api/app.py: post_gather()
   │
   ├─▶ deps.get_state()  (returns shared AppState with clients)
   │
   ├─▶ services/gather.py: gather(req, atomic, embed, llm, resolver)
   │
   │      stage 1 context  → mapping.detect_language_simple(claim)
   │                         (~0 ms)
   │
   │      stage 2 retrieval → atomic.search(claim, mode="semantic", limit=100)
   │                         → filter to Type/Claim only
   │                         (~300 ms; atomic embeds query via BGE-M3 internally)
   │
   │      stage 3 rerank   → atomic.get_atom(...) for top-15 claim atoms
   │                         → embed.rerank(claim, claim_texts)
   │                         (~1000 ms; calls reranker via BGE cross-encoder)
   │
   │      stage 4 nli      → mapping.parse_claim_atom_body() per top result
   │                         → nli.classify_batch(llm, claim, evidence_texts)
   │                         (parallel calls to qwen3.5, ~3-4 sec; one per parent doc)
   │
   │      stage 5 evidence → atomic.get_atom_by_source_url() per parent doc
   │                         → mapping.evidence_from_parent() builds EvidenceItem
   │                         (~400 ms parallel)
   │
   ├─▶ Returns GatherResponse with brain_meta.cache_status set based on top
   │   relevance score: >=0.6 HIT, 0.3-0.6 PARTIAL, <0.3 MISS.
   │
   ▼
Didi backend receives JSON, branches on cache_status.

Total: ~5-6 sec with NLI, ~1.5 sec without.

Scenario 2 — operator time: fresh-server deployment

operator                                   what runs
═══════                                    ══════════
ssh server, cd ~/didibrain                 (none)
cp .env.example .env, vim .env             (none)
./scripts/bootstrap_deploy.sh
   │
   ├─ preflight                            docker, python3, curl available
   ├─ docker compose build brain-api       brain_api/Dockerfile builds the image
   ├─ docker compose up -d                 starts postgres, atomic-server, brain-api
   ├─ wait_healthy x3                      polls docker inspect for "healthy"
   ├─ python3 -m venv .venv                local Python venv
   ├─ pip install ...                      operator deps in venv
   ├─ python scripts/02_bootstrap_atomic.py
   │    │
   │    ├─ atomic.health()                 verify atomic responding
   │    ├─ atomic.setup_status()           is the instance claimed?
   │    ├─ atomic._request POST /api/setup/claim   if not, claim and get token
   │    ├─ patch_env_token()               write ATOMIC_TOKEN= back to .env
   │    └─ atomic.set_setting() x8         provider=openai_compat, BGE URLs, dim, etc.
   │
   ├─ python scripts/04_seed_taxonomy.py
   │    │
   │    ├─ atomic.list_tags()              what's already there
   │    ├─ for each path in TAXONOMY: atomic.create_tag() if missing
   │    └─ resolver.save()                 write shared/_tag_ids.json (host-only)
   │
   ├─ python scripts/05_import_wikipedia_seed.py    (optional, on by default)
   │    │
   │    ├─ for each title in SEED_EN + SEED_RO:
   │    │    fetch_article() → atomic.create_atom() with proper tag_ids
   │    └─ Atomic chunks + embeds in background via BGE-M3
   │
   ├─ python scripts/07_run_extraction.py           (optional, on by default)
   │    │
   │    ├─ list Type/Document atoms
   │    ├─ for each not in extractor/_extracted.json:
   │    │    fetch full atom → extract.extract_claims_from_atom() (qwen3.5)
   │    │    push.push_claim() per valid claim → atomic.create_atom()
   │    └─ save state every doc
   │
   └─ smoke test: curl /health + curl /v1/gather, parse cache_status

Scenario 3 — background time: Didi falls back to web module, posts result back

Didi backend
   │
   │ POST /v1/gather → cache_status=MISS
   │
   ├─ Didi backend calls live web module (expensive)
   │
   │ POST /v1/ingest with the web-module result
   ▼
brain_api/app.py: post_ingest()
   │
   ├─▶ services/ingest.py: ingest(req, atomic, resolver, background)
   │
   │      for each evidence in req.evidence:
   │         atomic.get_atom_by_source_url()  (dedup check)
   │         if not exists:
   │            evidence_to_markdown(evidence)
   │            build_tag_ids_for_evidence()  (Type/Document + Credibility from
   │                                           credibility_score + Language)
   │            atomic.create_atom()           (synchronous)
   │
   │      if req.run_extraction and created_ids:
   │         background.add_task(_run_extraction_background, created_ids)
   │
   ├─▶ Returns IngestResponse immediately (extraction is fire-and-forget)
   │
   ▼
   FastAPI BackgroundTasks: _run_extraction_background(created_ids)
   │
   └─▶ extractor.batch.run_batch(only_atom_ids=set(created_ids))
          ├─ runs qwen3.5 claim extraction
          ├─ pushes Type/Claim atoms
          └─ saves extractor/_extracted.json

Effect: next time Didi asks the same claim, brain returns HIT and Didi doesn't need to call the web module.

State files reference

DidiBrain has three host-only JSON state files (gitignored). They are all idempotency ledgers — never primary data — so they can be regenerated from Atomic without loss.

File	Owner	Purpose	Recreatable?
`shared/_tag_ids.json`	`scripts/04_seed_taxonomy.py`	Maps canonical paths like `Country/Romania` → atom tag UUIDs. Used by every script that creates atoms.	yes — re-run script 04
`extractor/_extracted.json`	`scripts/07_run_extraction.py` (and brain_api ingest)	Per-document extraction log: which docs have been processed, with which prompt version, how many claims came out. Skip already-done docs on rerun.	yes — delete and re-run; will re-extract everything
`lint/_contradictions.json`	`scripts/10_run_lint.py`	Per-pair verdict ledger with the EQUIVALENT / CONTRADICTORY / INCOMPARABLE labels. Skip already-done pairs on rerun.	yes — delete and re-run; will re-classify everything

The brain_api container refreshes TagResolver in-memory at startup by calling atomic.list_tags(), so it does not bake any of these files into the image. It does, however, mount shared/_tag_ids.json read-only via the compose volumes: entry — background extraction (triggered by /v1/ingest) instantiates TagResolver directly, bypassing the lifespan refresh, so it relies on that mounted file to resolve canonical tag UUIDs. The other two ledgers (extractor/_extracted.json, lint/_contradictions.json) are host-only and not used by the container.

Environment variables — full table

All loaded by shared/config.py via Pydantic Settings from .env at the project root. The brain_api container also receives these via env_file: ../.env in docker-compose, with ATOMIC_URL overridden to http://atomic-server:8080 (Docker DNS).

Var	Default	What it controls	Required?
`LLM_ROUTER_URL`	`http://localhost:14011`	URL of the unified LLM router (vLLM + llamacpp behind it)	yes (set to `10.11.10.17:14011` in our setup)
`LLM_ROUTER_API_KEY`	empty	optional bearer	no
`LLM_VLLM_URL`	`http://localhost:14001`	direct vLLM URL (fallback only, currently unused)	no
`LLM_LLAMACPP_URLS`	empty	comma-separated direct llamacpp URLs (fallback)	no
`MODEL_REASONING`	`qwen3.5`	model id for extraction, NLI, gather (live working model)	yes
`MODEL_REASONING_BACKEND`	`vllm`	router backend hint	yes
`MODEL_FAST`	`qwen3.5`	fast model id (enabled — the live working model)	no
`MODEL_FAST_BACKEND`	`vllm`	fast backend hint	no
`MODEL_FAST_ENABLED`	`true`	fast model enabled in routing (live: `true`, qwen3.5 active)	no
`MODEL_VISION`	empty	vision model id (vision disabled; no vision model deployed)	no
`MODEL_VISION_URL`	empty	direct vision endpoint	no
`MODEL_VISION_ENABLED`	`false`	enable vision in routing	no
`EMBEDDING_URL`	`http://10.11.10.15:14100`	BGE-M3 vLLM endpoint (`didiAI-embeddings-api`)	yes
`EMBEDDING_API_KEY`	empty	optional bearer	no
`EMBEDDING_MODEL`	`BAAI/bge-m3`	model id sent in requests	yes
`EMBEDDING_DIM`	`1024`	vector dimension (must match Atomic's setting)	yes
`EMBEDDING_MAX_TOKENS`	`8192`	input length cap	yes
`RERANKER_URL`	`http://10.11.10.15:14200`	BGE-reranker-v2-m3 endpoint (`didiAI-rerank-api`)	yes
`RERANKER_API_KEY`	empty	optional bearer	no
`RERANKER_MODEL`	`BAAI/bge-reranker-v2-m3`	model id sent in requests	yes
`ATOMIC_URL`	`http://localhost:8088`	atomic-server URL (host scripts use this; brain_api container overrides to docker DNS)	yes
`ATOMIC_TOKEN`	empty	API token, set by `02_bootstrap_atomic.py`	yes (auto-populated)
`POSTGRES_USER`	`atomic`	Postgres user	yes
`POSTGRES_PASSWORD`	`atomic_dev_changeme`	Postgres password (override on server!)	yes
`POSTGRES_DB`	`atomic`	Postgres database name	yes
`POSTGRES_PORT`	`5434`	host-side port mapping	yes
`LOG_LEVEL`	`INFO`	structlog filter level	no

Container-only env vars set inside the Dockerfile (not in .env):

Var	Set to	Why
`BRAIN_API_HOST`	`0.0.0.0`	so Docker port mapping reaches uvicorn
`BRAIN_API_PORT`	`8090`	container internal port
`PYTHONIOENCODING`	`utf-8`	UTF-8 logging on Linux base image

Versioned prompts

Three prompts. Each has its own version number. Bumping a version means re-running the relevant batch (extractor or lint) to re-process what's been done.

Prompt file	Used by	Output	Version
`extractor/prompts/claim_extraction_v1.md`	`extractor.extract.extract_claims_from_atom`	JSON list of `{claim, quote, stance, confidence}` from a document	v1
`brain_api/prompts/nli_v1.md`	`brain_api.services.nli.classify_one`	JSON `{label: SUPPORTS/CONTRADICTS/NEUTRAL, confidence}`	v1
`lint/prompts/pair_nli_v1.md`	`lint.detector.classify_pair`	JSON `{label: EQUIVALENT/CONTRADICTORY/INCOMPARABLE, confidence}`	v1

To bump: copy the file, rename to _v2.md, update the constant PROMPT_VERSION = "v2" in the Python file, re-run the relevant batch. The state files (extractor/_extracted.json, lint/_contradictions.json) include prompt_version per record, so re-runs only re-process items at the old version.

When something breaks — start here

Something with /v1/gather failing?
   │
   ├─ check brain_api/app.py route handler — it just delegates
   ├─ → services/gather.py — the 5-stage pipeline
   ├─ → services/mapping.py — atom shape conversion
   └─ → shared/atomic_api.py + embedding_client.py + llm_client.py — upstream calls
       │
       └─ run scripts/01_sanity_full.py first to verify upstream stack

Atomic returning 500?
   │
   ├─ docker logs didibrain-atomic — likely BGE unreachable
   └─ curl http://10.11.10.15:14100/v1/models on host — upstream reachability check

Claim extraction acting weird?
   │
   ├─ extractor/_extracted.json — see what's been processed
   ├─ extractor/extract.py — substring quote validation may be rejecting
   └─ extractor/prompts/claim_extraction_v1.md — prompt drift?

Lint pass slow / hanging?
   │
   ├─ lint/runner.py MAX_PARALLEL — must match llamacpp backend count (we have 2)
   ├─ lint/detector.py PER_CALL_TIMEOUT_S — currently 30s
   └─ lint/_contradictions.json saved every 25 — Ctrl-C safe to interrupt

Atomic doesn't have my tag?
   │
   ├─ shared/taxonomy.py — is the path defined in TAXONOMY?
   ├─ shared/_tag_ids.json — is it in the cache?
   └─ run scripts/04_seed_taxonomy.py to refresh both

For deeper troubleshooting see STATUS.md "Troubleshooting" section.

Verification Cache (added 2026-04-23)

What

A relational layer alongside the knowledge graph (Atomic/atoms) that stores the LLM verification result produced by didi-backend after it runs NLI + verdict analysis on evidence. This lets backend skip its expensive LLM call on subsequent requests for the same (claim, tier) pair.

Backend owns the prompt + model + status logic. Brain is a pure cache: it stores the opaque JSON blob backend produces, and returns it 1:1 on read. Zero prompt sync between the two sides.

Table: `brain_verification_cache`

Lives in the same Postgres instance atomic-server uses — we share the DB connection with brain_ prefix for trivial isolation. Asyncpg pool connects directly (doesn't go through atomic-server).

Column summary:

Column	Purpose
`claim_hash`	`sha256(normalize_claim(claim_text))` — normalization strips diacritics, lowercases, collapses whitespace, trims trailing `.?!`
`tier`	`'free'` or `'premium'` — separate cache per tier (different LLMs used)
`evidence_hash`	`sha256(sorted(lowercased_urls))` — metadata only, not part of the key
`evidence_urls`	jsonb array of URLs the verification was written for
`prompt_hash`	Backend's current prompt hash; used for staleness detection on read
`framework_version`	Backend's current thresholds hash; used for staleness on read
`verification_processed`	jsonb — UI-ready payload (status + sources + reasoning)
`verification_raw`	jsonb — raw LLM output; lets backend recompute when framework changes
`expires_at`	TTL, default 30 days from write

Unique key: (claim_hash, tier). UPSERT last-wins.

Why not (claim_hash, evidence_hash, tier)? Because the URLs backend has at write-time rarely match what brain's /v1/gather returns at read-time (brain uses its own semantic retrieval, which may rank differently than backend's original source list). Including evidence in the key made the cache unreachable. v2 drops it.

Flow

Write (backend → brain, fire-and-forget)

backend runs its LLM verification call
     │
     ▼
POST /v1/verification_cache
  claim, evidence_urls, tier,
  model, prompt_hash, framework_version,
  verification_raw, verification_processed
     │
     ▼
brain normalizes + hashes + UPSERTs row (claim_hash, tier)
     │
     ▼
200 OK (cached: true, claim_hash, evidence_hash, expires_at)

Read (via /v1/gather extension)

backend wants to check if verification exists for this claim
     │
     ▼
POST /v1/gather
  claim, include_verification: true, tier,
  prompt_hash, framework_version
     │
     ▼
brain runs its normal gather (retrieval + rerank + evidence)
     │
     ▼
brain queries brain_verification_cache (claim_hash, tier)
     │
     ▼
returns GatherResponse with brain_meta.verification_staleness:
  - miss           → null; backend runs LLM + writes cache
  - stale_prompt   → null; backend runs LLM + writes cache
  - stale_framework→ returns verification_raw; backend recomputes status locally (zero LLM)
  - fresh          → returns verification_processed; direct to UI

See CONTRACT_VERIFICATION_CACHE.md for the full field-by-field spec.

Files

brain_api/db.py — asyncpg pool + idempotent schema migrations (runs in lifespan)
brain_api/services/verification_cache.py — normalize, hash, upsert, lookup, freshness decider
brain_api/schemas.py — GatherRequest extension, VerificationCacheWriteRequest/Response, BrainMeta extension
brain_api/app.py — POST /v1/verification_cache endpoint wired in lifespan
brain_api/services/gather.py — cache lookup after evidence build, attach to brain_meta

Glossary

Term	Meaning
atom	The unit of storage in Atomic. Markdown content + metadata + tags. Two flavors: `Type/Document` (a full source article) and `Type/Claim` (an atomic factual claim extracted from a document).
Atomic	The Rust knowledge-graph backend (kenforthewin/atomic). DidiBrain uses its REST API. Runs as `didibrain-atomic` container.
brain_api	Our FastAPI service. The HTTP face DidiBrain shows to Didi backend.
chunk	A piece of an atom's content split for embedding. Atomic does this internally; we never see chunks directly.
HIT / PARTIAL / MISS	`brain_meta.cache_status` returned by `/v1/gather`. Tells Didi backend whether the brain had relevant knowledge or it should fall back to web module.
Lint pass	Background contradiction-detection job. Reads existing claim atoms, finds contradictory pairs cross-corpus.
NLI	Natural Language Inference. The "decide if A supports / contradicts / is neutral toward B" classification. We do it twice: stance-vs-query (gather) and pair-equivalence (lint).
prompt_version	A string like "v1" pinned per prompt file. State files record it so we can bump prompts and re-process selectively.
reranker	Cross-encoder (BGE-reranker-v2-m3). Takes (query, document) pairs and outputs a precise relevance score. Used after embedding retrieval to refine top-K.
stance_in_source	What the source itself asserts about a claim (ASSERTS/REPORTS/REFUTES/QUESTIONS/NEUTRAL). Set during extraction.
stance_vs_query	What an evidence claim says about Didi user's input claim (SUPPORTS/CONTRADICTS/NEUTRAL). Set during gather's NLI stage.
TagResolver	Helper that maps canonical paths like `Country/Romania` to Atomic tag UUIDs. Loaded from `_tag_ids.json` (scripts) or refreshed from atomic at startup (brain_api container).
TAXONOMY	The canonical tree in `shared/taxonomy.py`. Single source of truth for what tags exist.

36 KiB Raw Permalink Blame History Unescape Escape

DidiBrain — Architecture & "Who Does What"

TL;DR — "WHO DOES WHAT" in one table

The big picture in one diagram

Module deep-dives

shared/ — the library every other module imports

brain_api/ — the HTTP service

extractor/ — Document → Claim transformation

lint/ — cross-corpus contradiction detection

scripts/ — operator CLI tools

infra/ — containerization

Data flow — three concrete scenarios

Scenario 1 — request time: Didi calls POST /v1/gather

Scenario 2 — operator time: fresh-server deployment

Scenario 3 — background time: Didi falls back to web module, posts result back

State files reference

Environment variables — full table

Versioned prompts

When something breaks — start here

Verification Cache (added 2026-04-23)

What

Table: brain_verification_cache

Flow

Write (backend → brain, fire-and-forget)

Read (via /v1/gather extension)

Files

Glossary

36 KiB

Raw Permalink Blame History

`shared/` — the library every other module imports

`brain_api/` — the HTTP service

`extractor/` — Document → Claim transformation

`lint/` — cross-corpus contradiction detection

`scripts/` — operator CLI tools

`infra/` — containerization

Scenario 1 — request time: Didi calls `POST /v1/gather`

Table: `brain_verification_cache`