DidiBrain

A claim-verification knowledge cache for Didi, a disinformation analysis application. DidiBrain sits between Didi's backend and Didi's live web-gathering module and speaks the same HTTP contract, answering from pre-ingested atoms + multilingual semantic retrieval + LLM stance classification — typically in 1.5-6 seconds instead of 20+ seconds for a live web crawl.

Built on top of Atomic, a Rust knowledge-graph backend. DidiBrain adds the scraping, claim-level extraction, reranker integration, NLI stance layer, HTTP contract, and the contradiction audit job on top.

What it is, in one paragraph

Didi's backend currently verifies claims by calling a live POST /v1/gather endpoint that does web search + fetch + score. That's slow (~23 s) and expensive on premium engines. DidiBrain exposes the same HTTP contract but serves responses from a locally-grown knowledge graph:

Atoms (documents and atomic claims) are stored in a Postgres + pgvector backend with multilingual embeddings (BGE-M3, 1024-dim).
Claim extraction runs Qwen3.5 over every ingested document to pull out verifiable atomic claims with source quotes and stance.
Retrieval uses vector kNN + BGE-reranker-v2-m3 cross-encoder for precision.
NLI stance vs query classifies each piece of evidence as SUPPORTS, CONTRADICTS, or NEUTRAL relative to the input claim — giving Didi the signal it really needs for a disinfo verdict.
brain_meta.cache_status tells Didi HIT / PARTIAL / MISS so the backend can cleanly fall back to the expensive web module when the brain doesn't have relevant knowledge yet.

The brain is also self-populating: whenever Didi falls back to the web module and gets fresh evidence, it can POST the result to /v1/ingest and DidiBrain will create new atoms + queue claim extraction in the background. Next time a similar claim arrives, it's a HIT.

Tech stack

Layer	Tech
Brain storage	Atomic (Rust) on Postgres 16 + pgvector
Embeddings	`BAAI/bge-m3` via vLLM OpenAI-compat (1024 dim, 8K ctx, multilingual)
Reranker	`BAAI/bge-reranker-v2-m3` cross-encoder via vllm-rerank-api
LLM (reasoning)	`qwen3.5` via vLLM through an OpenAI-compat router (`MODEL_FAST` enabled)
Service	Python 3.12, FastAPI, Uvicorn, Pydantic v2, httpx, structlog, tenacity
Deployment	Docker + docker-compose (4 services: api, atomic, postgres, scheduler)

Quick start — local dev

Assumes Docker Desktop running and .env filled with reachable upstream endpoints.

cd didibrain

# 1. Copy env template and fill upstream URLs
cp .env.example .env
# Edit LLM_ROUTER_URL, EMBEDDING_URL, RERANKER_URL to reachable addresses

# 2. Bring up the stack (atomic + postgres + brain_api)
docker compose -f infra/docker-compose.yml --env-file .env up -d --build

# 3. Create a Python venv for operator scripts
python3 -m venv .venv
.venv/bin/pip install httpx pydantic pydantic-settings structlog \
                      python-dotenv tenacity rich

# 4. Bootstrap Atomic (claim instance, configure BGE-M3 provider)
.venv/bin/python scripts/02_bootstrap_atomic.py

# 5. Seed the canonical tag taxonomy
.venv/bin/python scripts/04_seed_taxonomy.py

# 6. (Optional) Import seed Wikipedia corpus — first topic: vaccines
.venv/bin/python scripts/05_import_wikipedia_seed.py

# 7. (Optional) Extract atomic claims from the imported documents
.venv/bin/python scripts/07_run_extraction.py

# 8. Verify it works
curl -fsS http://localhost:8090/health
.venv/bin/python scripts/09_brain_api_demo.py

Open the interactive Swagger UI at http://localhost:8090/docs to poke the full route set live (~30 routes).

Production deploy — fresh Linux server

A single bash script does the whole thing:

scp -r didibrain/ user@server:~/
ssh user@server
cd ~/didibrain
cp .env.example .env
vim .env                           # set LLM/BGE/reranker URLs for this network
./scripts/bootstrap_deploy.sh      # preflight → build → up → seed → smoke test

The script is idempotent — re-run it any time to pick up from wherever it was interrupted. See scripts/bootstrap_deploy.sh for the exact flow and optional env flags (BRAIN_IMPORT_CORPUS=0 to skip the seed import, etc.).

Alternatively, run the numbered scripts/XX_*.py files in order by hand — they are all idempotent and explicit.

HTTP API — the contract

Brain-API speaks Didi's existing web-gathering module contract, 1:1 at the response-shape level, with additive brain_meta fields that older backends safely ignore.

Endpoint	Purpose	Typical latency
`GET /health`	liveness	<10 ms
`GET /docs`	interactive Swagger UI	—
`GET /redoc`	read-only API docs	—
`GET /openapi.json`	machine-readable schema	—
`POST /v1/search`	flat list of doc-level search results	200-400 ms
`POST /v1/fetch`	look up atoms by URL → extracted text	30-100 ms
`POST /v1/gather`	claim → ranked evidence with NLI stance	5-6 s (1.5 s without NLI)
`POST /v1/image-search`	stub (always empty list)	<5 ms
`POST /v1/ingest`	populate brain from web-module output	variable (async extraction)
`POST /v1/verification_cache`	write claim verification result (claims cache)	<50 ms
`POST /v1/analysis_atom/lookup`	read cached analysis result (techniques / ai_tampered / claims)	<50 ms
`POST /v1/analysis_atom`	write analysis atom (silver/bronze by confidence)	<50 ms
`PATCH /v1/analysis_atom/{id}`	promote atom to gold (moderator review)	<50 ms
`GET /v1/analysis_atom/stats`	per-tier/component counts + 24h hit rate	<20 ms
`POST /v1/canonicalize`	temporal claim disambiguation (Pilon 7)	LLM-bound
`POST /v1/cache/invalidate`	mass invalidation with `dry_run` (Pilon 8)	variable
`GET /v1/cache/audit_log`	paginated audit browser	<50 ms
`GET/PATCH /v1/fact_status/*`	versioned fact-status layer (list/detail/versions/override)	<50 ms

This is the full set served by brain_api (~30 routes including the FastAPI auto docs); the /v1/gather, /v1/search, /v1/fetch, /v1/ingest, /v1/image-search group is the web-gathering contract, the rest are the brain-owned cache + freshness-defense layers. See INDEX.md for the canonical route list.

The response from /v1/gather matches the existing web-module shape exactly plus an additive brain_meta object on the top level and inside each evidence[].provenance. The key signal to branch on:

resp = requests.post(f"{BRAIN_URL}/v1/gather", json={"claim": text}).json()
if resp["brain_meta"]["cache_status"] == "MISS":
    resp = requests.post(f"{WEB_MODULE_URL}/v1/gather", json={"claim": text}).json()
    # optional self-populating:
    requests.post(f"{BRAIN_URL}/v1/ingest",
                  json={"claim": text, "evidence": resp["evidence"]})

See STATUS.md for the full field reference.

Architecture

Didi backend
   │
   │  POST /v1/gather {claim}
   ▼
┌────────────────────────────────────────────────────┐
│  brain_api :8090  (FastAPI, Uvicorn)                │
│                                                      │
│  stage context    → language detection               │
│  stage retrieval  → Atomic semantic search (top 50)  │
│  stage rerank     → BGE cross-encoder (top 15)       │
│  stage nli        → Qwen3.5 stance vs query          │
│  stage evidence   → group by parent doc, shape       │
└──────┬──────────────────────────┬──────────────────┘
       │                          │
       │ HTTP (docker DNS)        │ HTTP (docker DNS)
       ▼                          ▼
┌────────────────┐        ┌───────────────────────┐
│ atomic-server  │        │  BGE-M3 embeddings    │
│  :8080 (8088)  │◀──SQL──│  & BGE reranker       │
└──────┬─────────┘        │  (vLLM)               │
       │                  └───────────────────────┘
       ▼
┌────────────────┐        ┌───────────────────────┐
│ postgres       │        │  qwen3.5              │
│ + pgvector     │        │  via LLM router       │
│  :5432 (5434)  │        │  (vLLM)               │
└────────────────┘        └───────────────────────┘

Project structure

didibrain/
├── infra/docker-compose.yml       # 4-service stack (api, atomic, postgres, scheduler)
├── brain_api/                     # FastAPI service (the main deliverable)
├── shared/                        # config, clients, taxonomy (reused everywhere)
├── extractor/                     # claim extraction (host jobs + /v1/ingest)
├── lint/                          # cross-corpus contradiction detection
├── scripts/                       # operator CLI tools (numbered 01-11)
│   └── bootstrap_deploy.sh        # fresh-server bootstrap script
├── STATUS.md                      # end-of-session snapshot + troubleshooting
└── README.md                      # this file

Detailed file-by-file rundown is in STATUS.md.

Status

Upstream stack validation (LLM + BGE + reranker sanity gate)
Atomic running on Postgres + pgvector in compose
Canonical tag taxonomy (79 tags, 7 root namespaces)
First Wikipedia import (19 documents, EN+RO, vaccines topic)
Claim extraction (513 atoms, 1.2% hallucination filter)
Document-level retrieval validated (cross-lingual cosine 0.88-0.94)
Claim-level retrieval validated
brain_api HTTP service with Didi contract (full route set, ~30 routes)
brain_api dockerized (self-sufficient, taxonomy auto-refresh)
NLI stance vs query in /v1/gather
Lint pass contradiction detection (code + smoke test)
Deployment bootstrap script
Corpus expansion — more topics, more sources (next session)
Full Lint run on diverse corpus
Bearer auth on brain_api (when exposing beyond loopback)
Optional atomic-web frontend service

Operational notes

Resource footprint: ~200 MB RAM total across the 4 containers; brain-api idles at ~6% CPU, spikes to ~20% during a /v1/gather with NLI.
Image size: brain-api Docker image is ~253 MB (Python 3.12-slim base).
Upstream connectivity: the LLM router, BGE-M3 embeddings, and the BGE reranker are reached over Docker DNS as didiAI-llm-api:14011, didiAI-embeddings-api:14100, and didiAI-rerank-api:14200 on the shared didi-network. If those upstreams are unreachable, /v1/gather returns 500 because Atomic cannot embed the query. Confirm upstream reachability before debugging anything else when search starts failing.
Startup: brain_api refreshes the TagResolver from Atomic at startup; in addition shared/_tag_ids.json is mounted into the container (compose volume) so background extraction — which instantiates TagResolver directly, bypassing the lifespan refresh — can still resolve canonical tag UUIDs.
Idempotency: every operator script (taxonomy seeder, Wikipedia importer, claim extractor, Lint pass) is idempotent via state files or URL-based dedup. Re-running is always safe.

License & upstream

DidiBrain itself is private (not open source). It builds on top of Atomic (MIT). The Atomic upstream clone at D:\didi_brain\atomic\ is untouched — git pull from upstream is always safe and doesn't conflict with anything in this repo.

12 KiB Raw Blame History