# DidiBrain

A claim-verification knowledge cache for **Didi**, a disinformation analysis
application. DidiBrain sits between Didi's backend and Didi's live
web-gathering module and speaks the same HTTP contract, answering from
pre-ingested atoms + multilingual semantic retrieval + LLM stance
classification — typically in 1.5-6 seconds instead of 20+ seconds for a
live web crawl.

Built on top of [Atomic](https://github.com/kenforthewin/atomic), a Rust
knowledge-graph backend. DidiBrain adds the scraping, claim-level extraction,
reranker integration, NLI stance layer, HTTP contract, and the contradiction
audit job on top.

---

## What it is, in one paragraph

Didi's backend currently verifies claims by calling a live
`POST /v1/gather` endpoint that does web search + fetch + score. That's
slow (~23 s) and expensive on premium engines. DidiBrain exposes the same
HTTP contract but serves responses from a locally-grown **knowledge graph**:

- **Atoms** (documents and atomic claims) are stored in a Postgres +
  pgvector backend with multilingual embeddings (BGE-M3, 1024-dim).
- **Claim extraction** runs Qwen 3.5 397B over every ingested document to
  pull out verifiable atomic claims with source quotes and stance.
- **Retrieval** uses vector kNN + BGE-reranker-v2-m3 cross-encoder for
  precision.
- **NLI stance vs query** classifies each piece of evidence as SUPPORTS,
  CONTRADICTS, or NEUTRAL relative to the input claim — giving Didi the
  signal it really needs for a disinfo verdict.
- **`brain_meta.cache_status`** tells Didi HIT / PARTIAL / MISS so the
  backend can cleanly fall back to the expensive web module when the brain
  doesn't have relevant knowledge yet.

The brain is also **self-populating**: whenever Didi falls back to the web
module and gets fresh evidence, it can POST the result to `/v1/ingest`
and DidiBrain will create new atoms + queue claim extraction in the
background. Next time a similar claim arrives, it's a HIT.

## Tech stack

| Layer | Tech |
|---|---|
| **Brain storage** | [Atomic](https://github.com/kenforthewin/atomic) (Rust) on Postgres 16 + pgvector |
| **Embeddings** | `BAAI/bge-m3` via vLLM OpenAI-compat (1024 dim, 8K ctx, multilingual) |
| **Reranker** | `BAAI/bge-reranker-v2-m3` cross-encoder via vllm-rerank-api |
| **LLM (reasoning)** | `Qwen3.5-397B-A17B` (MoE) via llama.cpp through an OpenAI-compat router |
| **Service** | Python 3.12, FastAPI, Uvicorn, Pydantic v2, httpx, structlog, tenacity |
| **Deployment** | Docker + docker-compose (3 services: api, atomic, postgres) |

## Quick start — local dev

Assumes Docker Desktop running and `.env` filled with reachable upstream
endpoints.

```bash
cd didibrain

# 1. Copy env template and fill upstream URLs
cp .env.example .env
# Edit LLM_ROUTER_URL, EMBEDDING_URL, RERANKER_URL to reachable addresses

# 2. Bring up the stack (atomic + postgres + brain_api)
docker compose -f infra/docker-compose.yml --env-file .env up -d --build

# 3. Create a Python venv for operator scripts
python3 -m venv .venv
.venv/bin/pip install httpx pydantic pydantic-settings structlog \
                      python-dotenv tenacity rich

# 4. Bootstrap Atomic (claim instance, configure BGE-M3 provider)
.venv/bin/python scripts/02_bootstrap_atomic.py

# 5. Seed the canonical tag taxonomy
.venv/bin/python scripts/04_seed_taxonomy.py

# 6. (Optional) Import seed Wikipedia corpus — first topic: vaccines
.venv/bin/python scripts/05_import_wikipedia_seed.py

# 7. (Optional) Extract atomic claims from the imported documents
.venv/bin/python scripts/07_run_extraction.py

# 8. Verify it works
curl -fsS http://localhost:8090/health
.venv/bin/python scripts/09_brain_api_demo.py
```

Open the interactive Swagger UI at **http://localhost:8090/docs** to poke
the 5 endpoints live.

## Production deploy — fresh Linux server

A single `bash` script does the whole thing:

```bash
scp -r didibrain/ user@server:~/
ssh user@server
cd ~/didibrain
cp .env.example .env
vim .env                           # set LLM/BGE/reranker URLs for this network
./scripts/bootstrap_deploy.sh      # preflight → build → up → seed → smoke test
```

The script is idempotent — re-run it any time to pick up from wherever it
was interrupted. See `scripts/bootstrap_deploy.sh` for the exact flow and
optional env flags (`BRAIN_IMPORT_CORPUS=0` to skip the seed import, etc.).

Alternatively, run the numbered `scripts/XX_*.py` files in order by hand —
they are all idempotent and explicit.

## HTTP API — the contract

Brain-API speaks **Didi's existing web-gathering module contract**, 1:1 at
the response-shape level, with additive `brain_meta` fields that older
backends safely ignore.

| Endpoint | Purpose | Typical latency |
|---|---|---|
| `GET /health` | liveness | <10 ms |
| `GET /docs` | interactive Swagger UI | — |
| `GET /redoc` | read-only API docs | — |
| `GET /openapi.json` | machine-readable schema | — |
| `POST /v1/search` | flat list of doc-level search results | 200-400 ms |
| `POST /v1/fetch` | look up atoms by URL → extracted text | 30-100 ms |
| `POST /v1/gather` | claim → ranked evidence with NLI stance | 5-6 s (1.5 s without NLI) |
| `POST /v1/image-search` | stub (always empty list) | <5 ms |
| `POST /v1/ingest` | populate brain from web-module output | variable (async extraction) |

The response from `/v1/gather` matches the existing web-module shape
exactly plus an additive `brain_meta` object on the top level and inside
each `evidence[].provenance`. The key signal to branch on:

```python
resp = requests.post(f"{BRAIN_URL}/v1/gather", json={"claim": text}).json()
if resp["brain_meta"]["cache_status"] == "MISS":
    resp = requests.post(f"{WEB_MODULE_URL}/v1/gather", json={"claim": text}).json()
    # optional self-populating:
    requests.post(f"{BRAIN_URL}/v1/ingest",
                  json={"claim": text, "evidence": resp["evidence"]})
```

See `STATUS.md` for the full field reference.

## Architecture

```
Didi backend
   │
   │  POST /v1/gather {claim}
   ▼
┌────────────────────────────────────────────────────┐
│  brain_api :8090  (FastAPI, Uvicorn)                │
│                                                      │
│  stage context    → language detection               │
│  stage retrieval  → Atomic semantic search (top 50)  │
│  stage rerank     → BGE cross-encoder (top 15)       │
│  stage nli        → Qwen 397B stance vs query        │
│  stage evidence   → group by parent doc, shape       │
└──────┬──────────────────────────┬──────────────────┘
       │                          │
       │ HTTP (docker DNS)        │ HTTPS (VPN)
       ▼                          ▼
┌────────────────┐        ┌───────────────────────┐
│ atomic-server  │        │  BGE-M3 embeddings    │
│  :8080 (8088)  │◀──SQL──│  & BGE reranker       │
└──────┬─────────┘        │  (vLLM)               │
       │                  └───────────────────────┘
       ▼
┌────────────────┐        ┌───────────────────────┐
│ postgres       │        │  Qwen 3.5 397B-A17B   │
│ + pgvector     │        │  via LLM router       │
│  :5432 (5434)  │        │  (llama.cpp + vLLM)   │
└────────────────┘        └───────────────────────┘
```

## Project structure

```
didibrain/
├── infra/docker-compose.yml       # 3-service stack
├── brain_api/                     # FastAPI service (the main deliverable)
├── shared/                        # config, clients, taxonomy (reused everywhere)
├── extractor/                     # claim extraction (host jobs + /v1/ingest)
├── lint/                          # cross-corpus contradiction detection
├── scripts/                       # operator CLI tools (numbered 01-11)
│   └── bootstrap_deploy.sh        # fresh-server bootstrap script
├── STATUS.md                      # end-of-session snapshot + troubleshooting
└── README.md                      # this file
```

Detailed file-by-file rundown is in `STATUS.md`.

## Status

- [x] Upstream stack validation (LLM + BGE + reranker sanity gate)
- [x] Atomic running on Postgres + pgvector in compose
- [x] Canonical tag taxonomy (79 tags, 7 root namespaces)
- [x] First Wikipedia import (19 documents, EN+RO, vaccines topic)
- [x] Claim extraction (513 atoms, 1.2% hallucination filter)
- [x] Document-level retrieval validated (cross-lingual cosine 0.88-0.94)
- [x] Claim-level retrieval validated
- [x] brain_api HTTP service with Didi contract (5 endpoints)
- [x] brain_api dockerized (self-sufficient, taxonomy auto-refresh)
- [x] NLI stance vs query in `/v1/gather`
- [x] Lint pass contradiction detection (code + smoke test)
- [x] Deployment bootstrap script
- [ ] Corpus expansion — more topics, more sources (next session)
- [ ] Full Lint run on diverse corpus
- [ ] Bearer auth on brain_api (when exposing beyond loopback)
- [ ] Optional `atomic-web` frontend service

## Operational notes

- **Resource footprint**: ~200 MB RAM total across the 3 containers; brain-api
  idles at ~6% CPU, spikes to ~20% during a `/v1/gather` with NLI.
- **Image size**: brain-api Docker image is ~253 MB (Python 3.12-slim base).
- **VPN dependency**: BGE and the LLM router live on a VPN-routed 10.11.10.x
  network. If the VPN drops, `/v1/gather` returns 500 because Atomic cannot
  embed the query. Confirm upstream reachability before debugging anything
  else when search starts failing.
- **Self-sufficient startup**: brain_api pulls the current taxonomy from
  Atomic at startup, so there is no baked `_tag_ids.json` in the image and
  the container is portable across environments.
- **Idempotency**: every operator script (taxonomy seeder, Wikipedia
  importer, claim extractor, Lint pass) is idempotent via state files or
  URL-based dedup. Re-running is always safe.

## License & upstream

DidiBrain itself is private (not open source). It builds on top of
[Atomic](https://github.com/kenforthewin/atomic) (MIT). The Atomic
upstream clone at `D:\didi_brain\atomic\` is untouched — `git pull`
from upstream is always safe and doesn't conflict with anything in
this repo.