255 lines
12 KiB
Markdown
255 lines
12 KiB
Markdown
# DidiBrain
|
|
|
|
A claim-verification knowledge cache for **Didi**, a disinformation analysis
|
|
application. DidiBrain sits between Didi's backend and Didi's live
|
|
web-gathering module and speaks the same HTTP contract, answering from
|
|
pre-ingested atoms + multilingual semantic retrieval + LLM stance
|
|
classification — typically in 1.5-6 seconds instead of 20+ seconds for a
|
|
live web crawl.
|
|
|
|
Built on top of [Atomic](https://github.com/kenforthewin/atomic), a Rust
|
|
knowledge-graph backend. DidiBrain adds the scraping, claim-level extraction,
|
|
reranker integration, NLI stance layer, HTTP contract, and the contradiction
|
|
audit job on top.
|
|
|
|
---
|
|
|
|
## What it is, in one paragraph
|
|
|
|
Didi's backend currently verifies claims by calling a live
|
|
`POST /v1/gather` endpoint that does web search + fetch + score. That's
|
|
slow (~23 s) and expensive on premium engines. DidiBrain exposes the same
|
|
HTTP contract but serves responses from a locally-grown **knowledge graph**:
|
|
|
|
- **Atoms** (documents and atomic claims) are stored in a Postgres +
|
|
pgvector backend with multilingual embeddings (BGE-M3, 1024-dim).
|
|
- **Claim extraction** runs Qwen3.5 over every ingested document to
|
|
pull out verifiable atomic claims with source quotes and stance.
|
|
- **Retrieval** uses vector kNN + BGE-reranker-v2-m3 cross-encoder for
|
|
precision.
|
|
- **NLI stance vs query** classifies each piece of evidence as SUPPORTS,
|
|
CONTRADICTS, or NEUTRAL relative to the input claim — giving Didi the
|
|
signal it really needs for a disinfo verdict.
|
|
- **`brain_meta.cache_status`** tells Didi HIT / PARTIAL / MISS so the
|
|
backend can cleanly fall back to the expensive web module when the brain
|
|
doesn't have relevant knowledge yet.
|
|
|
|
The brain is also **self-populating**: whenever Didi falls back to the web
|
|
module and gets fresh evidence, it can POST the result to `/v1/ingest`
|
|
and DidiBrain will create new atoms + queue claim extraction in the
|
|
background. Next time a similar claim arrives, it's a HIT.
|
|
|
|
## Tech stack
|
|
|
|
| Layer | Tech |
|
|
|---|---|
|
|
| **Brain storage** | [Atomic](https://github.com/kenforthewin/atomic) (Rust) on Postgres 16 + pgvector |
|
|
| **Embeddings** | `BAAI/bge-m3` via vLLM OpenAI-compat (1024 dim, 8K ctx, multilingual) |
|
|
| **Reranker** | `BAAI/bge-reranker-v2-m3` cross-encoder via vllm-rerank-api |
|
|
| **LLM (reasoning)** | `qwen3.5` via vLLM through an OpenAI-compat router (`MODEL_FAST` enabled) |
|
|
| **Service** | Python 3.12, FastAPI, Uvicorn, Pydantic v2, httpx, structlog, tenacity |
|
|
| **Deployment** | Docker + docker-compose (4 services: api, atomic, postgres, scheduler) |
|
|
|
|
## Quick start — local dev
|
|
|
|
Assumes Docker Desktop running and `.env` filled with reachable upstream
|
|
endpoints.
|
|
|
|
```bash
|
|
cd didibrain
|
|
|
|
# 1. Copy env template and fill upstream URLs
|
|
cp .env.example .env
|
|
# Edit LLM_ROUTER_URL, EMBEDDING_URL, RERANKER_URL to reachable addresses
|
|
|
|
# 2. Bring up the stack (atomic + postgres + brain_api)
|
|
docker compose -f infra/docker-compose.yml --env-file .env up -d --build
|
|
|
|
# 3. Create a Python venv for operator scripts
|
|
python3 -m venv .venv
|
|
.venv/bin/pip install httpx pydantic pydantic-settings structlog \
|
|
python-dotenv tenacity rich
|
|
|
|
# 4. Bootstrap Atomic (claim instance, configure BGE-M3 provider)
|
|
.venv/bin/python scripts/02_bootstrap_atomic.py
|
|
|
|
# 5. Seed the canonical tag taxonomy
|
|
.venv/bin/python scripts/04_seed_taxonomy.py
|
|
|
|
# 6. (Optional) Import seed Wikipedia corpus — first topic: vaccines
|
|
.venv/bin/python scripts/05_import_wikipedia_seed.py
|
|
|
|
# 7. (Optional) Extract atomic claims from the imported documents
|
|
.venv/bin/python scripts/07_run_extraction.py
|
|
|
|
# 8. Verify it works
|
|
curl -fsS http://localhost:8090/health
|
|
.venv/bin/python scripts/09_brain_api_demo.py
|
|
```
|
|
|
|
Open the interactive Swagger UI at **http://localhost:8090/docs** to poke
|
|
the full route set live (~30 routes).
|
|
|
|
## Production deploy — fresh Linux server
|
|
|
|
A single `bash` script does the whole thing:
|
|
|
|
```bash
|
|
scp -r didibrain/ user@server:~/
|
|
ssh user@server
|
|
cd ~/didibrain
|
|
cp .env.example .env
|
|
vim .env # set LLM/BGE/reranker URLs for this network
|
|
./scripts/bootstrap_deploy.sh # preflight → build → up → seed → smoke test
|
|
```
|
|
|
|
The script is idempotent — re-run it any time to pick up from wherever it
|
|
was interrupted. See `scripts/bootstrap_deploy.sh` for the exact flow and
|
|
optional env flags (`BRAIN_IMPORT_CORPUS=0` to skip the seed import, etc.).
|
|
|
|
Alternatively, run the numbered `scripts/XX_*.py` files in order by hand —
|
|
they are all idempotent and explicit.
|
|
|
|
## HTTP API — the contract
|
|
|
|
Brain-API speaks **Didi's existing web-gathering module contract**, 1:1 at
|
|
the response-shape level, with additive `brain_meta` fields that older
|
|
backends safely ignore.
|
|
|
|
| Endpoint | Purpose | Typical latency |
|
|
|---|---|---|
|
|
| `GET /health` | liveness | <10 ms |
|
|
| `GET /docs` | interactive Swagger UI | — |
|
|
| `GET /redoc` | read-only API docs | — |
|
|
| `GET /openapi.json` | machine-readable schema | — |
|
|
| `POST /v1/search` | flat list of doc-level search results | 200-400 ms |
|
|
| `POST /v1/fetch` | look up atoms by URL → extracted text | 30-100 ms |
|
|
| `POST /v1/gather` | claim → ranked evidence with NLI stance | 5-6 s (1.5 s without NLI) |
|
|
| `POST /v1/image-search` | stub (always empty list) | <5 ms |
|
|
| `POST /v1/ingest` | populate brain from web-module output | variable (async extraction) |
|
|
| `POST /v1/verification_cache` | write claim verification result (claims cache) | <50 ms |
|
|
| `POST /v1/analysis_atom/lookup` | read cached analysis result (techniques / ai_tampered / claims) | <50 ms |
|
|
| `POST /v1/analysis_atom` | write analysis atom (silver/bronze by confidence) | <50 ms |
|
|
| `PATCH /v1/analysis_atom/{id}` | promote atom to gold (moderator review) | <50 ms |
|
|
| `GET /v1/analysis_atom/stats` | per-tier/component counts + 24h hit rate | <20 ms |
|
|
| `POST /v1/canonicalize` | temporal claim disambiguation (Pilon 7) | LLM-bound |
|
|
| `POST /v1/cache/invalidate` | mass invalidation with `dry_run` (Pilon 8) | variable |
|
|
| `GET /v1/cache/audit_log` | paginated audit browser | <50 ms |
|
|
| `GET/PATCH /v1/fact_status/*` | versioned fact-status layer (list/detail/versions/override) | <50 ms |
|
|
|
|
This is the full set served by `brain_api` (~30 routes including the FastAPI
|
|
auto docs); the `/v1/gather`, `/v1/search`, `/v1/fetch`, `/v1/ingest`,
|
|
`/v1/image-search` group is the web-gathering contract, the rest are the
|
|
brain-owned cache + freshness-defense layers. See `INDEX.md` for the
|
|
canonical route list.
|
|
|
|
The response from `/v1/gather` matches the existing web-module shape
|
|
exactly plus an additive `brain_meta` object on the top level and inside
|
|
each `evidence[].provenance`. The key signal to branch on:
|
|
|
|
```python
|
|
resp = requests.post(f"{BRAIN_URL}/v1/gather", json={"claim": text}).json()
|
|
if resp["brain_meta"]["cache_status"] == "MISS":
|
|
resp = requests.post(f"{WEB_MODULE_URL}/v1/gather", json={"claim": text}).json()
|
|
# optional self-populating:
|
|
requests.post(f"{BRAIN_URL}/v1/ingest",
|
|
json={"claim": text, "evidence": resp["evidence"]})
|
|
```
|
|
|
|
See `STATUS.md` for the full field reference.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Didi backend
|
|
│
|
|
│ POST /v1/gather {claim}
|
|
▼
|
|
┌────────────────────────────────────────────────────┐
|
|
│ brain_api :8090 (FastAPI, Uvicorn) │
|
|
│ │
|
|
│ stage context → language detection │
|
|
│ stage retrieval → Atomic semantic search (top 50) │
|
|
│ stage rerank → BGE cross-encoder (top 15) │
|
|
│ stage nli → Qwen3.5 stance vs query │
|
|
│ stage evidence → group by parent doc, shape │
|
|
└──────┬──────────────────────────┬──────────────────┘
|
|
│ │
|
|
│ HTTP (docker DNS) │ HTTP (docker DNS)
|
|
▼ ▼
|
|
┌────────────────┐ ┌───────────────────────┐
|
|
│ atomic-server │ │ BGE-M3 embeddings │
|
|
│ :8080 (8088) │◀──SQL──│ & BGE reranker │
|
|
└──────┬─────────┘ │ (vLLM) │
|
|
│ └───────────────────────┘
|
|
▼
|
|
┌────────────────┐ ┌───────────────────────┐
|
|
│ postgres │ │ qwen3.5 │
|
|
│ + pgvector │ │ via LLM router │
|
|
│ :5432 (5434) │ │ (vLLM) │
|
|
└────────────────┘ └───────────────────────┘
|
|
```
|
|
|
|
## Project structure
|
|
|
|
```
|
|
didibrain/
|
|
├── infra/docker-compose.yml # 4-service stack (api, atomic, postgres, scheduler)
|
|
├── brain_api/ # FastAPI service (the main deliverable)
|
|
├── shared/ # config, clients, taxonomy (reused everywhere)
|
|
├── extractor/ # claim extraction (host jobs + /v1/ingest)
|
|
├── lint/ # cross-corpus contradiction detection
|
|
├── scripts/ # operator CLI tools (numbered 01-11)
|
|
│ └── bootstrap_deploy.sh # fresh-server bootstrap script
|
|
├── STATUS.md # end-of-session snapshot + troubleshooting
|
|
└── README.md # this file
|
|
```
|
|
|
|
Detailed file-by-file rundown is in `STATUS.md`.
|
|
|
|
## Status
|
|
|
|
- [x] Upstream stack validation (LLM + BGE + reranker sanity gate)
|
|
- [x] Atomic running on Postgres + pgvector in compose
|
|
- [x] Canonical tag taxonomy (79 tags, 7 root namespaces)
|
|
- [x] First Wikipedia import (19 documents, EN+RO, vaccines topic)
|
|
- [x] Claim extraction (513 atoms, 1.2% hallucination filter)
|
|
- [x] Document-level retrieval validated (cross-lingual cosine 0.88-0.94)
|
|
- [x] Claim-level retrieval validated
|
|
- [x] brain_api HTTP service with Didi contract (full route set, ~30 routes)
|
|
- [x] brain_api dockerized (self-sufficient, taxonomy auto-refresh)
|
|
- [x] NLI stance vs query in `/v1/gather`
|
|
- [x] Lint pass contradiction detection (code + smoke test)
|
|
- [x] Deployment bootstrap script
|
|
- [ ] Corpus expansion — more topics, more sources (next session)
|
|
- [ ] Full Lint run on diverse corpus
|
|
- [ ] Bearer auth on brain_api (when exposing beyond loopback)
|
|
- [ ] Optional `atomic-web` frontend service
|
|
|
|
## Operational notes
|
|
|
|
- **Resource footprint**: ~200 MB RAM total across the 4 containers; brain-api
|
|
idles at ~6% CPU, spikes to ~20% during a `/v1/gather` with NLI.
|
|
- **Image size**: brain-api Docker image is ~253 MB (Python 3.12-slim base).
|
|
- **Upstream connectivity**: the LLM router, BGE-M3 embeddings, and the BGE
|
|
reranker are reached over Docker DNS as `didiAI-llm-api:14011`,
|
|
`didiAI-embeddings-api:14100`, and `didiAI-rerank-api:14200` on the shared
|
|
`didi-network`. If those upstreams are unreachable, `/v1/gather` returns 500
|
|
because Atomic cannot embed the query. Confirm upstream reachability before
|
|
debugging anything else when search starts failing.
|
|
- **Startup**: brain_api refreshes the `TagResolver` from Atomic at startup;
|
|
in addition `shared/_tag_ids.json` is mounted into the container (compose
|
|
volume) so background extraction — which instantiates `TagResolver`
|
|
directly, bypassing the lifespan refresh — can still resolve canonical tag
|
|
UUIDs.
|
|
- **Idempotency**: every operator script (taxonomy seeder, Wikipedia
|
|
importer, claim extractor, Lint pass) is idempotent via state files or
|
|
URL-based dedup. Re-running is always safe.
|
|
|
|
## License & upstream
|
|
|
|
DidiBrain itself is private (not open source). It builds on top of
|
|
[Atomic](https://github.com/kenforthewin/atomic) (MIT). The Atomic
|
|
upstream clone at `D:\didi_brain\atomic\` is untouched — `git pull`
|
|
from upstream is always safe and doesn't conflict with anything in
|
|
this repo.
|