# DidiBrain — Status Snapshot **Last updated:** 2026-04-23 (integration session — web-api cache + backend verification cache) **Working dir:** `/home/admin365/ml-projects/modules/didi_brain/` **Related docs:** - `CONTRACT_VERIFICATION_CACHE.md` — contract backend↔brain pentru verification cache (v2, current) - `../../CHANGES_2026-04-23.md` — session changelog cu lista completă de schimbări - `../web/BRAIN_INTEGRATION.md` — integrarea web-api ↔ brain --- ## TL;DR DidiBrain is **SHIP READY** și **INTEGRATED**. Pe lângă contractul inițial HTTP (web-gathering compatibility), brain servește acum și ca: - **Cache layer pentru web-api** — premium ingests + read pentru ambele tiers - **Verification cache pentru didi-backend** — post-LLM stance/verdict storage cu 4 staleness states (fresh / stale_framework / stale_prompt / miss) **Trei containere, zero regressions pe funcționalitatea v1.** Tabelul relational nou `brain_verification_cache` trăiește în același Postgres cu atomic-server, prefix `brain_` pentru izolare. ## Current state in 30 seconds ``` 3 containers running continuously, all healthy didibrain-api brain_api FastAPI service :8090 52 MB RAM didibrain-atomic atomic-server (API only) :8088 83 MB RAM didibrain-postgres pgvector pg16 :5434 60 MB RAM (+ brain_verification_cache table) Brain contents (grow continuously via web-api ingest + manual bootstrap) 79 tags (canonical taxonomy, 7 root namespaces) 19+ documents (initial Wikipedia vaccines seed + anything premium adds) 509+ claims (extracted by Qwen 397B, substring validated) Verification cache (new in v2 — Apr 23) brain_verification_cache UNIQUE (claim_hash, tier) TTL 30 days, max payload 64KB, UPSERT last-wins 4 staleness states exposed via brain_meta Query latency (measured end-to-end) /v1/gather with NLI ~5-6 s /v1/gather no NLI ~1.5-2 s /v1/search ~200-400 ms /v1/fetch ~30-100 ms /v1/image-search ~5 ms (stub) /v1/ingest variable (atom create + optional async extract) ``` ## What it does Brain-API speaks **Didi's existing web-gathering module contract**. Didi's backend can call it like calling the web module today, and switch between them transparently based on `brain_meta.cache_status`: ``` Didi backend ┌───────────────────┐ │ │ DidiBrain │ │ POST /v1/gather {claim} │ (this repo) │ ├────────────────────────────────▶│ │ │ │ 1 semantic search│ │ │ 2 BGE rerank │ │ │ 3 NLI stance │ │ │ 4 aggregate │ │◀────────────────────────────────┤ │ │ GatherResponse └───────────────────┘ │ (brain_meta.cache_status = HIT | PARTIAL | MISS) │ if cache_status == MISS: fallback = http.post(WEB_MODULE, {claim}) # optional: POST /v1/ingest {fallback.evidence} # so the brain self-populates from the expensive web module ``` ## Architecture ``` ┌─────────────────────────────────────────────────────────────────┐ │ DidiBrain stack │ │ │ │ ┌──────────────────────┐ │ │ │ brain-api :8090 │ FastAPI + Uvicorn, 5 v1 endpoints │ │ │ (didibrain-api) │ speaks Didi contract 1:1 │ │ └──────────┬───────────┘ │ │ │ HTTP (docker DNS) │ │ ┌──────────▼───────────┐ ┌─────────────────────┐ │ │ │ atomic-server │ │ postgres + pgvector│ │ │ │ :8080 (8088 ext) │◀─┤ :5432 (5434 ext) │ │ │ │ (didibrain-atomic) │ │ (didibrain-postgres│ │ │ └──────┬───────┬───────┘ └─────────────────────┘ │ │ │ │ │ └─────────┼───────┼───────────────────────────────────────────────┘ │ │ │ │ HTTPS │ ▼ │ ┌────────────────────────────────┐ │ │ BGE-M3 embeddings │ 10.11.10.15:8200 │ │ (vLLM OpenAI-compat) │ 1024 dim, 8K ctx, multilingual │ └────────────────────────────────┘ │ │ ┌────────────────────────────────┐ │ │ BGE-reranker-v2-m3 │ 10.11.10.15:8100 │ │ (cross-encoder) │ precision boost │ └────────────────────────────────┘ │ │ ┌────────────────────────────────┐ └─▶│ Qwen3.5-397B-A17B │ 10.11.10.17:14011 (router) │ via LLM router │ round-robin to .18 and .19 │ (llama.cpp + vLLM backends) │ claim extraction, NLI └────────────────────────────────┘ ``` All upstream endpoints live on a VPN-routed 10.11.10.x network. The Docker containers reach them through WSL2 NAT (Docker Desktop) or host networking (Linux server). ## Repo layout ``` didibrain/ ├── .env # real endpoints + token (gitignored) ├── .env.example # template ├── .dockerignore ├── .gitignore ├── pyproject.toml ├── README.md # project landing ├── STATUS.md # this file ├── AUDIT.md # initial upstream Atomic audit │ ├── infra/ │ └── docker-compose.yml # 3-service stack (+ optional atomic-web) │ ├── brain_api/ # HTTP service — the main deliverable │ ├── Dockerfile │ ├── requirements.txt │ ├── app.py # FastAPI routes │ ├── schemas.py # Pydantic v2 models — Didi contract │ ├── deps.py # app state (clients) │ ├── run.py # uvicorn entry │ ├── prompts/ │ │ └── nli_v1.md │ └── services/ │ ├── mapping.py # atom → EvidenceItem / FetchedPage │ ├── gather.py # /v1/gather pipeline │ ├── search.py # /v1/search │ ├── fetch.py # /v1/fetch │ ├── ingest.py # /v1/ingest + background extraction │ └── nli.py # stance vs query classification │ ├── shared/ # libraries used by brain_api + scripts + lint │ ├── config.py # Pydantic Settings, LlmRole, model routing │ ├── logging.py # structlog, UTF-8 stdout, httpx silencer │ ├── llm_client.py # async OpenAI-compat wrapper │ ├── embedding_client.py # BGE-M3 embed + reranker │ ├── atomic_api.py # typed REST client for atomic-server │ └── taxonomy.py # canonical TAXONOMY tree + TagResolver │ ├── extractor/ # claim extraction service (host + background) │ ├── extract.py # single-doc extraction + substring quote validation │ ├── push.py # claim atom creation with hash URL fragment │ ├── batch.py # batch orchestrator with state file │ ├── _state.py # idempotency log │ └── prompts/ │ └── claim_extraction_v1.md │ ├── lint/ # background contradiction detection │ ├── runner.py # orchestrator │ ├── detector.py # single-pair NLI │ ├── pairs.py # candidate generation via find_similar │ ├── reporter.py # rich table output │ ├── _state.py # PairVerdict ledger (idempotent) │ └── prompts/ │ └── pair_nli_v1.md │ └── scripts/ # operator CLI tools (host Python venv) ├── 01_sanity_full.py # upstream stack validator (LLM+embed+rerank) ├── 02_bootstrap_atomic.py # claim instance, configure provider ├── 03_sanity_atomic.py # brain end-to-end (create→embed→search→cleanup) ├── 04_seed_taxonomy.py # taxonomy seeder (idempotent) ├── 05_import_wikipedia_seed.py # seed-list Wikipedia import ├── 06_validate_queries.py # smoke test doc-level retrieval ├── 07_run_extraction.py # claim extraction batch ├── 08_validate_claims.py # smoke test claim-level retrieval ├── 09_brain_api_demo.py # brain_api contract test (all 5 endpoints) ├── 10_run_lint.py # Lint pass runner (--limit --force) ├── 11_show_contradictions.py # read state file, render top contradictions └── bootstrap_deploy.sh # fresh-server deploy orchestrator ``` ## HTTP API summary — Didi contract All responses include an additive `brain_meta` object with `cache_status`, `api_version`, `implementation`, `evidence_sources`, and `total_claim_atoms_matched`. Unknown fields are safely ignored by older backends. | Endpoint | Purpose | Latency | Notes | |---|---|---|---| | `GET /health` | liveness | <10 ms | container healthcheck | | `GET /docs` | Swagger UI | — | interactive tester (FastAPI auto) | | `GET /redoc` | ReDoc | — | read-only doc view | | `GET /openapi.json` | contract JSON | — | machine-readable schema | | `POST /v1/search` | flat search results | ~200-400 ms | no rerank, no NLI | | `POST /v1/fetch` | URL lookup → text | ~30-100 ms | returns `not_in_brain` for misses | | `POST /v1/gather` | claim → ranked evidence | ~5-6 s with NLI | **the main Didi entry point** | | `POST /v1/gather` with `run_nli:false` | claim → evidence | ~1.5-2 s | skip stance classification for speed | | `POST /v1/image-search` | stub | <5 ms | always empty list | | `POST /v1/ingest` | populate from web module | variable | creates atoms + optional async extraction | ### /v1/gather response shape (key fields) ```json { "request_id": "uuid", "claim": "original claim text", "evidence": [ { "url": "https://source.example", "title": "...", "publisher": "example.com", "published_at": "2024-...", "retrieved_at": "2026-04-11T...", "summary": "the matching claim text, canonical single sentence", "full_text": "full parent document content", "full_text_hash": "sha256", "relevance_score": 0.98, "credibility_score": 0.70, "provenance": { "extraction_method": "brain", "fallback_chain": [], "brain_meta": { "parent_atom_id": "uuid", "best_claim_text": "...", "best_claim_stance_in_source": "ASSERTS|REPORTS|REFUTES|QUESTIONS|NEUTRAL", "stance_vs_query": "SUPPORTS|CONTRADICTS|NEUTRAL|UNKNOWN", "nli_confidence": 0.95, "reranker_score": 0.98, "embedding_similarity": 0.74 } } } ], "evidence_stats": { ... }, "search_context": { "primary_country": "Global", "detected_language": "en", ... }, "stages": [ { "stage": "context", "success": true, "duration_ms": 0 }, { "stage": "retrieval", "success": true, "duration_ms": 300 }, { "stage": "rerank", "success": true, "duration_ms": 1000 }, { "stage": "nli", "success": true, "duration_ms": 3800 }, { "stage": "evidence", "success": true, "duration_ms": 300 } ], "total_evidence_items": 5, "execution_time_ms": 5400, "brain_meta": { "cache_status": "HIT", "api_version": "v1", "implementation": "didibrain", "evidence_sources": 5, "total_claim_atoms_matched": 42 } } ``` The key signal for Didi backend: ```python response = requests.post(f"{BRAIN_URL}/v1/gather", json={"claim": text}) data = response.json() if data["brain_meta"]["cache_status"] == "MISS": # brain does not have relevant knowledge — fall back to live web module data = requests.post(f"{WEB_MODULE_URL}/v1/gather", json={"claim": text}).json() # optional: pump it into the brain so next time we HIT requests.post(f"{BRAIN_URL}/v1/ingest", json={"evidence": data["evidence"]}) ``` ## Deployment on a server — cheat sheet Full automated flow: ```bash scp -r didibrain/ user@server:~/ ssh user@server cd ~/didibrain cp .env.example .env vim .env # fill LLM_ROUTER_URL / EMBEDDING_URL / RERANKER_URL ./scripts/bootstrap_deploy.sh ``` Manual step-by-step is documented in `scripts/bootstrap_deploy.sh` comments or in README.md under "Production deploy". ### What to change in .env when moving machines Only the upstream endpoint URLs may need updating: ```bash LLM_ROUTER_URL=http://10.11.10.17:14011 # if router is on VPN, unchanged EMBEDDING_URL=http://10.11.10.15:8200 # if BGE is on VPN, unchanged RERANKER_URL=http://10.11.10.15:8100 # if reranker is on VPN, unchanged ``` Everything else (`ATOMIC_URL`, ports, model names, Postgres creds) is either handled by Docker DNS automatically or comes from the same file. ## Health checklist — run after any deploy/restart ```bash # 1. Containers healthy docker ps --format '{{.Names}} {{.Status}}' | grep didibrain # expected: 3 lines, all "Up ... (healthy)" # 2. Liveness curl -fsS http://localhost:8090/health # expected: {"status":"ok","service":"didibrain-api","version":"0.1.0"} # 3. Full stack sanity (LLM + embed + rerank + atomic) .venv/bin/python scripts/01_sanity_full.py # expected: 8/8 PASS, total <10s # 4. Brain end-to-end (create atom → embed → search → cleanup) .venv/bin/python scripts/03_sanity_atomic.py # expected: 5/5 PASS # 5. Real Didi-style query curl -s -X POST http://localhost:8090/v1/gather \ -H 'Content-Type: application/json' \ -d '{"claim":"vaccines cause autism","max_evidence":3,"run_nli":false}' \ | jq '.brain_meta.cache_status, .total_evidence_items' # expected: "HIT" and a positive number (if the vaccines corpus is loaded) ``` ## Known issues that DO NOT block deploy 1. **`edges_status column does not exist`** — atomic-server logs this warning at startup due to an upstream Postgres schema mismatch. The semantic_edges feature in Atomic is partially broken, but **nothing we use depends on it**: vector search, reranker, claim extraction, and the Didi contract all work. Cosmetic — safe to ignore until upstream Atomic fixes the migration. 2. **`/api/embeddings/status`** returns 500 — same root cause as #1. We never call this endpoint; we read `embedding_status` per atom via `list_atoms` or `get_atom` instead. 3. **Qwen 35B (vLLM on :14001)** — thinking mode stuck ON via the router and safety alignment refuses disinfo-extraction tasks. Disabled in config (`MODEL_FAST_ENABLED=false`); the whole pipeline runs on Qwen 397B. If a future session un-sticks 35B, Lint pass could speed up ~4x by using it. 4. **Gemma 31B endpoint (10.11.10.16:8001)** — port unreachable (host pings OK). Not used by any feature; listed as optional in `.env`. 5. **Atomic's React UI not running** — we only deploy `atomic-server` (API), not `atomic-web` (React frontend). `http://localhost:8088/` returns 404 for this reason. Brain_api has its own Swagger UI at `/docs` which covers dev-testing needs. If visual atom/tag/wiki browsing is needed, add an `atomic-web` service to compose (5-minute task). ## Troubleshooting — things that have actually gone wrong | Symptom | Root cause | Fix | |---|---|---| | `/health` timeouts | BGE endpoint unreachable (VPN down?) | `curl http://10.11.10.15:8200/v1/models` on host; bring VPN back | | `/v1/gather` returns 500 silently | Same as above — Atomic can't embed the query | Same fix | | Post-reboot: brain_api empty reply | uvicorn bound to 127.0.0.1 inside container (not 0.0.0.0) | Rebuild — Dockerfile now sets `BRAIN_API_HOST=0.0.0.0` | | `LLM_ROUTER_URL` points to localhost but nothing there | Router is actually on 10.11.10.17:14011 (not on the dev box) | Update `.env`; validated in session 2 | | Claim extraction returns zero claims on a doc | `list_atoms` returns summary without `content`; must call `get_atom` | Fixed in `extractor/batch.py` | | NLI responses come back as NEUTRAL with confidence 0.0 | MAX_PARALLEL > backend concurrency → timeouts | Set `MAX_PARALLEL=2` to match 2 llama.cpp instances | | Wikipedia API 403 on every request | Non-compliant User-Agent | Use `DidiBrain/0.1 (https://github.com/didibrain; didibrain@local.test)` | ## What is NOT done (all optional, not blockers) - **Bearer auth on brain_api** — skipped intentionally. Add when exposing beyond loopback on a server with a public IP. - **Full Lint pass on vaccines corpus** — skipped. Wikipedia is too internally consistent for Lint to surface many contradictions; rerun after corpus diversification (B). - **Corpus expansion (B)** — next session. Wikipedia seeds for RO elections 2024, Russia-Ukraine war, climate, COVID general. Also RSS feeds and possibly Playwright adapters for non-Wiki sources. - **`/v1/contradictions` endpoint in brain_api** — currently contradictions are only visible via `scripts/11_show_contradictions.py`. Expose over HTTP when Didi needs programmatic access. - **Atomic React UI (`atomic-web`)** — optional visual browsing, add if useful for operators. - **Monitoring / metrics** — no Prometheus exporter yet. - **Upstream schema fix for `edges_status`** — cosmetic, PR to kenforthewin/atomic. ## Session log (condensed) ### Session 1 (2026-04-10, ~6 hours) - Full audit of upstream Atomic repo - Validated LLM/embed/rerank stack (8 sanity checks PASS) - Built `shared/` + `extractor/` + scripts 01-08 - Stood up Atomic + Postgres in compose - Imported 19 Wikipedia vaccines articles, EN + RO - Extracted 513 claims via Qwen 397B (1.2% hallucination drop) - Validated claim-level retrieval (cross-lingual RO↔EN confirmed) ### Session 2 (2026-04-11, ~4 hours) - Re-onboarded state after VPN restart (fixed LLM router URL) - Built `brain_api/` FastAPI service with 5-endpoint Didi contract - Dockerized brain_api (self-sufficient startup, taxonomy refresh from atomic) - Added NLI stance-vs-query pass in `/v1/gather` (additive fields in `brain_meta`) - Fixed NLI concurrency (MAX_PARALLEL=2 matches llama.cpp backends) - Built `lint/` contradiction detection module with idempotent state file - Smoke-tested Lint (371 pairs, 1 genuine contradiction found at 0.95 confidence) - Wrote deployment bootstrap script - Wrote this STATUS.md and the new README.md ## Next session — pick-up plan Brain is in a stable end-of-session state. To resume: 1. Verify infra is alive (`docker ps | grep didibrain`) 2. If it is not, `docker compose -f infra/docker-compose.yml --env-file .env up -d` 3. Run `.venv/bin/python scripts/01_sanity_full.py` to confirm upstream stack 4. Decide: corpus expansion (B), Lint full run, new feature, or deploy to server **Natural next step is corpus expansion.** The code paths are complete; everything else is about feeding the brain more knowledge and then running the existing scripts over the new data.