448 lines
22 KiB
Markdown
448 lines
22 KiB
Markdown
# DidiBrain — Status Snapshot
|
|
|
|
**Last updated:** 2026-04-23 (integration session — web-api cache + backend verification cache)
|
|
|
|
> **Note (current):** the platform is **LIVE and functional**. Since this
|
|
> snapshot the stack grew to **four containers** (added `didibrain-scheduler`),
|
|
> the analysis-atom cache (v2, gold/silver/bronze) and the Cache Freshness
|
|
> Defense layer (feeder + auditor + watcher) shipped, and the live working
|
|
> model is **`qwen3.5`** (vLLM, `MODEL_FAST` enabled). Sections below that
|
|
> still say "3 containers / 5 endpoints / 397B" are corrected inline.
|
|
**Working dir:** `/home/admin365/ml-projects/modules/didi_brain/`
|
|
**Related docs:**
|
|
- `CONTRACT_VERIFICATION_CACHE.md` — contract backend↔brain pentru verification cache (v2, current)
|
|
- `../../CHANGES_2026-04-23.md` — session changelog cu lista completă de schimbări
|
|
- `../web/BRAIN_INTEGRATION.md` — integrarea web-api ↔ brain
|
|
|
|
---
|
|
|
|
## TL;DR
|
|
|
|
DidiBrain is **SHIP READY** și **INTEGRATED**. Pe lângă contractul inițial HTTP
|
|
(web-gathering compatibility), brain servește acum și ca:
|
|
- **Cache layer pentru web-api** — premium ingests + read pentru ambele tiers
|
|
- **Verification cache pentru didi-backend** — post-LLM stance/verdict storage
|
|
cu 4 staleness states (fresh / stale_framework / stale_prompt / miss)
|
|
|
|
- **Analysis atom cache (v2)** — techniques + ai_tampered + claims full-component
|
|
results, 3-tier gold/silver/bronze (since 2026-05-01)
|
|
- **Cache Freshness Defense** — `didibrain-scheduler` (feeder + auditor + watcher)
|
|
keeps cached verdicts fresh
|
|
|
|
**Patru containere, zero regressions pe funcționalitatea v1.** Tabelele relationale
|
|
noi `brain_verification_cache` + `brain_analysis_atom` (și layerul fact_status)
|
|
trăiesc în același Postgres cu atomic-server, prefix `brain_` pentru izolare.
|
|
|
|
## Current state in 30 seconds
|
|
|
|
```
|
|
4 containers running continuously, all healthy
|
|
|
|
didibrain-api brain_api FastAPI service :8090 52 MB RAM
|
|
didibrain-atomic atomic-server (API only) :8088 (8080 internal) 83 MB RAM
|
|
didibrain-postgres pgvector pg16 :5434 60 MB RAM
|
|
(+ brain_verification_cache
|
|
+ brain_analysis_atom tables)
|
|
didibrain-scheduler feeder + auditor + watcher (no port)
|
|
|
|
Brain contents (grow continuously via web-api ingest + manual bootstrap)
|
|
79 tags (canonical taxonomy, 7 root namespaces)
|
|
19+ documents (initial Wikipedia vaccines seed + anything premium adds)
|
|
509+ claims (extracted by qwen3.5, substring validated)
|
|
|
|
Verification cache (new in v2 — Apr 23)
|
|
brain_verification_cache UNIQUE (claim_hash, tier)
|
|
TTL 30 days, max payload 64KB, UPSERT last-wins
|
|
4 staleness states exposed via brain_meta
|
|
|
|
Query latency (measured end-to-end)
|
|
/v1/gather with NLI ~5-6 s
|
|
/v1/gather no NLI ~1.5-2 s
|
|
/v1/search ~200-400 ms
|
|
/v1/fetch ~30-100 ms
|
|
/v1/image-search ~5 ms (stub)
|
|
/v1/ingest variable (atom create + optional async extract)
|
|
```
|
|
|
|
## What it does
|
|
|
|
Brain-API speaks **Didi's existing web-gathering module contract**. Didi's
|
|
backend can call it like calling the web module today, and switch between
|
|
them transparently based on `brain_meta.cache_status`:
|
|
|
|
```
|
|
Didi backend ┌───────────────────┐
|
|
│ │ DidiBrain │
|
|
│ POST /v1/gather {claim} │ (this repo) │
|
|
├────────────────────────────────▶│ │
|
|
│ │ 1 semantic search│
|
|
│ │ 2 BGE rerank │
|
|
│ │ 3 NLI stance │
|
|
│ │ 4 aggregate │
|
|
│◀────────────────────────────────┤ │
|
|
│ GatherResponse └───────────────────┘
|
|
│ (brain_meta.cache_status = HIT | PARTIAL | MISS)
|
|
│
|
|
if cache_status == MISS:
|
|
fallback = http.post(WEB_MODULE, {claim})
|
|
# optional: POST /v1/ingest {fallback.evidence}
|
|
# so the brain self-populates from the expensive web module
|
|
```
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ DidiBrain stack │
|
|
│ │
|
|
│ ┌──────────────────────┐ │
|
|
│ │ brain-api :8090 │ FastAPI + Uvicorn, full route set │
|
|
│ │ (didibrain-api) │ speaks Didi contract 1:1 │
|
|
│ └──────────┬───────────┘ │
|
|
│ │ HTTP (docker DNS) │
|
|
│ ┌──────────▼───────────┐ ┌─────────────────────┐ │
|
|
│ │ atomic-server │ │ postgres + pgvector│ │
|
|
│ │ :8080 (8088 ext) │◀─┤ :5432 (5434 ext) │ │
|
|
│ │ (didibrain-atomic) │ │ (didibrain-postgres│ │
|
|
│ └──────┬───────┬───────┘ └─────────────────────┘ │
|
|
│ │ │ │
|
|
└─────────┼───────┼───────────────────────────────────────────────┘
|
|
│ │
|
|
│ │ HTTPS
|
|
│ ▼
|
|
│ ┌────────────────────────────────┐
|
|
│ │ BGE-M3 embeddings │ 10.11.10.15:14100
|
|
│ │ (vLLM OpenAI-compat) │ 1024 dim, 8K ctx, multilingual
|
|
│ └────────────────────────────────┘
|
|
│
|
|
│ ┌────────────────────────────────┐
|
|
│ │ BGE-reranker-v2-m3 │ 10.11.10.15:14200
|
|
│ │ (cross-encoder) │ precision boost
|
|
│ └────────────────────────────────┘
|
|
│
|
|
│ ┌────────────────────────────────┐
|
|
└─▶│ qwen3.5 │ 10.11.10.17:14011 (router)
|
|
│ via LLM router │ round-robin to .18 and .19
|
|
│ (vLLM backends) │ claim extraction, NLI
|
|
└────────────────────────────────┘
|
|
```
|
|
|
|
All upstream endpoints (LLM router, BGE-M3, reranker) are reached over Docker
|
|
DNS on the shared external `didi-network` as `didiAI-llm-api:14011`,
|
|
`didiAI-embeddings-api:14100`, and `didiAI-rerank-api:14200` (the 10.11.10.x
|
|
addresses above are the host-side equivalents).
|
|
|
|
## Repo layout
|
|
|
|
```
|
|
didibrain/
|
|
├── .env # real endpoints + token (gitignored)
|
|
├── .env.example # template
|
|
├── .dockerignore
|
|
├── .gitignore
|
|
├── pyproject.toml
|
|
├── README.md # project landing
|
|
├── STATUS.md # this file
|
|
├── AUDIT.md # initial upstream Atomic audit
|
|
│
|
|
├── infra/
|
|
│ └── docker-compose.yml # 4-service stack (api, atomic, postgres, scheduler)
|
|
│
|
|
├── brain_api/ # HTTP service — the main deliverable
|
|
│ ├── Dockerfile
|
|
│ ├── requirements.txt
|
|
│ ├── app.py # FastAPI routes
|
|
│ ├── schemas.py # Pydantic v2 models — Didi contract
|
|
│ ├── deps.py # app state (clients)
|
|
│ ├── run.py # uvicorn entry
|
|
│ ├── prompts/
|
|
│ │ └── nli_v1.md
|
|
│ └── services/
|
|
│ ├── mapping.py # atom → EvidenceItem / FetchedPage
|
|
│ ├── gather.py # /v1/gather pipeline
|
|
│ ├── search.py # /v1/search
|
|
│ ├── fetch.py # /v1/fetch
|
|
│ ├── ingest.py # /v1/ingest + background extraction
|
|
│ └── nli.py # stance vs query classification
|
|
│
|
|
├── shared/ # libraries used by brain_api + scripts + lint
|
|
│ ├── config.py # Pydantic Settings, LlmRole, model routing
|
|
│ ├── logging.py # structlog, UTF-8 stdout, httpx silencer
|
|
│ ├── llm_client.py # async OpenAI-compat wrapper
|
|
│ ├── embedding_client.py # BGE-M3 embed + reranker
|
|
│ ├── atomic_api.py # typed REST client for atomic-server
|
|
│ └── taxonomy.py # canonical TAXONOMY tree + TagResolver
|
|
│
|
|
├── extractor/ # claim extraction service (host + background)
|
|
│ ├── extract.py # single-doc extraction + substring quote validation
|
|
│ ├── push.py # claim atom creation with hash URL fragment
|
|
│ ├── batch.py # batch orchestrator with state file
|
|
│ ├── _state.py # idempotency log
|
|
│ └── prompts/
|
|
│ └── claim_extraction_v1.md
|
|
│
|
|
├── lint/ # background contradiction detection
|
|
│ ├── runner.py # orchestrator
|
|
│ ├── detector.py # single-pair NLI
|
|
│ ├── pairs.py # candidate generation via find_similar
|
|
│ ├── reporter.py # rich table output
|
|
│ ├── _state.py # PairVerdict ledger (idempotent)
|
|
│ └── prompts/
|
|
│ └── pair_nli_v1.md
|
|
│
|
|
└── scripts/ # operator CLI tools (host Python venv)
|
|
├── 01_sanity_full.py # upstream stack validator (LLM+embed+rerank)
|
|
├── 02_bootstrap_atomic.py # claim instance, configure provider
|
|
├── 03_sanity_atomic.py # brain end-to-end (create→embed→search→cleanup)
|
|
├── 04_seed_taxonomy.py # taxonomy seeder (idempotent)
|
|
├── 05_import_wikipedia_seed.py # seed-list Wikipedia import
|
|
├── 06_validate_queries.py # smoke test doc-level retrieval
|
|
├── 07_run_extraction.py # claim extraction batch
|
|
├── 08_validate_claims.py # smoke test claim-level retrieval
|
|
├── 09_brain_api_demo.py # brain_api contract test (core web-gathering endpoints)
|
|
├── 10_run_lint.py # Lint pass runner (--limit --force)
|
|
├── 11_show_contradictions.py # read state file, render top contradictions
|
|
└── bootstrap_deploy.sh # fresh-server deploy orchestrator
|
|
```
|
|
|
|
## HTTP API summary — Didi contract
|
|
|
|
All responses include an additive `brain_meta` object with `cache_status`,
|
|
`api_version`, `implementation`, `evidence_sources`, and
|
|
`total_claim_atoms_matched`. Unknown fields are safely ignored by older
|
|
backends.
|
|
|
|
| Endpoint | Purpose | Latency | Notes |
|
|
|---|---|---|---|
|
|
| `GET /health` | liveness | <10 ms | container healthcheck |
|
|
| `GET /docs` | Swagger UI | — | interactive tester (FastAPI auto) |
|
|
| `GET /redoc` | ReDoc | — | read-only doc view |
|
|
| `GET /openapi.json` | contract JSON | — | machine-readable schema |
|
|
| `POST /v1/search` | flat search results | ~200-400 ms | no rerank, no NLI |
|
|
| `POST /v1/fetch` | URL lookup → text | ~30-100 ms | returns `not_in_brain` for misses |
|
|
| `POST /v1/gather` | claim → ranked evidence | ~5-6 s with NLI | **the main Didi entry point** |
|
|
| `POST /v1/gather` with `run_nli:false` | claim → evidence | ~1.5-2 s | skip stance classification for speed |
|
|
| `POST /v1/image-search` | stub | <5 ms | always empty list |
|
|
| `POST /v1/ingest` | populate from web module | variable | creates atoms + optional async extraction |
|
|
| `POST /v1/verification_cache` | write claim verification result | <50 ms | claims cache (v2) |
|
|
| `POST /v1/analysis_atom/lookup` | read cached analysis | <50 ms | techniques / ai_tampered / claims |
|
|
| `POST /v1/analysis_atom` | write analysis atom | <50 ms | silver/bronze by `llm_confidence` |
|
|
| `PATCH /v1/analysis_atom/{id}` | promote to gold | <50 ms | moderator review |
|
|
| `GET /v1/analysis_atom/stats` | per-tier/component counts | <20 ms | 24h hit rate |
|
|
| `POST /v1/canonicalize` | temporal claim disambiguation | LLM-bound | Pilon 7 |
|
|
| `POST /v1/cache/invalidate` | mass invalidation (`dry_run`) | variable | Pilon 8 |
|
|
| `GET /v1/cache/audit_log` | audit browser | <50 ms | paginated |
|
|
| `GET/PATCH /v1/fact_status/*` | versioned fact-status layer | <50 ms | list/detail/versions/override |
|
|
|
|
(Full set ~30 routes incl. FastAPI auto docs; see `INDEX.md` for the canonical list.)
|
|
|
|
### /v1/gather response shape (key fields)
|
|
|
|
```json
|
|
{
|
|
"request_id": "uuid",
|
|
"claim": "original claim text",
|
|
"evidence": [
|
|
{
|
|
"url": "https://source.example",
|
|
"title": "...",
|
|
"publisher": "example.com",
|
|
"published_at": "2024-...",
|
|
"retrieved_at": "2026-04-11T...",
|
|
"summary": "the matching claim text, canonical single sentence",
|
|
"full_text": "full parent document content",
|
|
"full_text_hash": "sha256",
|
|
"relevance_score": 0.98,
|
|
"credibility_score": 0.70,
|
|
"provenance": {
|
|
"extraction_method": "brain",
|
|
"fallback_chain": [],
|
|
"brain_meta": {
|
|
"parent_atom_id": "uuid",
|
|
"best_claim_text": "...",
|
|
"best_claim_stance_in_source": "ASSERTS|REPORTS|REFUTES|QUESTIONS|NEUTRAL",
|
|
"stance_vs_query": "SUPPORTS|CONTRADICTS|NEUTRAL|UNKNOWN",
|
|
"nli_confidence": 0.95,
|
|
"reranker_score": 0.98,
|
|
"embedding_similarity": 0.74
|
|
}
|
|
}
|
|
}
|
|
],
|
|
"evidence_stats": { ... },
|
|
"search_context": { "primary_country": "Global", "detected_language": "en", ... },
|
|
"stages": [
|
|
{ "stage": "context", "success": true, "duration_ms": 0 },
|
|
{ "stage": "retrieval", "success": true, "duration_ms": 300 },
|
|
{ "stage": "rerank", "success": true, "duration_ms": 1000 },
|
|
{ "stage": "nli", "success": true, "duration_ms": 3800 },
|
|
{ "stage": "evidence", "success": true, "duration_ms": 300 }
|
|
],
|
|
"total_evidence_items": 5,
|
|
"execution_time_ms": 5400,
|
|
"brain_meta": {
|
|
"cache_status": "HIT",
|
|
"api_version": "v1",
|
|
"implementation": "didibrain",
|
|
"evidence_sources": 5,
|
|
"total_claim_atoms_matched": 42
|
|
}
|
|
}
|
|
```
|
|
|
|
The key signal for Didi backend:
|
|
|
|
```python
|
|
response = requests.post(f"{BRAIN_URL}/v1/gather", json={"claim": text})
|
|
data = response.json()
|
|
if data["brain_meta"]["cache_status"] == "MISS":
|
|
# brain does not have relevant knowledge — fall back to live web module
|
|
data = requests.post(f"{WEB_MODULE_URL}/v1/gather", json={"claim": text}).json()
|
|
# optional: pump it into the brain so next time we HIT
|
|
requests.post(f"{BRAIN_URL}/v1/ingest", json={"evidence": data["evidence"]})
|
|
```
|
|
|
|
## Deployment on a server — cheat sheet
|
|
|
|
Full automated flow:
|
|
|
|
```bash
|
|
scp -r didibrain/ user@server:~/
|
|
ssh user@server
|
|
cd ~/didibrain
|
|
cp .env.example .env
|
|
vim .env # fill LLM_ROUTER_URL / EMBEDDING_URL / RERANKER_URL
|
|
./scripts/bootstrap_deploy.sh
|
|
```
|
|
|
|
Manual step-by-step is documented in `scripts/bootstrap_deploy.sh` comments
|
|
or in README.md under "Production deploy".
|
|
|
|
### What to change in .env when moving machines
|
|
|
|
Only the upstream endpoint URLs may need updating:
|
|
|
|
```bash
|
|
LLM_ROUTER_URL=http://10.11.10.17:14011 # didiAI-llm-api, unchanged
|
|
EMBEDDING_URL=http://10.11.10.15:14100 # didiAI-embeddings-api, unchanged
|
|
RERANKER_URL=http://10.11.10.15:14200 # didiAI-rerank-api, unchanged
|
|
```
|
|
|
|
Everything else (`ATOMIC_URL`, ports, model names, Postgres creds) is either
|
|
handled by Docker DNS automatically or comes from the same file.
|
|
|
|
## Health checklist — run after any deploy/restart
|
|
|
|
```bash
|
|
# 1. Containers healthy
|
|
docker ps --format '{{.Names}} {{.Status}}' | grep didibrain
|
|
# expected: 3 lines, all "Up ... (healthy)"
|
|
|
|
# 2. Liveness
|
|
curl -fsS http://localhost:8090/health
|
|
# expected: {"status":"ok","service":"didibrain-api","version":"0.1.0"}
|
|
|
|
# 3. Full stack sanity (LLM + embed + rerank + atomic)
|
|
.venv/bin/python scripts/01_sanity_full.py
|
|
# expected: 8/8 PASS, total <10s
|
|
|
|
# 4. Brain end-to-end (create atom → embed → search → cleanup)
|
|
.venv/bin/python scripts/03_sanity_atomic.py
|
|
# expected: 5/5 PASS
|
|
|
|
# 5. Real Didi-style query
|
|
curl -s -X POST http://localhost:8090/v1/gather \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{"claim":"vaccines cause autism","max_evidence":3,"run_nli":false}' \
|
|
| jq '.brain_meta.cache_status, .total_evidence_items'
|
|
# expected: "HIT" and a positive number (if the vaccines corpus is loaded)
|
|
```
|
|
|
|
## Known issues that DO NOT block deploy
|
|
|
|
1. **`edges_status column does not exist`** — atomic-server logs this warning
|
|
at startup due to an upstream Postgres schema mismatch. The semantic_edges
|
|
feature in Atomic is partially broken, but **nothing we use depends on it**:
|
|
vector search, reranker, claim extraction, and the Didi contract all work.
|
|
Cosmetic — safe to ignore until upstream Atomic fixes the migration.
|
|
|
|
2. **`/api/embeddings/status`** returns 500 — same root cause as #1. We never
|
|
call this endpoint; we read `embedding_status` per atom via `list_atoms`
|
|
or `get_atom` instead.
|
|
|
|
3. **Fast model `qwen3.5`** — now **enabled** (`MODEL_FAST_ENABLED=true`) and
|
|
is the live working model for the whole pipeline (extraction, NLI, gather)
|
|
via the LLM router at `didiAI-llm-api:14011` (vLLM). (Historically the fast
|
|
model was disabled and the pipeline ran on a separate reasoning model; that
|
|
is no longer the case.)
|
|
|
|
4. **Atomic's React UI not running** — we only deploy `atomic-server` (API),
|
|
not `atomic-web` (React frontend). `http://localhost:8088/` returns 404
|
|
for this reason. Brain_api has its own Swagger UI at `/docs` which covers
|
|
dev-testing needs. If visual atom/tag/wiki browsing is needed, add an
|
|
`atomic-web` service to compose (5-minute task).
|
|
|
|
## Troubleshooting — things that have actually gone wrong
|
|
|
|
| Symptom | Root cause | Fix |
|
|
|---|---|---|
|
|
| `/health` timeouts | BGE endpoint unreachable | `curl http://10.11.10.15:14100/v1/models` on host; restore upstream reachability |
|
|
| `/v1/gather` returns 500 silently | Same as above — Atomic can't embed the query | Same fix |
|
|
| Post-reboot: brain_api empty reply | uvicorn bound to 127.0.0.1 inside container (not 0.0.0.0) | Rebuild — Dockerfile now sets `BRAIN_API_HOST=0.0.0.0` |
|
|
| `LLM_ROUTER_URL` points to localhost but nothing there | Router is actually on 10.11.10.17:14011 (not on the dev box) | Update `.env`; validated in session 2 |
|
|
| Claim extraction returns zero claims on a doc | `list_atoms` returns summary without `content`; must call `get_atom` | Fixed in `extractor/batch.py` |
|
|
| NLI responses come back as NEUTRAL with confidence 0.0 | MAX_PARALLEL > backend concurrency → timeouts | Set `MAX_PARALLEL=2` to match 2 llama.cpp instances |
|
|
| Wikipedia API 403 on every request | Non-compliant User-Agent | Use `DidiBrain/0.1 (https://github.com/didibrain; didibrain@local.test)` |
|
|
|
|
## What is NOT done (all optional, not blockers)
|
|
|
|
- **Bearer auth on brain_api** — skipped intentionally. Add when exposing
|
|
beyond loopback on a server with a public IP.
|
|
- **Full Lint pass on vaccines corpus** — skipped. Wikipedia is too
|
|
internally consistent for Lint to surface many contradictions; rerun
|
|
after corpus diversification (B).
|
|
- **Corpus expansion (B)** — next session. Wikipedia seeds for RO elections
|
|
2024, Russia-Ukraine war, climate, COVID general. Also RSS feeds and
|
|
possibly Playwright adapters for non-Wiki sources.
|
|
- **`/v1/contradictions` endpoint in brain_api** — currently contradictions
|
|
are only visible via `scripts/11_show_contradictions.py`. Expose over
|
|
HTTP when Didi needs programmatic access.
|
|
- **Atomic React UI (`atomic-web`)** — optional visual browsing, add if
|
|
useful for operators.
|
|
- **Monitoring / metrics** — no Prometheus exporter yet.
|
|
- **Upstream schema fix for `edges_status`** — cosmetic, PR to kenforthewin/atomic.
|
|
|
|
## Session log (condensed)
|
|
|
|
### Session 1 (2026-04-10, ~6 hours)
|
|
- Full audit of upstream Atomic repo
|
|
- Validated LLM/embed/rerank stack (8 sanity checks PASS)
|
|
- Built `shared/` + `extractor/` + scripts 01-08
|
|
- Stood up Atomic + Postgres in compose
|
|
- Imported 19 Wikipedia vaccines articles, EN + RO
|
|
- Extracted 513 claims via qwen3.5 (1.2% hallucination drop)
|
|
- Validated claim-level retrieval (cross-lingual RO↔EN confirmed)
|
|
|
|
### Session 2 (2026-04-11, ~4 hours)
|
|
- Re-onboarded state after VPN restart (fixed LLM router URL)
|
|
- Built `brain_api/` FastAPI service with the Didi web-gathering contract
|
|
- Dockerized brain_api (self-sufficient startup, taxonomy refresh from atomic)
|
|
- Added NLI stance-vs-query pass in `/v1/gather` (additive fields in `brain_meta`)
|
|
- Fixed NLI concurrency (MAX_PARALLEL=2 matches llama.cpp backends)
|
|
- Built `lint/` contradiction detection module with idempotent state file
|
|
- Smoke-tested Lint (371 pairs, 1 genuine contradiction found at 0.95 confidence)
|
|
- Wrote deployment bootstrap script
|
|
- Wrote this STATUS.md and the new README.md
|
|
|
|
## Next session — pick-up plan
|
|
|
|
Brain is in a stable end-of-session state. To resume:
|
|
|
|
1. Verify infra is alive (`docker ps | grep didibrain`)
|
|
2. If it is not, `docker compose -f infra/docker-compose.yml --env-file .env up -d`
|
|
3. Run `.venv/bin/python scripts/01_sanity_full.py` to confirm upstream stack
|
|
4. Decide: corpus expansion (B), Lint full run, new feature, or deploy to server
|
|
|
|
**Natural next step is corpus expansion.** The code paths are complete;
|
|
everything else is about feeding the brain more knowledge and then running
|
|
the existing scripts over the new data.
|