Livrare LOT 1 - Didi
This commit is contained in:
commit
5380c3fc63
990 changed files with 133308 additions and 0 deletions
424
ai_platform/modules/didi_brain/STATUS.md
Normal file
424
ai_platform/modules/didi_brain/STATUS.md
Normal file
|
|
@ -0,0 +1,424 @@
|
|||
# DidiBrain — Status Snapshot
|
||||
|
||||
**Last updated:** 2026-04-23 (integration session — web-api cache + backend verification cache)
|
||||
**Working dir:** `/home/admin365/ml-projects/modules/didi_brain/`
|
||||
**Related docs:**
|
||||
- `CONTRACT_VERIFICATION_CACHE.md` — contract backend↔brain pentru verification cache (v2, current)
|
||||
- `../../CHANGES_2026-04-23.md` — session changelog cu lista completă de schimbări
|
||||
- `../web/BRAIN_INTEGRATION.md` — integrarea web-api ↔ brain
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
DidiBrain is **SHIP READY** și **INTEGRATED**. Pe lângă contractul inițial HTTP
|
||||
(web-gathering compatibility), brain servește acum și ca:
|
||||
- **Cache layer pentru web-api** — premium ingests + read pentru ambele tiers
|
||||
- **Verification cache pentru didi-backend** — post-LLM stance/verdict storage
|
||||
cu 4 staleness states (fresh / stale_framework / stale_prompt / miss)
|
||||
|
||||
**Trei containere, zero regressions pe funcționalitatea v1.** Tabelul relational
|
||||
nou `brain_verification_cache` trăiește în același Postgres cu atomic-server,
|
||||
prefix `brain_` pentru izolare.
|
||||
|
||||
## Current state in 30 seconds
|
||||
|
||||
```
|
||||
3 containers running continuously, all healthy
|
||||
|
||||
didibrain-api brain_api FastAPI service :8090 52 MB RAM
|
||||
didibrain-atomic atomic-server (API only) :8088 83 MB RAM
|
||||
didibrain-postgres pgvector pg16 :5434 60 MB RAM
|
||||
(+ brain_verification_cache table)
|
||||
|
||||
Brain contents (grow continuously via web-api ingest + manual bootstrap)
|
||||
79 tags (canonical taxonomy, 7 root namespaces)
|
||||
19+ documents (initial Wikipedia vaccines seed + anything premium adds)
|
||||
509+ claims (extracted by Qwen 397B, substring validated)
|
||||
|
||||
Verification cache (new in v2 — Apr 23)
|
||||
brain_verification_cache UNIQUE (claim_hash, tier)
|
||||
TTL 30 days, max payload 64KB, UPSERT last-wins
|
||||
4 staleness states exposed via brain_meta
|
||||
|
||||
Query latency (measured end-to-end)
|
||||
/v1/gather with NLI ~5-6 s
|
||||
/v1/gather no NLI ~1.5-2 s
|
||||
/v1/search ~200-400 ms
|
||||
/v1/fetch ~30-100 ms
|
||||
/v1/image-search ~5 ms (stub)
|
||||
/v1/ingest variable (atom create + optional async extract)
|
||||
```
|
||||
|
||||
## What it does
|
||||
|
||||
Brain-API speaks **Didi's existing web-gathering module contract**. Didi's
|
||||
backend can call it like calling the web module today, and switch between
|
||||
them transparently based on `brain_meta.cache_status`:
|
||||
|
||||
```
|
||||
Didi backend ┌───────────────────┐
|
||||
│ │ DidiBrain │
|
||||
│ POST /v1/gather {claim} │ (this repo) │
|
||||
├────────────────────────────────▶│ │
|
||||
│ │ 1 semantic search│
|
||||
│ │ 2 BGE rerank │
|
||||
│ │ 3 NLI stance │
|
||||
│ │ 4 aggregate │
|
||||
│◀────────────────────────────────┤ │
|
||||
│ GatherResponse └───────────────────┘
|
||||
│ (brain_meta.cache_status = HIT | PARTIAL | MISS)
|
||||
│
|
||||
if cache_status == MISS:
|
||||
fallback = http.post(WEB_MODULE, {claim})
|
||||
# optional: POST /v1/ingest {fallback.evidence}
|
||||
# so the brain self-populates from the expensive web module
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ DidiBrain stack │
|
||||
│ │
|
||||
│ ┌──────────────────────┐ │
|
||||
│ │ brain-api :8090 │ FastAPI + Uvicorn, 5 v1 endpoints │
|
||||
│ │ (didibrain-api) │ speaks Didi contract 1:1 │
|
||||
│ └──────────┬───────────┘ │
|
||||
│ │ HTTP (docker DNS) │
|
||||
│ ┌──────────▼───────────┐ ┌─────────────────────┐ │
|
||||
│ │ atomic-server │ │ postgres + pgvector│ │
|
||||
│ │ :8080 (8088 ext) │◀─┤ :5432 (5434 ext) │ │
|
||||
│ │ (didibrain-atomic) │ │ (didibrain-postgres│ │
|
||||
│ └──────┬───────┬───────┘ └─────────────────────┘ │
|
||||
│ │ │ │
|
||||
└─────────┼───────┼───────────────────────────────────────────────┘
|
||||
│ │
|
||||
│ │ HTTPS
|
||||
│ ▼
|
||||
│ ┌────────────────────────────────┐
|
||||
│ │ BGE-M3 embeddings │ 10.11.10.15:8200
|
||||
│ │ (vLLM OpenAI-compat) │ 1024 dim, 8K ctx, multilingual
|
||||
│ └────────────────────────────────┘
|
||||
│
|
||||
│ ┌────────────────────────────────┐
|
||||
│ │ BGE-reranker-v2-m3 │ 10.11.10.15:8100
|
||||
│ │ (cross-encoder) │ precision boost
|
||||
│ └────────────────────────────────┘
|
||||
│
|
||||
│ ┌────────────────────────────────┐
|
||||
└─▶│ Qwen3.5-397B-A17B │ 10.11.10.17:14011 (router)
|
||||
│ via LLM router │ round-robin to .18 and .19
|
||||
│ (llama.cpp + vLLM backends) │ claim extraction, NLI
|
||||
└────────────────────────────────┘
|
||||
```
|
||||
|
||||
All upstream endpoints live on a VPN-routed 10.11.10.x network. The Docker
|
||||
containers reach them through WSL2 NAT (Docker Desktop) or host networking
|
||||
(Linux server).
|
||||
|
||||
## Repo layout
|
||||
|
||||
```
|
||||
didibrain/
|
||||
├── .env # real endpoints + token (gitignored)
|
||||
├── .env.example # template
|
||||
├── .dockerignore
|
||||
├── .gitignore
|
||||
├── pyproject.toml
|
||||
├── README.md # project landing
|
||||
├── STATUS.md # this file
|
||||
├── AUDIT.md # initial upstream Atomic audit
|
||||
│
|
||||
├── infra/
|
||||
│ └── docker-compose.yml # 3-service stack (+ optional atomic-web)
|
||||
│
|
||||
├── brain_api/ # HTTP service — the main deliverable
|
||||
│ ├── Dockerfile
|
||||
│ ├── requirements.txt
|
||||
│ ├── app.py # FastAPI routes
|
||||
│ ├── schemas.py # Pydantic v2 models — Didi contract
|
||||
│ ├── deps.py # app state (clients)
|
||||
│ ├── run.py # uvicorn entry
|
||||
│ ├── prompts/
|
||||
│ │ └── nli_v1.md
|
||||
│ └── services/
|
||||
│ ├── mapping.py # atom → EvidenceItem / FetchedPage
|
||||
│ ├── gather.py # /v1/gather pipeline
|
||||
│ ├── search.py # /v1/search
|
||||
│ ├── fetch.py # /v1/fetch
|
||||
│ ├── ingest.py # /v1/ingest + background extraction
|
||||
│ └── nli.py # stance vs query classification
|
||||
│
|
||||
├── shared/ # libraries used by brain_api + scripts + lint
|
||||
│ ├── config.py # Pydantic Settings, LlmRole, model routing
|
||||
│ ├── logging.py # structlog, UTF-8 stdout, httpx silencer
|
||||
│ ├── llm_client.py # async OpenAI-compat wrapper
|
||||
│ ├── embedding_client.py # BGE-M3 embed + reranker
|
||||
│ ├── atomic_api.py # typed REST client for atomic-server
|
||||
│ └── taxonomy.py # canonical TAXONOMY tree + TagResolver
|
||||
│
|
||||
├── extractor/ # claim extraction service (host + background)
|
||||
│ ├── extract.py # single-doc extraction + substring quote validation
|
||||
│ ├── push.py # claim atom creation with hash URL fragment
|
||||
│ ├── batch.py # batch orchestrator with state file
|
||||
│ ├── _state.py # idempotency log
|
||||
│ └── prompts/
|
||||
│ └── claim_extraction_v1.md
|
||||
│
|
||||
├── lint/ # background contradiction detection
|
||||
│ ├── runner.py # orchestrator
|
||||
│ ├── detector.py # single-pair NLI
|
||||
│ ├── pairs.py # candidate generation via find_similar
|
||||
│ ├── reporter.py # rich table output
|
||||
│ ├── _state.py # PairVerdict ledger (idempotent)
|
||||
│ └── prompts/
|
||||
│ └── pair_nli_v1.md
|
||||
│
|
||||
└── scripts/ # operator CLI tools (host Python venv)
|
||||
├── 01_sanity_full.py # upstream stack validator (LLM+embed+rerank)
|
||||
├── 02_bootstrap_atomic.py # claim instance, configure provider
|
||||
├── 03_sanity_atomic.py # brain end-to-end (create→embed→search→cleanup)
|
||||
├── 04_seed_taxonomy.py # taxonomy seeder (idempotent)
|
||||
├── 05_import_wikipedia_seed.py # seed-list Wikipedia import
|
||||
├── 06_validate_queries.py # smoke test doc-level retrieval
|
||||
├── 07_run_extraction.py # claim extraction batch
|
||||
├── 08_validate_claims.py # smoke test claim-level retrieval
|
||||
├── 09_brain_api_demo.py # brain_api contract test (all 5 endpoints)
|
||||
├── 10_run_lint.py # Lint pass runner (--limit --force)
|
||||
├── 11_show_contradictions.py # read state file, render top contradictions
|
||||
└── bootstrap_deploy.sh # fresh-server deploy orchestrator
|
||||
```
|
||||
|
||||
## HTTP API summary — Didi contract
|
||||
|
||||
All responses include an additive `brain_meta` object with `cache_status`,
|
||||
`api_version`, `implementation`, `evidence_sources`, and
|
||||
`total_claim_atoms_matched`. Unknown fields are safely ignored by older
|
||||
backends.
|
||||
|
||||
| Endpoint | Purpose | Latency | Notes |
|
||||
|---|---|---|---|
|
||||
| `GET /health` | liveness | <10 ms | container healthcheck |
|
||||
| `GET /docs` | Swagger UI | — | interactive tester (FastAPI auto) |
|
||||
| `GET /redoc` | ReDoc | — | read-only doc view |
|
||||
| `GET /openapi.json` | contract JSON | — | machine-readable schema |
|
||||
| `POST /v1/search` | flat search results | ~200-400 ms | no rerank, no NLI |
|
||||
| `POST /v1/fetch` | URL lookup → text | ~30-100 ms | returns `not_in_brain` for misses |
|
||||
| `POST /v1/gather` | claim → ranked evidence | ~5-6 s with NLI | **the main Didi entry point** |
|
||||
| `POST /v1/gather` with `run_nli:false` | claim → evidence | ~1.5-2 s | skip stance classification for speed |
|
||||
| `POST /v1/image-search` | stub | <5 ms | always empty list |
|
||||
| `POST /v1/ingest` | populate from web module | variable | creates atoms + optional async extraction |
|
||||
|
||||
### /v1/gather response shape (key fields)
|
||||
|
||||
```json
|
||||
{
|
||||
"request_id": "uuid",
|
||||
"claim": "original claim text",
|
||||
"evidence": [
|
||||
{
|
||||
"url": "https://source.example",
|
||||
"title": "...",
|
||||
"publisher": "example.com",
|
||||
"published_at": "2024-...",
|
||||
"retrieved_at": "2026-04-11T...",
|
||||
"summary": "the matching claim text, canonical single sentence",
|
||||
"full_text": "full parent document content",
|
||||
"full_text_hash": "sha256",
|
||||
"relevance_score": 0.98,
|
||||
"credibility_score": 0.70,
|
||||
"provenance": {
|
||||
"extraction_method": "brain",
|
||||
"fallback_chain": [],
|
||||
"brain_meta": {
|
||||
"parent_atom_id": "uuid",
|
||||
"best_claim_text": "...",
|
||||
"best_claim_stance_in_source": "ASSERTS|REPORTS|REFUTES|QUESTIONS|NEUTRAL",
|
||||
"stance_vs_query": "SUPPORTS|CONTRADICTS|NEUTRAL|UNKNOWN",
|
||||
"nli_confidence": 0.95,
|
||||
"reranker_score": 0.98,
|
||||
"embedding_similarity": 0.74
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"evidence_stats": { ... },
|
||||
"search_context": { "primary_country": "Global", "detected_language": "en", ... },
|
||||
"stages": [
|
||||
{ "stage": "context", "success": true, "duration_ms": 0 },
|
||||
{ "stage": "retrieval", "success": true, "duration_ms": 300 },
|
||||
{ "stage": "rerank", "success": true, "duration_ms": 1000 },
|
||||
{ "stage": "nli", "success": true, "duration_ms": 3800 },
|
||||
{ "stage": "evidence", "success": true, "duration_ms": 300 }
|
||||
],
|
||||
"total_evidence_items": 5,
|
||||
"execution_time_ms": 5400,
|
||||
"brain_meta": {
|
||||
"cache_status": "HIT",
|
||||
"api_version": "v1",
|
||||
"implementation": "didibrain",
|
||||
"evidence_sources": 5,
|
||||
"total_claim_atoms_matched": 42
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The key signal for Didi backend:
|
||||
|
||||
```python
|
||||
response = requests.post(f"{BRAIN_URL}/v1/gather", json={"claim": text})
|
||||
data = response.json()
|
||||
if data["brain_meta"]["cache_status"] == "MISS":
|
||||
# brain does not have relevant knowledge — fall back to live web module
|
||||
data = requests.post(f"{WEB_MODULE_URL}/v1/gather", json={"claim": text}).json()
|
||||
# optional: pump it into the brain so next time we HIT
|
||||
requests.post(f"{BRAIN_URL}/v1/ingest", json={"evidence": data["evidence"]})
|
||||
```
|
||||
|
||||
## Deployment on a server — cheat sheet
|
||||
|
||||
Full automated flow:
|
||||
|
||||
```bash
|
||||
scp -r didibrain/ user@server:~/
|
||||
ssh user@server
|
||||
cd ~/didibrain
|
||||
cp .env.example .env
|
||||
vim .env # fill LLM_ROUTER_URL / EMBEDDING_URL / RERANKER_URL
|
||||
./scripts/bootstrap_deploy.sh
|
||||
```
|
||||
|
||||
Manual step-by-step is documented in `scripts/bootstrap_deploy.sh` comments
|
||||
or in README.md under "Production deploy".
|
||||
|
||||
### What to change in .env when moving machines
|
||||
|
||||
Only the upstream endpoint URLs may need updating:
|
||||
|
||||
```bash
|
||||
LLM_ROUTER_URL=http://10.11.10.17:14011 # if router is on VPN, unchanged
|
||||
EMBEDDING_URL=http://10.11.10.15:8200 # if BGE is on VPN, unchanged
|
||||
RERANKER_URL=http://10.11.10.15:8100 # if reranker is on VPN, unchanged
|
||||
```
|
||||
|
||||
Everything else (`ATOMIC_URL`, ports, model names, Postgres creds) is either
|
||||
handled by Docker DNS automatically or comes from the same file.
|
||||
|
||||
## Health checklist — run after any deploy/restart
|
||||
|
||||
```bash
|
||||
# 1. Containers healthy
|
||||
docker ps --format '{{.Names}} {{.Status}}' | grep didibrain
|
||||
# expected: 3 lines, all "Up ... (healthy)"
|
||||
|
||||
# 2. Liveness
|
||||
curl -fsS http://localhost:8090/health
|
||||
# expected: {"status":"ok","service":"didibrain-api","version":"0.1.0"}
|
||||
|
||||
# 3. Full stack sanity (LLM + embed + rerank + atomic)
|
||||
.venv/bin/python scripts/01_sanity_full.py
|
||||
# expected: 8/8 PASS, total <10s
|
||||
|
||||
# 4. Brain end-to-end (create atom → embed → search → cleanup)
|
||||
.venv/bin/python scripts/03_sanity_atomic.py
|
||||
# expected: 5/5 PASS
|
||||
|
||||
# 5. Real Didi-style query
|
||||
curl -s -X POST http://localhost:8090/v1/gather \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"claim":"vaccines cause autism","max_evidence":3,"run_nli":false}' \
|
||||
| jq '.brain_meta.cache_status, .total_evidence_items'
|
||||
# expected: "HIT" and a positive number (if the vaccines corpus is loaded)
|
||||
```
|
||||
|
||||
## Known issues that DO NOT block deploy
|
||||
|
||||
1. **`edges_status column does not exist`** — atomic-server logs this warning
|
||||
at startup due to an upstream Postgres schema mismatch. The semantic_edges
|
||||
feature in Atomic is partially broken, but **nothing we use depends on it**:
|
||||
vector search, reranker, claim extraction, and the Didi contract all work.
|
||||
Cosmetic — safe to ignore until upstream Atomic fixes the migration.
|
||||
|
||||
2. **`/api/embeddings/status`** returns 500 — same root cause as #1. We never
|
||||
call this endpoint; we read `embedding_status` per atom via `list_atoms`
|
||||
or `get_atom` instead.
|
||||
|
||||
3. **Qwen 35B (vLLM on :14001)** — thinking mode stuck ON via the router and
|
||||
safety alignment refuses disinfo-extraction tasks. Disabled in config
|
||||
(`MODEL_FAST_ENABLED=false`); the whole pipeline runs on Qwen 397B. If a
|
||||
future session un-sticks 35B, Lint pass could speed up ~4x by using it.
|
||||
|
||||
4. **Gemma 31B endpoint (10.11.10.16:8001)** — port unreachable (host pings
|
||||
OK). Not used by any feature; listed as optional in `.env`.
|
||||
|
||||
5. **Atomic's React UI not running** — we only deploy `atomic-server` (API),
|
||||
not `atomic-web` (React frontend). `http://localhost:8088/` returns 404
|
||||
for this reason. Brain_api has its own Swagger UI at `/docs` which covers
|
||||
dev-testing needs. If visual atom/tag/wiki browsing is needed, add an
|
||||
`atomic-web` service to compose (5-minute task).
|
||||
|
||||
## Troubleshooting — things that have actually gone wrong
|
||||
|
||||
| Symptom | Root cause | Fix |
|
||||
|---|---|---|
|
||||
| `/health` timeouts | BGE endpoint unreachable (VPN down?) | `curl http://10.11.10.15:8200/v1/models` on host; bring VPN back |
|
||||
| `/v1/gather` returns 500 silently | Same as above — Atomic can't embed the query | Same fix |
|
||||
| Post-reboot: brain_api empty reply | uvicorn bound to 127.0.0.1 inside container (not 0.0.0.0) | Rebuild — Dockerfile now sets `BRAIN_API_HOST=0.0.0.0` |
|
||||
| `LLM_ROUTER_URL` points to localhost but nothing there | Router is actually on 10.11.10.17:14011 (not on the dev box) | Update `.env`; validated in session 2 |
|
||||
| Claim extraction returns zero claims on a doc | `list_atoms` returns summary without `content`; must call `get_atom` | Fixed in `extractor/batch.py` |
|
||||
| NLI responses come back as NEUTRAL with confidence 0.0 | MAX_PARALLEL > backend concurrency → timeouts | Set `MAX_PARALLEL=2` to match 2 llama.cpp instances |
|
||||
| Wikipedia API 403 on every request | Non-compliant User-Agent | Use `DidiBrain/0.1 (https://github.com/didibrain; didibrain@local.test)` |
|
||||
|
||||
## What is NOT done (all optional, not blockers)
|
||||
|
||||
- **Bearer auth on brain_api** — skipped intentionally. Add when exposing
|
||||
beyond loopback on a server with a public IP.
|
||||
- **Full Lint pass on vaccines corpus** — skipped. Wikipedia is too
|
||||
internally consistent for Lint to surface many contradictions; rerun
|
||||
after corpus diversification (B).
|
||||
- **Corpus expansion (B)** — next session. Wikipedia seeds for RO elections
|
||||
2024, Russia-Ukraine war, climate, COVID general. Also RSS feeds and
|
||||
possibly Playwright adapters for non-Wiki sources.
|
||||
- **`/v1/contradictions` endpoint in brain_api** — currently contradictions
|
||||
are only visible via `scripts/11_show_contradictions.py`. Expose over
|
||||
HTTP when Didi needs programmatic access.
|
||||
- **Atomic React UI (`atomic-web`)** — optional visual browsing, add if
|
||||
useful for operators.
|
||||
- **Monitoring / metrics** — no Prometheus exporter yet.
|
||||
- **Upstream schema fix for `edges_status`** — cosmetic, PR to kenforthewin/atomic.
|
||||
|
||||
## Session log (condensed)
|
||||
|
||||
### Session 1 (2026-04-10, ~6 hours)
|
||||
- Full audit of upstream Atomic repo
|
||||
- Validated LLM/embed/rerank stack (8 sanity checks PASS)
|
||||
- Built `shared/` + `extractor/` + scripts 01-08
|
||||
- Stood up Atomic + Postgres in compose
|
||||
- Imported 19 Wikipedia vaccines articles, EN + RO
|
||||
- Extracted 513 claims via Qwen 397B (1.2% hallucination drop)
|
||||
- Validated claim-level retrieval (cross-lingual RO↔EN confirmed)
|
||||
|
||||
### Session 2 (2026-04-11, ~4 hours)
|
||||
- Re-onboarded state after VPN restart (fixed LLM router URL)
|
||||
- Built `brain_api/` FastAPI service with 5-endpoint Didi contract
|
||||
- Dockerized brain_api (self-sufficient startup, taxonomy refresh from atomic)
|
||||
- Added NLI stance-vs-query pass in `/v1/gather` (additive fields in `brain_meta`)
|
||||
- Fixed NLI concurrency (MAX_PARALLEL=2 matches llama.cpp backends)
|
||||
- Built `lint/` contradiction detection module with idempotent state file
|
||||
- Smoke-tested Lint (371 pairs, 1 genuine contradiction found at 0.95 confidence)
|
||||
- Wrote deployment bootstrap script
|
||||
- Wrote this STATUS.md and the new README.md
|
||||
|
||||
## Next session — pick-up plan
|
||||
|
||||
Brain is in a stable end-of-session state. To resume:
|
||||
|
||||
1. Verify infra is alive (`docker ps | grep didibrain`)
|
||||
2. If it is not, `docker compose -f infra/docker-compose.yml --env-file .env up -d`
|
||||
3. Run `.venv/bin/python scripts/01_sanity_full.py` to confirm upstream stack
|
||||
4. Decide: corpus expansion (B), Lint full run, new feature, or deploy to server
|
||||
|
||||
**Natural next step is corpus expansion.** The code paths are complete;
|
||||
everything else is about feeding the brain more knowledge and then running
|
||||
the existing scripts over the new data.
|
||||
Loading…
Add table
Add a link
Reference in a new issue