Livrare LOT 1 - Didi

ai_platform/modules/didi_brain/INDEX.md

@@ -0,0 +1,306 @@
+# DidiBrain - Index
+
+Knowledge atom storage + claim verification cache + analysis atom cache pentru platforma DIDI. FastAPI service care vorbeste contractul `web-gathering` 1:1 (drop-in replacement / cache layer pentru `/v1/gather`) si in plus expune doua cache-uri proprii: `brain_verification_cache` (claims, din 2026-04-23) si `brain_analysis_atom` (techniques + ai_tampered + claims, NEW v2 din 2026-05-01).
+
+**Stack**: Python 3.11+, FastAPI, Pydantic v2, asyncpg, BGE-M3 1024-dim embeddings (vLLM), BGE-reranker-v2-m3 (vLLM cross-encoder), Qwen3.5-397B-A17B (llama.cpp via OpenAI-compat router)
+**Storage**: Atomic (Rust knowledge graph) on Postgres 16 + pgvector. Brain owns its own tables prefixed `brain_*` alongside Atomic's tables.
+**URL**: `http://10.11.10.12:8090` (production-local, since brain cutover 2026-05-01 — anterior pe `10.11.10.13:8090`)
+**Container**: `didibrain-api` (alongside `didibrain-atomic` and `didibrain-postgres`)
+
+---
+
+## Ce face
+
+DidiBrain sta intre Didi backend (agent-v3) si modulele live de web-gathering. Vorbeste acelasi contract HTTP pentru `/v1/gather`, raspunzand din knowledge atoms pre-ingerate + semantic search + cross-encoder reranking + NLI stance vs query. Tipic 1.5-6s in loc de 20+s pentru un crawl live. Plus detine 2 cache-uri proprii pentru rezultate LLM ale didi-backend: `brain_verification_cache` (per-claim verification, since 2026-04-23) si `brain_analysis_atom` (rezultate full-component pe techniques + ai_tampered + claims, since 2026-05-01 v2).
+
+---
+
+## API endpoints
+
+| Method | Path | Ce face |
+|---|---|---|
+| GET | `/health` | Liveness probe |
+| POST | `/v1/search` | Flat list of doc-level search results |
+| POST | `/v1/fetch` | Look up atoms by URL → return extracted text |
+| POST | `/v1/gather` | Full claim → ranked evidence pipeline (with optional verification cache lookup) |
+| POST | `/v1/image-search` | Stub, always returns empty list |
+| POST | `/v1/ingest` | Populate brain from web-module output |
+| POST | `/v1/verification_cache` | Write claim verification result (by didi-backend after LLM run) |
+| POST | `/v1/analysis_atom/lookup` | NEW v2 — find cached analysis result by `(content_hash, component, prompt_hash)`. Tier-agnostic. Bronze never served. |
+| POST | `/v1/analysis_atom` | NEW v2 — write atom (rejects `tier=free`). Auto cache_tier silver/bronze din `llm_confidence` (>=60 silver). Gold preservation in upsert. |
+| PATCH | `/v1/analysis_atom/{atom_id}` | NEW v2 — promote to gold (after moderator review). `human_validated=true`, `expires_at=NULL`. |
+| GET | `/v1/analysis_atom/stats` | NEW v2 — counts per tier/component, hit rate 24h |
+
+Plus FastAPI auto: `/docs` (Swagger UI), `/redoc`, `/openapi.json`.
+
+---
+
+## Structura fisiere
+
+```
+brain_api/
+  app.py                  # FastAPI app + lifespan + 11 endpoints
+  db.py                   # asyncpg pool + _SCHEMA_SQL (creates brain_verification_cache + brain_analysis_atom tables idempotent)
+  schemas.py              # Pydantic v2 models (request/response for all endpoints)
+  deps.py                 # AppState (atomic, embed, llm, resolver) singleton
+  run.py                  # CLI entry (uvicorn launcher)
+  Dockerfile
+  requirements.txt
+  prompts/                # LLM prompts for verification + extraction (nli_v1.md etc.)
+  services/
+    fetch.py              # /v1/fetch
+    gather.py             # /v1/gather (most complex — semantic search + reranking + NLI + verification cache integration)
+    ingest.py             # /v1/ingest
+    search.py             # /v1/search
+    verification_cache.py # claims cache (since 2026-04-23 v2 schema)
+    analysis_atom.py      # NEW 2026-05-01 — techniques+ai_tampered+claims cache, 3 tiers (gold/silver/bronze)
+    nli.py                # NLI stance classification
+    mapping.py            # taxonomy / tag mapping helpers (atom → EvidenceItem / FetchedPage)
+
+shared/                   # (one level up, used by brain_api)
+  atomic_api.py           # AtomicClient (typed REST client for atomic-server)
+  embedding_client.py     # BGE-M3 embed + reranker
+  llm_client.py           # async OpenAI-compat wrapper
+  config.py               # Pydantic Settings, LlmRole, model routing
+  taxonomy.py             # canonical TAXONOMY tree + TagResolver
+  logging.py              # structlog setup
+
+extractor/                # claim extraction layer (separate package)
+  extract.py              # single-doc extraction + substring quote validation
+  push.py                 # claim atom creation
+  batch.py                # batch orchestrator with state file
+  prompts/
+
+scripts/                  # operator CLI tools (numbered 01-11 + bootstrap_deploy.sh)
+infra/                    # docker-compose.yml + deploy
+lint/                     # custom lint pass (cross-corpus contradiction detection)
+reports/                  # generated audit reports
+```
+
+---
+
+## Brain-owned PG schema
+
+Doua tabele alaturi de cele ale Atomic, in acelasi Postgres (prefix `brain_*` pentru izolare).
+
+### brain_verification_cache (since 2026-04-23 v2)
+
+Cache pentru verificare claims (one row per `(claim_hash, tier)`, last-wins UPSERT). `evidence_urls` + `evidence_hash` stocate ca METADATA (NU in unique key) pentru ca backend sa poata calcula overlap. TTL 30 zile default.
+
+| Coloana | Tip | Scop |
+|---|---|---|
+| `id` | uuid PK | `gen_random_uuid()` |
+| `claim_hash` | text NOT NULL | `sha256(normalized claim)` |
+| `tier` | text CHECK (`free`\|`premium`) | |
+| `evidence_hash` | text NOT NULL | `sha256(canonical urls)`, metadata only |
+| `evidence_urls` | jsonb NOT NULL | metadata only |
+| `model` | text NULL | LLM used |
+| `prompt_hash` | text NOT NULL | `sha256(system+user_template)[:12]` |
+| `framework_version` | text NULL | sha256 of relevant Redis configs |
+| `schema_name` | text default `'didi-v1'` | |
+| `verification_raw` | jsonb NULL | raw LLM output |
+| `verification_processed` | jsonb NOT NULL | mapped to canonical |
+| `created_at` | timestamptz default `now()` | |
+| `updated_at` | timestamptz default `now()` | |
+| `expires_at` | timestamptz NOT NULL | TTL 30d default |
+
+UNIQUE: `(claim_hash, tier)`. Indexes: `idx_bvc_lookup`, `idx_bvc_expires`, `idx_bvc_prompt`.
+
+**Migration v1 → v2**: vechiul UNIQUE `(claim_hash, evidence_hash, tier)` facea cache-ul practic nereachable (URL-urile la read difera de cele la write). v2 muta `evidence_hash` afara din unique key. Migrarea e idempotenta (rulata la fiecare connect).
+
+### brain_analysis_atom (NEW 2026-05-01)
+
+Cache pentru rezultate LLM full-component (techniques, ai_tampered, claims-aggregator). 3-tier system gold/silver/bronze.
+
+| Coloana | Tip | Scop |
+|---|---|---|
+| `atom_id` | bigserial PK | |
+| `content_hash` | text NOT NULL | `sha256(normalized text)` |
+| `content_preview` | text NULL | first 200 chars for debug |
+| `component` | text CHECK (`techniques`\|`ai_tampered`\|`claims`) | |
+| `tier` | text CHECK (`free`\|`premium`) | informational only on row |
+| `prompt_hash` | text NOT NULL | |
+| `framework_version` | text NULL | |
+| `model_used` | text NULL | |
+| `result_processed` | jsonb NOT NULL | canonical mapped output |
+| `result_raw` | jsonb NULL | |
+| `llm_confidence` | numeric NULL | 0-100, drives silver vs bronze |
+| `cache_tier` | text CHECK (`gold`\|`silver`\|`bronze`) default `'silver'` | |
+| `human_validated` | bool default false | true after moderator review |
+| `human_corrections` | jsonb NULL | diff applied by moderator |
+| `validator_user_id` | text NULL | keycloak_id |
+| `validated_at` | timestamptz NULL | |
+| `hit_count` | integer default 0 | incremented on lookup hit |
+| `last_hit_at` | timestamptz NULL | |
+| `created_at`, `updated_at` | timestamptz | |
+| `expires_at` | timestamptz NULL | NULL=never (gold). Silver=90d, Bronze=30d |
+
+UNIQUE: `(content_hash, component, prompt_hash)` — `tier` OUT of unique key (write only `premium`, read tier-agnostic). Indexes: `idx_baa_lookup`, `idx_baa_gold` (partial WHERE `cache_tier='gold'`), `idx_baa_expires`, `idx_baa_prompt`.
+
+**Cache tier rules**:
+- **gold**: `human_validated=true`. Set by didi-backend moderation flow (PATCH endpoint). Survives prompt change. Forever (`expires_at=NULL`).
+- **silver**: LLM result, `llm_confidence >= 60`. Default fresh write. TTL 90d.
+- **bronze**: LLM result, `llm_confidence < 60`. Stored for audit but **NEVER served on lookup**. TTL 30d.
+
+**Gold preservation**: ON CONFLICT upsert preserves gold (won't downgrade to silver if a fresh write comes in — `result_processed`, `cache_tier`, `llm_confidence`, `expires_at` all CASE-WHEN protected).
+
+**Lookup logic** (`services/analysis_atom.py::lookup`): SELECT WHERE `content_hash=$1 AND component=$2 AND cache_tier IN ('gold','silver') AND (expires_at IS NULL OR expires_at > now())` ORDER BY `(cache_tier='gold') DESC, updated_at DESC` LIMIT 1. Apoi `decide_freshness()` pe baza `prompt_hash` + `framework_version`. Gold = always `fresh`.
+
+---
+
+## How didi-backend (agent-v3) uses brain
+
+3 puncte de integrare:
+
+1. **Claims verification cache** — claims executor cheama `gatherFromBrain()` cu `include_verification=true`. Brain returneaza evidence + cached verification (if any) + staleness. Backend decide fresh / stale_framework / stale_prompt / miss path. Backend scrie via fire-and-forget POST `/v1/verification_cache`. Detalii contract: `CONTRACT_VERIFICATION_CACHE.md`. Helper centralizat: `agent-v3/src/shared/brain/client.ts`.
+
+2. **Analysis atom cache** (NEW 2026-05-01) — techniques si ai_tampered executors cheama `lookupAnalysisAtom()` **inainte** de LLM. Pe hit (gold sau silver+fresh), skip LLM, return cached. Dupa LLM run, fire-and-forget `writeAnalysisAtomAsync()` (only `tier=premium`). Dupa ce moderator rezolva sesiunea cu corectii, `patchAnalysisAtomGold()` promoveaza silver → gold.
+
+3. **Gather pipeline** — agent-v3 inca foloseste `/v1/gather` pentru live web-augmented retrieval (pre-ingerat + semantic search + reranker + NLI). Branch-ul pe `brain_meta.cache_status` = `HIT|PARTIAL|MISS` decide fallback la web module.
+
+---
+
+## Configuration
+
+- Settings via `shared/config.py` cu `pydantic-settings`
+- Env vars: `postgres_dsn`, `atomic_url`, `llm_router_url`, `embedding_url`, `reranker_url`, `verification_cache_ttl_days` (default 30), `verification_cache_max_payload_kb` (default 64)
+- Internal Atomic URL: `http://didibrain-atomic:8080` (Docker DNS)
+- Upstream LLM/embed/rerank pe VPN 10.11.10.x (10.11.10.17:14011 router, 10.11.10.15:8200 BGE-M3, 10.11.10.15:8100 reranker)
+- **No auth in v1** (binds to internal Docker network)
+
+---
+
+## Deployment
+
+- Docker compose la `infra/docker-compose.yml`
+- 3 containere: `didibrain-api` (FastAPI), `didibrain-atomic` (Rust KG), `didibrain-postgres` (PG 16 + pgvector)
+- Schema bootstrap on startup (idempotent — `_SCHEMA_SQL` ruleaza la fiecare connect, all DDL is `IF NOT EXISTS` + ALTER guards)
+- TTL cleanup: cron job in `scripts/` sterge expired silver/bronze atoms
+- Resource footprint: ~200 MB RAM total cross 3 containere; brain-api idle ~6% CPU, spike ~20% during `/v1/gather` cu NLI
+
+**Rebuild**:
+```bash
+cd /home/admin365/didi_mono/ai_platform/modules/didi_brain
+docker compose -f infra/docker-compose.yml --env-file .env up -d --build
+```
+
+---
+
+## Bootstrap & lint
+
+- `scripts/01_sanity_full.py` — upstream stack validator (LLM + embed + rerank, 8 checks)
+- `scripts/02_bootstrap_atomic.py` — claim atomic instance, configure BGE-M3 provider
+- `scripts/03_sanity_atomic.py` — brain end-to-end (create → embed → search → cleanup)
+- `scripts/04_seed_taxonomy.py` — seed canonical tag taxonomy (79 tags, 7 namespaces)
+- `scripts/05_import_wikipedia_seed.py` — seed-list Wikipedia import
+- `scripts/06_validate_queries.py` — smoke test doc-level retrieval
+- `scripts/07_run_extraction.py` — claim extraction batch (Qwen 397B)
+- `scripts/08_validate_claims.py` — smoke test claim-level retrieval
+- `scripts/09_brain_api_demo.py` — brain_api contract test (all endpoints)
+- `scripts/10_run_lint.py` — Lint pass runner (contradiction detection)
+- `scripts/11_show_contradictions.py` — render top contradictions
+- `scripts/bootstrap_deploy.sh` — fresh-server deploy orchestrator
+- `lint/` — custom lint pass (cross-corpus contradiction detection via NLI)
+
+---
+
+## Ce NU face
+
+- No image storage (image-search endpoint always returns empty)
+- No auth in v1 (intern only, binds to Docker internal network)
+- No multi-tenant (single brain instance, multiple consumers)
+- No long-term archive — atoms expire on TTL (gold = forever, silver = 90d, bronze = 30d)
+- No live web crawling (delegata la modulele M17 / web — brain serveste din corpus pre-ingest)
+
+---
+
+## Related docs (in this module)
+
+- `README.md` — project landing, quick start, architecture diagram
+- `STATUS.md` — end-of-session snapshot + troubleshooting + session log
+- `CONTRACT_VERIFICATION_CACHE.md` — contract backend ↔ brain pentru verification cache (v2)
+- `CHANGES_2026-04-23.md` — session changelog cu integrare web-api + verification cache v2
+- `ARCHITECTURE.md` — diagrame + pipeline detail
+- `AUDIT.md` — initial upstream Atomic audit
+
+---
+
+## Cache Freshness Defense (2026-05-04 → 2026-05-05) — 11/11 piloni live
+
+Whole-system response to "what happens if a cached verdict goes stale because the world changed?". Strict additive — zero breaking changes on the contracts agent-v3 already calls.
+
+### New services in `brain_api/services/`
+
+| Module | Pilon | Role |
+|--------|-------|------|
+| `classifier.py` | 1 | LLM single-call classify of a claim → `{volatility, topic_codes, entity_bindings, estimated_validity_hours}`. Caps clamped per tier (volatile≤48h, evolving≤720h, stable≤26280h). Conservative defaults on LLM failure. |
+| `cache_judge.py` | 2 | NLI judge run on cache hits — given a claim, cached truth direction (TRUE/FALSE/MIXED), and current top-3 evidence, decides KEEP_CACHE / INVALIDATE / NEEDS_FULL_RECHECK. Cheap-path skip for stable+young. Audit-pass bonus raises invalidate threshold for atoms that survived multiple audits. |
+| `fact_status.py` | 11 | Versioned (subject, predicate, object) knowledge layer. `register_facts_from_bindings`, `assert_fact_truth` (opens new fact_version when truth flips), `lock_fact`/`unlock_fact` for moderator overrides, `check_fact_validity` for gather-time lookup. Plus admin: `list_facts_admin`, `list_audit_log`. |
+| `canonicalizer.py` | 7 | LLM rewrites a claim with relative time markers (`azi`, `alegerile`) anchored to the current date. Same surface text at different times produces different cache keys. Passthrough on failure. |
+| `invalidation.py` | 8 | `InvalidateFilter` (topic_codes / entity_canonicals / claim_pattern / since / invalidate_gold / dry_run) → bulk UPDATE on both cache tables. Audit log row written per non-dry-run execution. |
+| `topic_volatility.py` | D1 | HTTP polls didiFramework `/api/sensitive-topics?active=true` (cache 60s) for admin-configured per-topic TTL/recency. `reconcile_with_classifier` picks min(LLM_estimate, admin_TTL). |
+
+### DB schema (alongside Atomic in same Postgres, prefix `brain_*`)
+
+```
+ALTER brain_analysis_atom + brain_verification_cache
+  ADD volatility ('volatile'|'evolving'|'stable')
+  ADD topic_codes text[]
+  ADD entity_bindings jsonb         -- list of {subject, predicate, object, confidence}
+  ADD ttl_hours_used integer
+  ADD last_audited_at timestamptz
+  ADD audit_history jsonb           -- last-50 JudgeVerdict snapshots
+  ADD consecutive_audit_passes integer
+
+CREATE brain_fact_status               -- current truth per (subject, predicate, object)
+CREATE brain_fact_version              -- temporal versioning with valid_from/valid_to
+CREATE brain_audit_log                 -- mass invalidations + judge decisions + truth changes
+```
+
+GIN indexes on `topic_codes`. Partial indexes on `cache_tier IN ('gold','silver') AND volatility != 'stable'` for the auditor's fast candidate query.
+
+### New endpoints (`brain_api/app.py`)
+
+```
+POST /v1/canonicalize                — Pilon 7 temporal disambiguation
+POST /v1/cache/invalidate            — Pilon 8 mass invalidation (with dry_run)
+GET  /v1/cache/audit_log             — paginated audit browser
+GET  /v1/fact_status/list            — paginated fact browser (admin)
+GET  /v1/fact_status/{fact_id}       — single fact detail
+GET  /v1/fact_status/{fact_id}/versions  — full timeline
+PATCH /v1/fact_status/{fact_id}      — moderator override (set_truth + lock + notes)
+```
+
+Existing upsert paths (`POST /v1/analysis_atom`, `POST /v1/verification_cache`) now run the classifier internally and persist volatility/topic_codes/entity_bindings on every write. Lookup paths (`/v1/gather` with `include_verification=true`) consult `brain_fact_status` and demote staleness to `stale_evidence` when bound facts have flipped.
+
+### `gather.py` recency reranking (Pilon 3+4)
+
+`_apply_recency` blends rerank score with age decay using volatility-driven profiles:
+
+| Volatility | w_recency | half_life_days | hard cut |
+|---|---|---|---|
+| volatile | 0.50 | 3 | 7 days |
+| evolving | 0.30 | 30 | (none) |
+| stable | 0.00 | ∞ | (none) |
+| (none / hint absent) | 0.15 | 30 | (none) |
+
+`combined = (1 - w_recency) * rerank + w_recency * exp(-age_days/half_life)`. Volatile evidence older than `recency_window_days` is hard-dropped.
+
+### `didibrain-scheduler` container (`scheduler/`)
+
+Single image, four asyncio tasks under `_supervised` (auto-restart with exponential backoff):
+
+- **feeder** — RSS pull at volatility-aware intervals (15 min volatile / 6 h evolving / 24 h stable). Default feeds in `config.py:DEFAULT_*_FEEDS`; override via `FEED_VOLATILE_URLS`/`FEED_EVOLVING_URLS`/`FEED_STABLE_URLS`/`FEED_BREAKING_URLS` env. Feeds POST to `/v1/ingest` with `run_extraction=true`.
+- **auditor** — daily sweep of `brain_analysis_atom` rows with `cache_tier IN ('gold','silver') AND volatility != 'stable' AND last_audited_at < now()-1d`. For each: confidence-decay cheap path → judge → `apply_judge_verdict`. Sequential (not parallel) to avoid LLM router saturation.
+- **watcher** — 5-min poll on the breaking-news feeds. Per item: classify with LLM → `_build_invalidation_targets` → `POST /v1/cache/invalidate` with `topic_codes` + `entity_canonicals`. In-process LRU of seen URLs (cap 2000). End-to-end target latency: <30s from RSS publish to cache invalidation.
+- **heartbeat** — touches `/tmp/scheduler.healthy` every 30s for the docker healthcheck.
+
+Compose entry in `infra/docker-compose.yml` builds via `scheduler/Dockerfile`. Imports `brain_api.*` directly so the auditor uses the same DB pool + `apply_judge_verdict` helper as the API container (separate process, separate pool to the same Postgres).
+
+### Live state at write-time
+
+47 articles ingested in first cycle. Watcher invalidated 4 breaking stories live (Iran/Hormuz, Nigerian elections, Ukraine drones, etc.) within seconds of feed publication. Auditor scheduled 24h after first start.
+
+### Memory pointer
+
+`~/.claude/projects/-home-admin365/memory/MEMORY.md` § "Cache Freshness Defense" — one-liner status + counts.